Kafka Consumer

    This sub-pipeline must start with an Injector transform.

    You can define the number of messages to accept for processing, as well as the specific data formats to stream activity data and system metrics.

    You can set up this step to collect monitored events, track user consumption of data streams, and monitor alerts.

    Kafka records are stored within topics, and consist of a category to which the records are published. Topics are divided into a set of logs known as partitions.

    Kafka scales topic consumption by distributing partitions among a consumer group.

    A consumer group is a set of consumers sharing a common group identifier.

    Since the Kafka Consumer transform continuously ingests streaming data, you may want to use the Abort transform in your parent or sub-pipeline to stop consuming records from Kafka for specific workflows.

    For example, you can run the parent pipeline on a timed schedule, or abort the sub-pipeline if sensor data exceeds a preset range.

    Setup

    OptionDescription

    Bootstrap servers

    Comma separated list of bootstrap servers in a Kafka cluster

    Topics

    Enter the name of each Kafka topic from which you want to consume streaming data (messages).

    Consumer group

    When part of a consumer group, each consumer is assigned a subset of the partitions from topics it has subscribed to, which locks those partitions. Each instance of a Kafka Consumer transform will only run a single consumer thread.

    Use this tab to determine how many messages to consume before processing. You can specify message count and/or a specific amount of time.

    OptionDescription

    Duration (ms)

    Specify a time in milliseconds. This value is the amount of time the transform will spend collecting records prior to the execution of the pipeline. If set to a value of ‘0’, then Number of records triggers consumption.

    Number of records

    Specify a number. After every ‘X’ number of records, the specified pipeline will be executed and these ‘X’ records will be passed to the pipeline. If set to a value of ‘0’ then Duration triggers consumption.

    Offset management

    Choose when to commit

    • when record read

    • when batch completed

    Fields

    Return fields from a transform in the sub pipeline for further processing in this pipeline.

    Use this tab to configure the property formats of the Kafka consumer broker sources. A few of the most common property formats have been included for your convenience. You can enter any desired Kafka property. For further information on these input names, see the Apache Kafka documentation site: https://kafka.apache.org/documentation/.

    The options that are included by default are:

    NNameValue

    auto.offset.reset

    latest

    ssl.key.password

    ssl.keystore.location

    ssl.keystore.password

    ssl.truststore.location

    ssl.truststore.password

    Error handling

    This feature is very useful in any case where we have a record in the Kafka queue that is stopping processing of further records (because it generates an error during processing) and we want to “remove it” from the queue so that we can process the remaining records in the queue. The failing record can be saved somewhere (for example saved to a file or to a DB table) for further investigation.