Kafka Multitopic Consumer

Supported pipeline types:

  • Data Collector
The Kafka Multitopic Consumer origin reads data from multiple topics in an Apache Kafka cluster. The origin can use multiple threads to enable parallel processing of data. For information about supported versions, see Supported Systems and VersionsSupported Systems and Versions in the Data Collector documentation.

When you configure a Kafka Multitopic Consumer, you configure the consumer group name and the brokers to use. You also specify the topics to process and the number of threads to use. In Kafka, make sure that the partition assignment strategy is configured appropriately.

You can configure the origin to produce a single record when a message includes multiple objects. You can add additional Kafka configuration properties as needed. You can configure the origin to use Kafka security features. You can also configure the origin to capture Kafka message keys and store them in the record.

When processing Avro data, you can configure the Kafka Multitopic Consumer to work with the Confluent Schema Registry. The Confluent Schema Registry is a distributed storage layer for Avro schemas which uses Kafka as its underlying storage mechanism.

Kafka Multitopic Consumer includes record header attributes that enable you to use information about the record in pipeline processing.

You can also use a connectionconnection to configure the origin.