MapR Multitopic Streams Consumer

Supported pipeline types:

  • Data Collector
The MapR Multitopic Streams Consumer origin reads data from multiple topics in a MapR Streams cluster. The origin can use multiple threads to enable parallel processing of data. For information about supported versions, see Supported Systems and VersionsSupported Systems and Versions in the Data Collector documentation.
Tip: Data Collector provides several MapR origins to address different needs. For a quick comparison chart to help you choose the right one, see Comparing MapR Origins.

When you configure a MapR Multitopic Streams Consumer, you configure the consumer group name, the topics to process, and the number of threads to use.

You can configure the origin to produce a single record when a message includes multiple objects. And you can add additional MapR Streams and supported Kafka configuration properties as needed.

When processing Avro data, you can configure the MapR Multitopic Streams Consumer to work with the Confluent Schema Registry. The Confluent Schema Registry is a distributed storage layer for Avro schemas which uses MapR Streams as its underlying storage mechanism.

MapR Multitopic Streams Consumer includes record header attributes that enable you to use information about the record in pipeline processing.

Before you use any MapR stage in a pipeline, you must perform additional steps to enable Data Collector to process MapR data. For more information, see MapR PrerequisitesMapR Prerequisites in the Data Collector documentation.