Kafka Message Keys

You can enable the Kafka origin and destination and the MapR Event Store origin and destination to process Kafka message keys. These stages cannot process message keys when the stage is configured to process delimited data.

Note: To process Kafka message keys with the JSON data format, Apache Spark version 2.4 or later must be installed on the Transformer machine and on each node in the cluster.
Stages handle Kafka message keys as follows:
Origins
The Kafka origin and the MapR Event Store origin can capture the message keys included in each Kafka message and store them in generated records. The origin writes the message key to a String field named key.
You can process data in the key field like any other String field.
To include message key data in records, enable the Include Message Keys property. The property appears on the Kafka tab of the Kafka origin, and on the MapR Event Store tab of the MapR Event Store origin.
Destinations
The Kafka destination and the MapR Event Store destination can pass values in a specified field to the destination system as Kafka message keys.
As a result, you can configure a pipeline that reads from and writes to Kafka or MapR Streams to pass the original message keys associated with each message. This allows Kafka or MapR Streams to reuse the original keys instead of generating new message keys for the data.
When writing data to Kafka or MapR Streams that originated from other systems, you can still configure the destinations to use data from a specified field as Kafka message keys.
To pass data to Kafka as message keys:
  1. On the Kafka tab of the Kafka destination, or on the MapR Event Store tab of the MapR Event Store destination, select the Send Message Keys property.
  2. In the Key Field property, specify the field that contains the message key data.

    To pass message key data included in records by a Kafka or MapR Event Store origin, use the default key field.

The specified key field is then used as a Kafka message key, and is not included as part of the message value.