Kafka Message Keys

You can enable the Kafka origin and destination to process Kafka message keys. Kafka stages cannot process message keys when the stage is configured to process delimited data.

Note: To process Kafka message keys with the JSON data format, Apache Spark version 2.4 or later must be installed on the Transformer machine and on each node in the cluster.
The Kafka stages can handle Kafka message keys as follows:
Kafka origin
The Kafka origin can capture the message keys included in each Kafka message and store them in generated records. The origin writes the message key to a String field named key.
You can process data in the key field like any other String field.
To include message key data in records, on the Kafka tab of the Kafka origin, select Include Message Keys.
Kafka destination
The Kafka destination can pass values in a specified field to Kafka as Kafka message keys.
As a result, you can configure a pipeline that reads from and writes to Kafka to pass the original message keys associated with each message to Kafka. This allows Kafka to reuse the original keys instead of generating new message keys for the data.
When writing data to Kafka that originated from other systems, you can still configure the Kafka destination to use data from a specified field as Kafka message keys.
To pass data to Kafka as message keys:
  1. On the Kafka tab of the Kafka destination, select the Send Message Keys property.
  2. In the Key Field property, specify the field that contains the message key data.

    To pass message key data included in records by a Kafka origin, use the default key field.