Kafka Message Keys

You can enable the Kafka origin and destination to process Kafka message keys. These stages cannot process message keys when configured to process delimited data.

Stages handle Kafka message keys as follows:
Origins
The Kafka origin can capture the message keys included in each Kafka message and store them in generated records. The origin writes the message key to a String field named key.
You can process data in the key field like any other String field.
To include message key data in records, enable the Include Message Keys property on the Kafka tab of the Kafka origin.
Destinations
The Kafka destination can pass values in a specified field to the destination system as Kafka message keys.
As a result, you can configure a pipeline that reads from and writes to Kafka to pass the original message keys associated with each message. This allows Kafka to reuse the original keys instead of generating new message keys for the data.
When writing data to Kafka that originated from other systems, you can still configure the destinations to use data from a specified field as Kafka message keys.
To pass data to Kafka as message keys:
  1. On the Kafka tab of the Kafka destination, select the Send Message Keys property.
  2. In the Key Field property, specify the field that contains the message key data.

    To pass message key data included in records by a Kafka origin, use the default key field.

The specified key field is then used as a Kafka message key, and is not included as part of the message value.