Pipeline Statistics

Use the following format: <host>:<port>.

When the SDC RPC destination cannot write the batch within the configured number of retries, it fails the batch.

Default is 3.

The value that you enter increases exponentially after each retry. For example, if you set the back off period to 10, the SDC RPC destination attempts the first retry after waiting 10 milliseconds, attempts the second retry after waiting 100 milliseconds, and attempts the third retry after waiting 1,000 milliseconds. Set to 0 to retry immediately.

Default is 0.

You cannot define an expression that evaluates to the ID.

The SDC RPC destination retries the connection based on the Retries Per Batch property.

Default is 5000 milliseconds.

Must be stored in the Data Collector resources directory, $SDC_RESOURCES, on each Data Collector machine that runs a pipeline instance for the job.

The SDC RPC destination retries the write based on the Retries Per Batch property.

Default is 2000 milliseconds.

To ensure a pipeline can connect to Kafka in case a specified broker goes down, list as many brokers as possible.

You cannot define an expression that evaluates to the topic name.

• Round Robin - Takes turns writing to different partitions.
• Random - Writes to partitions randomly.
• Expression - Uses an expression to write data to different partitions. Writes records to the partitions specified by the results of the expression.
  Note: The expression results are written to a specified Kafka message key attribute, overwriting any existing values.
• Default - Uses an expression to extract a partition key from the record. Writes records to partitions based on a hash of the partition key.

Define the expression to evaluate to the partition where you want statistics written. Partition numbers start with 0.

Optionally, click Ctrl + Space Bar for help with creating the expression.

Use the property names and values as expected by Kafka. Do not use the broker.list property.

To use a ZooKeeper quorum, enter a comma-separated list.

<host>:<port>, <host2>:<port2>, .../<chroot_path>

Enter an expression that specifies the attribute where the message keys are stored.

${record:attribute('<message key attribute name>'}

${avro:decode(record:attribute('avroKeySchema'),base64:decodeBytes(record:attribute('<messsage key attribute name')))}

For more information, about working with Kafka message keys, see Kafka Message Keys.

• None (Security Protocol=PLAINTEXT) - Uses no authentication or encryption.
• SSL/TLS Encryption (Security Protocol=SSL)
• SSL/TLS Encryption and Authentication (Security Protocol=SSL)
• SASL Authentication (Security Protocol=SASL_PLAINTEXT)
• SASL Authentication on SSL/TLS (Security Protocol=SASL_SSL)
• Custom Authentication (Security Protocol=CUSTOM)

Enabling security requires completing several prerequisite tasks and configuring additional security properties, as described in Security in Kafka Stages.

For more information about working with Kafka message key values, see Kafka Message Keys.

• AWS Keys - Authenticates using an AWS access key pair.
• Instance Profile - Authenticates using an instance profile associated with the Data Collector EC2 instance.

Amazon resource name (ARN) of the role to assume, entered in the following format:

arn:aws:iam::<account_id>:role/<role_name>

Where <account_id> is the ID of your AWS account and <role_name> is the name of the role to assume. You must create and attach an IAM trust policy to this role that allows the role to be assumed.

Available when assuming another role.

Optional name for the session created by assuming a role. Overrides the default unique identifier for the session.

Available when assuming another role.

Maximum number of seconds for each session created by assuming a role. The session is refreshed if the pipeline continues to run for longer than this amount of time.

Set to a value between 3,600 seconds and 43,200 seconds.

Available when assuming another role.

Sets a session tag to record the name of the currently logged in StreamSets user that starts the pipeline or the job for the pipeline. AWS IAM verifies that the user account set in the session tag can assume the specified role.

Select only when the IAM trust policy attached to the role to be assumed uses session tags and restricts the session tag values to specific user accounts.

When cleared, the connection does not set a session tag.

Available when assuming another role.

You cannot define an expression that evaluates to the stream name.

• Random - Generates a random partition key.

• Expression - Uses the result of an expression as the partition key.

Use for the expression partition strategy.

When you add a configuration property, enter the exact property name and the value. The pipeline does not validate the property names or values.

You cannot define an expression that evaluates to the topic name.

• Round Robin - Takes turns writing to different partitions.
• Random - Writes to partitions randomly.
• Expression - Uses an expression to write data to different partitions. Writes records to the partitions specified by the results of the expression.
  Note: The expression results are written to a specified Kafka message key attribute, overwriting any existing values.
• Default - Uses an expression to extract a partition key from the record. Writes records to partitions based on a hash of the partition key.

Define the expression to evaluate to the partition where you want statistics written. Partition numbers start with 0.

Optionally, click Ctrl + Space Bar for help with creating the expression.

Use the property names and values as expected by MapR Streams. You can use MapR Streams properties and the set of Kafka properties supported by MapR Streams.