MQTT

Available when using an authoring Data Collector version 5.0.0 or later.

To create an MQTT connection, the Basic stage library, streamsets-datacollector-basic-lib, must be installed on the selected authoring Data Collector.

For a description of the MQTT connection properties, see MQTT Connection Properties.

After you create an MQTT connection, you can use the connection in the following stages and locations:
Engine Stages and Locations
Data Collector 5.0.0 or later
  • MQTT Subscriber origin
  • MQTT Publisher destination
  • Write to MQTT error record handling configured for a pipeline

MQTT Connection Properties

When creating an MQTT connection, configure the following properties on the MQTT tab:
MQTT Property Description
Broker URL MQTT Broker URL. Enter in the following format:
<tcp | ssl>://<hostname>:<port>

Use ssl for secure connections to the broker.

For example:
tcp://localhost:1883
For high availability MQTT clusters without a load balancer, specify a list of brokers from the cluster, separated by commas and without spaces. The stage connects to the first available broker, trying in the order listed. For example:
tcp://hostA:1883,tcp://hostB:1883,tcp://hostC:1883
Client ID MQTT Client ID. The ID must be unique across all clients connecting to the same broker.
You can define an expression that evaluates to the client ID. For example, enter the following expression to use the unique pipeline ID as the client ID:
${pipeline:id()}

If a pipeline includes multiple MQTT stages and you want to use the unique pipeline ID as the client ID for both stages, prefix the client ID with a string like this:

sub-${pipeline:id()} and pub-${pipeline:id()} 
Otherwise, all stages will use the same client ID. This can cause problems, such as messages disappearing.
Use Credentials Enables entering credentials on the Credentials tab. Use when the MQTT broker requires a user name and password.

If you enabled credentials, specify a user name and password on the Credentials tab.

On the TLS tab, optionally configure the following SSL/TLS properties:

TLS Property Description
Use TLS Enables the use of TLS.
Use Remote Keystore Enables loading the contents of the keystore from a remote credential store or from values entered in the stage properties.

For more information, see the Data Collector documentation.

Private Key Private key used in the remote keystore. Enter a credential function that returns the key or enter the contents of the key.

For more information about credential functions, see the Data Collector documentation.

Certificate Chain Each PEM certificate used in the remote keystore. Enter a credential function that returns the certificate or enter the contents of the certificate.

Using simple or bulk edit mode, click the Add icon to add additional certificates.

For more information about credential functions, see the Data Collector documentation.

Keystore File Path to the local keystore file. Enter an absolute path to the file or enter the following expression to define the file stored in the Data Collector resources directory: ${runtime:resourcesDirPath()}/keystore.jks

By default, no keystore is used.

Keystore Type Type of keystore to use. Use one of the following types:
  • Java Keystore File (JKS)
  • PKCS #12 (p12 file)

Default is Java Keystore File (JKS).

Keystore Password Password to the keystore file. A password is optional, but recommended.
Tip: To secure sensitive information, you can use credential stores or runtime resources.
Keystore Key Algorithm Algorithm to manage the keystore.

Default is SunX509.

Use Remote Truststore Enables loading the contents of the truststore from a remote credential store or from values entered in the stage properties.

For more information, see the Data Collector documentation.

Trusted Certificates Each PEM certificate used in the remote truststore. Enter a credential function that returns the certificate or enter the contents of the certificate.

Using simple or bulk edit mode, click the Add icon to add additional certificates.

Truststore File Path to the local truststore file. Enter an absolute path to the file or enter the following expression to define the file stored in the Data Collector resources directory: ${runtime:resourcesDirPath()}/truststore.jks

By default, no truststore is used.

Truststore Type Type of truststore to use. Use one of the following types:
  • Java Keystore File (JKS)
  • PKCS #12 (p12 file)

Default is Java Keystore File (JKS).

Truststore Password Password to the truststore file. A password is optional, but recommended.
Tip: To secure sensitive information, you can use credential stores or runtime resources.
Truststore Trust Algorithm Algorithm to manage the truststore.

Default is SunX509.

Use Default Protocols Uses the default TLSv1.2 transport layer security (TLS) protocol. To use a different protocol, clear this option.
Transport Protocols TLS protocols to use. To use a protocol other than the default TLSv1.2, click the Add icon and enter the protocol name. You can use simple or bulk edit mode to add protocols.
Note: Older protocols are not as secure as TLSv1.2.
Cipher Suites Uses a default cipher suite for the SSL/TLS handshake. To use a different cipher suite, clear this option.
Cipher Suites Cipher suites to use. To use a cipher suite that is not a part of the default set, click the Add icon and enter the name of the cipher suite. You can use simple or bulk edit mode to add cipher suites.

Enter the Java Secure Socket Extension (JSSE) name for the additional cipher suites that you want to use.