Security in Kafka Stages

You can configure the Kafka origin and destination to use one of the following options to connect securely to Kafka:

The SASL authentication options provide two SASL mechanisms: PLAIN or GSSAPI (Kerberos).

Enabling security requires completing several prerequisite tasks in addition to configuring security properties in the stage.

Note: At this time, you cannot enable Kafka security when the pipeline runs on a Databricks cluster.

Prerequisite Tasks

Before enabling security for a Kafka stage, complete the following prerequisite tasks for the security method that you want to use:

SSL/TLS
Complete the following prerequisite tasks before using SSL/TLS to connect to Kafka:
  • Make sure Kafka is configured for SSL/TLS as described in the Kafka documentation.
  • Store the SSL truststore and keystore files in the same location on the Transformer machine and on each node in the Spark cluster.
SASL with the PLAIN mechanism
Complete the following prerequisite tasks before using SASL with the PLAIN mechanism to connect to Kafka:
  • Make sure Kafka is configured for SASL authentication with the PLAIN mechanism as described in the Kafka documentation.
  • Define the user name and password credentials in a JAAS configuration file, as described in Providing PLAIN Credentials.
  • Store the JAAS configuration file in the same location on the Transformer machine and on each node in the Spark cluster.
SASL with the GSSAPI (Kerberos) mechanism

Complete the following prerequisite tasks before using SASL with the GSSAPI (Kerberos) mechanism to connect to Kafka:

  • Make sure Kafka is configured for SASL authentication with the GSSAPI (Kerberos) mechanism as described in the Kafka documentation.
  • For pipelines that run on a Hadoop YARN cluster configured for Kerberos, make sure that Kerberos authentication is enabled for Transformer.
  • Define the Kerberos credentials in a JAAS configuration file, as described in Providing Kerberos Credentials.
  • Store the JAAS configuration and Kafka keytab files in the same location on the Transformer machine and on each node in the Spark cluster.

SASL Authentication Credentials

When using SASL authentication to connect to Kafka, you must provide credentials in a Java Authentication and Authorization Service (JAAS) file. The contents of the JAAS file depend on whether you use the PLAIN or GSSAPI (Kerberos) SASL mechanism.

Providing PLAIN Credentials

To connect to Kafka using SASL authentication with the PLAIN mechanism, provide the credentials in a Java Authentication and Authorization Service (JAAS) file.

Create a JAAS file on the Transformer machine. You can define a single JAAS file for Transformer. As a result, every Kafka connection in every pipeline that uses SASL authentication with the PLAIN mechanism uses the same credentials.

Add the configuration properties required for Kafka clients based on whether Transformer uses LDAP authentication:
LDAP authentication disabled
If Transformer does not use LDAP authentication, create a separate JAAS configuration file on the Transformer machine. Add the following KafkaClient login section to the file:
KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required 
    username="<username>" 
    password="<password>";
};
Then modify the TRANSFORMER_JAVA_OPTS environment variable to include the following option that defines the path to the JAAS configuration file:
-Djava.security.auth.login.config=<JAAS config path>/kafka_client_jaas.conf

Modify environment variables using the method required by your installation type.

LDAP authentication enabled
If LDAP authentication is enabled, add the properties to the JAAS configuration file used by Transformer - the $TRANSFORMER_CONF/ldap-login.conf file. Add the following KafkaClient login section to the end of the ldap-login.conf file:
KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required 
    username="<username>" 
    password="<password>";
};

Providing Kerberos Credentials

To connect to Kafka using SASL authentication with the GSSAPI (Kerberos) mechanism, provide the Kerberos credentials in a JAAS configuration file.

Create a JAAS file on the Transformer machine. You can define a single JAAS file for Transformer. As a result, every Kafka connection in every pipeline that uses SASL authentication with the Kerberos mechanism uses the same credentials.

Add the configuration properties required for Kafka clients based on whether Transformer uses LDAP authentication:
LDAP authentication disabled
If Transformer does not use LDAP authentication, create a separate JAAS configuration file on the Transformer machine. Add the following KafkaClient login section to the file:
KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="<keytab path>"
    principal="<principal name>/<host name>@<realm>";
};
For example:
KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/etc/security/keytabs/kafka_client.keytab"
    principal="kafka/node-01.cluster@EXAMPLE.COM";
};
Then modify the TRANSFORMER_JAVA_OPTS environment variable to include the following option that defines the path to the JAAS configuration file:
-Djava.security.auth.login.config=<JAAS config path>/kafka_client_jaas.conf

Modify environment variables using the method required by your installation type.

LDAP authentication enabled
If LDAP authentication is enabled, add the properties to the JAAS configuration file used by Transformer - the $TRANSFORMER_CONF/ldap-login.conf file. Add the following KafkaClient login section to the end of the ldap-login.conf file:
KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="<keytab path>"
    principal="<principal name>/<host name>@<realm>";
};
For example:
KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/etc/security/keytabs/kafka_client.keytab"
    principal="kafka/node-01.cluster@EXAMPLE.COM";
};

Enabling SSL/TLS Encryption

When the Kafka cluster uses the Kafka SSL security protocol, enable the Kafka stage to use SSL/TLS encryption.

Before you enable Kafka stages to use SSL/TLS, make sure that you have performed all necessary prerequisite tasks. Then, perform the following steps to enable the Kafka stages to use SSL/TLS to connect to Kafka.

  1. On the Kafka tab of the stage, configure each Kafka broker URI to use the SSL/TLS port.

    The default SSL/TLS port number is 9093.

  2. On the Security tab, configure the following properties:
    Security Property Description
    Security Option Set to SSL/TLS Encryption (Security Protocol=SSL).
    Truststore Type Type of truststore to use. Use one of the following types:
    • Java Keystore File (JKS)
    • PKCS #12 (p12 file)

    Default is Java Keystore File (JKS).
    Truststore File Absolute path to the truststore file stored in the same location on the Transformer machine and on each node in the cluster.

    For example, you might enter the following path: /var/private/ssl/kafka.client.truststore.jks
    Truststore Password Password to the truststore file.
    Tip: To secure sensitive information, you can use credential stores or runtime resources.
    Enabled Protocols Comma-separated list of protocols used to connect to the Kafka brokers. Ensure that at least one of these protocols is enabled in the Kafka brokers.
    Note: Older protocols are not as secure as TLSv1.2.

Enabling SSL/TLS Encryption and Authentication

When the Kafka cluster uses the Kafka SSL security protocol and requires client authentication, enable the Kafka stage to use SSL/TLS encryption and authentication.

Before you enable a Kafka stage to use SSL/TLS encryption and authentication, make sure that you have performed all necessary prerequisite tasks. Then, perform the following steps to enable the stage to use SSL/TLS encryption and authentication to connect to Kafka.

  1. On the Kafka tab of the stage, configure each Kafka broker URI to use the SSL/TLS port.

    The default SSL/TLS port number is 9093.

  2. On the Security tab, configure the following properties:
    Security Property Description
    Security Option Set to SSL/TLS Encryption and Authentication (Security Protocol=SSL).
    Truststore Type Type of truststore to use. Use one of the following types:
    • Java Keystore File (JKS)
    • PKCS #12 (p12 file)

    Default is Java Keystore File (JKS).
    Truststore File Absolute path to the truststore file stored in the same location on the Transformer machine and on each node in the cluster.

    For example, you might enter the following path: /var/private/ssl/kafka.client.truststore.jks
    Truststore Password Password to the truststore file.
    Tip: To secure sensitive information, you can use credential stores or runtime resources.
    Keystore Type Type of keystore to use. Use one of the following types:
    • Java Keystore File (JKS)
    • PKCS #12 (p12 file)

    Default is Java Keystore File (JKS).
    Keystore File Absolute path to the keystore file stored in the same location on the Transformer machine and on each node in the cluster.

    For example, you might enter the following path: /var/private/ssl/kafka.client.keystore.jks
    Keystore Password Password to the keystore file.
    Key Password Password for the key in the keystore file.
    Tip: To secure sensitive information, you can use credential stores or runtime resources.
    Enabled Protocols Comma-separated list of protocols used to connect to the Kafka brokers. Ensure that at least one of these protocols is enabled in the Kafka brokers.
    Note: Older protocols are not as secure as TLSv1.2.

Enabling SASL Authentication

When the Kafka cluster uses the Kafka SASL_PLAINTEXT security protocol, enable the Kafka stage to use SASL authentication.

Before you enable Kafka stages to use SASL authentication, make sure that you have performed all necessary prerequisite tasks.

  1. Create a Java Authentication and Authorization Service (JAAS) file on the Transformer machine to provide the required credentials.

    The contents of the JAAS configuration file depend on whether you use the PLAIN or GSSAPI (Kerberos) SASL mechanism and depend on your Transformer authentication type. For details, see Providing PLAIN Credentials or Providing Kerberos Credentials.

  2. On the Security tab of the stage, configure the following properties:
    Security Property Description
    Security Option Set to SASL Authentication (Security Protocol=SASL_PLAINTEXT).
    SASL Mechanism SASL mechanism to use:
    • PLAIN
    • GSSAPI (Kerberos)
    Kerberos Service Name Kerberos service principal name that the Kafka brokers run as.

    Available when using the GSSAPI (Kerberos) mechanism.
  3. In the pipeline properties, on the Cluster tab, add the following additional configuration properties under Extra Spark Configuration:
    • spark.driver.extraJavaOptions
    • spark.executor.extraJavaOptions

    Set both properties to the path to the JAAS configuration file on the Transformer machine. For example:

Enabling SASL Authentication on SSL/TLS

When the Kafka cluster uses the SASL_SSL security protocol, enable the Kafka stage to use SASL authentication on SSL/TLS.

Before you enable Kafka stages to use SASL authentication on SSL/TLS, make sure that you have performed all necessary prerequisite tasks.

  1. Create a Java Authentication and Authorization Service (JAAS) file on the Transformer machine to provide the required credentials.

    The contents of the JAAS configuration file depend on whether you use the PLAIN or GSSAPI (Kerberos) SASL mechanism and depend on your Transformer authentication type. For details, see Providing PLAIN Credentials or Providing Kerberos Credentials.

  2. On the Kafka tab of the stage, configure each Kafka broker URI to use the SSL/TLS port.

    The default SSL/TLS port number is 9093.

  3. On the Security tab of the stage, configure the following properties:
    Security Property Description
    Security Option Set to SASL Authentication on SSL/TLS (Security Protocol=SASL_SSL).
    SASL Mechanism SASL mechanism to use:
    • PLAIN
    • GSSAPI (Kerberos)
    Kerberos Service Name Kerberos service principal name that the Kafka brokers run as.

    Available when using the GSSAPI (Kerberos) mechanism.
    Truststore Type Type of truststore to use. Use one of the following types:
    • Java Keystore File (JKS)
    • PKCS #12 (p12 file)

    Default is Java Keystore File (JKS).
    Truststore File Absolute path to the truststore file stored in the same location on the Transformer machine and on each node in the cluster.

    For example, you might enter the following path: /var/private/ssl/kafka.client.truststore.jks
    Truststore Password Password to the truststore file.
    Tip: To secure sensitive information, you can use credential stores or runtime resources.
    Enabled Protocols Comma-separated list of protocols used to connect to the Kafka brokers. Ensure that at least one of these protocols is enabled in the Kafka brokers.
    Note: Older protocols are not as secure as TLSv1.2.
  4. In the pipeline properties, on the Cluster tab, add the following additional configuration properties under Extra Spark Configuration:
    • spark.driver.extraJavaOptions
    • spark.executor.extraJavaOptions

    Set both properties to the path to the JAAS configuration file on the Transformer machine. For example:

Enabling Custom Authentication

To specify security requirements with custom properties, enable the Kafka stage to use custom authentication.

With custom authentication, you specify custom properties that contain the information required by the security protocol rather than using the properties in the stage. For example, you can enable custom authentication and then configure custom properties required for the SASL_SSL security protocol rather than enabling SASL Authentication on SSL/TLS.

Before enabling custom authentication, complete any necessary prerequisites for the security methods you are using, as described in the Kafka documentation. For example, if using SSL/TLS to connect to Kafka, you must make sure Kafka is configured for SSL/TLS.

To enable custom authentication, configure the following properties on the Security tab:
Security Property Description
Security Option Set to Custom Authentication (Security Protocol=CUSTOM).
Custom Security Properties Additional security properties to use. In simple or bulk edit mode, click the Add icon to add properties. Define the security property name and value.

Use the property names and values as expected for Kafka clients. For more information, see the Kafka documentation.

For example, to configure the SASL_SSL security protocol with the PLAIN mechanism, in Custom Security Properties you configure properties to set the protocol, to connect to the Java Authentication and Authorization Service (JAAS), and to use SSL/TLS:
Name Value
security.protocol SASL_SSL
sasl.mechanism PLAIN
sasl.jaas.config 
org.apache.kafka.common.security.plain.PlainLoginModule required \
   username="<user name for JAAS>"\
   password="<password>";
ssl.truststore.location <full path to the client truststore jks file>
ssl.truststore.password <password>