Enabling HTTPS

To secure communications to the Control Hub UI and REST API, enable both Control Hub and the separate Admin tool to use HTTPS. HTTPS requires an SSL/TLS certificate.

By default, both Control Hub and the Admin tool use HTTP. StreamSets recommends using HTTPS in a production environment. HTTPS requires SSL/TLS certificates.

Prerequisites

Before you enable HTTPS for Control Hub, complete the following requirements:

Obtain access to OpenSSL and Java keytool
If you do not have a keystore file that includes an SSL/TLS certificate signed by a certificate authority (CA), you can request a certificate and create the keystore file using the following tools:
  • OpenSSL - Use OpenSSL to create a Certificate Signing Request (CSR) that you send to the CA of your choice, as well as to create the keystore and truststore files. For more information, see the OpenSSL documentation.
  • Java keytool - You can also use Java keytool to create a CSR and to create keystore and truststore files. Java keytool is part of the Java Development Kit (JDK). For more information, see the keytool documentation.
Generate an SSL/TLS certificate and private key signed by a certificate authority (CA)
Control Hub provides a self-signed certificate that you can use. However, web browsers generally issue a warning for self-signed certificates. StreamSets strongly recommends that you generate a private key and public certificate signed by a trusted CA.
Important: The signed certificate must include the Control Hub fully qualified domain name (FQDN).
To obtain a certificate from a trusted CA, you must provide proof that you are the owner of the domain name for which you are requesting the certificate. Use OpenSSL or keytool to generate a key pair and then submit a Certificate Signing Request (CSR) to the CA. The exact procedure depends on the CA that you choose to use - see the documentation provided by the CA.

Step 1. Create a Keystore File

Create a keystore file that includes the private key and public certificate signed by the CA. A keystore is used to verify the identity of the client upon a request from an SSL/TLS server.

StreamSets recommends creating a keystore in the PKCS #12 (p12 file) format. In most cases, a CA issues certificates in PEM format. Use OpenSSL to directly import the certificate into a PKCS #12 keystore.

  1. Use the following command to import the certificate and private key issued in PEM format to a PKCS #12 keystore: 
    openssl pkcs12 -export -in <PEM certificate> -inkey <private key> -out <keystore filename> -name <keystore name>

    You will be prompted to create a password for the keystore file.

    For example, the following command converts the certificate mycompany.pem and private key mycompany.key to the PKCS #12 keystore file named mycompany.p12:
    openssl pkcs12 -export -in mycompany.pem -inkey mycompany.key -out mycompany.p12 -name mycompany
  2. Store the keystore file in the Control Hub configuration directory, $DPM_CONF.
    For example, for an RPM installation of Control Hub, we'd store our mycompany.p12 keystore file in the following directory:
    /etc/dpm/mycompany.p12
  3. Store the keystore password in the default password text file used by Control Hub, $DPM_CONF/keystore-password.txt.

Step 2. Create a Truststore File (Optional)

A truststore file contains certificates from trusted CAs that an SSL/TLS client uses to verify the identity of an SSL/TLS server.

By default, Control Hub uses the Java truststore file located in $JAVA_HOME/jre/lib/security/cacerts. If your certificate is signed by a trusted CA that is included in the default Java truststore file, you do not need to create a truststore file and can skip this step.

If your certificate was signed by a private CA or not trusted by the default Java truststore, you must create a custom truststore file or modify a copy of the default Java truststore file to add the root and intermediate CA certificates to the truststore file. For example, if your organization generates its own certificates, you must add the root and intermediate certificates for your organization to the truststore file.

In these steps, we show how to modify a copy of the default truststore file to add an additional CA to the list of trusted CAs. If you prefer to create a custom truststore file, see the keytool documentation.

You can create the following types of truststores for Control Hub:
  • Java keystore file (JKS)
  • PKCS #12 (p12 file)
Note: If you've already configured Control Hub to use a custom truststore file for secure connections to an LDAP server, then simply add this additional CA to the same modified truststore file that you created in Step 2. Configure Secure Connections for LDAP.
  1. Use the following command to set the JAVA_HOME environment variable: 
    export JAVA_HOME=<Java home directory>
  2. Use the following command to set the DPM_CONF environment variable: 
    export DPM_CONF=<Control Hub configuration directory>

    For example, for an RPM installation use:

    export DPM_CONF=/etc/dpm
  3. Use the following command to copy the default Java truststore file to the Control Hub configuration directory: 
    cp "${JAVA_HOME}/jre/lib/security/cacerts" "${DPM_CONF}/truststore.jks"
  4. Use the following keytool command to import the CA certificate into the truststore file: 
    keytool -import -file <CA certificate> -trustcacerts -noprompt -alias <CA alias> -storepass <password> -keystore "${DPM_CONF}/truststore.jks"
    For example:
    keytool -import -file  mycompany.pem -trustcacerts -noprompt -alias MyCorporateCA -storepass changeit -keystore "${DPM_CONF}/truststore.jks"

Step 3. Configure Control Hub to Use HTTPS

Modify the Control Hub configuration files to configure Control Hub and the Admin tool to use a secure port and your keystore file. If you created a custom truststore file or modified a copy of the default Java truststore file, configure Control Hub to use that truststore file.

  1. To define the secure port and keystore file, configure the following properties in the Control Hub configuration file, $DPM_CONF/dpm.properties:
    Control Hub HTTPS Property Description
    http.port Set to -1 to enable the HTTPS protocol.
    https.port Secure port number for Control Hub. For example, 19631.

    Any number besides -1 enables the secure port number.
    https.keystore.path Path and name of the keystore file that contains the private key and public certificate signed by a CA. Enter an absolute path or a path relative to the $DPM_CONF directory.

    In our example, we'd configure the property as follows:

    https.keystore.path=mycompany.p12
    Note: Default is keystore.jks in the $DPM_CONF directory which provides a self-signed certificate that you can use. However, StreamSets strongly recommends that you generate a certificate signed by a trusted CA, as described in Prerequisites.
    https.keystore.password Password to open the keystore file. To protect the password, store the password in an external location and then use a function to retrieve the password.
    In our example, we added the password to the default password text file used by Control Hub. So we'd configure the property as follows:
    https.keystore.password=${file("keystore-password.txt")}
  2. In the $DPM_CONF/common-to-all-apps.properties file, modify the dpm.base.url property to use the HTTPS protocol and the same secure port that you just defined.
    For example, if you defined the Control Hub secure port number as 19631, define the Control Hub base URL property as follows:
    dpm.base.url=https://sch.acme.com:19631
  3. For an installation started as a service on operating systems that use the systemd init system, edit the dpm.socket file to use the same secure port that you just defined.
    The location of the dpm.socket file depends on how you installed Control Hub:
    • From the RPM package - /usr/lib/systemd/system/dpm.socket
    • From the tarball - etc/systemd/system/dpm.socket
    For example, if you defined the Control Hub secure port number as 19631, modify these lines in the file as follows:
    [Socket]
ListenStream=19631
ListenStream=0.0.0.0:19631
  4. Use the following command to reload the systemd manager configuration: 
    systemctl daemon-reload
  5. To define the secure port and the keystore file for the Admin tool, configure the following properties in the Control Hub configuration file, $DPM_CONF/dpm.properties:
    Admin Tool HTTPS Property Description
    admin.http.port Set to -1 to enable the HTTPS protocol for the Admin tool.
    admin.https.port Secure port number for the Admin tool, unique from the port number defined for Control Hub. For example, 19632.

    Any number besides -1 enables the secure port number.
    admin.https.keystore.path Path and name of the keystore file that contains the private key and public certificate signed by a CA. Use the same keystore file that you specified above for Control Hub.
    admin.https.keystore.password Password to open the keystore file. Use the same password that you specified above for Control Hub.
  6. Optionally if you created a custom truststore file or modified a copy of the default Java truststore file, define the following options in the DPM_JAVA_OPTS environment variable:
    • javax.net.ssl.trustStore - Path to the truststore file on the Control Hub machine.
    • javax.net.ssl.trustStorePassword - Truststore password.

    Modify environment variables in the file required by your installation type.

    For example, define the options as follows:
    export DPM_JAVA_OPTS="${DPM_JAVA_OPTS} -Djavax.net.ssl.trustStore=/etc/dpm/truststore.jks -Djavax.net.ssl.trustStorePassword=mypassword 
-server -Xmx1024m -Xms1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./dpm-$$-outOfMem.hprof -XX:ErrorFile=./dpm-$$-error.log"

    Or to avoid saving the password in the export command, save the password in a text file and then define the truststore password option as follows: -Djavax.net.ssl.trustStorePassword=$(cat passwordfile.txt)

  7. Run the Control Hub security command to regenerate the Control Hub SAML metadata with the updated URL.

    To configure SAML authentication for an organization, users must download the Control Hub SAML metadata and keys, and then use that information to register Control Hub as a service provider with an IdP. The Control Hub URL is included in that metadata.

    Run the following command from the $DPM_HOME directory:

    bin/streamsets dpmcli security createSamlConfig -u <system administrator> -p <password>

    Where <system administrator> is the Control Hub system administrator account.

  8. Shut down and then launch Control Hub again to enable the changes.

Step 4. Configure the Truststore File for Clients (Optional)

If the Control Hub SSL/TLS certificate was signed by a private CA, then you also must configure Control Hub clients - registered Data Collectors and Transformers - to use a modified truststore file that includes the certificate for the private CA.

By default, both Data Collector and Transformer use the Java truststore file located in $JAVA_HOME/jre/lib/security/cacerts. If the certificate generated for Control Hub was signed by a trusted CA that is included in this truststore file, you can skip this step.

Configuring the Truststore for Data Collectors

Data Collector can use truststore files in the JKS or PKCS #12 format.

  1. Store the truststore file that you created in Step 2. Create a Truststore File (Optional) on each Data Collector machine that you plan to register with Control Hub.
    For example, store the file in the $SDC_CONF directory on each Data Collector machine.
  2. To configure Data Collector to use the truststore file, define the following options in the SDC_JAVA_OPTS environment variable:
    • javax.net.ssl.trustStore - Path to the truststore file stored on the Data Collector machine.
    • javax.net.ssl.trustStorePassword - Truststore password.

    Modify environment variables using the method required by your installation type.

    For example, define the options as follows:
    export SDC_JAVA_OPTS="-Djavax.net.ssl.trustStore=/etc/sdc/truststore.jks -Djavax.net.ssl.trustStorePassword=mypassword 
-Xmx1024m  -Xms1024m -server ${SDC_JAVA_OPTS}"

    Or to avoid saving the password in the export command, save the password in a text file and then define the truststore password option as follows: -Djavax.net.ssl.trustStorePassword=$(cat passwordfile.txt)

  3. Restart Data Collector to enable the changes.

Configuring the Truststore for Transformers

Transformer can use truststore files in the JKS or PKCS #12 format.

  1. Store the truststore file that you created in Step 2. Create a Truststore File (Optional) on each Transformer machine that you plan to register with Control Hub.
    For example, store the file in the $TRANSFORMER_CONF directory on each Transformer machine.
  2. To configure Transformer to use the truststore file, uncomment and define the following properties in the Transformer configuration file, $TRANSFORMER_CONF/transformer.properties:
    Transformer Property Description
    https.truststore.path Path to the truststore file stored on the Transformer machine.
    https.truststore.password Truststore password.
  3. Restart Transformer to enable the changes.