Register Transformer

To register Transformer with Control Hub, you generate an authentication token and modify the Transformer configuration files.

The method you use to register Transformer depends on the Transformer installation type:
Tarball installation
You can register Transformer from the Transformer UI, from the command line interface, or from Control Hub.
RPM installation
You must register Transformer from Control Hub.

A registered Transformer communicates with Control Hub at regular intervals. If Transformer cannot connect to Control Hub, due to a network or system outage, then Transformer uses the Control Hub disconnected mode.

Before you register Transformer, ensure that you have enabled HTTPS for Transformer.

Registering from Control Hub

When you register Transformer from Control Hub, you generate an authentication token. Then, you edit Transformer configuration files to register the token with Transformer and to enable communication with Control Hub.

When you register from Control Hub, you can generate multiple authentication tokens at one time.

  1. Log in to Control Hub using your Control Hub user account.
  2. In the Navigation panel, click Administration > Transformers.
    Control Hub displays the Transformers that have already been registered.
  3. Click the Generate Authentication Tokens icon .
  4. Enter the number of tokens to generate.
  5. Click Generate.
    The Authentication Tokens window displays each generated token.
  6. Record the generated tokens.
    You can copy the tokens from the window. Or, you can click Download to download all generated tokens to a JSON file named authTokens.json.
    Note: If you close the window before recording the tokens, you cannot retrieve the tokens. You can delete unregistered authentication tokens, as described in Deleting Unregistered Tokens.
  7. Click Close in the Authentication Tokens window.
  8. Complete the following steps for each Transformer that you want to register:
    1. Open the $TRANSFORMER_CONF/application-token.txt file for the Transformer, and copy a token into the file.
      Each Transformer must use a unique authentication token.
    2. Open the $TRANSFORMER_CONF/dpm.properties file for the Transformer and edit the following properties:
      Property Description
      dpm.enabled Set to true.
      dpm.base.url URL to access Control Hub:
      • For Control Hub cloud, set to https://cloud.streamsets.com.
      • For Control Hub on-premises, set to the Control Hub URL provided by your system administrator. For example, https://<hostname>:18631.
    3. Restart the Transformer.

Registering from Transformer

For a Transformer tarball installation, you can register Transformer with Control Hub using the Transformer UI.

Note: For a Transformer RPM installation, you must use Control Hub to register Transformer.

When you register from the Transformer UI, Transformer generates the authentication token and modifies the configuration files for you. You can also create Control Hub user accounts and groups during the registration process.

  1. Log in to Transformer using the following default credentials: admin/admin.
  2. Click Administration > Enable Control Hub.
  3. Enter the following information in the Enable StreamSets Control Hub window:
    Property Description
    Control Hub Base URL URL to access Control Hub:
    • For Control Hub cloud, set to https://cloud.streamsets.com.
    • For Control Hub on-premises, set to the Control Hub URL provided by your system administrator. For example, https://<hostname>:18631.
    Control Hub User ID Enter your Control Hub user ID using the following format:
    <ID>@<organization ID>
    Control Hub User Password Enter the password for your Control Hub user account.
    Labels for this Transformer Assign a label to this Transformer. Labels that you assign here are defined in the Control Hub configuration file, $TRANSFORMER_CONF/dpm.properties. To remove these labels after you register the Transformer, you must modify the configuration file.

    Default is all, which you can use to run a job on any of the registered Transformers.

    For more information about labels, see Labels Overview.
  4. Click Enable Control Hub.
  5. Optionally, you can choose to create a Control Hub user account and group for each Transformer user account and group.
    The Create Control Hub Groups and Users window maps all existing Transformer user accounts and groups to Control Hub user accounts and groups. You can remove users or groups and can edit the IDs, names, and email addresses as needed.

    When you have finished reviewing the users and groups to create, click Create. Each new user is assigned a default set of Control Hub roles. Groups are not assigned any roles. After you log in to Control Hub, change those role assignments as needed to secure the integrity of your organization and data.

  6. Restart Transformer.

Registering from the Command Line Interface

For a Transformer tarball installation, you can register Transformer with Control Hub using the command line interface.

Note: For a Transformer RPM installation, you must use Control Hub to register Transformer.

When you register Transformer from the command line interface, Transformer generates the authentication token and modifies the configuration files for you. Transformer must be running before you can use the command line interface.

Start Transformer, and then use the system enableDPM command to register the Transformer.

Use the command from the $TRANSFORMER_DIST directory as follows:

bin/streamsets cli \
(-U <sdcURL> | --url <sdcURL>) \
[(-D <dpmURL> | --dpmURL <dpmURL>)] \
[(-a <sdcAuthType> | --auth-type <sdcAuthType>)] \
[(-u <sdcUser> | --user <sdcUser>)] \
[(-p <sdcPassword> | --password <sdcPassword>)] \
system enableDPM \
(--dpmUrl <dpmBaseURL>) \
(--dpmUser <dpmUserID>) \
(--dpmPassword <dpmUserPassword>) \
[(--labels <labels>)]

When using the system enableDPM command, the following basic options are required:

Basic Option Description
-U <sdcURL>

or

--url <sdcURL>		 Required. URL of the Transformer.

The default URL is http://localhost:19630.
-D <dpmURL>

or

--dpmURL <dpmURL>

 Required. URL to access Control Hub:
  • For Control Hub cloud, set to https://cloud.streamsets.com.
  • For Control Hub on-premises, set to the Control Hub URL provided by your system administrator. For example, https://<hostname>:18631.

The following table describes the enableDPM options:

Enable DPM Option Description
--dpmUrl <dpmBaseURL> URL to access Control Hub:
  • For Control Hub cloud, set to https://cloud.streamsets.com.
  • For Control Hub on-premises, set to the Control Hub URL provided by your system administrator. For example, https://<hostname>:18631.
--dpmUser <dpmUserID> Required. Enter your Control Hub user ID using the following format:
<ID>@<organization ID>
--dpmPassword <dpmUserPassword> Required. Enter the password for your Control Hub user account.
--labels <labels> Required. Assign a label to this Transformer. You can enter multiple labels separated by commas. Labels that you assign here are defined in the Control Hub configuration file, $TRANSFORMER_CONF/dpm.properties. To remove these labels after you register the Transformer, you must modify the configuration file.

For more information about labels, see Labels Overview.
For example, the following command registers a Transformer with Control Hub and assigns three labels to the Transformer:
bin/streamsets cli -U http://localhost:19630 -D https://cloud.streamsets.com system enableDPM --dpmUrl https://cloud.streamsets.com --dpmUser alison@MyOrg --dpmPassword MyPassword --labels Finance,Accounting,Development

Restart Transformer to apply the changes.

Using a Publicly Accessible URL

If you register Transformer when it is installed on a cloud computing platform such as Amazon Elastic Compute Cloud (EC2), configure Transformer to use a publicly accessible URL.

When you register Transformer with Control Hub, Transformer sends its URL to Control Hub in the format http://<hostname>:<http.port>, where <hostname> is the value defined in the http.bindHost property in the Transformer configuration file, $TRANSFORMER_CONF/transformer.properties. If the host name is not defined in http.bindHost, Transformer runs the following command to determine the host name: hostname -f

For most cloud computing platforms, the hostname -f command returns the private IP address of the machine on the cloud platform. Control Hub includes the private IP address in the Transformer URL displayed in Control Hub. However, when you click the Transformer URL, you cannot access Transformer because you must use a public IP address to access a cloud machine.

To access Transformer installed on a cloud computing platform from Control Hub, uncomment the transformer.base.http.url property in the Transformer configuration file, $TRANSFORMER_CONF/transformer.properties, and then configure it to use the publicly accessible URL to that Transformer.

After modifying the configuration file, restart Transformer for the changes to take effect.

Using a Proxy Server

You can configure each registered Transformer to use an authenticated HTTP or HTTPS proxy server for outbound requests made to Control Hub. Define the proxy properties in the TRANSFORMER_JAVA_OPTS environment variable.

Modify environment variables using the method required by your installation type.

Add the following Java options to the TRANSFORMER_JAVA_OPTS environment variable:

  • https.proxyUser
  • https.proxyPassword
  • https.proxyHost
  • https.proxyPort

If the proxy server uses HTTP instead of HTTPS, use http.<property name> for each property.

For example, to configure a registered Transformer to use an HTTPS proxy server on host 138.0.0.1 and port 3138, define TRANSFORMER_JAVA_OPTS as follows: 

export TRANSFORMER_JAVA_OPTS="${TRANSFORMER_JAVA_OPTS} -Xmx1024m -Xms1024m -Dhttps.proxyUser=MyName -Dhttps.proxyPassword=MyPsswrd -Dhttps.proxyHost=138.0.0.1 -Dhttps.proxyPort=3138 -server"
Note: Oracle JDK disabled HTTP proxy authentication for HTTPS URLs in JDK 8 update 111. If Transformer runs on a machine with Java 8u111 or later, consider using an HTTPS proxy server. Or as a workaround, consider adding the following Java property to the TRANSFORMER_JAVA_OPTS environment variable, setting the property to an empty string:
-Djdk.http.auth.tunneling.disabledSchemes=''

However, use this workaround with caution since it exposes credentials by sending them through an unencrypted proxy.

Deleting Unregistered Tokens

Delete unregistered authentication tokens when you used Control Hub to generate the tokens, but did not copy or download the tokens from the Generate Authentication Token window.

Control Hub lists the number of unregistered authentication tokens in the Transformer Administration view as follows:

  1. In the Navigation panel, click Administration > Transformers.
  2. Click the Toggle Filter Column icon () to view the number of unregistered authentication tokens.
  3. Click the More icon , and then click Delete Unregistered Authentication Tokens.

Deactivating a Registered Transformer

Deactivate a registered Transformer when you want to temporarily prevent users from using that Transformer.

When you deactivate a Transformer, Control Hub deactivates the authentication token for the Transformer. Shut down the Transformer before you deactivate it. You cannot start a deactivated Transformer.

  1. In Control Hub, stop all jobs running on the Transformer.
  2. Log into Transformer and shut it down.
  3. In Control Hub, click Administration > Transformers in the Navigation panel.
  4. Select the registered Transformer to display its details.
  5. Click Deactivate.
    The Transformer Administration view uses a gray Transformer icon for any deactivated Transformer.
  6. To activate the Transformer again, view the Transformer details in Control Hub and click Activate. Then, start the Transformer.

Regenerating a Token

You can regenerate an authentication token for a Transformer. You might need to regenerate a token to replace a token that has been compromised or to follow your organization's security policy.

When you regenerate an authentication token for a Transformer, you replace the previous authentication token with a new one. You must copy the new token into the $TRANSFORMER_CONF/application_token.txt file for the Transformer.

  1. In the Navigation panel, click Administration > Transformers.
  2. Select a registered Transformer to display its details.
  3. Click Regenerate Authentication Token.
    The Authentication Tokens window displays the regenerated token.
  4. Record the regenerated token.
    You can copy the token from the window. Or, you can click Download to download the token to a JSON file named authTokens.json.
    Note: If you close the window before recording the token, you cannot retrieve the token. You can delete unregistered authentication tokens, as described in Deleting Unregistered Tokens.
  5. To register the Transformer with the newly generated token, copy the token to the $TRANSFORMER_CONF/application-token.txt file and restart the Transformer.