Upgrading a Highly Available Environment

For a highly available production environment, upgrade the additional Control Hub instances and update the load balancer used by Control Hub.

Note: You can run a single Control Hub instance in a development environment. If you are upgrading a development environment, you can skip this section.

Step 1. Upgrade Additional Control Hub Instances

When you upgrade additional Control Hub instances for a highly available environment, use the shortened upgrade process described here.

Important: The minimum requirements for Control Hub can change with each version. Before you upgrade to a new Control Hub version, verify that the machine meets the latest minimum requirements as described in Installation Requirements.
  1. Install the new version of Control Hub on the same machine as the previous version or on a separate machine.

    Use one of the following installation methods to install the new version of

    Control Hub:
  2. If you installed the RPM package on the same machine as the previous version, rename the previous and new versions of the configuration files.

    When you install the new RPM package on the same machine as the previous version, the configuration files are written to the same default directory as the previous version, /etc/dpm. The new versions of the configuration files are renamed with the following extension: .rpmnew. For example, the new version of the Control Hub configuration file is renamed to dpm.properties.rpmnew.

    1. In the working $DPM_CONF directory, /etc/dpm by default, rename all previous configuration files with the following extension: .old.
    2. Remove the following extension from all new configuration files: .rpmnew.
  3. Download the JDBC driver for the relational database instance that you are using:
  4. Copy the driver to the following directory:

    $DPM_HOME/extra-lib

    For example, copy the driver to the following directory in an RPM installation:

    /opt/streamsets-dpm/extra-lib

  5. Set the DPM_HOME and DPM_CONF environment variables.
    • Use the following command to set the DPM_HOME environment variable:
      export DPM_HOME=<home directory>

      For example:

      export DPM_HOME=/opt/streamsets-dpm
    • Use the following command to set the DPM_CONF environment variable:
      export DPM_CONF=<configuration directory>

      For example:

      export DPM_CONF=/etc/dpm
  6. Copy all files from the $DPM_CONF directory in the initial Control Hub instance that you upgraded to the $DPM_CONF directory in this additional Control Hub instance.
    By copying all configuration files, you ensure that this additional Control Hub instance connects to the same load balancer, databases, SMTP account, and system Data Collector as the initially upgraded Control Hub instance.
  7. Update the configuration file for each Control Hub application to define the same unique component ID used by the previous version. Each Control Hub instance of the same version must use a unique component ID.

    We recommend configuring the new version to use the same component ID as the previous version. However, you can configure the new version to use a different component ID. For example, if you install the new version on a different machine and your policy is to set the component ID to the IP address of the machine, you'd want to configure a different component ID for the new version. For example: <application name>199.57.90.24.

    Modify the value of the dpm.componentId property in these files located in the Control Hub configuration directory, $DPM_CONF:
    • connection-app.properties
    • dpm.properties
    • dynamic_preview-app.properties
    • jobrunner-app.properties
    • messaging-app.properties
    • notification-app.properties
    • pipelinestore-app.properties
    • policy-app.properties
    • provisioning-app.properties
    • reporting-app.properties
    • scheduler-app.properties
    • sdp_classification-app.properties
    • security-app.properties
    • sla-app.properties
    • timeseries-app.properties
    • topology-app.properties
  8. Modify and then run the security script to generate a unique authentication token for each Control Hub application.
    1. In the command prompt, set the DPM_ADMIN_USER and DPM_ADMIN_PASSWORD environment variables.
      • Use the following command to set the DPM_ADMIN_USER environment variable:
        export DPM_ADMIN_USER=<user name>

        For example:

        export DPM_ADMIN_USER=admin@admin
      • Use the following command to set the DPM_ADMIN_PASSWORD environment variable:
        export DPM_ADMIN_PASSWORD=<password>

        For example:

        export DPM_ADMIN_PASSWORD=mypassword
    2. Use a text editor to modify the $DPM_HOME/dev/02-initsecurity.sh script to comment out the last line in the script as follows: 
      # create SAML configuration
# "${DPM_DIST}/bin/streamsets" dpmcli security createSamlConfig -u "${DPM_ADMIN_USER}" -p "${DPM_ADMIN_PASSWORD}"
    3. Save and close the script.
    4. Use the following command to run the security script from the $DPM_HOME directory: 
      dev/02-initsecurity.sh <component ID>
      For example, if you defined the component ID for this installation as <application name>002, use the following command:
      dev/02-initsecurity.sh 002

      You do not need to specify the default component ID of 000.

  9. If you installed the new version of Control Hub on a separate machine, make sure that the Control Hub instance can access the front end of the load balancer to communicate with the other Control Hub instances.
Repeat these steps for each additional Control Hub instance that you upgrade.

Step 2. Update the Load Balancer

If you are upgrading multiple Control Hub instances in a highly available production environment, update the load balancer for Control Hub.

Configure the following information to update the load balancer:
Control Hub backend definition
If upgrading from version 2.7.x, define the following new backend definitions for Control Hub:
/dynamic_preview
/policy
/provisioning
/reporting
/scheduler
/sdp_classification
If upgrading from version 3.0.x or 3.1.x, define the following new backend definitions for Control Hub:
/dynamic_preview
/policy
/reporting
/scheduler
/sdp_classification
If upgrading from version 3.2.x or 3.3.x, define the following new backend definitions for Control Hub:
/dynamic_preview
/policy
/sdp_classification
If upgrading from version 3.12.x, define the following new backend definition for Control Hub:
/dynamic_preview
IP address and port numbers for each Control Hub instance if they differ from the previous version
If you installed the new version of Control Hub on separate machines or if you changed the port numbers during the upgrade process, update the IP address and port numbers for each Control Hub instance. If needed, configure both of the following port numbers for each Control Hub instance:
  • Control Hub port number. Default is 18631.
  • Control Hub Admin tool port number. Default is 18632.

After you finish updating the load balancer, reload the updated configuration.