Completing Post Upgrade Tasks

In some situations, you must complete tasks within Control Hub after you upgrade.

Upgrade to MySQL 8.x

Starting with version 3.56.1, Control Hub supports MySQL version 8.x for the relational database that stores metadata written by Control Hub applications.

Although Control Hub continues to support MySQL 5.6 and 5.7, these earlier MySQL versions have reached end of life. After you upgrade Control Hub to version 3.56.x, you might consider upgrading to MySQL 8.x as a post-upgrade task.

To upgrade to MySQL 8.x, complete the following general steps:

  1. Upgrade all Control Hub instances while still using MySQL 5.6 or 5.7.

    The upgrade process renames several database columns that are required to support MySQL 8.x.

  2. Upgrade the MySQL 5.6 or 5.7 databases to version 8.x. For instructions, see the MySQL documentation.
  3. Modify the following properties in the Control Hub application configuration files to use the connection URL, user name, and password of the MySQL 8.x database:
    • db.openjpa.ConnectionURL
    • db.openjpa.ConnectionUserName
    • db.openjpa.ConnectionPassword

    Modify the property values in all of the following files located in the Control Hub configuration directory, $DPM_CONF:

    • connection-app.properties
    • dpm.properties
    • dynamic_preview-app.properties
    • jobrunner-app.properties
    • messaging-app.properties
    • notification-app.properties
    • pipelinestore-app.properties
    • policy-app.properties
    • provisioning-app.properties
    • reporting-app.properties
    • scheduler-app.properties
    • sdp_classification-app.properties
    • security-app.properties
    • sla-app.properties
    • timeseries-app.properties
    • topology-app.properties
  4. Restart Control Hub for the changes to take effect.

Disable Logging for Non-Delivered Events

Control Hub version 3.51.0 includes a known issue where the logging of non-delivered events might cause out-of-memory errors to occur when the Messaging application retains a large number of requests that have not been retrieved.

To avoid these out-of-memory errors, StreamSets recommends disabling this logging by setting the non.delivered.events.checker.task.enabled property to false in the $DPM_CONF/messaging-app.properties configuration file.

This issue will be fixed in a future release.

Assign New Roles to Upgraded Users

Control Hub introduced the following new roles in these versions:
  • 3.0.0 introduced the Provisioning Operator role.
  • 3.2.0 introduced the Reporting Operator and Scheduler Operator roles.
  • 3.5.0 introduced the Classification Administrator and Policy Manager roles.
  • 3.10.0 introduced the Control Hub Authentication role.
  • 3.19.0 introduced the Connection Editor and Connection User roles.

Before upgraded users can complete the tasks that require these roles, you must assign the appropriate new roles to their user accounts or to the groups to which they belong. Users and groups are not assigned new roles during the upgrade process.

For a description of each role, see Role Descriptions.

  1. In the Navigation panel, click Administration > Groups or Administration > Users.
  2. Click the name of a group or user.
  3. Select the required new roles as needed, and then click Save.

Update Provisioning Agents

Due to an important security fix, StreamSets recommends that you update all existing Provisioning Agents to use the StreamSets Control Agent Docker image version 3.17.0 or later.

If an existing Provisioning Agent uses latest as the Control Agent Docker image version and the imagePullPolicy attribute in the Provisioning Agent YAML specification is set to Always, redeploy the Provisioning Agent so that it is automatically updated to use version 3.17.0.

If an existing Provisioning Agent uses a specific Control Agent Docker image version, update the Provisioning Agent YAML specification file to use the Control Agent Docker image version 3.17.0 or to use latest. For more information, see Applying Changes to Provisioning Agents.

Update Deployments for Provisioned Data Collectors

Starting with the StreamSets Control Agent Docker image version 3.10.0, the Control Agent requires that each YAML specification file that defines a deployment use the Kubernetes API version apps/v1. Previously, the Control Agent Docker image required that each YAML specification file use the API version extensions/v1beta1. Kubernetes has deprecated the extensions/v1beta1 version.

If you upgrade a Provisioning Agent that uses a Control Agent Docker image version earlier than 3.10.0, you must update all deployment YAML specification files to use apps/v1 before you redeploy the Provisioning Agent.

To upgrade a Provisioning Agent and then update deployments, complete the following steps:

  1. Stop all deployments.
    1. In the Navigation panel, select Execute > Deployments.
    2. Select each deployment, and then click the Stop icon.
  2. Upgrade the Provisioning Agent to use the Control Agent Docker image version 3.17.0 or later. If using latest as the Control Agent Docker image version, you can skip this step.
    1. Modify the YAML specification file that defines the Provisioning Agent to use the 3.17.0 or later image of the StreamSets Control Agent.
    2. Run the appropriate Helm or Kubernetes command to apply the changes to the running Provisioning Agent application in the Kubernetes pod.

      For more information, see Applying Changes to Provisioning Agents.

  3. Update all deployments.
    1. In the Navigation panel, select Execute > Deployments.
    2. Select a deployment, and then click the Edit icon.
    3. In the Edit Deployment page, modify the API version to use for the deployment in the YAML Specification property as follows:
      apiVersion: apps/v1

      Modify the YAML specification so that it meets all other requirements to create a deployment using the Kubernetes API version apps/v1. For example, the apps/v1 version requires that you define a selector attribute for the deployment. For more information, see the Kubernetes documentation.

    4. Save and then restart the deployment.

Update the Data Collector Version Range

By default, StreamSets Control Hub can work with registered Data Collectors from version 2.1.0.0 to the current version of Control Hub. If you want Control Hub to work with later versions of Data Collector, you must update the Data Collector version range.

For example, if you upgrade Control Hub on-premises to version 3.3.0 and you upgrade registered Data Collectors to version 3.5.0, you must update the maximum Data Collector version that can work with Control Hub. As a best practice, configure the maximum Data Collector version to 3.5.10 to ensure that Data Collector upgrades to later minor versions, such as 3.5.1 or 3.5.2, will continue to work with Control Hub.

You can specify official Data Collector release versions or Snapshot versions installed from a nightly build.
  1. Log in to Control Hub as the system administrator.
  2. In the Navigation panel, click Administration > Data Collectors.
  3. Click the Component Version Range icon: .
  4. Enter the minimum and maximum Data Collector versions that can work with Control Hub.
  5. If you enter a Snapshot version for the minimum version, enter the oldest acceptable build date for that version using the following format:
    yyyy-MM-dd'T'HH:mm'Z'
    For example:
    2016-08-22T06:30Z

    Enter ANY to allow any build date from the specified Snapshot version.

    Control Hub checks the build date only when the Data Collector version matches the specified minimum version.

Modify the System Sample Pipeline Organization

With version 3.0.0, you can configure another organization besides the default system organization with an ID of admin to manage system sample pipelines. If you modified the organization that manages system sample pipelines, configure the new Control Hub version to use the same organization to manage system sample pipelines.

Note: If you are upgrading from version 2.7.x or if you did not modify the system sample pipeline organization, you can skip this task.
Modify the following property in the $DPM_CONF/pipelinestore-app.properties file to use the same organization as the previous version:
pipeline.templates.organization

Restart Control Hub for the changed property to take effect.

For more information, see System Sample Pipelines.

Delete the Previous Applications

During the upgrade process, if you configured the upgraded Control Hub applications to use a different component ID from the previous version, delete the previous Control Hub applications from the Control Hub UI.

Note: If you configured the upgraded Control Hub applications to use the same component ID as the previous version, you can skip this task.
  1. In the Navigation panel, click Administration > SCH Apps.
  2. Select the previous Control Hub applications, and then click the Deactivate icon: .
  3. With the previous Control Hub applications still selected, click the Delete icon: .