Upgrading Engines

Once you save a deployment, you cannot change the engine version. As a result, to upgrade to a newer engine version, you must create a new deployment. At this time, you must manually configure the new deployment to use the same options as the previous deployment.

The steps that you use to upgrade an engine version depend on following deployment types:

Upgrading Engines for Self-Managed Deployments

Use the following steps to upgrade an engine version for a self-managed deployment.

Important: Before upgrading, verify that the machines that run the engine instances meet the installation requirements for the latest engine version. The minimum requirements for an engine can change with each version. See the minimum Data Collector requirements or the minimum Transformer requirements.
  1. Create a new self-managed deployment that uses the new engine version.
    1. Configure the remaining deployment properties to use the same values as the previous deployment.
      Important: By default, Control Hub creates an engine label using the name of the deployment. Be sure to delete the default label and then define the same engine labels as the previous deployment. That way, existing jobs can run pipelines on engines managed by this new deployment.
    2. In the Review & Launch step of the New Deployment window, click Exit.
  2. Stop all jobs running pipelines on engines managed by the previous deployment.
  3. Shut down all engine instances managed by the previous deployment.
  4. Stop the previous deployment.
  5. Start the new deployment.
  6. Launch engine instances for the new deployment.
  7. Start all jobs that were running on engines managed by the previous deployment.

    As long as you assign the same labels to the new deployment, the jobs will start on engines managed by the new deployment.

  8. Optionally, delete the previous deployment.

Upgrading Engines for Control Hub-Managed Deployments

Use the following steps to upgrade an engine version for a Control Hub-managed deployment, such as an Amazon EC2 or Kubernetes deployment.

  1. For the same parent environment, create a new Control Hub-managed deployment that uses the new engine version.
    1. Configure the remaining deployment properties to use the same values as the previous deployment.
      Important: By default, Control Hub creates an engine label using the name of the deployment. Be sure to delete the default label and then define the same engine labels as the previous deployment. That way, existing jobs can run pipelines on engines managed by this new deployment.
    2. In the Review & Launch step of the New Deployment window, click Launch Deployment to provision resources in the external system and launch StreamSets engine instances on those resources.
  2. Stop the previous deployment.

    As long as you assign the same labels to the new deployment and the jobs are enabled for pipeline failover, the jobs will start on engines managed by the new deployment.

    For Data Collector jobs that are not enabled for pipeline failover, synchronize the jobs so that the jobs start on the new Data Collector instances.

    For Transformer jobs that are not enabled for pipeline failover, stop all jobs running pipelines on engines managed by the previous deployment. Then start the jobs again so that the jobs start on the new Transformer instances.

  3. Optionally, delete the previous deployment.