Managing Deployments

When you manage deployments, you can filter the list of deployments and start, stop, or edit deployments. When you edit a deployment, you can add additional stage libraries or edit the engine proxy properties.

After editing an active deployment to update the engine configuration, you must restart all engine instances belonging to the deployment for the changes to take effect.

When needed, you can delete a deployment. You can also share deployments with other users and groups, as described in Permissions.

To upgrade an engine version, you must create a new deployment.

Filtering Deployments

In the Deployments view, you can filter the list of displayed deployments by type, state, tag, or engine type. You can also search for deployments by name.

  1. In the Navigation panel, click Set Up > Deployments.
  2. If the Filter column does not display, click the Toggle Filter Column icon: .

    The following image displays the Filter column in the Deployments view:

  3. To search for deployments by name, enter text in the search field, and then press Return.
  4. Select a type, state, tag, or engine type to additionally filter the list of deployments.
  5. Select the Keep Filter Persistent checkbox to retain the filter when you return to the view.
    Tip: To share the applied filter, copy the URL and send it to another user in your organization.

Starting Deployments

Start a deployment when you want to launch engine instances for the deployment.

You can start a deployment when you create it, or you can start it at a later time.

When you start a new self-managed deployment, Control Hub generates the installation script that you run to install and launch engine instances. When you restart an existing self-managed deployment, you must manually restart each existing engine instance from the command line.

When you start a Control Hub-managed deployment, Control Hub connects to the external system represented by the parent environment and automatically provisions the resources needed to run the engine type, ensuring that the resources meet engine requirements. Engine instances are then automatically deployed and launched on those resources.

You can start a deployment that is in the Deactivated or Deactivation Error state.

  1. In the Control Hub Navigation panel, click Set Up > Deployments.
  2. Select the deactivated deployments, and then click the Start icon: .
  3. Click OK to confirm.
  4. For an existing self-managed deployment, restart each existing engine instance from the command line.

Stopping Deployments

Stop a deployment when you want to temporarily prevent engine instances that belong to that deployment from running.

When you stop a self-managed deployment, running engine instances belonging to that deployment are shut down so that they no longer use computing resources. After you restart the deployment, you must manually start each engine instance from the command line.

When you stop a Control Hub-managed deployment, all provisioned resources in the external system are deleted so that you no longer incur any costs from the cloud service provider for that deployment. For example, if you stop an Amazon EC2 deployment, the provisioned AWS CloudFormation stack, LaunchConfig template, and auto scaling group are deleted in addition to the EC2 instances that run StreamSets engines.
Note: You can also edit an active Control Hub-managed deployment to set the number of desired instances to 0 to temporarily prevent engine instances from running. In this case, the remaining resources, such as the AWS CloudFormation stack, LaunchConfig template, and auto scaling group provisioned for an Amazon EC2 deployment continue to exist and to incur minimal costs from the cloud service provider.
You can stop a deployment that meets the following conditions:
  • The deployment is in the Active or Activation Error state.
  • No engine instances belonging to the deployment are currently running pipelines.
  1. In the Control Hub Navigation panel, click Set Up > Deployments.
  2. Select the active deployments, and then click the Stop icon: .
  3. Click OK to confirm.

    It can take a few minutes for a deployment to stop. Self-managed deployments wait to stop until all running engine instances are successfully shut down. Control Hub-managed deployments wait to stop until all running engine instances are successfully shut down and all provisioned resources in the external system are deleted.

Editing Deployments

You can edit deployments when they are in any state except for the transient Activating and Deactivating states.

When you edit an active self-managed deployment that updates most engine configuration properties, Control Hub prompts you to restart the engine instances for the changes to take effect.

When you edit a Control Hub-managed deployment that updates most engine configuration properties, Control Hub might delete the provisioned instances in your cloud account and then provision new instances. For more information, see one of the following topics:
Important: You cannot edit an active deployment to update engine proxy properties. You first must stop the deployment to change the proxy properties.

You cannot change all properties when you edit a deployment. You must create a new deployment to change the following properties:

  • Deployment type
  • Environment that the deployment belongs to
  • Engine version

To edit a deployment, locate the deployment in the Deployments view. In the Actions column, click the More icon () and then click Edit.

Updating Stage Libraries

Add stage libraries to a deployment when you want to use functionality provided by the stage library, such as a stage or credential store.

Remove stage libraries from a deployment when you no longer needed the stage library functionality.

To update stage libraries, you edit the deployment, then restart associated engines. During the restart, the added stage libraries are installed on the engine and the removed stage libraries are uninstalled from the engine.

  1. In the Control Hub Navigation panel, click Set Up > Deployments.
  2. In the Actions column of the deployment, click the More icon () and then click Edit.
  3. To bypass the Define Deployment step, click Save & Next.
  4. In the Configure Engine step, click <number> stage libraries selected next to the Stage Libraries property.
  5. In the Select Stage Libraries window, select the stage libraries to add or remove in one of the following ways:
    • Select from stages - Click the Select from Stages tab and then filter or search for the stages or credential stores to add or remove. After selecting a stage that is included in multiple stage libraries, select the specific stage library version to add.
    • Select from stage libraries - Click the Select from Stage Libraries tab and then select the stage libraries to add. Clear the stage libraries to remove.
  6. In the Select Stage Libraries window, click the Summary tab to review the selected stage libraries by file name.
  7. To save your selections and return to the Edit Deployment window, click Ok.
  8. To save all changes and finish editing the deployment, click Save & Next until you reach the Review step.
  9. Click Restart Engines to restart all engine instances for the changes to take effect.

    If engine instances are not running, click Exit. When the engines start, Control Hub replicates the updated stage libraries to each instance.

Editing Engine Proxy Properties

When you edit a deactivated deployment, you can define proxy properties that configure engine instances to use a proxy server for outbound network requests.

The required steps depend on the deployment type:
Note: To define proxy properties for an existing Kubernetes deployment, you must complete additional steps. For instructions, see Using a Proxy Server for Kubernetes.

Editing for Self-Managed Deployments

To define proxy properties for an existing self-managed deployment, you must stop the deployment. After editing the deployment, you start the deployment again and then run the updated engine installation script that includes proxy properties defined as environment variables.

  1. In the Control Hub Navigation panel, click Set Up > Deployments.
  2. Select the active self-managed deployment, and then click the Stop icon: .
  3. Click OK to confirm, and then click Close.
  4. In the Actions column of the deployment, click the More icon () and then click Edit.
  5. To bypass the Define Deployment step, click Save & Next.
  6. In the Configure Engine step, click Click here to configure, next to the Advanced Configuration property.
  7. In the Proxy tab, edit the engine proxy properties as needed.

    For a description of each property, see the Data Collector documentation or the Transformer documentation.

  8. To save your changes and return to the deployment wizard, click Save, and then click OK.
  9. To save all changes and finish editing the deployment, click Save & Next until you reach the Review step.
  10. Click Start & Generate Install Script to start the deployment and generate the updated engine installation script that includes the proxy environment variables.
  11. Select whether the installation script runs an engine instance as a foreground or background process.
  12. Click the Copy to Clipboard icon () to copy the generated command.
  13. Open a command prompt on the engine machine.
  14. Paste and then run the updated installation script command.

    When prompted by the script, you can enter the same download and installation directory as the original engine installation, and then choose to replace the existing folder.

    The script starts the engine instance using the required proxy environment variables.

Editing for Cloud Service Provider Deployments

To define proxy properties for an existing cloud service provider deployment, such as an Amazon EC2 or GCE deployment, you must stop the deployment. When you stop a cloud service provider deployment, the provisioned resources in your cloud account are deleted.

After editing the deployment to modify the proxy properties, you start the deployment again. Control Hub provisions resources in the external system again, deploying and launching new engine instances that use the updated proxy properties.

  1. In the Control Hub Navigation panel, click Set Up > Deployments.
  2. Select the active deployment, and then click the Stop icon: .
  3. Click OK to confirm, and then click Close.
  4. In the Actions column of the deployment, click the More icon () and then click Edit.
  5. To bypass the Define Deployment step, click Save & Next.
  6. In the Configure Engine step, click Click here to configure, next to the Advanced Configuration property.
  7. In the Proxy tab, edit the engine proxy properties as needed.

    For a description of each property, see the Data Collector documentation or the Transformer documentation.

  8. To save your changes and return to the deployment wizard, click Save, and then click OK.
  9. To save all changes and finish editing the deployment, click Save & Next until you reach the Review step.
  10. Click Launch Deployment to start the deployment, provision resources in the external system, and launch new engine instances that use the updated proxy properties.

Restarting Engines

When you edit an active deployment to update the engine configuration, Control Hub prompts you to restart all engine instances. If you choose not to restart the engine instances while editing the deployment, you can restart the engine instances at a later time from the Deployments view.

During the restart process, the engine instances shut down and then automatically restart.

  1. In the Navigation panel, click Set Up > Deployments.
  2. Select an active deployment.
  3. In the Actions column, click the More icon () and then click Restart Engines.
  4. Click Ok to confirm.

    It can take a few minutes for engine instances to shut down and then restart.

  5. Click Close.

Deleting Deployments

Delete a deployment when you no longer want the deployed engine instances to run pipelines.

When you delete a self-managed deployment, all engine instances belonging to that deployment are shut down, but the engine installation files are not deleted. You must manually remove the files from the host machines.

When you delete a Control Hub-managed deployment, all resources provisioned for that deployment are deleted. For example, when you delete an Amazon EC2 deployment, all provisioned EC2 instances are deleted in addition to the AWS CloudFormation stack created for that deployment.

You can delete a deployment that meets the following conditions:
  • The deployment is in the Setup Incomplete, Deactivated, or Deactivation Error state.
  • No engine instances belonging to the deployment are currently running pipelines.
  1. In the Control Hub Navigation panel, click Set Up > Deployments.
  2. Select the deployments, and then click the Delete icon: .
  3. Confirm the deletion.
  4. For self-managed deployments, locate and then delete the installation directory on each machine where you launched an engine instance.

    The installation directory is named streamsets-<engine type>-<version>.

    For example, streamsets-datacollector-4.4.0.