Wait for Pipelines (deprecated)

The Wait for Pipelines processor is an orchestration stage that you use in orchestration pipelines. Orchestration stages perform tasks, such as schedule and start pipelines and Control Hub jobs, that you can use to create an orchestrated workflow across the StreamSets platform.

Use this processor when you want to wait for pipelines that were started upstream to complete, before performing other orchestration tasks. The Wait for Pipelines processor can wait for pipelines that are running on the specified Data Collector, Data Collector Edge, or Transformer execution engine. To wait for pipelines on multiple engines, use additional processors.

For example, you might use a Wait for Pipelines processor to wait for SDC Edge pipelines to complete that were started by a Start Pipelines origin. If you also wanted to wait for Data Collector pipelines to complete, you would use another Wait for Pipelines processor to perform that task.

The Wait for Pipelines processor checks the status of all pipelines listed in incoming orchestration records that run on the specified execution engine. When the pipelines complete, the processor updates the pipeline status details in the record and passes a single orchestration record downstream.

When you configure the Wait for Pipelines processor, you define the URL of the execution engine that runs the pipelines. For an execution engine registered with Control Hub, you specify the Control Hub URL, so the processor starts the pipelines through Control Hub. Then, you specify how long to wait between each pipeline status checks.

You also configure the user name and password to run the pipeline and can optionally configure SSL/TLS properties.