Start Pipelines (deprecated)

Data Collector

The Start Pipelines origin starts one or more pipelines in parallel when the pipeline starts.

Important: This stage is deprecated and may be removed in a future release.

The Start Pipelines origin is an orchestration stage that you use in orchestration pipelines. Orchestration stages perform tasks, such as schedule and start pipelines and Control Hub jobs, that you can use to create an orchestrated workflow across the StreamSets platform.

The Start Pipelines origin can start pipelines that run on any StreamSets execution engine, such as Data Collector, Data Collector Edge, or Transformer. The origin can start pipelines that run on the execution engine specified in the stage. To start pipelines on a different execution engine, you can use a Start Pipelines processor.

The origin generates a record that contains a list of the started pipelines and information about those pipelines. You can pass the record to an orchestration stage to trigger another task. Or, you can pass it to a non-orchestration stage to perform other processing. For example, you might use a Stream Selector processor to pass the record to different stages based on a pipeline completion status.

When you configure the Start Pipelines origin, you define the URL of the execution engine that runs the pipelines. You specify the names or IDs of the pipelines to start along with any runtime parameters to use. For an execution engine registered with Control Hub, you specify the Control Hub URL, so the origin starts the pipelines through Control Hub.

You can configure the origin to reset the origins in the pipelines when possible, and to run the pipelines in the background. When running pipelines in the background, the origin immediately passes its generated record downstream instead of waiting for the pipelines to finish.

You also configure the user name and password to run the pipeline and can optionally configure SSL/TLS properties.