Start Jobs

Supported pipeline types:

  • Data Collector
The Start Jobs origin starts one or more Control Hub jobs in parallel when the pipeline starts. The origin can also start job instances from a job template.

The Start Jobs origin is an orchestration stage that you use in orchestration pipelines. Orchestration stages perform tasks, such as schedule and start pipelines and Control Hub jobs, that you can use to create an orchestrated workflow across the StreamSets platform.

The Start Jobs origin can start jobs that run on any StreamSets execution engine, including Data Collector, Data Collector Edge, and Transformer. The origin generates an orchestration record that contains the list of started jobs and additional details about those jobs. You can pass the record to an orchestration stage to trigger another task. Or, you can pass it to a non-orchestration stage to perform other processing. For example, you might use a scripting processor to add information to the orchestration record.

When you configure the Start Jobs origin, you specify the Control Hub URL, and the names or IDs of the jobs or job template to start. You can also specify runtime parameters for each job or job instance.

You can configure the origin to reset the origins in the jobs when possible, and to run the jobs in the background. When running jobs in the background, the origin immediately passes its generated record downstream instead of waiting for the jobs to finish.

You also configure the user name and password to run the job and can optionally configure SSL/TLS properties.