Start Jobs

The Start Jobs processor is an orchestration stage that you use in orchestration pipelines. Orchestration stages perform tasks, such as schedule and start pipelines and Control Hub jobs, that you can use to create an orchestrated workflow across the StreamSets platform. For example, an orchestration pipeline can use the Cron Scheduler origin to generate a record every weekday at 9 AM that triggers the Start Jobs processor, which starts a set of Control Hub jobs.

After performing its task, the Start Jobs processor updates the orchestration record, adding details about the jobs that it started. Then, it passes the record downstream. You can pass the record to an orchestration stage to trigger another task. Or, you can pass it to a non-orchestration stage to perform other processing.

When you configure the Start Jobs processor, you specify the Control Hub URL, and the names or IDs of the jobs or job template to start. You can also specify runtime parameters for each job or job instance.

You can configure the processor to reset the origins in the jobs when possible, and to run the jobs in the background. When running jobs in the background, the processor immediately updates and passes the input record downstream instead of waiting for the jobs to finish.

You also configure the user name and password to run the job and can optionally configure SSL/TLS properties.