Wait for Jobs

The Wait for Jobs processor waits for one or more jobs or job instances to complete.

The Wait for Jobs processor is an orchestration stage that you use in orchestration pipelines. Orchestration stages perform tasks, such as schedule and start pipelines and Control Hub jobs, that you can use to create an orchestrated workflow across the StreamSets platform.

Use this processor when you want to wait for jobs that were started upstream to complete, before performing other orchestration tasks. For example, you might use this processor to wait for jobs to complete that were started by a Start Jobs origin, before starting additional related jobs with a Start Jobs processor.

The Wait for Jobs processor checks the status of all jobs listed in the incoming orchestration records. When the jobs complete, the processor updates the jobs status details in the record and passes a single orchestration record downstream.

When you configure the Wait for Jobs processor, you specify the Control Hub URL where the jobs or job instances run. Then, you specify how long to wait between each job status checks. You also configure the Control Hub user name and password to use to monitor jobs, and you can optionally configure SSL/TLS properties.