DataOps Platform - Control Hub Guide

Home
Run
Job Instances

Learn
Set Up
Build
Run
- Job Templates
- Job Instances
  - Job Instances Overview
  - Number of Pipeline Instances
    For Data Collector jobs, you can manually scale out pipeline processing by increasing the number of pipeline instances that Control Hub runs for a job.
  - Data Collector Pipeline Failover
    You can enable a Data Collector job for pipeline failover. Enable pipeline failover to minimize downtime due to unexpected pipeline failures and to help you achieve high availability.
  - Transformer Pipeline Failover
    You can enable a Transformer job for pipeline failover for some cluster types. Enable pipeline failover to prevent Spark applications from failing due to an unexpected Transformer shutdown.
  - Runtime Parameters
    Runtime parameters are parameters that you define for a pipeline and call from within that same pipeline. When you define a runtime parameter, you enter the default value to use. When you create or edit a job that includes a pipeline with runtime parameters, you can specify another value to override the default. When the job starts, the value replaces the default value of the runtime parameter.
  - Job Tags
    A job tag identifies similar jobs. Use job tags to easily search and filter jobs when viewing them in the Job Instances view. For example, you might want to group jobs by the origin system or by the test or production environment.
  - Managing Jobs
  - Monitoring Jobs
    After you start a job, you can monitor the statistics, error information, logs, and alerts about all remote pipeline instances run from the job. During draft runs or job runs for Data Collector pipelines, you can also capture and review a snapshot of the data being processed.
  - Troubleshooting
- Scheduled Tasks
- Draft Runs
Monitor
Manage
Search

Job Instances

Job Instances Overview
Number of Pipeline Instances
For Data Collector jobs, you can manually scale out pipeline processing by increasing the number of pipeline instances that Control Hub runs for a job.
Data Collector Pipeline Failover
You can enable a Data Collector job for pipeline failover. Enable pipeline failover to minimize downtime due to unexpected pipeline failures and to help you achieve high availability.
Transformer Pipeline Failover
You can enable a Transformer job for pipeline failover for some cluster types. Enable pipeline failover to prevent Spark applications from failing due to an unexpected Transformer shutdown.
Runtime Parameters
Runtime parameters are parameters that you define for a pipeline and call from within that same pipeline. When you define a runtime parameter, you enter the default value to use. When you create or edit a job that includes a pipeline with runtime parameters, you can specify another value to override the default. When the job starts, the value replaces the default value of the runtime parameter.
Job Tags
A job tag identifies similar jobs. Use job tags to easily search and filter jobs when viewing them in the Job Instances view. For example, you might want to group jobs by the origin system or by the test or production environment.
Managing Jobs
Monitoring Jobs
After you start a job, you can monitor the statistics, error information, logs, and alerts about all remote pipeline instances run from the job. During draft runs or job runs for Data Collector pipelines, you can also capture and review a snapshot of the data being processed.
Troubleshooting

© 2023 StreamSets, Inc.