Resource Thresholds

Control Hub monitors the following resources used by each engine:

CPU load
Memory used
Number of running pipelines

For each engine, you can define a maximum threshold for each of these resources. When starting, balancing, or synchronizing jobs for Data Collector pipelines, Control Hub runs pipeline instances only on Data Collector engines that have not exceeded any resource thresholds. Similarly, when starting jobs for Transformer pipelines, Control Hub runs pipeline instances only on Transformer engines that have not exceeded any resource thresholds.

When multiple matching engines have not exceeded their resource thresholds, Control Hub prioritizes engines that are currently running the fewest number of pipelines. For example, you start a job and two matching Data Collector engines have the maximum CPU load set to 100%. Data Collector A is currently using 70% of the CPU and running 1 pipeline. Data Collector B is currently using 50% of the CPU and running 3 pipelines. Control Hub starts a pipeline instance for the job on Data Collector A.

When all matching engines have exceeded their resource thresholds, Control Hub randomly places jobs in a queue, giving the jobs a red active status. The job details display the following warning message:

JOBRUNNER_72 - Insufficient <Data Collector or Transformer> resources to run job. All matching <Data Collectors or Transformers> [<URLs>] have reached their maximum CPU usage limits.

When a matching engine no longer exceeds its resource thresholds, Control Hub randomly assigns a job from the queue to that engine, changing the job status to green active and running a pipeline instance on that engine.