Resource Thresholds

Control Hub monitors the following resources used by each Transformer:
  • CPU load
  • Memory used
  • Number of running pipelines
For each Transformer, you can define a maximum threshold for each of these resources. When starting jobs, Control Hub runs pipeline instances only on Transformers that have not exceeded any resource thresholds.
Note: If you start multiple jobs at the exact same time using the scheduler or using the Control Hub REST API, the number of pipelines running on a Transformer can exceed the configured resource threshold. If exceeding the resource threshold is not acceptable, you can enable an organization property that synchronizes the start of multiple jobs.

When multiple matching Transformers have not exceeded their resource thresholds, Control Hub prioritizes Transformers that are currently running the fewest number of pipelines. For example, you start a job and two matching Transformers have the maximum CPU load set to 100%. Transformer A is currently using 70% of the CPU and running 1 pipeline. Transformer B is currently using 50% of the CPU and running 3 pipelines. Control Hub starts a pipeline instance for the job on Transformer A.

When all matching Transformers have exceeded their resource thresholds, Control Hub randomly places jobs in a queue, giving the jobs a red active status. The job details display the following warning message:
JOBRUNNER_72 - Insufficient Transformer resources to run job. All matching Transformers [<URLs>] have reached their maximum CPU usage limits.

When a matching Transformer no longer exceeds its resource thresholds, Control Hub randomly assigns a job from the queue to that Transformer, changing the job status to green active and running a pipeline instance on that Transformer.

Defining Resource Thresholds

You can define the resource thresholds for any registered Transformer.

  1. In the navigation panel, click Execute > Transformers.
  2. Expand the details for the Transformer that you want to configure.
  3. Configure the following threshold values:
    Threshold Description
    Max CPU Load (%) Maximum percentage of CPU that can be used. Default is 100.
    Max Memory (%) Maximum percentage of memory that can be used. Default is 100.
    Max Running Pipeline Count Maximum number of pipelines that can be running. Default is 1,000,000.
  4. Click Save.