Resource Thresholds
- CPU load
- Memory used
- Number of running pipelines
All engine instances belonging to the deployment inherit the same resource threshold values. For advanced use cases, some organizations can override the resource thresholds for individual engine instances.
When multiple matching engines have not exceeded their resource thresholds, Control Hub prioritizes engines that are currently running the fewest number of pipelines. For example, you start a job and two matching Data Collector engines have the maximum CPU load set to 100%. Data Collector A is currently using 70% of the CPU and running 1 pipeline. Data Collector B is currently using 50% of the CPU and running 3 pipelines. Control Hub starts a pipeline instance for the job on Data Collector A.
JOBRUNNER_72 - Insufficient <Data Collector or Transformer> resources to run job. All matching <Data Collectors or Transformers> [<URLs>] have reached their maximum CPU usage limits.
When a matching engine no longer exceeds its resource thresholds, Control Hub randomly assigns a job from the queue to that engine, changing the job status to green active and running a pipeline instance on that engine.
Editing Resource Thresholds for a Deployment
You can edit the maximum resource thresholds for a deployment.
- In the Control Hub Navigation panel, click .
-
In the Actions column of the deployment, click the
More icon (
) and then click Edit.
- In the Edit Deployment dialog box, expand the Configure Deployment section.
-
Modify the following threshold values:
Threshold Description Max CPU Load (%) Maximum percentage of CPU that can be used by each engine instance.
Default is 80.
Max Memory (%) Maximum percentage of memory that can be used by each engine instance. Default is 100.
Max Running Pipeline Count Maximum number of pipelines that can be running on each engine instance. Default is 1,000,000.
- Click Save.
Overriding Resource Thresholds
You define resource thresholds for a deployment, and then all engine instances belonging to that deployment inherit the same resource threshold values.
For advanced use cases, some organizations can override the resource thresholds for individual engine instances. However, when an engine restarts, overridden values are lost and the engine inherits the resource thresholds set for the deployment.
- In the Navigation panel, click .
- Click an engine type tab.
- Locate the engine that you want to configure.
-
In the Actions column, click the
More icon (
) and then click Edit.
- Select Override Resource Thresholds.
-
Override the following threshold values:
Threshold Description Max CPU Load (%) Maximum percentage of CPU that can be used. Max Memory (%) Maximum percentage of memory that can be used. Max Running Pipeline Count Maximum number of pipelines that can be running. - Click Save.