Registered Data Collectors (Control Hub)

Once a Data Collector instance is started, it can be registered with an organization on a Control Hub instance. Those instances are referred to as Registered Data Collectors, and are only accessible from within the organization they were registered with.

Updating Data Collector Resource Thresholds

Control Hub can limit the amount of resources consumed on a Data Collector instance, regardless of the workload, to ensure that no Data Collector is ever overloaded. To check the resource thresholds configured for a given streamsets.sdk.sch_models.DataCollector instance, you can reference the max_cpu_load, max_memory_used, and max_pipelines_running attributes:

data_collector = sch.data_collectors[0]
data_collector.max_cpu_load
data_collector.max_memory_used
data_collector.max_pipelines_running

Output:

# data_collector.max_cpu_load
100.0

# data_collector.max_memory_used
1000000000000

# data_collector.max_pipelines_running
1000000000000

To set new values for the resource thresholds, you can use the streamsets.sdk.ControlHub.update_data_collector_resource_thresholds() method to pass in the streamsets.sdk.sch_models.DataCollector instance and values you wish to set:

sch.update_data_collector_resource_thresholds(data_collector,
                                              max_cpu_load=51.5,
                                              max_memory_used=550,
                                              max_pipelines_running=25)
data_collector.max_cpu_load

Output:

51.5