Running Multiple Concurrent Pipelines

By default, Data Collector can run approximately 22 standalone pipelines concurrently. If you plan to run a larger number of pipelines at the same time, increase the thread pool size.

The runner.thread.pool.size property in the Data Collector configuration fileconfiguration properties, determines the number of threads in the pool that are available to run standalone pipelines. One running pipeline requires five threads, and pipelines share threads in the pool.

  1. Use a text editor to open sdc.properties.In Control Hub, edit the deployment. In the Configure Engine section, click Advanced Configuration. Then, click Data Collector Configuration.
  2. Calculate the approximate runner thread pool size by multiplying the number of running pipelines by 2.2.
  3. Set the runner.thread.pool.size property to your calculated value.
  4. To enable the changes, restart Data Collector.Save the changes to the deployment and restart all engine instances.