Spark Executors

A Transformer pipeline runs on one or more Spark executors.

When you run a local pipeline, Spark uses a single executor to run the pipeline. When you run a cluster pipeline, Spark uses as many executors as required to run the pipeline, up to the maximum number of executors allowed. The maximum number of Spark executors is determined by your account type.

When you have an enterprise account for StreamSets Control Hub and you register Transformer with Control Hub, Transformer can use an unlimited number of Spark executors for each pipeline.

If you want to limit executor usage in the cluster, you can tune the performance of a specific pipeline to decrease the maximum number of executors that Spark uses for that pipeline.