Spark Executors

A Transformer pipeline runs on one or more Spark executors.

When you run a local pipeline, Spark uses a single executor to run the pipeline. When you run a cluster pipeline, Spark uses as many executors as required to run the pipeline, up to the maximum number of executors allowed. The maximum number of Spark executors is determined by your account type.

When you have an enterprise account for StreamSets Control Hub and you register Transformer with Control Hub, Transformer can use an unlimited number of Spark executors for each pipeline.

When Transformer is not registered with Control Hub, the maximum number of Spark executors for each pipeline is determined by the Transformer installation type and activation code:
Cloud service provider installation
When you install Transformer through a cloud service provider marketplace, the maximum number of Spark executors is determined by your selected Transformer offer. Each offer includes a valid activation code. To increase the number of executors, migrate to another Transformer offer in the marketplace or contact StreamSets for a custom offer.
Installation package downloaded from the StreamSets website
When you download a Transformer installation package from the StreamSets website, Transformer requires an activation code. The maximum number of Spark executors is determined by the activation code:
  • If you have a free account, the activation code allows one Spark executor for each pipeline.
  • If you have an enterprise account for Transformer, the activation code allows more than one Spark executor for each pipeline, based on the license agreement.

To increase the number of executors, contact StreamSets to update your activation code.

If you want to limit executor usage in the cluster, you can tune the performance of a specific pipeline to decrease the maximum number of executors that Spark uses for that pipeline.