Deployment Mode

Cluster pipelines on Hadoop YARN can use one of the following deployment modes:

Client

In client deployment mode, the Spark driver program is launched on the local machine outside of the cluster. Use client mode when the Transformer machine is physically co-located with the cluster worker machines.

Cluster

In cluster deployment mode, the Spark driver program is launched remotely on one of the worker nodes inside the cluster. Use cluster mode when the Transformer machine is physically located far from the worker machines. In this case, using cluster mode minimizes network latency between the drivers and the executors.

For more information about deployment modes, see the Apache Spark documentation.