Extra Spark Configuration
When you create a pipeline, you can define extra Spark configuration properties that determine how the pipeline runs on Spark. Transformer passes the configuration properties to Spark when it launches the Spark application.
You can add any additional Spark configuration property, as described in the Spark configuration documentation.
You can also add the following extra configuration property provided by Transformer. This
is not a Spark configuration property:
| Configuration Property | Description |
|---|---|
| spark.home | Overrides the SPARK_HOME environment variable set on the
machine. For example, let's say that multiple Spark versions
are installed locally on the Transformer machine. You can add the |