Scala, Spark, and Java JDK Requirements

Transformer requires that the appropriate versions of Scala, Spark, and Java JDK are installed.

Scala match requirement

To run cluster pipelines, the Scala version on the clusters must match the Scala version prebuilt in Transformer. If you install Spark on the Transformer machine, the Scala version prebuilt in the Spark installation must also match the Scala version prebuilt in Transformer.

You can view the Scala version prebuilt with Transformer by clicking the Help icon in the upper right corner of the user interface, then clicking About.

When you configure a deployment, you select the Transformer engine version to use. The version indicates the Scala version that Transformer is prebuilt with. For example, you might choose between Transformer 4.1.0 prebuilt with Scala 2.11 or Scala 2.12.

You can see the Scala version prebuilt in Transformer when you view deployment details on the Deployments page. Or, on the Engines page, select the engine and click the Configuration tab.

Spark requirement

The Spark version that you install on clusters or the Transformer machine depends on the Transformer installation that you use:

For Transformer prebuilt with Scala 2.11, install Spark 2.x prebuilt with Scala 2.11.
In general, most Spark 2.x installation packages are prebuilt with Scala 2.11. However, most Spark 2.4.2 installation packages are prebuilt with Scala 2.12.x instead.
For Transformer prebuilt with Scala 2.12, install Spark 3.x, which is prebuilt with Scala 2.12.

Java JDK requirement

The Java Development Kit (JDK) version that you must install on the Transformer machine depends on the Transformer installation package that you useScala version associated with the Transformer engine version for the deployment:

Scala 2.11 - Requires Java JDK 8.
Scala 2.12 - Requires Java JDK 11.