Choosing an Installation PackageEngine Version

StreamSets provides Transformer installation packagesengine versions prebuilt with different versions of Scala.

You can use Transformer prebuilt with the following Scala versions:

Choose an installation packageengine version based on the clusters that you want to run pipelines on and the Transformer features that you want to use.

The Scala version that Transformer is built with determines the Java JDK version that must be installed on the Transformer machine and the Spark versions that you can use with Transformer. The Spark version that you choose determines the cluster types and the Transformer features that you can use.

For example, Amazon EMR 6.1 clusters use Spark 3.x. To run Transformer pipelines on those clusters, you use an installation package for Transformer prebuilt with Scala 2.12. And since Transformer prebuilt with Scala 2.12 requires Java JDK 11, you install that JDK version on the Transformer machine.

For more information, see Cluster Compatibility Matrix, Scala, Spark, and Java JDK Requirements, and Spark Versions and Available Features.