General Installation Requirements

Choose a Transformer engine version based on the clusters that you want to run pipelines on and the Transformer features that you want to use.

The Scala version that Transformer is built with determines the Java JDK version that must be installed on the Transformer machine and the Spark versions that you can use with Transformer. The Spark version that you choose determines the cluster types and the Transformer features that you can use.

For example, Amazon EMR 7.5 clusters use Spark 3.5. To run Transformer pipelines on those clusters, you use Transformer prebuilt with Scala 2.12. And since Transformer prebuilt with Scala 2.12 requires Java JDK 11, you install that JDK version on the Transformer machine.

For more information, see Cluster Compatibility Matrix, Scala, Spark, and Java JDK Requirements, and Spark Versions and Available Features.

Also note the other Transformer requirements in this section.

Choosing an Engine Version

You can use Transformer prebuilt with the following Scala version:

  • Scala 2.12 - Use with Spark 3.x. Requires Java JDK 11.

The Scala version that Transformer is built with determines the Java JDK version that must be installed on the Transformer machine and the Spark versions that you can use with Transformer. The Spark version that you choose determines the cluster types and the Transformer features that you can use.

For example, Amazon EMR 7.5 clusters use Spark 3.5. To run Transformer pipelines on those clusters, you use Transformer prebuilt with Scala 2.12. And since Transformer prebuilt with Scala 2.12 requires Java JDK 11, you install that JDK version on the Transformer machine.

For more information, see Cluster Compatibility Matrix, Scala, Spark, and Java JDK Requirements, and Spark Versions and Available Features.

Cluster Compatibility Matrix

The following matrix shows the Transformer Scala version that is required for supported cluster and underlying Spark versions.

You can use this matrix to determine the Transformer engine version to use in your deployment.

Cluster Type Supported Cluster Versions Cluster Underlying Spark Version Transformer Scala Version
Amazon EMR 6.8 and later 6.x 3.x 1 Scala 2.12
7.x 3.x 1 Scala 2.12
Amazon EMR Serverless 6.9.0 and later 6.x 3.x 1 Scala 2.12
7.x 3.x 1 Scala 2.12
Cloudera Data Engineering 1.3.3 and later 1.3.x 3.x 1 Scala 2.12
Databricks 11.3 3.3.0 Scala 2.12
12.2 3.3.2 Scala 2.12
13.3 3.4.1 Scala 2.12
14.3 3.5.0 Scala 2.12
Google Dataproc 2.1 3.3.0 Scala 2.12
2.2 3.5.0 Scala 2.12
Hadoop YARN 2

Cloudera distribution

 CDP Private Cloud Base 7.1.9 3.3.0 Scala 2.12
Spark Standalone Cluster 3 NA NA Any

1 Support for Spark 3.0 - 3.2 has been deprecated.

2 Before using a Hadoop YARN cluster, complete all required tasks.

3 Spark Standalone clusters are supported for development workloads only.

Scala, Spark, and Java JDK Requirements

Transformer requires that the appropriate versions of Scala, Spark, and Java JDK are installed.

Scala match requirement
To run cluster pipelines, the Scala version on the clusters must match the Scala version prebuilt in Transformer. If you install Spark on the Transformer machine, the Scala version prebuilt in the Spark installation must also match the Scala version prebuilt in Transformer.

When you configure a deployment, you select the Transformer engine version to use. The version indicates the Scala version that Transformer is prebuilt with.

At this time, you can use Transformer prebuilt with Scala 2.12.

You can see the Scala version prebuilt in Transformer when you view deployment details on the Deployments page. Or, on the Engines page, select the engine and click the Configuration tab.

Spark requirement
The Spark version that you install on clusters or the Transformer machine depends on the Transformer installation that you use.
For Transformer prebuilt with Scala 2.12, install Spark 3.x, which is prebuilt with Scala 2.12.
Java JDK requirement
The Java Development Kit (JDK) version that you must install on the Transformer machine depends on the Scala version associated with the Transformer engine version for the deployment.
Install Java JDK 11 to use Transformer prebuilt with Scala 2.12.

Spark Versions and Available Features

The Spark version on a cluster determines the Transformer features that you can use in pipelines that the cluster runs. The Spark version that you install on the Transformer machine determines the features that you can use in local and standalone pipelines.

Transformer does not need a local Spark installation to run cluster pipelines. However, Transformer does require a local Spark installation to perform certain tasks, such as using embedded Spark libraries to preview or validate pipelines, and starting pipelines in client deployment mode on Hadoop YARN clusters.

Transformer supports Spark 3.x, which provides access to all Transformer features. Spark 3.0 - 3.2 are deprecated at this time.

Important: IBM StreamSets does not provide support for Spark installations.

Spark Shuffle Service Requirement

To run a pipeline on a Spark cluster, Transformer requires that the Spark external shuffle service be enabled on the cluster.

Most Spark clusters have the external shuffle service enabled by default. However, Hortonworks clusters do not.

Before you run a pipeline on a Hortonworks cluster, enable the Spark external shuffle service on Hortonworks clusters. Enable the shuffle service on other clusters as needed.

When you run a pipeline on a cluster without the Spark external shuffle service enabled, the following error is written to the Spark log:
org.apache.spark.SparkException: Dynamic allocation of executors requires the external shuffle service. You may enable this through spark.shuffle.service.enabled.

For more information about enabling the Spark external shuffle service, see the Spark documentation.

Default Port

Transformer uses either HTTP or HTTPS that runs over the TCP protocol. Configure network routes and firewalls so that the Spark cluster can reach the Transformer IP address.

For example, if your Transformer is installed on EC2 and you run pipelines on EMR, make sure that the EMR cluster can access Transformer on EC2.

The default port number that Transformer uses depends on the configured protocol:
  • HTTP - Default is 19630.
  • HTTPS - Default depends on the configuration. For more information, see Enabling HTTPS.

Hadoop YARN Requirements

Before using a Hadoop YARN cluster, complete the following requirements:

Directories

When using a Hadoop YARN cluster, the following directories must exist:
Spark node local directories
The Spark yarn.nodemanager.local-dir configuration parameter in the yarn-site.xml file defines one or more directories that must exist on each Spark node.
The value of the configuration parameter should be available in the cluster manager user interface. By default, the property is set to ${hadoop.tmp.dir}/nm-local-dir.
The specified directories must meet all of the following requirements for each node of the cluster:
  • Exist on each node of the cluster.
  • Be owned by YARN.
  • Have read permission granted to the Transformer proxy user.
HDFS application resource directories
Spark stores resources for all Spark applications started by Transformer in the HDFS home directory of the Transformer proxy user. Home directories are named after the Transformer proxy user, as follows:
/user/<Transformer proxy user name>
Ensure that both of the following requirements are met:
  • Each resource directory exists on HDFS.
  • Each Transformer proxy user has read and write permission on their resource directory.
For example, you might use the following command to add a Transformer user, tx, to a spark user group:
usermod -aG spark tx
Then, you can use the following commands to create the /user/tx directory and ensure that the spark user group has the correct permissions to access the directory:
sudo -u hdfs hdfs dfs -mkdir /user/tx
sudo -u hdfs hdfs dfs -chown tx:spark /user/tx
sudo -u hdfs hdfs dfs -chmod -R 775 /user/tx

JDBC Driver

When you run pipelines on older distributions of Hadoop YARN clusters, the cluster can have an older JDBC driver on the classpath that takes precedence over the JDBC driver required for the pipeline. This can be a problem for PostgreSQL and SQL Server JDBC drivers.

When a pipeline encounters this issue, it generates a SQLFeatureNotSupportedException error, such as:

java.sql.SQLFeatureNotSupportedException: This operation is not supported.

To avoid this issue, update the PostgreSQL and SQL Server JDBC drivers on the cluster to the latest available versions.

Memory

When you run pipelines on a Hadoop YARN cluster, the Spark submit process continues to run until the pipeline finishes, which uses memory on the Transformer machine. This memory usage can cause pipelines to indefinitely remain in a running or stopping status when the Transformer machine has limited memory or when a large number of pipelines start on a single Transformer.

To avoid this issue, run the following command on each Transformer machine to decrease the amount of memory available to the Spark submit process:
export SPARK_SUBMIT_OPTS="-Xmx64m"