Release Notes

• You can run pipelines on the following cluster versions:
  • Databricks 14.3
  • EMR 7.0.0
  • EMR Serverless 7.0.0
  • Hadoop YARN, Cloudera version CDP Private Cloud Base 7.1.x with Spark 3.x

• You can configure an EMR pipeline to terminate a cluster provisioned by AWS Service Catalog after the pipeline stops.

• The following properties are available when you configure a Snowflake stage to use a Control Hub Snowflake connection. You can use them to override the value in the related connection property:
  • Override Warehouse
  • Override Database
  • Override Schema
• The following property is available when you configure connection information in the Snowflake stage. Use it to enable using a private link URL:
  • Use Private Link Snowflake URL
• Snowflake stages include Transformer-provided stage libraries for Spark 3.0 - 3.4. Previously, Transformer-provided stage libraries were for Spark 3.1. As a result, Snowflake stages in existing pipelines are updated to use the Transformer-provided stage library for Spark 3.1.

Pipelines include the following new advanced properties:

• Cache Level - Determines how or where Spark caches data during a pipeline run.
• Cache Replicas - Determines how many replicas of cached data to create.

This enhancement does not affect existing pipelines. Existing pipelines have the Cache Level property set to Memory and Disk to ensure the previous behavior.

Microsoft retired Azure Data Lake Storage Gen1 in February 2024. As a result, this release includes the following changes:

• ADLS Gen1 origin and destination - These stages, which were deprecated in an earlier version, have been removed.

• Delta Lake origin, processor, and destination - These stages no longer support processing data stored on Azure Data Lake Storage Gen1.

• Databricks 13.3 support - You can now run pipelines on Databricks 13.3 clusters.
• External libraries on Databricks, Dataproc, EMR, and EMR Serverless clusters - Transformer fully manages staging directories for these clusters. When you add, remove, or update external libraries as external resources on Transformer, Transformer now automatically updates the cluster staging directories when you run a related pipeline.

  Previously, Transformer only updated these cluster staging directories when you added external libraries or updated external libraries with the same base file name. Now, Transformer compares the external libraries installed on Transformer against cluster staging directories and updates these directories to match, as needed.

• EMR clusters

  • AWS Service Catalog support - You can configure an EMR pipeline to run on a cluster provisioned by AWS Service Catalog.

  • Define Cluster Start Option property - Use this property to specify whether the pipeline runs on an existing cluster, a cluster provisioned by Transformer, or a cluster provisioned by AWS Service Catalog.

    In previous releases, you used the Provision a New Cluster property to specify whether the pipeline ran on an existing cluster or a cluster provisioned by Transformer. This update does not require changes to existing pipelines.

• You can configure a connection to run a job on a cluster provisioned by AWS Service Catalog.
• The new Define Cluster Start Option property allows you to specify whether the pipeline runs on an existing cluster, a cluster provisioned by Transformer, or a cluster provisioned by AWS Service Catalog.

  Previously, you used the Provision a New Cluster property to specify whether the pipeline ran on an existing cluster or a cluster provisioned by Transformer. This update does not require changes to existing pipelines.

Transformer includes a new driver callback URL property, transformer.driver.callback.url. This property defines the cluster callback URL for Spark to communicate with Transformer. Transformer uses the specified callback URL for all pipelines, unless overridden by the existing Cluster Callback URL property defined in individual pipelines.

In previous releases the specified Transformer base URL, transformer.base.http.url, which defines how Control Hub communicates with Transformer, also acted as the cluster callback URL.

This enhancement is not a behavior change. However, if you previously configured the Cluster Callback URL property in pipelines as a workaround to avoid using the Transformer base URL as the cluster callback URL, you can now simply define the new driver cluster callback URL Transformer property.

However, note that the Cluster Callback URL pipeline property, when defined, takes precedence over all other possible URLs. For more information, see Understanding the Spark Cluster Callback URL.

• Library support:
  • Delta Lake 2.4.0 support - Transformer-provided libraries for Delta Lake stages have been upgraded from Delta Lake version 0.7.0 to version 2.4.0, which supports Spark 3.4.x. This change can have upgrade impact.
  • Amazon support for Hadoop 3.3.4 - Transformer provides AWS Transformer-provided libraries for Hadoop 3.3.4 for Amazon S3 stages.
• Deprecated stages - The ADLS Gen1 origin and destination have been deprecated with this release and may be removed in a future release. We recommend that you avoid using these stages. For suggested alternatives, see Deprecated Functionality.

• Docker image upgrades - The Docker images for Transformer 5.7.0 include upgraded Spark versions, as follows:

  • streamsets/transformer:scala-2.11_5.7.0 now uses Spark 2.4.8, upgraded from Spark 2.4.5.

  • streamsets/transformer:scala-2.12_5.7.0 now uses Spark 3.4.1, upgraded from Spark 3.0.1.

• Ludicrous input metrics - When a pipeline runs in Ludicrous processing mode, Transformer provides both input and output statistics.

  In previous releases, Transformer provided only output statistics by default, but you could enable the generation of input statistics as needed. With this update, the Collect Input Metrics pipeline property has been removed since input metrics are now always generated in Ludicrous mode.

• The destination is available for use with Spark 3.0.x and later, except for version 3.2.x.
• The destination includes the following new properties for performing a bulk copy:
  • Reliability Level
  • Isolation Level
  • Schema Check
• The Auto-Create Table property, which was available when performing a bulk copy, has been removed. It was unnecessary because the destination always creates a table as needed when performing a bulk copy.
• The Azure SQL destination includes a Microsoft JDBC driver for SQL Server with the destination. The destination uses Microsoft JDBC driver for SQL Server version 8 or later. The destination also requires version 8 or later of the driver starting with this release. This change can have upgrade impact.

• Databricks 12.2 support - You can run pipelines on a Databricks 12.2 cluster.
• Dataproc 2.1 support - You can run pipelines on a Google Dataproc 2.1 cluster.

• Unity Catalog origin - Use the new origin to read from a Databricks Unity Catalog managed table.
• Unity Catalog destination - Use the new destination to write to a Databricks Unity Catalog managed or external table.

• Pipeline retry properties - Use the following properties to specify how Transformer tries to start pipelines that fail to start:
  • Retry Pipeline on Error - Enables trying to start a pipeline again after it fails to start.
  • Retry Attempts - Number of times to try to start a pipeline that fails to start. The default is 3.
• Field selection - When a property includes a magnifying glass icon, you can use preview data to select fields instead of typing field names:
  • After previewing data, stages provide autocomplete suggestions for field names in properties that accept multiple fields.
  • After selecting fields, you can drag the fields to change the order in which they appear.
  • In some stages, after you select one or more fields to use, you can then enter an expression that includes the field name or enter additional field names.

Due to end-of-life declarations, the following cluster versions are deprecated with this release and are no longer tested with Transformer:

• Azure HDInsight 4.0
• CDH 6.x
• Databricks 5.x - 8.x
• HDP
• SQL Server 2019 Big Data Clusters

• Amazon EMR - You can specify a cluster using the cluster name and tags, instead of using the cluster ID.
• Amazon EMR Serverless - You can specify an application using the application name and tags, instead of using the application ID.
• Amazon EMR and EMR Serverless - You can configure the following Transformer configuration properties to define how Transformer retries monitoring EMR and EMR Serverless pipelines:

  • transformer.emr.monitoring.max.retry

  • transformer.emr.monitoring.retry.base.backoff

  • transformer.emr.monitoring.retry.max.backoff

• Databricks enhancements:
  • You can configure the transformer.databricks.external.resources.cache Transformer configuration property to cache runtime resource files for reuse.
  • You can configure the following Transformer configuration properties to define how Transformer retries starting Databricks pipelines:
    • transformer.databricks.run.max.retries
    • transformer.databricks.run.retry.interval
  • When needed, you can configure Transformer to lock the Databricks workspace when it starts a pipeline. This can prevent timeout errors when multiple Transformer engines try to start pipelines that require uploading a large volume of runtime resource files to staging areas.
  • Amazon EMR, Amazon EMR Serverless, Databricks, and Dataproc clusters - When you update an external library used by these clusters for Transformer, Transformer can update the external libraries in the cluster staging directories, so you do not need to manually remove older versions.

• Amazon EMR - You can specify a cluster using the cluster name and tags, instead of using the cluster ID. Available with authoring Data Collector 5.5.0 or later.
• Amazon EMR Serverless - You can specify an application using the application name and tags, instead of using the application ID. Available with authoring Data Collector 5.5.0 or later.

• Transformer Docker image - The Docker images for Transformer 5.4.0, streamsets/transformer:scala-2.11_5.4.0 and streamsets/transformer:scala-2.12_5.4.0, use Ubuntu 22.04 LTS (Jammy Jellyfish) as a parent image. This change can have upgrade impact.

In previous releases, Transformer Docker images used Alpine Linux as a parent image. Due to limitations in Alpine Linux, with this release Transformer Docker images use Ubuntu 22.04 LTS (Jammy Jellyfish) as the parent image.

• Amazon EMR Serverless - Transformer can run pipelines on an Amazon EMR Serverless application.
• Databricks clusters
  • You can include the Google Big Query origin and destination and the Google Cloud Storage origin and destination in pipelines that you run on a Databricks cluster.
  • You can run pipelines on a Databricks 11.3 cluster.
• Hadoop YARN clusters - You can run pipelines on a MapR 7.0 distribution of Hadoop YARN. The distribution requires Ezmeral Ecosystem Pack (EEP) 8.1, which includes Spark 3.2.0. Map R is now called HPE Ezmeral Data Fabric.

• Data Formats property - The Whole Directory and MapR FS origins have an Additional Data Format Configuration property on the Data Formats tab. You can use this property to enter other data format parameters.
• Google stages - You can include the Google Big Query origin and destination and the Google Cloud Storage origin and destination in pipelines that you run on a Databricks cluster.
• Slowly Changing Dimension processor - Improved logic produces consistent and expected results in the following existing features:
  • Null handling - When enabled, the processor replaces null values in change records.
  • Records without changes - The processor discards and change record that does not contain a change.
  • Tracking fields - Change records do not require tracking fields.
  • Timestamp basis field - The processor supports three categories for the timestamp basis field:
    • Same name as tracking field
    • Data field in master dimension
    • Extra field in change record
  • Non-Type 2 updates - In a Type 2 dimension, a change record can trigger a non-Type 2 update.
  • New records - For change records without a matching master record, the processor inserts a new record

• Amazon EMR Serverless - You can use an Amazon EMR Serverless connection when configuring a pipeline to run on an Amazon EMR Serverless application.

• Thycotic Secret Server credential store - You can use Thycotic Secret Server as a credential store for Transformer.
• Support bundles - You can generate a support bundle when Transformer uses the default WebSocket tunneling communication method.

  Earlier Transformer versions require using direct engine REST APIs to generate support bundles.

• In the documentation that discusses slowly changing file dimensions, the terms “grouped file dimension” and “ungrouped file dimension” replace the terms “partitioned file dimension” and “unpartitioned file dimension.”

This can adversely affect communicating with Snowflake when Snowflake connection information specified in a Snowflake stage or connection, such as a URL, includes underscores. When needed, you can bypass this behavior by setting the allowUnderscoresInHost driver property to true. For more information and alternate solutions, see this Snowflake community article.

• Support for multiple tables - These origins can read from multiple tables. The Table property in previous releases has become the Tables property. With one of these origins configured to read from multiple tables, a pipeline processes multiple batches, in both batch and streaming mode.
• Batch headers - Pipelines with these origins generate batch headers for each batch and the origin writes the jdbc.table attribute in the header. The attribute stores the name of the table that the origin reads for the batch.

• Data Formats property - The ADLS Gen1, ADLS Gen2, Amazon S3, File, and Google Cloud Storage origins have an Additional Data Format Configuration property on the Data Formats tab. You can use this property to enter other data format parameters.
• File-based destinations - The ADLS Gen1, ADLS Gen2, Amazon S3, File, and Google Cloud Storage destinations generate a validation error when writing in the Text data format if the Partition by Fields property is enabled.
• Security option for Kafka stages - Kafka stages support the Custom Authentication (Security Protocol=CUSTOM) security option. Use the option to specify custom properties that contain the information required by a security protocol, rather than using predefined properties associated with other security options.
• Snowflake destination - When merging data, the destination supports multiple join keys.

• Amazon EMR clusters - You can set the Amazon EMR runtime role with the Execution Role property.
• Google Dataproc clusters
  • In pipelines running on Dataproc 2.0.40 clusters, you can include the Amazon Redshift destination and the Avro data format in Amazon S3 stages and Google Cloud Storage stages.
  • You can access details about the Dataproc job run for a Transformer pipeline in one of the following ways:
    • After the corresponding Control Hub job completes, view the job run summary from the job History tab which displays the Dataproc Job URL. Use the URL to access the Dataproc job in the Google Cloud Console.
    • Log into the Google Cloud Console and view the list of jobs run on the Dataproc cluster. Filter the jobs by the streamsets-transformer-pipeline-id or streamsets-transformer-pipeline-name label which are applied to all Dataproc jobs run for Transformer pipelines.

• Expression language - Batch functions retrieve the value of an attribute in a batch header. You can use the functions in specific properties of destination stages.
• Credential stores
  • Google Secret Manager - You can use Google Secret Manager as a credential store for Transformer.
  • Property for migration - You can use a new credentialStores.usePortableGroups credential stores property to migrate pipelines that access credential stores from one Control Hub organization to another. Contact StreamSets Support before enabling this option.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

If you use Transformer prebuilt with Scala 2.12, then after you upgrade to 5.2.0, review how pipelines use the generated files and make necessary changes to account for the new initial declaration.

After you upgrade to 5.2.0, review your SDK for Python code for these origins and replace origin.table with origin.tables.

• Databricks 10.4 - You can run pipelines on Databricks 10.4 clusters.
• Dataproc 2.0.40 - You can run pipelines on Dataproc 2.0.40 clusters. However, with this version pipelines cannot include the Amazon Redshift destination or the Avro data format in Amazon S3 stages or Google Cloud Storage stages.

• Maximum number of partitions - The Number of Partitions property in previous releases has become the Maximum Number of Partitions property. When the pipeline runs, the origin creates up to the specified number of partitions. This allows the pipeline to run if the origin cannot create the specified number of partitions.

  In previous versions, if an origin cannot create the specified number of partitions, the pipeline fails.

• Automatic partition selection - When you configure an origin to skip offset tracking, the origin attempts to select a logical partition column for the read. For more information, see “Partition Column Selection” in the documentation for the origin.

  In previous versions, when an offset column could not be found to be used for partitioning, the pipeline fails.

• Cloudera Data Engineering cluster support - Transformer now supports running pipelines on Cloudera Data Engineering virtual clusters.
• Cloudera CDP Private Cloud Base 7.1.x support - Transformer now supports Hadoop YARN clusters on Cloudera CDP Private Cloud Base 7.1.x.
• EMR connection retry properties - You can configure the following new properties to define how Transformer retries a failed request or throttling error for an EMR cluster:
  • Max Retries
  • Retry Base Delay
  • Throttling Retry Base Delay
  • Max Backoff

• New JSON Parser processor - Use the JSON Parser processor to parse a JSON object embedded in a string field.

• Amazon S3 origin property rename - The Bucket property is now named Bucket and Path. It has always allowed entering a path that includes the asterisk (*) and question mark (?) wildcards.
• New empty dataframe behavior for JDBC Table origins - When there is no data to be read, the following origins now pass the table schema in an empty dataframe:
  • JDBC Table origin
  • MySQL JDBC Table origin
  • Oracle JDBC Table origin
  • PostgreSQL JDBC Table origin

  In previous releases, these origins passed an empty schema with empty dataframes. This change has no upgrade impact because it includes a new Use Empty Schemas property that passes an empty schema with empty dataframes.

  To preserve backward compatibility, the Use Empty Schemas property is enabled for all upgraded pipelines. For new pipelines, this property is disabled by default.

• Partition Base Path origin property - The following origins now allow specifying a base path for partitions in a Partition Base Path property:
  • ADLS Gen1 origin
  • ADLS Gen2 origin
  • Amazon S3 origin
  • File origin
  • Google Cloud Storage origin
  • MapR FS origin
• Skip Empty Batches destination property - The following destinations can now skip writing empty batches when you select the Skip Empty Batches property:
  • ADLS Gen1 destination
  • ADLS Gen2 destination
  • Amazon S3 destination
  • File destination
  • Google Cloud Storage destination
  • MapR FS destination

• Amazon EMR:
  • You can specify a SSH EC2 Key ID property for the EC2 SSH key to be used on nodes of the cluster.
  • You can configure pipelines on an EMR cluster to assume another role.
• Databricks:
  • You can run pipelines on Databricks 9.1 LTS clusters.
  • You can access the Databricks job details when you view a job run summary from the job History tab.
• Google Dataproc 2.0 - You can run pipelines on Google Dataproc 2.0.

• Amazon S3 destination - When configuring the destination, you can now use the s3 URI scheme, in addition to the s3a scheme. Best practice is to use s3 with EMR clusters and s3a with all other clusters.
• Field Replacer processor - Use Spark SQL expressions to generate new values for specified fields. You can use quotation marks to specify a string.
• Google Big Query origin - The origin can now read from Google BigQuery views.

• Amazon EMR cluster connections include the following enhancements:
  • You can configure Amazon EMR cluster connections to assume another role.
  • You can specify a SSH EC2 Key ID property for the EC2 SSH key to be used on nodes of the cluster.

• CyberArk credential store support - You can use CyberArk as a credential store for Transformer.
• Cluster URL access - When monitoring a Control Hub job for a Databricks pipeline, when you view the job summary, you can now access the Databricks cluster job URL.