Release Notes

For help with Transformer for Snowflake, join the StreamSets Community and use the Transformer for Snowflake tag when you post your question. To provide suggestions and feedback, contact productfeedback@streamsets.com.

May 2023

The following Transformer for Snowflake release occurred in May 2023.

May 12, 2023

This release includes several enhancements, a behavior change, and a fixed issue.

New Features and Enhancements

Data preview enhancements
  • You can preview data for incomplete pipelines, as long as all required fields are defined.
  • You can run preview up to a specific stage in the pipeline instead of the entire pipeline.
  • Preview displays Snowflake data types instead of StreamSets data types.
  • Preview provides as much preview data as possible when errors cause a preview request to fail to complete.
Column selection
  • To specify column names, you can use preview data to select columns instead of typing column names.
  • After previewing data, stages provide autocompletion suggestion for column names in properties that accept multiple columns.
  • After selecting multiple columns, you can drag the columns to change the order in which they appear.
  • In some stages, after you select a column to use, you can then enter an expression that includes the column name or enter additional column names.
Table selection
In the Snowflake Table origin and destination, when you can explore your data and select the table that you want to use, you can explore the entire warehouse instead of being limited to tables within the default database or schema.
Apply Function processor enhancements
  • The Column property replaces the Column Expression property.

    Use the Column property to specify the column to apply the function to. You can also specify a regular expression that applies the function to a set of columns.

  • The Output Column property replaces the Add Prefix to New Output Columns property and the Add Suffix to New Output Columns property.

    Existing pipelines merge any prefix and suffix values defined in the old properties into the Output Column property using $0 to represent the original column name. If existing pipelines use $0 in the old prefix and suffix properties, you should review those pipelines. For more information, see Behavior Change.

Column Type Converter processor change
The Column property replaces the Expression property. The behavior of the property has not changed.
Join processor enhancements
The processor allows specifying a custom join by columns or condition. You can also specify prefixes for columns with matching names.
Union processor enhancement
The processor includes a Union All option that performs a Snowflake Union All operation. The Union All operation includes all rows from all incoming streams and can include duplicate rows.

Behavior Change

Review Apply Function processor pipelines
This release includes a new Output Column property that replaces the following properties:
  • Add Prefix to New Output Columns
  • Add Suffix to New Output Columns
Existing Apply Function processor pipelines populate the new Output Column property based on values specified in the two replaced properties, with $0 representing the original column name as follows: <previously-defined prefix>$0<previously-defined suffix>.
Since $0 is used as a variable, review and update pipelines that used $0 in the old prefix and suffix properties. Then, update the upgrade-generated value for the Output Column property, as needed.
For example, say you have a prefix defined as $0new_ and no defined suffix. After this release, the Output Column property replaces those properties and is defined as follows: $0new_$0.
So if an original column name is time, this results in an output column of timenew_time.
In this case, you would probably update the Output Column property to new_$0, removing the first $0. Then, the same output column is named new_time.

Fixed Issue

  • Exploring Snowflake data requires including the https:// protocol in the specified Snowflake URL instead of simply specifying the domain name.

April 2023

The following Transformer for Snowflake release occurred in April 2023.

April 21, 2023

This release includes several enhancements and fixed issues, a behavior change, and a known issue.

New Features and Enhancements

Snowflake pipeline defaults
Snowflake pipeline defaults defined in your StreamSets account settings are placed in the corresponding properties of new pipelines as static values.
For example, if you go to My Account and specify a database under Snowflake Pipeline Defaults, every new pipeline that you create includes the specified database in the Database property of the pipeline.
This is a behavior change, but does not require action for existing pipelines.
Support for PKCS#8 private keys
When specifying private key Snowflake credentials, you can now enter a PKCS#8 private key. Previously, only PKCS#1 private keys were supported.
When specifying a PKCS#8 private key, include the -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY----- key delimiters.

Behavior Change

Snowflake pipeline defaults no longer integrate with pipeline default parameters
Previously, Snowflake pipeline defaults were placed in the corresponding properties of new pipelines as pipeline default parameters. Now, Snowflake pipeline defaults are placed in pipelines as static values.
For example, if you specify DEV as a Snowflake pipeline default warehouse, new pipelines have the Warehouse pipeline property set to DEV.
Previously, the Warehouse pipeline property would be set to ${SNOWFLAKE_WAREHOUSE}, which was a pipeline default parameter, and the SNOWFLAKE_WAREHOUSE parameter would be set to DEV.
Note: Existing pipelines are not changed. You can continue to use pipeline default parameters: SNOWFLAKE_ACCOUNT_URL, SNOWFLAKE_WAREHOUSE, SNOWFLAKE_DATABASE, SNOWFLAKE_SCHEMA, and SNOWFLAKE_ROLE. However, since they are no longer integrated with Snowflake pipeline defaults, they now function like any user-defined runtime parameter.

Fixed Issues

  • You cannot explore your Snowflake data when your Snowflake credentials defined in Control Hub use the private key authentication method.

  • You cannot explore your Snowflake data to select a role when no warehouse is defined.
  • You cannot explore your Snowflake data using the Safari browser.

Known Issue

Please note the following known issue:
  • When you use the More > Use Snowflake Defaults menu option to apply Snowflake pipeline defaults to an existing pipeline, pipeline default parameters are placed in pipeline properties and are defined based on their corresponding Snowflake pipeline defaults, as they were before this release.

    The correct behavior is to place configured Snowflake pipeline defaults into the pipeline as static values, instead of parameters.

    Despite this issue, the pipeline will still use the configured Snowflake pipeline defaults as intended, and perform as expected.

March 2023

The following Transformer for Snowflake release occurred in March 2023.

March 29, 2023

This release includes several new features, enhancements, and a behavior change.

New Features and Enhancements

Explore your Snowflake data
As you build a pipeline, you can explore your Snowflake data to select the following types of Snowflake elements that you want to connect to or process:
  • Roles
  • Warehouses
  • Databases
  • Schemas
  • Tables and views
  • Columns
Control Hub displays all Snowflake elements that your Snowflake credentials grant you access to.
For example, to define the Snowflake schema that a pipeline uses, you click the Select Schema icon: . Control Hub displays a dialog box listing all Snowflake databases and schemas that you have access to. You can browse or search for the database and schema to use.
Previously, you had to type the names of the Snowflake elements in the pipeline and stage properties.
Pipeline and fragment preview
Previewing a Transformer for Snowflake pipeline or fragment includes the following enhancements:
  • When you preview a pipeline or fragment, Control Hub uses the default configuration and no longer displays the Preview Configuration dialog box. While running preview, you can click the Preview Configuration icon () to change the configuration and then run the preview again.
  • When you preview a pipeline or fragment, the data displays in table view by default.
  • Table view displays only output data by default. You can optionally choose to display both input and output data.
  • Table view no longer displays colors for different types of data and changed data. List view continues to display colors.
  • While running preview, you can click the Expand icon () to quickly expand the preview panel to view more preview data.
For information about using data preview, see the Control Hub documentation.
Terminology changes
StreamSets has implemented the following terminology changes to more closely align with Snowflake terminology:
  • The term "column" replaces the term "field."
  • The term "row" replaces the term "record."
  • The Expression Evaluator processor has been renamed to the Snowflake SQL Evaluator processor.
Join processor
The Join processor joins data based on matching values in specific columns. The specified columns can have identical or unique names. Previously, the Join processor joined data based on identical column names or based on a condition that you defined.
In addition, when the joined tables have identical column names, the processor automatically adds a t1_ or t2_ prefix to the column names in the generated output to avoid duplicate names. Previously, you optionally configured the processor to add prefixes to column names and then defined the text to add.
These behavior changes might require you to modify Join processor pipelines.

Behavior Changes

Review Join processor pipelines
With this release, the Join processor joins data based on matching values in specific columns only. The processor can no longer join data based on a condition that you define.
If you previously configured the processor to join by condition, review the pipelines that include the processor and specify the column names with matching values used to join the data.
In addition, when the joined tables have identical column names, the Join processor now includes both columns in the generated output, automatically adding a t1_ or t2_ prefix to each column name to avoid duplicate names. You can no longer specify the text to use for the prefix. Previously, when the join columns had identical names, the processor included a single column in the generated output, and you could optionally specify the text to use as a prefix.
If you previously configured the processor to join tables based on join columns with identical names, review the pipelines that include the processor and add a Column Remover processor if you want to remove one of the joined columns in the output or add a Column Renamer processor if you want to remove the t1_ or t2_ prefix from the column names.
If you previously configured the processor to add prefixes to column names, review the pipelines that include the processor and add a Column Renamer processor if you still require those prefixes in the column names.

February 2023

The following Transformer for Snowflake release occurred in February 2023.

February 10, 2023

This release includes several enhancements.

Enhancements

Apply Functions processor
You can use the Apply Functions processor to apply the REGEXP_REPLACE Snowflake function to a column.
Column Type Converter processor
The Column Type Converter processor can convert columns with the Varchar, Variant, or Binary data type to the Geography data type.

January 2023

The following Transformer for Snowflake release occurred in January 2023.

January 20, 2023

This release includes several fixed issues.

Fixed Issues

  • If a table column name uses non-standard characters, the Slowly Changing Dimension processor fails to process the columns correctly.
  • When the Apply Function processor uses the Round() and Truncate() functions and is configured to keep only two decimal digits, the processor incorrectly replaces the additional digits with zeros instead of removing them.
  • The JSON Parser processor converts the original String column that contains the JSON to be parsed into a Variant column.

2022

The following Transformer for Snowflake releases occurred in 2022.

December 2022

The following Transformer for Snowflake release occurred in December 2022.

December 16, 2022

This release includes the following enhancements:

JSON Parser processor
The JSON Parser processor can extract all elements from the JSON data, extracting all columns as Variant columns or inferring the data type based on the column value. Previously, to extract all elements, you had to individually specify the path to each element.
User-defined functions
When you define an inline user-defined function (UDF) in a pipeline, you can choose to define a permanent or temporary UDF. A permanent UDF is stored in your Snowflake account and available for other pipelines to call as a precompiled UDF. A temporary UDF is available for the duration of the current pipeline run only. The temporary UDF cannot be called from additional pipelines.
Previously, each inline UDF was created as a permanent UDF.
Job monitoring
When you monitor Snowflake jobs, pipeline log messages include more detailed messages for several processors that can help with troubleshooting pipeline processing at the stage level.

November 2022

The following Transformer for Snowflake release occurred in November 2022.

November 18, 2022

This release includes the following enhancement:

Job monitoring
Control Hub includes the following enhancement when you monitor Snowflake jobs:
  • Pipeline log messages now include the number of bytes scanned by the Snowflake queries, the name and size of the warehouse where the Snowflake queries were executed, and the aggregated duration of the query executions.

October 2022

The following Transformer for Snowflake releases occurred in October 2022.

October 26, 2022

This release includes the following enhancements:
Job monitoring
Control Hub includes the following enhancements when you monitor Snowflake jobs:
  • Viewing log messages and Snowflake queries in the job summary is now enabled for all organizations.
  • A job generates a unique Snowflake query for each Snowflake SQL Query processor. When a destination has an upstream Snowflake SQL Query processor and you select that destination in the pipeline canvas to view generated Snowflake queries, at least two unique Snowflake queries display for the destination.

October 5, 2022

This release includes a fixed issue.

Fixed Issue
  • In rare cases, a draft run of a Transformer for Snowflake pipeline fails to start and displays an error message that Transformer for Snowflake has stopped responding.

August 2022

The following Transformer for Snowflake release occurred in August 2022.

August 26, 2022

This release includes the following enhancements:

Snowflake Partner Connect

StreamSets is happy to announce that you can now use Snowflake Partner Connect to access Transformer for Snowflake.

Job monitoring
When you monitor Snowflake jobs in Control Hub, you can now view the following information in the job summary:
  • Input and output row count
  • Log messages
  • Snowflake queries run for the job

July 2022

The following Transformer for Snowflake release occurred in July 2022.

July 20, 2022

This release includes the following enhancements, a behavior change, and several fixed issues:
Apply Snowflake pipeline defaults
When you apply Snowflake pipeline defaults to an existing pipeline, only the defaults that are defined in your account are applied to the pipeline.
Snowflake settings display
An update to the Snowflake Settings tab in your account settings eliminates confusion about the properties to be saved.
Slowly Changing Dimension processor
  • The processor can now process multiple changes to the same row in the same batch. Changes are applied in the order of the rows in the batch. If necessary, you can use a Sort processor upstream to ensure rows are in the appropriate order.
  • The Behavior for New Columns property no longer includes the “Keep for inserted rows and populate values from change data for updated rows” option. This results in a behavior change.
  • When configured to process Type 1 changes, the processor no longer requires specifying a tracking column.
Behavior Change
New column behavior for the Slowly Changing Dimension processor

Previously, the Behavior for New Columns property included the "Keep for inserted rows and populate values from change data for updated rows" option. With this update, the option has been removed.

Pipelines that previously used the option have the Behavior for New Columns property set to "Keep for newest rows and set to null for previous rows".

Review Slowly Changing Dimension pipelines to ensure that this behavior is acceptable.

Fixed Issues
  • When new columns appear in change data and the Behavior for New Columns property is configured to keep new columns, the Slowly Changing Dimension processor fails with an exception.
  • The Snowflake Table destination sometimes generates errors and stops the pipeline when merging data.

June 2022

The following Transformer for Snowflake releases occurred in June 2022.

June 17, 2022

This release includes the following enhancement:
Apply Snowflake pipeline defaults to existing pipelines
Snowflake pipeline defaults defined in your account settings are automatically included in new pipelines. When you add or update Snowflake pipeline defaults in your account settings, existing pipelines do not automatically inherit those changes.
You can now apply Snowflake pipeline defaults to existing pipelines on a case-by-case basis.

June 8, 2022

With this release, StreamSets is happy to announce that Transformer for Snowflake is now generally available and can be used for production workloads.

June 3, 2022 (public preview)

This release includes the following enhancement and a fixed issue:
Snowflake credentials

When you view existing Snowflake credentials in the Control Hub user interface, you can now view the previously-saved Snowflake username and role.

Fixed Issue
  • When the Slowly Changing Dimension processor is configured to use a Timestamp expression, and the master data includes a column that is used in that expression, the timestamps are not set correctly.

May 2022

The following Transformer for Snowflake releases occurred in May 2022.

May 20, 2022 (public preview)

This release includes the following new feature.
Draft runs
You can start a draft run of a Transformer for Snowflake pipeline to quickly test the pipeline logic.
A draft run is the execution of a draft pipeline. Use draft runs for development purposes only.

May 6, 2022 (public preview)

This release includes several fixed issues.

Fixed Issues
  • The Snowflake Table origin incorrectly requires all columns in the Additional Where Clause property to be listed in the Columns to Read list.
  • The Column Remover processor does not perform actions correctly based on regular expressions that contain uppercase letters.
  • Data preview can fail for pipelines that include a Column Renamer processor that uses certain configuration options.
  • The Find and Replace in All Columns by String option in the Column Renamer processor does not produce the expected results.

Known Issues

There are no important known issues at this time.