Release Notes

For help with Transformer for Snowflake, join the IBM StreamSets Community and use the Transformer for Snowflake tag when you post your question. To provide suggestions and feedback, add a new idea at https://ibm-data-and-ai.ideas.ibm.com/?project=SSETS.

March 2025

March 21 (Version 5.11.0)

The Transformer for Snowflake 5.11.0 release occurred on March 21, 2025.

This release includes some minor bug fixes and the following updates:
Documentation updates
IBM now hosts the documentation for recent releases of StreamSets products on the IBM documentation website.
With this release, Transformer for Snowflake stages point to the IBM StreamSets for Snowflake 5.11.x documentation and Control Hub points to the IBM StreamSets Control Hub documentation.
Older release documentation that is not hosted by IBM remains available in a documentation S3 archive that you can access through redirected links.

February 2025

February 19 (Version 5.10.0)

The Transformer for Snowflake 5.10.0 release occurred on February 19, 2025.

New Features and Enhancements

UDF processor
The new UDF processor enables easily adding a scalar Snowflake UDF to a pipeline. Use this processor to include UDFs in new pipelines. You might also use it in existing pipelines to replace Apply Functions processors that implement Snowflake UDFs.
Stored Procedure executor
The new Stored Procedure executor enables running a Snowflake stored procedure after the executor receives data. Use this executor to perform a task after pipeline processing completes.
Stage selection enhancement

The Stage Selector dialog box, which adds stages to the pipeline canvas, includes a new Functions tab. The Functions tab displays all stored procedures and user-defined functions available to the role defined in the Role pipeline property.

Use the Functions tab to create UDF and Stored Procedure processors based on the selected function or procedure.

Azure Key Vault credential store support
When your organization uses deployed Transformer for Snowflake engines, you can use Azure Key Vault credential stores to store sensitive information.

Fixed Issues

Fixed issue for hosted engines
  • When you do not have the Account URL property defined in your Snowflake pipeline defaults, the Snowflake Credentials dialog box appears when you create a new pipeline. After you enter the account URL and save the change, the pipeline does not appear to save the specified URL in the pipeline properties. This is simply a display issue.
Fixed issue for deployed engines
  • Draft runs and job runs do not work correctly for pipelines that access AWS credential stores.
  • When you first edit an imported pipeline, pipeline task icons, such as Validate and Preview, are incorrectly available for use before you select an authoring engine for the pipeline.

January 2025

January 22 (Version 5.9.0)

The Transformer for Snowflake 5.9.0 release occurred on January 22, 2025.

New Features and Enhancements

Apply Functions processor
The Apply Functions processor can apply logical functions to your data. Logical functions include type predicates such as IS_ARRAY and IS_CHAR, and conditional expressions such as IN, BETWEEN, and EQUAL_NULL.
Memory management enhancements
This release includes general enhancements to Transformer for Snowflake memory management.

Known Issues

Known issue for hosted engines

  • When you do not have the Account URL property defined in your Snowflake pipeline defaults, the Snowflake Credentials dialog box appears when you create a new pipeline. After you enter the account URL and save the change, the pipeline does not appear to save the specified URL in the pipeline properties. This is simply a display issue.

    Workaround: To see the saved URL, navigate away from the pipeline properties by adding a stage to the pipeline. Then, return to the pipeline properties.

Known issues for deployed engines
  • Draft runs and job runs do not work correctly for pipelines that access AWS credential stores.
  • When you first edit an imported pipeline, pipeline task icons, such as Validate and Preview, are incorrectly available for use before you select an authoring engine for the pipeline.

2024

December 2024

December 13 (Version 5.8.0)

The Transformer for Snowflake 5.8.0 release occurred on December 13, 2024.

New Features and Enhancements
Processor enhancements

  • Apply Functions - The processor can apply geospatial functions to your data.

  • Column Type Converter - The processor can convert columns of data to the Geography and Geometry types. For information about valid data type conversions, see the Snowflake documentation.

IBM StreamSets account

Your account, previously known as your “StreamSets account”, is now known as your “IBM StreamSets account”. You can still access your account settings using the My Account icon.

When your organization runs pipelines with the hosted Transformer for Snowflake engine, you can also configure Snowflake account properties in your IBM StreamSets account, as before.

Known Issue
Known issue for deployed engines

  • When you first edit an imported pipeline, pipeline task icons, such as Validate and Preview, are incorrectly available for use before you select an authoring engine for the pipeline.

November 2024

November 15 (Version 5.7.0)

The Transformer for Snowflake 5.7.0 release occurred on November 15, 2024.

New Features and Enhancements
Pipeline validation enhancements
When you validate a pipeline, Transformer for Snowflake now provides warnings for the following errors:
  • Specified databases, schemas, tables, or columns do not exist.
  • Specified Snowflake SQL functions or user-defined functions do not exist, or the number or type of specified arguments do not match those of the specified function.
  • Invalid SQL expressions.
Also, when an error is related to a property that contains multiple fields, the pipeline canvas now highlights the field that generated the error.
Deployed Docker engine update
Docker images for Transformer for Snowflake 5.7.0 use RedHat UBI9 OpenJDK 11 as the parent image. This change can have upgrade impact.
Preview enhancement
Transformer for Snowflake now supports previewing Geography and Geometry data.
Upgrade Impact for Deployed Docker Engines
Review Dockerfiles for custom Docker images
In previous releases, Transformer for Snowflake Docker images used eclipse-temurin 11.0.21_9-jdk as the parent image. With this release, Transformer for Snowflake Docker images use RedHat UBI9 OpenJDK 11 as the parent image.
If you build custom Transformer for Snowflake images with earlier releases of streamsets/streamflake as the parent image, review your Dockerfiles. Make all required updates to ensure that they are compatible with RedHat UBI9 OpenJDK 11 before you build a new custom Transformer for Snowflake image.

October 2024

October 16 (Version 5.6.0)

The Transformer for Snowflake 5.6.0 release occurred on October 16, 2024.

New Features and Enhancements
Snowflake connection enhancement
Transformer for Snowflake pipelines now support OAuth authentication defined in Control Hub Snowflake connections.
For more information about Snowflake connections, see the Control Hub documentation.

September 2024

September 18 and 20 (Version 5.5.0)

The Transformer for Snowflake 5.5.0 release occurred on September 18, 2024 for the eu01.hub.streamsets.com instance. The release occurred on September 20, 2024 for all other instances.

New Features and Enhancements
LLM properties renamed
The following LLM processor properties have been renamed with this release:
  • LLM Summarize processor - The Source property is now the Source Column property.
  • LLM Translate processor - The Source Columns property is now the Source Column property, and the Output Columns property is the Output Column property.
Fixed Issues
  • When configured to overwrite a table by dropping it, the Snowflake Table destination incorrectly keeps all existing columns when the Data Drift Enabled property is also selected.

August 2024

August 16 (Version 5.4.0)

The Transformer for Snowflake 5.4.0 release occurred on August 16, 2024.

New Features and Enhancements
New Case processor
The Case processor returns values based on a sequence of conditions, like a cascading if-then-else statement.
New LLM Complete processor
The LLM Complete processor generates a customized response to an English-language prompt based on the selected large language model (LLM) and the Snowflake LLM Complete function.
Note: At this time, Snowflake charges differently for LLM processing. See the Snowflake consumption rates for details.
Data preview enhancement
A new Limit Stage Input preview property determines how preview provides data to stages in Transformer for Snowflake pipelines.
For more information, see the Control Hub documentation.

July 2024

July 24 (Version 5.3.0)

The Transformer for Snowflake 5.3.0 release occurred on July 24, 2024.

New Features and Enhancements
New LLM Extract Answer processor
The LLM Extract Answer processor uses a Snowflake Large Language Model (LLM) function to extract answers to questions from English-language text. It also generates a confidence score for the extracted answer.
Note: At this time, Snowflake charges differently for LLM processing. See the Snowflake consumption rates for details.
Product rename
Following the IBM acquisition of StreamSets, Transformer for Snowflake is part of what is now known as IBM StreamSets for Snowflake.
You can still find information about using IBM StreamSets for Snowflake in this Transformer for Snowflake documentation and in the Control Hub documentation.
For more information, see What is IBM StreamSets for Snowflake?
Fixed Issues
  • The Snowflake Table destination can generate errors when you configure it to write to a view.

June 2024

June 26 (Version 5.2.0)

The Transformer for Snowflake 5.2.0 release occurred on June 26, 2024.

New Features and Enhancements
New LLM processors
The following new processors use Snowflake Large Language Model (LLM) functions to process data:
Note: At this time, Snowflake charges differently for LLM processing. See the Snowflake consumption rates for details.
Snowflake Table origin enhancement
When you specify the columns for the origin to read, you can now rename those columns as needed.
Fixed Issues
Fixed issue for deployed Transformer for Snowflake engines
  • Proxy properties defined in the advanced properties for the deployment are ignored. As a result, engine communication with Control Hub or Snowflake that requires a proxy will fail.

May 2024

May 22 (Version 5.1.0)

The Transformer for Snowflake 5.1.0 release occurred on May 22, 2024.

Fixed Issues

  • When a pipeline includes multiple conditions, such as a Where clause in an origin and a condition in a Filter processor, the conditions are joined incorrectly.

    With this fix, multiple conditions are handled differently in pipelines, which is a behavior change.

  • When you configure a Filter processor with a comparison operator and value, then change the comparison operator to ISNULL or ISNOTNULL, the processor stores the previously defined value and attempts to use it, causing the pipeline to fail with a SQL syntax error.

  • When a pipeline is configured to use an invalid Snowflake URL, the job that runs the pipeline incorrectly shows an Active status instead of a Start Error status.
  • An Amazon EC2 deployment for a Transformer for Snowflake engine might fail to start the engine if the feature version for the parent environment is AWS_2021_06_01.
  • The Proxy tab for a Transformer for Snowflake deployment is empty.

Behavior Change
Review pipelines that include multiple conditions

With this release, when a pipeline includes multiple conditions, they are joined as expected. In previous releases, they were joined inappropriately. Review pipelines with multiple conditions to verify that the data is processed as expected.

For example, say you have an origin with the following condition defined in the Additional Where Clause property: monthlySales >= 10000. And you have a Filter processor with the following condition in the Where Clause property: country = "Spain" OR "Germany".

In previous releases, the pipeline merges the conditions incorrectly, as follows:

monthlySales >= 10000 AND country = "Spain" OR "Germany"

With this fix, the pipeline merges the conditions appropriately, as follows:

monthlySales >= 10000 AND (country = "Spain" OR "Germany")

Known Issue
Known issue for deployed Transformer for Snowflake engines
  • Proxy properties defined in the advanced properties for the deployment are ignored. As a result, engine communication with Control Hub or Snowflake that requires a proxy will fail. Do not use a proxy at this time.

April 2024

April 26 (Version 5.0.0)

The Transformer for Snowflake 5.0.0 release occurred on April 26, 2024.

New Features and Enhancements
Deployed Transformer for Snowflake engines
When needed, your organization can deploy Transformer for Snowflake engines to a private network, which can be on-premises or on the AWS cloud computing platform. This first release of the Transformer for Snowflake deployed engine is version 5.0.0.
By default, organizations use the hosted Transformer for Snowflake engine to run pipelines. The hosted engine suits the needs of most organizations and provides the easiest way to work with Transformer for Snowflake: You do not need to install or manage engines – you simply start configuring pipelines.
An organization can use either the hosted engine or deployed engines, not both. Deploying Transformer for Snowflake engines requires that your organization have the appropriate account agreement. For information about your account agreement, contact your IBM StreamSets account team.
For more information, see Hosted or Deployed Engines.
This feature includes the following enhancements:
  • AWS credential store support - When your organization uses deployed Transformer for Snowflake engines, you can use AWS credential stores to store sensitive information such as passwords or private keys.
  • Snowflake connection support - When your organization uses deployed Transformer for Snowflake engines, you use Snowflake connections to provide connection information for pipelines. As a result pipeline configuration properties differ depending on the engine type that your organization uses. For more information about connections, see the Control Hub documentation.
Known Issues

This release includes known issues that apply separately to hosted and deployed engines:

Known issues for hosted Transformer for Snowflake engines
  • When a pipeline is configured to use an invalid Snowflake URL, the job that runs the pipeline incorrectly shows an Active status instead of a Start Error status.
Known issues for deployed Transformer for Snowflake engines
  • An Amazon EC2 deployment for a Transformer for Snowflake engine might fail to start the engine if the feature version for the parent environment is AWS_2021_06_01.

    Workaround: Change the environment feature version to AWS_2023_12_15.

  • The Proxy tab for a Transformer for Snowflake deployment is empty.

    Workaround: Copy and paste the proxy properties from a deployment for a different engine type.

March 2024

March 27, 2024

This release includes a fixed issue.

Fixed Issue
  • When you create a job instance or template for a Transformer for Snowflake pipeline through the Run > Job Instances or Run > Job Template menu options, the job creation wizard requires specifying a deployment, which is not applicable for Transformer for Snowflake pipelines.

    Creating jobs through the pipeline canvas or the Pipelines view was not affected by this issue.

February 2024

February 21, 2024

This release includes a fixed issue.

Fixed Issue
  • The Apply Function processor unnecessarily changes the column order of incoming data.

January 2024

The following Transformer for Snowflake release occurred in January 2024.

January 24, 2024

This release includes a new feature, several fixed issues, and a behavior change.

New Feature
New Snowflake Notification executor
The new Snowflake Notification executor sends an email after the pipeline completes all writes. The executor uses a Snowflake email notification integration to send email to Snowflake users in your account when the specified condition has been met.
Fixed Issues

  • Preview fails to run if the pipeline includes a destination or executor after an invalid stage.

  • When the Join processor has the Join Command property set to Using, the processor adds a __dup__<n> suffix, where <n> is a number, to differentiate between duplicate column names in the output.

    With this fix, the processor uses a T1_ or T2_ prefix as with other join methods, which is a behavior change.

Behavior Change
Review pipelines that join data with the Using join command
With this release, when a Join processor is configured with the Join Command property set to Using, and when output includes duplicate column names, the processor adds a T1_ or T2_ prefix to those column names to avoid duplicate names: T1_ for the left input, and T2_ for the right input.
Previously, due to the fixed issue listed above, the processor added the following suffix to duplicate output column names instead: __dup__<n>, where <n> is a number.
Review pipelines that join data with the Using command if the output includes duplicate column names. Then, update downstream stages as needed to use the new column names for duplicate columns.
For example, say you have a pipeline that joins data with Using. Incoming data includes a STOREID column and an ADDRESS column from each input, and the processor is configured to join the data using the STOREID column. Before this release, the Join processor output included a STOREID column, an ADDRESS column from the first input, and an ADDRESS__dup__<n> column from the second input. After this release, the same output columns are named: STOREID, T1_ADDRESS, and T2_ADDRESS. So any downstream stages that reference the address columns must be updated to function as expected.

2023

The following Transformer for Snowflake releases occurred in 2023.

December 2023

The following Transformer for Snowflake release occurred in December 2023.

December 20, 2023

This release includes several enhancements and a fixed issue.

New Features and Enhancements
Snowflake View destination
The new Snowflake View destination creates a Snowflake view. The destination can create the view or replace an existing view. The destination creates a standard view by default, but you can configure it to create a secure, a materialized, or a secure materialized view.
Union processor enhancement

When you configure the Union processor to require the same schema for all incoming data, the processor adjusts the order of the columns for all data based on the column order of the first batch of data from the first input location.

Since the Snowflake Union operation can produce undesired results when columns are ordered differently, this enhancement provides uniformity in column order, when possible.

Fixed Issue

  • Draft run jobs do not stop when the pipeline includes an open output stream. With this change, draft runs require all output streams to be connected.

November 2023

The following Transformer for Snowflake release occurred in November 2023.

November 15, 2023

This release includes an enhancement and a fixed issue.

Enhancement
  • When you preview a pipeline or fragment that generates an error, Control Hub now provides preview data for the pipeline stages before the stage that generated the error. For example, if the pipeline includes an error in the fifth stage, Control Hub provides preview data for the first four stages in the pipeline.

    For more information about previewing Transformer for Snowflake pipelines, see the Control Hub documentation.

Fixed Issue

  • Draft run jobs do not stop when the pipeline includes an open output stream. With this change, draft runs require all output streams to be connected.

October 2023

The following Transformer for Snowflake release occurred in October 2023.

October 27, 2023

This release includes several enhancements.

New Features and Enhancements
Data preview enhancements
Data preview can run when a pipeline includes stages or branches that are not connected to an origin. The pipeline provides preview data for stages connected to a configured origin.
For more information, see the Control Hub documentation.
Join processor property names
For clarity, the following Join processor properties have been renamed:
  • The Join Using Condition property has been renamed to the Join Condition property.
  • The Input 1 property has been renamed to the Input 1 (Left) property.
  • The Input 2 property has been renamed to the Input 2 (Right) property.

This change has no upgrade impact.

September 2023

The following Transformer for Snowflake release occurred in September 2023.

September 27, 2023

This release includes several new features, enhancements, and fixed issues.

New Features and Enhancements
Snowflake credentials

  • Transformer for Snowflake validates Snowflake credentials before saving them to your account settings.

  • When you create a new pipeline, the Snowflake Credentials dialog box appears so you can specify the Snowflake account URL and user credentials to use, if you have not already saved them to your StreamSets account.

Stages

  • When using the Select Column from Schema icon in a processor, Transformer for Snowflake automatically loads the schema when needed.

  • The Aggregate processor includes an Any Value function that returns a random value from the group.

  • The Column Renamer processor can add a prefix or suffix to all columns.

  • The Snowflake Table origin and destination are automatically named for the selected table unless you manually override the name.

    In the August 2023 release, the Snowflake Table origin and destination became automatically named for the selected table. This release enables you to specify a different stage name using the Name property on the General tab.

    If you select a new table for the stage, the name of the selected table overrides the manually-entered stage name.

    Note: With this update, Snowflake Table origins and destinations in existing pipelines are automatically named for the selected table. Override the automatic stage names as desired. The functionality of the pipelines and associated jobs are not affected.
Fixed Issues

  • When you configure a custom join condition with wrong column names in the Join Using Condition property, the Join processor performs a cross join instead of the configured join type.

  • The Union processor can provide preview data for only one input, instead of displaying data for all inputs.

    With this fix, the preview batch for the Union processor includes rows for each input, when possible. For example, if the preview batch is ten records, and there are five input streams, preview displays two rows for each input.

  • When selecting a role, schema, or warehouse, connection errors display in two locations in the same dialog box.

August 2023

The following Transformer for Snowflake release occurred in August 2023.

August 23, 2023

This release includes several new features and enhancements, a behavior change, and a fixed issue.

New Features and Enhancements
Snowflake Table origin and destination enhancements
  • Default stage naming - When you specify a table to read in the Snowflake Table origin, the table name becomes the name of the origin. Similarly, when you specify a table to write to in the Snowflake Table destination, the table name becomes the name of the destination.
  • Improved table selection:
    • You can now specify a table to use by selecting the database, schema, and table from lists.
    • If you specify the schema, database, and table to use, then click the Select Table icon, the explorer window opens to the specified position.

With these enhancements, the Overwrite Database, Schema, and Table dialog box has been removed.

Additional stage enhancements
  • Apply function processor - You can now use all Snowflake date functions in the processor, as well as a new Transformer for Snowflake Age function.

    This enhancement includes a behavior change. For more information, see Behavior Change.

  • Unpivot processor - You can use several methods to specify the columns to unpivot. In addition to listing specific column names, you can now specify strings and regular expressions to match column names.
    This enhancement includes the following changes:
    • New Column Match Criteria property indicates how to specify the columns to unpivot.
    • Columns to Pivot property renamed to the Columns property. The property still allows you to list the columns to unpivot.
    • Several new properties allow you to define strings, prefixes, or suffixes in column names, or regular expressions that match column names.
Additional pipeline enhancement
Behavior Change
With this release, the Apply Functions processor supports using Snowflake date and time functions, as well as the Transformer for Snowflake Age function. With this enhancement, the Treat as Constant property is no longer available with the following functions:
  • DATEDIFF
  • NEXT_DAY
  • PREVIOUS_DAY

Apply Functions processors in existing pipelines are updated accordingly and do not require review.

However, when using date and time functions with a new Apply Functions processor, and when configuring the Comparison Date property, you must enclose constants for the listed functions in single quotation marks. Previously, you selected the Treat as Constant property, instead.

The Comparison Date property is available with the DATEDIFF, NEXT_DAY, and PREVIOUS_DAY functions, as well as the new AGE function.

For example, say you add a new Apply Functions processor to a pipeline and set the Function Type property to Date and Time and the Date and Time Function property to NEXT_DAY. To use a constant as a comparison, in the Comparison Date property, you specify the date to use in single quotation marks as follows: '2023/05/08'.

Fixed Issue
  • The Stream Selector processor inappropriately drops rows from the pipeline when a column used in a condition contains a null value.

July 2023

The following Transformer for Snowflake release occurred in July 2023.

July 21, 2023

This release includes several new features, enhancements, and changes.

New Features and Enhancements
Stages
  • New Null Handling processor - Replaces null values based on the specified function and related properties.
  • Apply Functions processor - When defining a regular expression to represent multiple output columns, you can use group variables, such as $1 and $2, to represent the groups defined in the corresponding expression that defines the input columns.
  • Snowflake SQL Evaluator processor - The processor has been renamed the Column Transformer processor.
  • Aggregate, Cube, and Rollup processors - These processors allow entering a regular expression in the Aggregate Column property. This allows you to specify more than one aggregate column for a calculation.

    When entering a regular expression in the Aggregate Column property, use $0 to represent the original column names in the Output Column Name property. For example $0_avg adds _avg to the original column names.

Snowflake credentials and pipeline defaults
  • If you have not defined Snowflake credentials or Snowflake pipeline defaults in your StreamSets account, when you configure a pipeline and perform a task that requires those details, you can specify them in convenient pop-up dialog boxes.

    Those details are then saved in your StreamSets account for future use.

    For example, say you create your first pipeline and click the Select Table icon in the pipeline. If you have not provided Snowflake credentials, a Snowflake Credentials dialog box appears so you can specify those details.

    You are prompted to select the role and warehouse to use, then you navigate to the appropriate database and schema, and select the table for the pipeline.

    Since you do not have Snowflake settings defined in your StreamSets account, the credentials, role, warehouse, database, and schema that you specified during pipeline creation are saved in your StreamSets account to be used in future pipelines.

  • You can also define Snowflake credentials in the pipeline using the Enter Snowflake Credentials option available in the More menu.
  • The Snowflake Credentials dialog box includes the following changes:

    • It includes the Snowflake account URL property.

    • It no longer includes the Role property. You can specify a default role as a Snowflake pipeline default.

    • It appears when you have not provided all of the required credentials when you configure a pipeline.

Pipeline configuration changes
  • The Warehouse and Working Default Schema properties are now required properties. Though, as always, when you create a pipeline, they are populated with associated Snowflake default properties stored in your StreamSets account.

    Pipelines and jobs provide validation messages to ensure that you define these properties.

  • The Role property now defaults to the Snowflake Public role.

    For existing pipelines, StreamSets recommends specifying a role for each pipeline.

June 2023

The following Transformer for Snowflake release occurred in June 2023.

June 14 and 21, 2023

This release includes several new features, enhancements, and fixed issues.

The release occurred in two phases across all Control Hub instances in the StreamSets platform. The first phase occurred on June 14, 2023, and the second phase occurred on June 21, 2023.

New Features and Enhancements
Stages
  • New Unpivot processor - Rotates a table, converting the specified columns into rows.
  • New Window Function processor - Performs window calculations and writes the results to a specified column. The processor performs calculations on windows or window frames.
  • Filter processor enhancements - The Filter Condition property has been replaced. You can now configure a set of properties to build a condition or enter the condition in a new Where Clause property.

    When building a condition, you can specify a column, select a comparison operator, and enter the value to use.

    Existing pipelines place the existing filter condition into the new Where Clause property.

Pipeline enhancements
  • When Snowflake credentials are not defined in your StreamSets account settings, you can now easily configure them in a pipeline when you explore your data.

    For example, if you explore your data to select a role for a pipeline, and you have not added your Snowflake username and password to your StreamSets account settings, a dialog box displays that enables you to specify that information.

  • The Default Schema pipeline property has been renamed to Pipeline Working Schema.
Fixed Issues
  • Timestamps display in the milliseconds instead of a standard date format, such as "May 31, 2023 11:00:00 AM".
  • In the Join processor, the column selector for input 2 displays columns from input 1.
  • Existing Join processor pipelines generate an error when you edit the processor to specify the columns to use when you define a join condition.
  • When using the Pivot processor to autocalculate pivot values, errors occur if the pivoted columns include NULL values.

May 2023

The following Transformer for Snowflake release occurred in May 2023.

May 12, 2023

This release includes several enhancements, a behavior change, and a fixed issue.

New Features and Enhancements
Data preview enhancements
  • You can preview data for incomplete pipelines, as long as all required fields are defined.
  • You can run preview up to a specific stage in the pipeline instead of the entire pipeline.
  • Preview displays Snowflake data types instead of StreamSets data types.
  • Preview provides as much preview data as possible when errors cause a preview request to fail to complete.
Column selection
  • To specify column names, you can use preview data to select columns instead of typing column names.
  • After previewing data, stages provide autocomplete suggestions for column names in properties that accept multiple columns.
  • After selecting multiple columns, you can drag the columns to change the order in which they appear.
  • In some stages, after you select a column to use, you can then enter a regular expression that includes the column name or enter additional column names.
Table selection
In the Snowflake Table origin and destination, when you can explore your data and select the table that you want to use, you can explore the entire warehouse instead of being limited to tables within the default database or schema.
Apply Function processor enhancements
  • The Column property replaces the Column Expression property.

    Use the Column property to specify the column to apply the function to. You can also specify a regular expression that applies the function to a set of columns.

  • The Output Column property replaces the Add Prefix to New Output Columns property and the Add Suffix to New Output Columns property.

    Existing pipelines merge any prefix and suffix values defined in the old properties into the Output Column property using $0 to represent the original column name. If existing pipelines use $0 in the old prefix and suffix properties, you should review those pipelines. For more information, see Behavior Change.

Column Type Converter processor change
The Column property replaces the Expression property. The behavior of the property has not changed.
Join processor enhancements
The processor allows specifying a custom join by columns or condition. You can also specify prefixes for columns with matching names.
Union processor enhancement
The processor includes a Union All option that performs a Snowflake Union All operation. The Union All operation includes all rows from all incoming streams and can include duplicate rows.
Behavior Change
Review Apply Function processor pipelines
This release includes a new Output Column property that replaces the following properties:
  • Add Prefix to New Output Columns
  • Add Suffix to New Output Columns
Existing Apply Function processor pipelines populate the new Output Column property based on values specified in the two replaced properties, with $0 representing the original column name as follows: <previously-defined prefix>$0<previously-defined suffix>.
Since $0 is used as a variable, review and update pipelines that used $0 in the old prefix and suffix properties. Then, update the upgrade-generated value for the Output Column property, as needed.
For example, say you have a prefix defined as $0new_ and no defined suffix. After this release, the Output Column property replaces those properties and is defined as follows: $0new_$0.
So if an original column name is time, this results in an output column of timenew_time.
In this case, you would probably update the Output Column property to new_$0, removing the first $0. Then, the same output column is named new_time.
Fixed Issue
  • Exploring Snowflake data requires including the https:// protocol in the specified Snowflake URL instead of simply specifying the domain name.

April 2023

The following Transformer for Snowflake release occurred in April 2023.

April 21, 2023

This release includes several enhancements and fixed issues, a behavior change, and a known issue.

New Features and Enhancements
Snowflake pipeline defaults
Snowflake pipeline defaults defined in your StreamSets account settings are placed in the corresponding properties of new pipelines as static values.
For example, if you go to My Account and specify a database under Snowflake Pipeline Defaults, every new pipeline that you create includes the specified database in the Database property of the pipeline.
This is a behavior change, but does not require action for existing pipelines.
Support for PKCS#8 private keys
When specifying private key Snowflake credentials, you can now enter a PKCS#8 private key. Previously, only PKCS#1 private keys were supported.
When specifying a PKCS#8 private key, include the -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY----- key delimiters.
Behavior Change
Snowflake pipeline defaults no longer integrate with pipeline default parameters
Previously, Snowflake pipeline defaults were placed in the corresponding properties of new pipelines as pipeline default parameters. Now, Snowflake pipeline defaults are placed in pipelines as static values.
For example, if you specify DEV as a Snowflake pipeline default warehouse, new pipelines have the Warehouse pipeline property set to DEV.
Previously, the Warehouse pipeline property would be set to ${SNOWFLAKE_WAREHOUSE}, which was a pipeline default parameter, and the SNOWFLAKE_WAREHOUSE parameter would be set to DEV.
Note: Existing pipelines are not changed. You can continue to use pipeline default parameters: SNOWFLAKE_ACCOUNT_URL, SNOWFLAKE_WAREHOUSE, SNOWFLAKE_DATABASE, SNOWFLAKE_SCHEMA, and SNOWFLAKE_ROLE. However, since they are no longer integrated with Snowflake pipeline defaults, they now function like any user-defined runtime parameter.
Fixed Issues
  • You cannot explore your Snowflake account when your Snowflake credentials defined in Control Hub use the private key authentication method.

  • You cannot explore your Snowflake account to select a role when no warehouse is defined.
  • You cannot explore your Snowflake account using the Safari browser.
Known Issue
Please note the following known issue:
  • When you use the More > Use Snowflake Defaults menu option to apply Snowflake pipeline defaults to an existing pipeline, pipeline default parameters are placed in pipeline properties and are defined based on their corresponding Snowflake pipeline defaults, as they were before this release.

    The correct behavior is to place configured Snowflake pipeline defaults into the pipeline as static values, instead of parameters.

    Despite this issue, the pipeline will still use the configured Snowflake pipeline defaults as intended, and perform as expected.

March 2023

The following Transformer for Snowflake release occurred in March 2023.

March 29, 2023

This release includes several new features, enhancements, and a behavior change.

New Features and Enhancements
Explore your Snowflake account
As you build a pipeline, you can explore your Snowflake account to select the following types of Snowflake elements that you want to connect to or process:
  • Roles
  • Warehouses
  • Databases
  • Schemas
  • Tables and views
  • Columns
Control Hub displays all Snowflake elements that your Snowflake credentials grant you access to.
For example, to define the Snowflake schema that a pipeline uses, you click the Select Schema icon: . Control Hub displays a dialog box listing all Snowflake databases and schemas that you have access to. You can browse or search for the database and schema to use.
Previously, you had to type the names of the Snowflake elements in the pipeline and stage properties.
Pipeline and fragment preview
Previewing a Transformer for Snowflake pipeline or fragment includes the following enhancements:
  • When you preview a pipeline or fragment, Control Hub uses the default configuration and no longer displays the Preview Configuration dialog box. While running preview, you can click the Preview Configuration icon () to change the configuration and then run the preview again.
  • When you preview a pipeline or fragment, the data displays in table view by default.
  • Table view displays only output data by default. You can optionally choose to display both input and output data.
  • Table view no longer displays colors for different types of data and changed data. List view continues to display colors.
  • While running preview, you can click the Expand icon () to quickly expand the preview panel to view more preview data.
For information about using data preview, see the Control Hub documentation.
Terminology changes
The following terms have changed to more closely align with Snowflake terminology:
  • The term "column" replaces the term "field."
  • The term "row" replaces the term "record."
  • The Expression Evaluator processor has been renamed to the Snowflake SQL Evaluator processor.
Join processor
The Join processor joins data based on matching values in specific columns. The specified columns can have identical or unique names. Previously, the Join processor joined data based on identical column names or based on a condition that you defined.
In addition, when the joined tables have identical column names, the processor automatically adds a t1_ or t2_ prefix to the column names in the generated output to avoid duplicate names. Previously, you optionally configured the processor to add prefixes to column names and then defined the text to add.
These behavior changes might require you to modify Join processor pipelines.
Behavior Changes
Review Join processor pipelines
With this release, the Join processor joins data based on matching values in specific columns only. The processor can no longer join data based on a condition that you define.
If you previously configured the processor to join by condition, review the pipelines that include the processor and specify the column names with matching values used to join the data.
In addition, when the joined tables have identical column names, the Join processor now includes both columns in the generated output, automatically adding a t1_ or t2_ prefix to each column name to avoid duplicate names. You can no longer specify the text to use for the prefix. Previously, when the join columns had identical names, the processor included a single column in the generated output, and you could optionally specify the text to use as a prefix.
If you previously configured the processor to join tables based on join columns with identical names, review the pipelines that include the processor and add a Column Remover processor if you want to remove one of the joined columns in the output or add a Column Renamer processor if you want to remove the t1_ or t2_ prefix from the column names.
If you previously configured the processor to add prefixes to column names, review the pipelines that include the processor and add a Column Renamer processor if you still require those prefixes in the column names.

February 2023

The following Transformer for Snowflake release occurred in February 2023.

February 10, 2023

This release includes several enhancements.

Enhancements
Apply Functions processor
You can use the Apply Functions processor to apply the REGEXP_REPLACE Snowflake function to a column.
Column Type Converter processor
The Column Type Converter processor can convert columns with the Varchar, Variant, or Binary data type to the Geography data type.

January 2023

The following Transformer for Snowflake release occurred in January 2023.

January 20, 2023

This release includes several fixed issues.

Fixed Issues
  • If a table column name uses non-standard characters, the Slowly Changing Dimension processor fails to process the columns correctly.
  • When the Apply Function processor uses the Round() and Truncate() functions and is configured to keep only two decimal digits, the processor incorrectly replaces the additional digits with zeros instead of removing them.
  • The JSON Parser processor converts the original String column that contains the JSON to be parsed into a Variant column.

2022

The following Transformer for Snowflake releases occurred in 2022.

December 2022

The following Transformer for Snowflake release occurred in December 2022.

December 16, 2022

This release includes the following enhancements:

JSON Parser processor
The JSON Parser processor can extract all elements from the JSON data, extracting all columns as Variant columns or inferring the data type based on the column value. Previously, to extract all elements, you had to individually specify the path to each element.
User-defined functions
When you define an inline user-defined function (UDF) in a pipeline, you can choose to define a permanent or temporary UDF. A permanent UDF is stored in your Snowflake account and available for other pipelines to call as a precompiled UDF. A temporary UDF is available for the duration of the current pipeline run only. The temporary UDF cannot be called from additional pipelines.
Previously, each inline UDF was created as a permanent UDF.
Job monitoring
When you monitor Snowflake jobs, pipeline log messages include more detailed messages for several processors that can help with troubleshooting pipeline processing at the stage level.

November 2022

The following Transformer for Snowflake release occurred in November 2022.

November 18, 2022

This release includes the following enhancement:

Job monitoring
Control Hub includes the following enhancement when you monitor Snowflake jobs:
  • Pipeline log messages now include the number of bytes scanned by the Snowflake queries, the name and size of the warehouse where the Snowflake queries were executed, and the aggregated duration of the query executions.

October 2022

The following Transformer for Snowflake releases occurred in October 2022.

October 26, 2022

This release includes the following enhancements:
Job monitoring
Control Hub includes the following enhancements when you monitor Snowflake jobs:
  • Viewing log messages and Snowflake queries in the job summary is now enabled for all organizations.
  • A job generates a unique Snowflake query for each Snowflake SQL Query processor. When a destination has an upstream Snowflake SQL Query processor and you select that destination in the pipeline canvas to view generated Snowflake queries, at least two unique Snowflake queries display for the destination.

October 5, 2022

This release includes a fixed issue.

Fixed Issue
  • In rare cases, a draft run of a Transformer for Snowflake pipeline fails to start and displays an error message that Transformer for Snowflake has stopped responding.

August 2022

The following Transformer for Snowflake release occurred in August 2022.

August 26, 2022

This release includes the following enhancements:

Snowflake Partner Connect

You can now use Snowflake Partner Connect to access Transformer for Snowflake.

Job monitoring
When you monitor Snowflake jobs in Control Hub, you can now view the following information in the job summary:
  • Input and output row count
  • Log messages
  • Snowflake queries run for the job

July 2022

The following Transformer for Snowflake release occurred in July 2022.

July 20, 2022

This release includes the following enhancements, a behavior change, and several fixed issues:
Apply Snowflake pipeline defaults
When you apply Snowflake pipeline defaults to an existing pipeline, only the defaults that are defined in your account are applied to the pipeline.
Snowflake settings display
An update to the Snowflake Settings tab in your account settings eliminates confusion about the properties to be saved.
Slowly Changing Dimension processor
  • The processor can now process multiple changes to the same row in the same batch. Changes are applied in the order of the rows in the batch. If necessary, you can use a Sort processor upstream to ensure rows are in the appropriate order.
  • The Behavior for New Columns property no longer includes the “Keep for inserted rows and populate values from change data for updated rows” option. This results in a behavior change.
  • When configured to process Type 1 changes, the processor no longer requires specifying a tracking column.
Behavior Change
New column behavior for the Slowly Changing Dimension processor

Previously, the Behavior for New Columns property included the "Keep for inserted rows and populate values from change data for updated rows" option. With this update, the option has been removed.

Pipelines that previously used the option have the Behavior for New Columns property set to "Keep for newest rows and set to null for previous rows".

Review Slowly Changing Dimension pipelines to ensure that this behavior is acceptable.

Fixed Issues
  • When new columns appear in change data and the Behavior for New Columns property is configured to keep new columns, the Slowly Changing Dimension processor fails with an exception.
  • The Snowflake Table destination sometimes generates errors and stops the pipeline when merging data.

June 2022

The following Transformer for Snowflake releases occurred in June 2022.

June 17, 2022

This release includes the following enhancement:
Apply Snowflake pipeline defaults to existing pipelines
Snowflake pipeline defaults defined in your account settings are automatically included in new pipelines. When you add or update Snowflake pipeline defaults in your account settings, existing pipelines do not automatically inherit those changes.
You can now apply Snowflake pipeline defaults to existing pipelines on a case-by-case basis.

June 8, 2022

With this release, Transformer for Snowflake is generally available and can be used for production workloads.

June 3, 2022 (public preview)

This release includes the following enhancement and a fixed issue:
Snowflake credentials

When you view existing Snowflake credentials in the Control Hub user interface, you can now view the previously-saved Snowflake username and role.

Fixed Issue
  • When the Slowly Changing Dimension processor is configured to use a Timestamp expression, and the master data includes a column that is used in that expression, the timestamps are not set correctly.

May 2022

The following Transformer for Snowflake releases occurred in May 2022.

May 20, 2022 (public preview)

This release includes the following new feature.
Draft runs
You can start a draft run of a Transformer for Snowflake pipeline to quickly test the pipeline logic.
A draft run is the execution of a draft pipeline. Use draft runs for development purposes only.

May 6, 2022 (public preview)

This release includes several fixed issues.

Fixed Issues
  • The Snowflake Table origin incorrectly requires all columns in the Additional Where Clause property to be listed in the Columns to Read list.
  • The Column Remover processor does not perform actions correctly based on regular expressions that contain uppercase letters.
  • Data preview can fail for pipelines that include a Column Renamer processor that uses certain configuration options.

  • The Find and Replace in All Columns by String option in the Column Renamer processor does not produce the expected results.