Release Notes
For help with Transformer for Snowflake, join the StreamSets Community and use the Transformer for
Snowflake
tag when you post your question. To provide suggestions and
feedback, contact productfeedback@streamsets.com.
May 2023
The following Transformer for Snowflake release occurred in May 2023.
May 12, 2023
This release includes several enhancements, a behavior change, and a fixed issue.
New Features and Enhancements
- Data preview enhancements
-
- You can preview data for incomplete pipelines, as long as all required fields are defined.
- You can run preview up to a specific stage in the pipeline instead of the entire pipeline.
- Preview displays Snowflake data types instead of StreamSets data types.
- Preview provides as much preview data as possible when errors cause a preview request to fail to complete.
- Column selection
-
- To specify column names, you can use preview data to select columns instead of typing column names.
- After previewing data, stages provide autocompletion suggestion for column names in properties that accept multiple columns.
- After selecting multiple columns, you can drag the columns to change the order in which they appear.
- In some stages, after you select a column to use, you can then enter an expression that includes the column name or enter additional column names.
- Table selection
- In the Snowflake Table origin and destination, when you can explore your data and select the table that you want to use, you can explore the entire warehouse instead of being limited to tables within the default database or schema.
- Apply Function processor enhancements
-
- The Column property replaces the Column Expression property.
Use the Column property to specify the column to apply the function to. You can also specify a regular expression that applies the function to a set of columns.
- The Output Column property replaces the Add Prefix to New Output
Columns property and the Add Suffix to New Output Columns property.
Existing pipelines merge any prefix and suffix values defined in the old properties into the Output Column property using
$0
to represent the original column name. If existing pipelines use$0
in the old prefix and suffix properties, you should review those pipelines. For more information, see Behavior Change.
- The Column property replaces the Column Expression property.
- Column Type Converter processor change
- The Column property replaces the Expression property. The behavior of the property has not changed.
- Join processor enhancements
- The processor allows specifying a custom join by columns or condition. You can also specify prefixes for columns with matching names.
- Union processor enhancement
- The processor includes a Union All option that performs a Snowflake Union All operation. The Union All operation includes all rows from all incoming streams and can include duplicate rows.
Behavior Change
- Review Apply Function processor pipelines
- This release includes a new Output Column property that replaces the
following properties:
- Add Prefix to New Output Columns
- Add Suffix to New Output Columns
Fixed Issue
- Exploring Snowflake data requires including the
https://
protocol in the specified Snowflake URL instead of simply specifying the domain name.
April 2023
The following Transformer for Snowflake release occurred in April 2023.
April 21, 2023
This release includes several enhancements and fixed issues, a behavior change, and a known issue.
New Features and Enhancements
- Snowflake pipeline defaults
- Snowflake pipeline defaults defined in your StreamSets account settings are placed in the corresponding properties of new pipelines as static values.
- Support for PKCS#8 private keys
- When specifying private key Snowflake credentials, you can now enter a PKCS#8 private key. Previously, only PKCS#1 private keys were supported.
Behavior Change
- Snowflake pipeline defaults no longer integrate with pipeline default parameters
- Previously, Snowflake pipeline defaults were placed in the corresponding properties of new pipelines as pipeline default parameters. Now, Snowflake pipeline defaults are placed in pipelines as static values.
Fixed Issues
- You cannot explore your Snowflake data when your Snowflake credentials
defined in Control Hub use the private key authentication method.
- You cannot explore your Snowflake data to select a role when no warehouse is
defined.
- You cannot explore your Snowflake data using the Safari browser.
Known Issue
- When you use the More > Use Snowflake Defaults menu option to apply
Snowflake pipeline defaults to an existing pipeline, pipeline default
parameters are placed in pipeline properties and are defined based on their
corresponding Snowflake pipeline defaults, as they were before this release.
The correct behavior is to place configured Snowflake pipeline defaults into the pipeline as static values, instead of parameters.
Despite this issue, the pipeline will still use the configured Snowflake pipeline defaults as intended, and perform as expected.
March 2023
The following Transformer for Snowflake release occurred in March 2023.
March 29, 2023
This release includes several new features, enhancements, and a behavior change.
New Features and Enhancements
- Explore your Snowflake data
- As you build a pipeline, you can explore your
Snowflake data to select the following types of Snowflake
elements that you want to connect to or process:
- Roles
- Warehouses
- Databases
- Schemas
- Tables and views
- Columns
- Pipeline and fragment preview
- Previewing a Transformer for Snowflake pipeline or fragment includes the
following enhancements:
- When you preview a pipeline or fragment, Control Hub uses the default configuration and no longer displays the Preview
Configuration dialog box. While running preview, you can click the
Preview Configuration icon (
) to change the configuration and then run the preview again.
- When you preview a pipeline or fragment, the data displays in table view by default.
- Table view displays only output data by default. You can optionally choose to display both input and output data.
- Table view no longer displays colors for different types of data and changed data. List view continues to display colors.
- While running preview, you can click the Expand icon (
) to quickly expand the preview panel to view more preview data.
- When you preview a pipeline or fragment, Control Hub uses the default configuration and no longer displays the Preview
Configuration dialog box. While running preview, you can click the
Preview Configuration icon (
- Terminology changes
- StreamSets has implemented the following terminology changes to more closely align
with Snowflake terminology:
- The term "column" replaces the term "field."
- The term "row" replaces the term "record."
- The Expression Evaluator processor has been renamed to the Snowflake SQL Evaluator processor.
- Join processor
- The Join processor joins data based on matching values in specific columns. The specified columns can have identical or unique names. Previously, the Join processor joined data based on identical column names or based on a condition that you defined.
Behavior Changes
- Review Join processor pipelines
- With this release, the Join processor joins data based on matching values in specific columns only. The processor can no longer join data based on a condition that you define.
February 2023
The following Transformer for Snowflake release occurred in February 2023.
February 10, 2023
This release includes several enhancements.
Enhancements
- Apply Functions processor
- You can use the Apply Functions processor to apply the
REGEXP_REPLACE
Snowflake function to a column. - Column Type Converter processor
- The Column Type Converter processor can convert columns with the Varchar, Variant, or Binary data type to the Geography data type.
January 2023
The following Transformer for Snowflake release occurred in January 2023.
January 20, 2023
This release includes several fixed issues.
Fixed Issues
- If a table column name uses non-standard characters, the Slowly Changing Dimension processor fails to process the columns correctly.
- When the Apply Function processor uses the
Round()
andTruncate()
functions and is configured to keep only two decimal digits, the processor incorrectly replaces the additional digits with zeros instead of removing them. - The JSON Parser processor converts the original String column that contains the JSON to be parsed into a Variant column.
2022
The following Transformer for Snowflake releases occurred in 2022.
December 2022
The following Transformer for Snowflake release occurred in December 2022.
December 16, 2022
This release includes the following enhancements:
- JSON Parser processor
- The JSON Parser processor can extract all elements from the JSON data, extracting all columns as Variant columns or inferring the data type based on the column value. Previously, to extract all elements, you had to individually specify the path to each element.
- User-defined functions
- When you define an inline user-defined function (UDF) in a pipeline, you can choose to define a permanent or temporary UDF. A permanent UDF is stored in your Snowflake account and available for other pipelines to call as a precompiled UDF. A temporary UDF is available for the duration of the current pipeline run only. The temporary UDF cannot be called from additional pipelines.
- Job monitoring
- When you monitor Snowflake jobs, pipeline log messages include more detailed messages for several processors that can help with troubleshooting pipeline processing at the stage level.
November 2022
The following Transformer for Snowflake release occurred in November 2022.
November 18, 2022
This release includes the following enhancement:
- Job monitoring
- Control Hub includes the following enhancement when you monitor
Snowflake jobs:
- Pipeline log messages now include the number of bytes scanned by the Snowflake queries, the name and size of the warehouse where the Snowflake queries were executed, and the aggregated duration of the query executions.
October 2022
The following Transformer for Snowflake releases occurred in October 2022.
October 26, 2022
- Job monitoring
- Control Hub includes the following enhancements when you monitor
Snowflake jobs:
- Viewing log messages and Snowflake queries in the job summary is now enabled for all organizations.
- A job generates a unique Snowflake query for each Snowflake SQL Query processor. When a destination has an upstream Snowflake SQL Query processor and you select that destination in the pipeline canvas to view generated Snowflake queries, at least two unique Snowflake queries display for the destination.
October 5, 2022
This release includes a fixed issue.
Fixed Issue
- In rare cases, a draft run of a Transformer for Snowflake pipeline fails to start and displays an error message that Transformer for Snowflake has stopped responding.
August 2022
The following Transformer for Snowflake release occurred in August 2022.
August 26, 2022
This release includes the following enhancements:
- Snowflake Partner Connect
-
StreamSets is happy to announce that you can now use Snowflake Partner Connect to access Transformer for Snowflake.
- Job monitoring
- When you monitor Snowflake jobs in Control Hub, you can now view the following information in the job summary:
- Input and output row count
- Log messages
- Snowflake queries run for the job
July 2022
The following Transformer for Snowflake release occurred in July 2022.
July 20, 2022
- Apply Snowflake pipeline defaults
- When you apply Snowflake pipeline defaults to an existing pipeline, only the defaults that are defined in your account are applied to the pipeline.
- Snowflake settings display
- An update to the Snowflake Settings tab in your account settings eliminates confusion about the properties to be saved.
- Slowly Changing Dimension processor
-
- The processor can now process multiple changes to the same row in the same batch. Changes are applied in the order of the rows in the batch. If necessary, you can use a Sort processor upstream to ensure rows are in the appropriate order.
- The Behavior for New Columns property no longer includes the “Keep for inserted rows and populate values from change data for updated rows” option. This results in a behavior change.
- When configured to process Type 1 changes, the processor no longer requires specifying a tracking column.
Behavior Change
- New column behavior for the Slowly Changing Dimension processor
-
Previously, the Behavior for New Columns property included the "Keep for inserted rows and populate values from change data for updated rows" option. With this update, the option has been removed.
Pipelines that previously used the option have the Behavior for New Columns property set to "Keep for newest rows and set to null for previous rows".
Review Slowly Changing Dimension pipelines to ensure that this behavior is acceptable.
Fixed Issues
- When new columns appear in change data and the Behavior for New Columns property is configured to keep new columns, the Slowly Changing Dimension processor fails with an exception.
- The Snowflake Table destination sometimes generates errors and stops the pipeline when merging data.
June 2022
The following Transformer for Snowflake releases occurred in June 2022.
June 17, 2022
- Apply Snowflake pipeline defaults to existing pipelines
- Snowflake pipeline defaults defined in your account settings are automatically included in new pipelines. When you add or update Snowflake pipeline defaults in your account settings, existing pipelines do not automatically inherit those changes.
June 8, 2022
With this release, StreamSets is happy to announce that Transformer for Snowflake is now generally available and can be used for production workloads.
June 3, 2022 (public preview)
- Snowflake credentials
-
When you view existing Snowflake credentials in the Control Hub user interface, you can now view the previously-saved Snowflake username and role.
Fixed Issue
- When the Slowly Changing Dimension processor is configured to use a Timestamp expression, and the master data includes a column that is used in that expression, the timestamps are not set correctly.
May 2022
The following Transformer for Snowflake releases occurred in May 2022.
May 20, 2022 (public preview)
- Draft runs
- You can start a draft run of a Transformer for Snowflake pipeline to quickly test the pipeline logic.
May 6, 2022 (public preview)
This release includes several fixed issues.
Fixed Issues
- The Snowflake Table origin incorrectly requires all columns in the Additional Where Clause property to be listed in the Columns to Read list.
- The Column Remover processor does not perform actions correctly based on regular expressions that contain uppercase letters.
- Data preview can fail for pipelines that include a Column Renamer processor that uses certain configuration options.
-
The Find and Replace in All Columns by String option in the Column Renamer processor does not produce the expected results.
Known Issues
There are no important known issues at this time.