Release Notes
For help with Transformer for Snowflake, join the StreamSets Community and use the Transformer for
Snowflake
tag when you post your question. To provide suggestions and
feedback, add a new idea at https://ibm-data-and-ai.ideas.ibm.com/?project=SSETS.
September 2024
September 18 and 20 (5.5.0)
The Transformer for Snowflake 5.5.0 release occurred on September 18, 2024 for the eu01.hub.streamsets.com instance. The release occurred on September 20, 2024 for all other instances.
New Features and Enhancements
- LLM properties renamed
- The following LLM processor properties have been renamed with this
release:
- LLM Summarize processor - The Source property is now the Source Column property.
- LLM Translate processor - The Source Columns property is now the Source Column property, and the Output Columns property is the Output Column property.
Fixed Issues
- When configured to overwrite a table by dropping it, the Snowflake Table destination incorrectly keeps all existing columns when the Data Drift Enabled property is also selected.
August 2024
August 16 (Version 5.4.0)
The Transformer for Snowflake 5.4.0 release occurred on August 16, 2024.
New Features and Enhancements
- New Case processor
- The Case processor returns values based on a sequence of conditions, like a cascading if-then-else statement.
- New LLM Complete processor
- The LLM Complete
processor generates a customized response to an
English-language prompt based on the selected large language model (LLM)
and the Snowflake LLM Complete function.Note: At this time, Snowflake charges differently for LLM processing. See the Snowflake consumption rates for details.
- Data preview enhancement
- A new Limit Stage Input preview property determines how preview provides data to stages in Transformer for Snowflake pipelines.
July 2024
July 24 (Version 5.3.0)
The Transformer for Snowflake 5.3.0 release occurred on July 24, 2024.
New Features and Enhancements
- New LLM Extract Answer processor
- The LLM Extract Answer
processor uses a Snowflake Large Language Model (LLM) function to
extract answers to questions from English-language text. It also generates a
confidence score for the extracted answer. Note: At this time, Snowflake charges differently for LLM processing. See the Snowflake consumption rates for details.
- Product rename
- Following the IBM acquisition of StreamSets, Transformer for Snowflake is part of what is now known as IBM StreamSets for Snowflake.
Fixed Issues
- The Snowflake Table destination can generate errors when you configure it to write to a view.
June 2024
June 26 (Version 5.2.0)
The Transformer for Snowflake 5.2.0 release occurred on June 26, 2024.
New Features and Enhancements
- New LLM processors
- The following new processors use Snowflake Large Language Model (LLM)
functions to process data:
- LLM Sentiment processor - Evaluates the sentiment of English language text in specified columns and provides a floating point score between -1 and 1 for the data.
- LLM Summarize processor - Summarizes English language text in specified columns.
- LLM Translate processor - Translates text in columns to other languages.
- Snowflake Table origin enhancement
- When you specify the columns for the origin to read, you can now rename those columns as needed.
Fixed Issues
- Fixed issue for deployed Transformer for Snowflake engines
-
- Proxy properties defined in the advanced properties for the deployment are ignored. As a result, engine communication with Control Hub or Snowflake that requires a proxy will fail.
May 2024
May 22 (Version 5.1.0)
The Transformer for Snowflake 5.1.0 release occurred on May 22, 2024.
Fixed Issues
-
When a pipeline includes multiple conditions, such as a Where clause in an origin and a condition in a Filter processor, the conditions are joined incorrectly.
With this fix, multiple conditions are handled differently in pipelines, which is a behavior change.
-
When you configure a Filter processor with a comparison operator and value, then change the comparison operator to ISNULL or ISNOTNULL, the processor stores the previously defined value and attempts to use it, causing the pipeline to fail with a SQL syntax error.
- When a pipeline is configured to use an invalid Snowflake URL, the job that runs the pipeline incorrectly shows an Active status instead of a Start Error status.
- An Amazon EC2 deployment for a Transformer for Snowflake engine might fail to start the engine if the feature version for the parent environment is AWS_2021_06_01.
- The Proxy tab for a Transformer for Snowflake deployment is empty.
Behavior Change
- Review pipelines that include multiple conditions
-
With this release, when a pipeline includes multiple conditions, they are joined as expected. In previous releases, they were joined inappropriately. Review pipelines with multiple conditions to verify that the data is processed as expected.
For example, say you have an origin with the following condition defined in the Additional Where Clause property:
monthlySales >= 10000
. And you have a Filter processor with the following condition in the Where Clause property:country = "Spain" OR "Germany"
.In previous releases, the pipeline merges the conditions incorrectly, as follows:
monthlySales >= 10000 AND country = "Spain" OR "Germany"
With this fix, the pipeline merges the conditions appropriately, as follows:
monthlySales >= 10000 AND (country = "Spain" OR "Germany")
Known Issue
- Known issue for deployed Transformer for Snowflake engines
-
- Proxy properties defined in the advanced properties for the deployment are ignored. As a result, engine communication with Control Hub or Snowflake that requires a proxy will fail. Do not use a proxy at this time.
April 2024
April 24 (Version 5.0.0)
The Transformer for Snowflake 5.0.0 release occurred on April 24, 2024.
New Features and Enhancements
- Deployed Transformer for Snowflake engines
- When needed, your organization can deploy Transformer for Snowflake engines to a private network, which can be on-premises or on the AWS cloud computing platform. This first release of the Transformer for Snowflake deployed engine is version 5.0.0.
Known Issues
This release includes known issues that apply separately to hosted and deployed engines:
- Known issues for hosted Transformer for Snowflake engines
-
- When a pipeline is configured to use an invalid Snowflake URL, the job that runs the pipeline incorrectly shows an Active status instead of a Start Error status.
- Known issues for deployed Transformer for Snowflake engines
-
- An Amazon EC2 deployment for a Transformer for Snowflake engine might fail to start the engine if the feature version
for the parent environment is AWS_2021_06_01.
Workaround: Change the environment feature version to AWS_2023_12_15.
- The Proxy tab for a Transformer for Snowflake deployment is empty.
Workaround: Copy and paste the proxy properties from a deployment for a different engine type.
- An Amazon EC2 deployment for a Transformer for Snowflake engine might fail to start the engine if the feature version
for the parent environment is AWS_2021_06_01.
March 2024
March 27, 2024
This release includes a fixed issue.
Fixed Issue
- When you create a job instance or template for a Transformer for Snowflake pipeline through the Run > Job Instances or Run > Job Template menu options,
the job creation wizard requires specifying a deployment, which is not
applicable for Transformer for Snowflake pipelines.
Creating jobs through the pipeline canvas or the Pipelines view was not affected by this issue.
February 2024
February 21, 2024
This release includes a fixed issue.
Fixed Issue
- The Apply Function processor unnecessarily changes the column order of incoming data.
January 2024
The following Transformer for Snowflake release occurred in January 2024.
January 24, 2024
This release includes a new feature, several fixed issues, and a behavior change.
New Feature
- New Snowflake Notification executor
- The new Snowflake Notification executor sends an email after the pipeline completes all writes. The executor uses a Snowflake email notification integration to send email to Snowflake users in your account when the specified condition has been met.
Fixed Issues
-
Preview fails to run if the pipeline includes a destination or executor after an invalid stage.
- When the Join processor has the Join Command property set to
Using
, the processor adds a__dup__<n>
suffix, where<n>
is a number, to differentiate between duplicate column names in the output.With this fix, the processor uses a
T1_
orT2_
prefix as with other join methods, which is a behavior change.
Behavior Change
- Review pipelines that join data with the Using join command
- With this release, when a Join processor is configured with the Join
Command property set to
Using
, and when output includes duplicate column names, the processor adds aT1_
orT2_
prefix to those column names to avoid duplicate names:T1_
for the left input, andT2_
for the right input.
2023
The following Transformer for Snowflake releases occurred in 2023.
December 2023
The following Transformer for Snowflake release occurred in December 2023.
December 20, 2023
This release includes several enhancements and a fixed issue.
New Features and Enhancements
- Snowflake View destination
- The new Snowflake View destination creates a Snowflake view. The destination can create the view or replace an existing view. The destination creates a standard view by default, but you can configure it to create a secure, a materialized, or a secure materialized view.
- Union processor enhancement
-
When you configure the Union processor to require the same schema for all incoming data, the processor adjusts the order of the columns for all data based on the column order of the first batch of data from the first input location.
Since the Snowflake
Union
operation can produce undesired results when columns are ordered differently, this enhancement provides uniformity in column order, when possible.
Fixed Issue
-
Draft run jobs do not stop when the pipeline includes an open output stream. With this change, draft runs require all output streams to be connected.
November 2023
The following Transformer for Snowflake release occurred in November 2023.
November 15, 2023
This release includes an enhancement and a fixed issue.
Enhancement
- When you preview a pipeline or fragment that generates an error, Control Hub now provides preview data for the pipeline stages before the stage that
generated the error. For example, if the pipeline includes an error in the
fifth stage, Control Hub provides preview data for the first four stages in the pipeline.
For more information about previewing Transformer for Snowflake pipelines, see the Control Hub documentation.
Fixed Issue
-
Draft run jobs do not stop when the pipeline includes an open output stream. With this change, draft runs require all output streams to be connected.
October 2023
The following Transformer for Snowflake release occurred in October 2023.
October 27, 2023
This release includes several enhancements.
New Features and Enhancements
- Data preview enhancements
- Data preview can run when a pipeline includes stages or branches that are not connected to an origin. The pipeline provides preview data for stages connected to a configured origin.
- Join processor property names
- For clarity, the following Join processor
properties have been renamed:
- The Join Using Condition property has been renamed to the Join Condition property.
- The Input 1 property has been renamed to the Input 1 (Left) property.
- The Input 2 property has been renamed to the Input 2 (Right) property.
This change has no upgrade impact.
September 2023
The following Transformer for Snowflake release occurred in September 2023.
September 27, 2023
This release includes several new features, enhancements, and fixed issues.
New Features and Enhancements
- Snowflake credentials
-
-
Transformer for Snowflake validates Snowflake credentials before saving them to your account settings.
-
When you create a new pipeline, the Snowflake Credentials dialog box appears so you can specify the Snowflake account URL and user credentials to use, if you have not already saved them to your StreamSets account.
-
- Stages
-
-
When using the Select Column from Schema icon in a processor, Transformer for Snowflake automatically loads the schema when needed.
-
The Aggregate processor includes an Any Value function that returns a random value from the group.
-
The Column Renamer processor can add a prefix or suffix to all columns.
- The Snowflake Table origin and destination are automatically named for the selected
table unless you manually override the name.
In the August 2023 release, the Snowflake Table origin and destination became automatically named for the selected table. This release enables you to specify a different stage name using the Name property on the General tab.
If you select a new table for the stage, the name of the selected table overrides the manually-entered stage name.
Note: With this update, Snowflake Table origins and destinations in existing pipelines are automatically named for the selected table. Override the automatic stage names as desired. The functionality of the pipelines and associated jobs are not affected.
-
Fixed Issues
-
When you configure a custom join condition with wrong column names in the Join Using Condition property, the Join processor performs a cross join instead of the configured join type.
- The Union processor can provide preview data for only one input, instead of
displaying data for all inputs.
With this fix, the preview batch for the Union processor includes rows for each input, when possible. For example, if the preview batch is ten records, and there are five input streams, preview displays two rows for each input.
-
When selecting a role, schema, or warehouse, connection errors display in two locations in the same dialog box.
August 2023
The following Transformer for Snowflake release occurred in August 2023.
August 23, 2023
This release includes several new features and enhancements, a behavior change, and a fixed issue.
New Features and Enhancements
- Snowflake Table origin and destination enhancements
-
- Default stage naming - When you specify a table to read in the Snowflake Table origin, the table name becomes the name of the origin. Similarly, when you specify a table to write to in the Snowflake Table destination, the table name becomes the name of the destination.
- Improved table selection:
- You can now specify a table to use by selecting the database, schema, and table from lists.
- If you specify the schema, database, and table to use, then click the Select Table icon, the explorer window opens to the specified position.
- Additional stage enhancements
-
- Apply function processor - You can now use all Snowflake date
functions in the processor, as well as a new Transformer for Snowflake Age function.
This enhancement includes a behavior change. For more information, see Behavior Change.
- Unpivot processor -
You can use several methods to specify the columns to unpivot. In
addition to listing specific column names, you can now specify
strings and regular expressions to match column names. This enhancement includes the following changes:
- New Column Match Criteria property indicates how to specify the columns to unpivot.
- Columns to Pivot property renamed to the Columns property. The property still allows you to list the columns to unpivot.
- Several new properties allow you to define strings, prefixes, or suffixes in column names, or regular expressions that match column names.
- Apply function processor - You can now use all Snowflake date
functions in the processor, as well as a new Transformer for Snowflake Age function.
- Additional pipeline enhancement
-
- Improved Pipeline Working Schema selection - Like with the Snowflake Table origin and destination, you can select the database and schema for the Pipeline Working Schema property from lists.
Behavior Change
- DATEDIFF
- NEXT_DAY
- PREVIOUS_DAY
Apply Functions processors in existing pipelines are updated accordingly and do not require review.
However, when using date and time functions with a new Apply Functions processor, and when configuring the Comparison Date property, you must enclose constants for the listed functions in single quotation marks. Previously, you selected the Treat as Constant property, instead.
The Comparison Date property is available with the DATEDIFF, NEXT_DAY, and PREVIOUS_DAY functions, as well as the new AGE function.
For example, say you add a new Apply Functions processor to a pipeline and set the
Function Type property to Date and Time
and the Date and Time
Function property to NEXT_DAY
. To use a constant as a comparison,
in the Comparison Date property, you specify the date to use in single quotation
marks as follows: '2023/05/08'
.
Fixed Issue
- The Stream Selector processor inappropriately drops rows from the pipeline when a column used in a condition contains a null value.
July 2023
The following Transformer for Snowflake release occurred in July 2023.
July 21, 2023
This release includes several new features, enhancements, and changes.
New Features and Enhancements
- Stages
-
- New Null Handling processor - Replaces null values based on the specified function and related properties.
- Apply
Functions processor - When defining a regular
expression to represent multiple output columns, you can use
group variables, such as
$1
and$2
, to represent the groups defined in the corresponding expression that defines the input columns. - Snowflake SQL Evaluator processor - The processor has been renamed the Column Transformer processor.
- Aggregate, Cube, and Rollup processors - These processors allow
entering a regular expression in the Aggregate Column property.
This allows you to specify more than one aggregate column for a
calculation.
When entering a regular expression in the Aggregate Column property, use
$0
to represent the original column names in the Output Column Name property. For example$0_avg
adds_avg
to the original column names.
- Snowflake credentials and pipeline defaults
-
- If you have not defined Snowflake
credentials or Snowflake pipeline
defaults in your StreamSets account, when you configure a pipeline and perform a task
that requires those details, you can specify them in convenient
pop-up dialog boxes.
Those details are then saved in your StreamSets account for future use.
For example, say you create your first pipeline and click the Select Table icon in the pipeline. If you have not provided Snowflake credentials, a Snowflake Credentials dialog box appears so you can specify those details.
You are prompted to select the role and warehouse to use, then you navigate to the appropriate database and schema, and select the table for the pipeline.
Since you do not have Snowflake settings defined in your StreamSets account, the credentials, role, warehouse, database, and schema that you specified during pipeline creation are saved in your StreamSets account to be used in future pipelines.
- You can also define Snowflake credentials in the pipeline using the Enter Snowflake Credentials option available in the More menu.
- The Snowflake Credentials dialog box includes the
following changes:
-
It includes the Snowflake account URL property.
-
It no longer includes the Role property. You can specify a default role as a Snowflake pipeline default.
-
It appears when you have not provided all of the required credentials when you configure a pipeline.
-
- If you have not defined Snowflake
credentials or Snowflake pipeline
defaults in your StreamSets account, when you configure a pipeline and perform a task
that requires those details, you can specify them in convenient
pop-up dialog boxes.
- Pipeline configuration changes
-
- The Warehouse and Working Default Schema properties are now
required properties. Though, as always, when you create a
pipeline, they are populated with associated Snowflake default
properties stored in your StreamSets account.
Pipelines and jobs provide validation messages to ensure that you define these properties.
- The Role property now defaults to the Snowflake Public role.
For existing pipelines, StreamSets recommends specifying a role for each pipeline.
- The Warehouse and Working Default Schema properties are now
required properties. Though, as always, when you create a
pipeline, they are populated with associated Snowflake default
properties stored in your StreamSets account.
June 2023
The following Transformer for Snowflake release occurred in June 2023.
June 14 and 21, 2023
This release includes several new features, enhancements, and fixed issues.
The release occurred in two phases across all Control Hub instances in the StreamSets platform. The first phase occurred on June 14, 2023, and the second phase occurred on June 21, 2023.
New Features and Enhancements
- Stages
-
- New Unpivot processor - Rotates a table, converting the specified columns into rows.
- New Window Function processor - Performs window calculations and writes the results to a specified column. The processor performs calculations on windows or window frames.
- Filter processor enhancements - The Filter Condition property
has been replaced. You can now configure a set of properties to
build a condition or enter the condition in a new Where Clause
property.
When building a condition, you can specify a column, select a comparison operator, and enter the value to use.
Existing pipelines place the existing filter condition into the new Where Clause property.
- Pipeline enhancements
-
- When Snowflake credentials are not defined in your StreamSets account settings, you can now easily configure them in a
pipeline when you explore your data.
For example, if you explore your data to select a role for a pipeline, and you have not added your Snowflake username and password to your StreamSets account settings, a dialog box displays that enables you to specify that information.
- The Default Schema pipeline property has been renamed to Pipeline Working Schema.
- When Snowflake credentials are not defined in your StreamSets account settings, you can now easily configure them in a
pipeline when you explore your data.
Fixed Issues
- Timestamps display in the milliseconds instead of a standard date format, such as "May 31, 2023 11:00:00 AM".
- In the Join processor, the column selector for input 2 displays columns from input 1.
- Existing Join processor pipelines generate an error when you edit the processor to specify the columns to use when you define a join condition.
- When using the Pivot processor to autocalculate pivot values, errors occur if the pivoted columns include NULL values.
May 2023
The following Transformer for Snowflake release occurred in May 2023.
May 12, 2023
This release includes several enhancements, a behavior change, and a fixed issue.
New Features and Enhancements
- Data preview enhancements
-
- You can preview data for incomplete pipelines, as long as all required fields are defined.
- You can run preview up to a specific stage in the pipeline instead of the entire pipeline.
- Preview displays Snowflake data types instead of StreamSets data types.
- Preview provides as much preview data as possible when errors cause a preview request to fail to complete.
- Column selection
-
- To specify column names, you can use preview data to select columns instead of typing column names.
- After previewing data, stages provide autocomplete suggestions for column names in properties that accept multiple columns.
- After selecting multiple columns, you can drag the columns to change the order in which they appear.
- In some stages, after you select a column to use, you can then enter a regular expression that includes the column name or enter additional column names.
- Table selection
- In the Snowflake Table origin and destination, when you can explore your data and select the table that you want to use, you can explore the entire warehouse instead of being limited to tables within the default database or schema.
- Apply Function processor enhancements
-
- The Column property replaces the Column Expression property.
Use the Column property to specify the column to apply the function to. You can also specify a regular expression that applies the function to a set of columns.
- The Output Column property replaces the Add Prefix to New Output
Columns property and the Add Suffix to New Output Columns property.
Existing pipelines merge any prefix and suffix values defined in the old properties into the Output Column property using
$0
to represent the original column name. If existing pipelines use$0
in the old prefix and suffix properties, you should review those pipelines. For more information, see Behavior Change.
- The Column property replaces the Column Expression property.
- Column Type Converter processor change
- The Column property replaces the Expression property. The behavior of the property has not changed.
- Join processor enhancements
- The processor allows specifying a custom join by columns or condition. You can also specify prefixes for columns with matching names.
- Union processor enhancement
- The processor includes a Union All option that performs a Snowflake Union All operation. The Union All operation includes all rows from all incoming streams and can include duplicate rows.
Behavior Change
- Review Apply Function processor pipelines
- This release includes a new Output Column property that replaces the
following properties:
- Add Prefix to New Output Columns
- Add Suffix to New Output Columns
Fixed Issue
- Exploring Snowflake data requires including the
https://
protocol in the specified Snowflake URL instead of simply specifying the domain name.
April 2023
The following Transformer for Snowflake release occurred in April 2023.
April 21, 2023
This release includes several enhancements and fixed issues, a behavior change, and a known issue.
New Features and Enhancements
- Snowflake pipeline defaults
- Snowflake pipeline defaults defined in your StreamSets account settings are placed in the corresponding properties of new pipelines as static values.
- Support for PKCS#8 private keys
- When specifying private key Snowflake credentials, you can now enter a PKCS#8 private key. Previously, only PKCS#1 private keys were supported.
Behavior Change
- Snowflake pipeline defaults no longer integrate with pipeline default parameters
- Previously, Snowflake pipeline defaults were placed in the corresponding properties of new pipelines as pipeline default parameters. Now, Snowflake pipeline defaults are placed in pipelines as static values.
Fixed Issues
- You cannot explore your Snowflake account when your Snowflake credentials
defined in Control Hub use the private key authentication method.
- You cannot explore your Snowflake account to select a role when no warehouse
is defined.
- You cannot explore your Snowflake account using the Safari browser.
Known Issue
- When you use the More > Use Snowflake Defaults menu option to apply
Snowflake pipeline defaults to an existing pipeline, pipeline default
parameters are placed in pipeline properties and are defined based on their
corresponding Snowflake pipeline defaults, as they were before this release.
The correct behavior is to place configured Snowflake pipeline defaults into the pipeline as static values, instead of parameters.
Despite this issue, the pipeline will still use the configured Snowflake pipeline defaults as intended, and perform as expected.
March 2023
The following Transformer for Snowflake release occurred in March 2023.
March 29, 2023
This release includes several new features, enhancements, and a behavior change.
New Features and Enhancements
- Explore your Snowflake account
- As you build a pipeline, you can explore your
Snowflake account to select the following types of Snowflake
elements that you want to connect to or process:
- Roles
- Warehouses
- Databases
- Schemas
- Tables and views
- Columns
- Pipeline and fragment preview
- Previewing a Transformer for Snowflake pipeline or fragment includes the
following enhancements:
- When you preview a pipeline or fragment, Control Hub uses the default configuration and no longer displays the Preview Configuration dialog box. While running preview, you can click the Preview Configuration icon () to change the configuration and then run the preview again.
- When you preview a pipeline or fragment, the data displays in table view by default.
- Table view displays only output data by default. You can optionally choose to display both input and output data.
- Table view no longer displays colors for different types of data and changed data. List view continues to display colors.
- While running preview, you can click the Expand icon () to quickly expand the preview panel to view more preview data.
- Terminology changes
- StreamSets has implemented the following terminology changes to more closely align
with Snowflake terminology:
- The term "column" replaces the term "field."
- The term "row" replaces the term "record."
- The Expression Evaluator processor has been renamed to the Snowflake SQL Evaluator processor.
- Join processor
- The Join processor joins data based on matching values in specific columns. The specified columns can have identical or unique names. Previously, the Join processor joined data based on identical column names or based on a condition that you defined.
Behavior Changes
- Review Join processor pipelines
- With this release, the Join processor joins data based on matching values in specific columns only. The processor can no longer join data based on a condition that you define.
February 2023
The following Transformer for Snowflake release occurred in February 2023.
February 10, 2023
This release includes several enhancements.
Enhancements
- Apply Functions processor
- You can use the Apply Functions processor to apply the
REGEXP_REPLACE
Snowflake function to a column. - Column Type Converter processor
- The Column Type Converter processor can convert columns with the Varchar, Variant, or Binary data type to the Geography data type.
January 2023
The following Transformer for Snowflake release occurred in January 2023.
January 20, 2023
This release includes several fixed issues.
Fixed Issues
- If a table column name uses non-standard characters, the Slowly Changing Dimension processor fails to process the columns correctly.
- When the Apply Function processor uses the
Round()
andTruncate()
functions and is configured to keep only two decimal digits, the processor incorrectly replaces the additional digits with zeros instead of removing them. - The JSON Parser processor converts the original String column that contains the JSON to be parsed into a Variant column.
2022
The following Transformer for Snowflake releases occurred in 2022.
December 2022
The following Transformer for Snowflake release occurred in December 2022.
December 16, 2022
This release includes the following enhancements:
- JSON Parser processor
- The JSON Parser processor can extract all elements from the JSON data, extracting all columns as Variant columns or inferring the data type based on the column value. Previously, to extract all elements, you had to individually specify the path to each element.
- User-defined functions
- When you define an inline user-defined function (UDF) in a pipeline, you can choose to define a permanent or temporary UDF. A permanent UDF is stored in your Snowflake account and available for other pipelines to call as a precompiled UDF. A temporary UDF is available for the duration of the current pipeline run only. The temporary UDF cannot be called from additional pipelines.
- Job monitoring
- When you monitor Snowflake jobs, pipeline log messages include more detailed messages for several processors that can help with troubleshooting pipeline processing at the stage level.
November 2022
The following Transformer for Snowflake release occurred in November 2022.
November 18, 2022
This release includes the following enhancement:
- Job monitoring
- Control Hub includes the following enhancement when you monitor
Snowflake jobs:
- Pipeline log messages now include the number of bytes scanned by the Snowflake queries, the name and size of the warehouse where the Snowflake queries were executed, and the aggregated duration of the query executions.
October 2022
The following Transformer for Snowflake releases occurred in October 2022.
October 26, 2022
- Job monitoring
- Control Hub includes the following enhancements when you monitor
Snowflake jobs:
- Viewing log messages and Snowflake queries in the job summary is now enabled for all organizations.
- A job generates a unique Snowflake query for each Snowflake SQL Query processor. When a destination has an upstream Snowflake SQL Query processor and you select that destination in the pipeline canvas to view generated Snowflake queries, at least two unique Snowflake queries display for the destination.
October 5, 2022
This release includes a fixed issue.
Fixed Issue
- In rare cases, a draft run of a Transformer for Snowflake pipeline fails to start and displays an error message that Transformer for Snowflake has stopped responding.
August 2022
The following Transformer for Snowflake release occurred in August 2022.
August 26, 2022
This release includes the following enhancements:
- Snowflake Partner Connect
-
StreamSets is happy to announce that you can now use Snowflake Partner Connect to access Transformer for Snowflake.
- Job monitoring
- When you monitor Snowflake jobs in Control Hub, you can now view the following information in the job summary:
- Input and output row count
- Log messages
- Snowflake queries run for the job
July 2022
The following Transformer for Snowflake release occurred in July 2022.
July 20, 2022
- Apply Snowflake pipeline defaults
- When you apply Snowflake pipeline defaults to an existing pipeline, only the defaults that are defined in your account are applied to the pipeline.
- Snowflake settings display
- An update to the Snowflake Settings tab in your account settings eliminates confusion about the properties to be saved.
- Slowly Changing Dimension processor
-
- The processor can now process multiple changes to the same row in the same batch. Changes are applied in the order of the rows in the batch. If necessary, you can use a Sort processor upstream to ensure rows are in the appropriate order.
- The Behavior for New Columns property no longer includes the “Keep for inserted rows and populate values from change data for updated rows” option. This results in a behavior change.
- When configured to process Type 1 changes, the processor no longer requires specifying a tracking column.
Behavior Change
- New column behavior for the Slowly Changing Dimension processor
-
Previously, the Behavior for New Columns property included the "Keep for inserted rows and populate values from change data for updated rows" option. With this update, the option has been removed.
Pipelines that previously used the option have the Behavior for New Columns property set to "Keep for newest rows and set to null for previous rows".
Review Slowly Changing Dimension pipelines to ensure that this behavior is acceptable.
Fixed Issues
- When new columns appear in change data and the Behavior for New Columns property is configured to keep new columns, the Slowly Changing Dimension processor fails with an exception.
- The Snowflake Table destination sometimes generates errors and stops the pipeline when merging data.
June 2022
The following Transformer for Snowflake releases occurred in June 2022.
June 17, 2022
- Apply Snowflake pipeline defaults to existing pipelines
- Snowflake pipeline defaults defined in your account settings are automatically included in new pipelines. When you add or update Snowflake pipeline defaults in your account settings, existing pipelines do not automatically inherit those changes.
June 8, 2022
With this release, StreamSets is happy to announce that Transformer for Snowflake is now generally available and can be used for production workloads.
June 3, 2022 (public preview)
- Snowflake credentials
-
When you view existing Snowflake credentials in the Control Hub user interface, you can now view the previously-saved Snowflake username and role.
Fixed Issue
- When the Slowly Changing Dimension processor is configured to use a Timestamp expression, and the master data includes a column that is used in that expression, the timestamps are not set correctly.
May 2022
The following Transformer for Snowflake releases occurred in May 2022.
May 20, 2022 (public preview)
- Draft runs
- You can start a draft run of a Transformer for Snowflake pipeline to quickly test the pipeline logic.
May 6, 2022 (public preview)
This release includes several fixed issues.
Fixed Issues
- The Snowflake Table origin incorrectly requires all columns in the Additional Where Clause property to be listed in the Columns to Read list.
- The Column Remover processor does not perform actions correctly based on regular expressions that contain uppercase letters.
- Data preview can fail for pipelines that include a Column Renamer processor that uses certain configuration options.
-
The Find and Replace in All Columns by String option in the Column Renamer processor does not produce the expected results.