Release Notes

New Features and Enhancements

Data preview enhancements

• You can preview data for incomplete pipelines, as long as all required fields are defined.
• You can run preview up to a specific stage in the pipeline instead of the entire pipeline.
• Preview displays Snowflake data types instead of StreamSets data types.
• Preview provides as much preview data as possible when errors cause a preview request to fail to complete.

Column selection

• To specify column names, you can use preview data to select columns instead of typing column names.
• After previewing data, stages provide autocompletion suggestion for column names in properties that accept multiple columns.
• After selecting multiple columns, you can drag the columns to change the order in which they appear.
• In some stages, after you select a column to use, you can then enter an expression that includes the column name or enter additional column names.

Table selection

In the Snowflake Table origin and destination, when you can explore your data and select the table that you want to use, you can explore the entire warehouse instead of being limited to tables within the default database or schema.

Apply Function processor enhancements

• The Column property replaces the Column Expression property.

  Use the Column property to specify the column to apply the function to. You can also specify a regular expression that applies the function to a set of columns.

• The Output Column property replaces the Add Prefix to New Output Columns property and the Add Suffix to New Output Columns property.

  Existing pipelines merge any prefix and suffix values defined in the old properties into the Output Column property using $0 to represent the original column name. If existing pipelines use $0 in the old prefix and suffix properties, you should review those pipelines. For more information, see Behavior Change.

Column Type Converter processor change

The Column property replaces the Expression property. The behavior of the property has not changed.

Join processor enhancements

The processor allows specifying a custom join by columns or condition. You can also specify prefixes for columns with matching names.

Union processor enhancement

The processor includes a Union All option that performs a Snowflake Union All operation. The Union All operation includes all rows from all incoming streams and can include duplicate rows.

Behavior Change

Review Apply Function processor pipelines

This release includes a new Output Column property that replaces the following properties:

• Add Prefix to New Output Columns
• Add Suffix to New Output Columns

Existing Apply Function processor pipelines populate the new Output Column property based on values specified in the two replaced properties, with $0 representing the original column name as follows: <previously-defined prefix>$0<previously-defined suffix>.

Since $0 is used as a variable, review and update pipelines that used $0 in the old prefix and suffix properties. Then, update the upgrade-generated value for the Output Column property, as needed.

For example, say you have a prefix defined as $0new_ and no defined suffix. After this release, the Output Column property replaces those properties and is defined as follows: $0new_$0.

So if an original column name is time, this results in an output column of timenew_time.

In this case, you would probably update the Output Column property to new_$0, removing the first $0. Then, the same output column is named new_time.

Fixed Issue

• Exploring Snowflake data requires including the https:// protocol in the specified Snowflake URL instead of simply specifying the domain name.

New Features and Enhancements

Behavior Change

Fixed Issues

Known Issue

New Features and Enhancements

Explore your Snowflake data

As you build a pipeline, you can explore your Snowflake data to select the following types of Snowflake elements that you want to connect to or process:

• Roles
• Warehouses
• Databases
• Schemas
• Tables and views
• Columns

Control Hub displays all Snowflake elements that your Snowflake credentials grant you access to.

For example, to define the Snowflake schema that a pipeline uses, you click the Select Schema icon: . Control Hub displays a dialog box listing all Snowflake databases and schemas that you have access to. You can browse or search for the database and schema to use.

Previously, you had to type the names of the Snowflake elements in the pipeline and stage properties.

Pipeline and fragment preview

Previewing a Transformer for Snowflake pipeline or fragment includes the following enhancements:

• When you preview a pipeline or fragment, Control Hub uses the default configuration and no longer displays the Preview Configuration dialog box. While running preview, you can click the Preview Configuration icon () to change the configuration and then run the preview again.
• When you preview a pipeline or fragment, the data displays in table view by default.
• Table view displays only output data by default. You can optionally choose to display both input and output data.
• Table view no longer displays colors for different types of data and changed data. List view continues to display colors.
• While running preview, you can click the Expand icon () to quickly expand the preview panel to view more preview data.

For information about using data preview, see the Control Hub documentation.

Terminology changes

StreamSets has implemented the following terminology changes to more closely align with Snowflake terminology:

• The term "column" replaces the term "field."
• The term "row" replaces the term "record."
• The Expression Evaluator processor has been renamed to the Snowflake SQL Evaluator processor.

Join processor

The Join processor joins data based on matching values in specific columns. The specified columns can have identical or unique names. Previously, the Join processor joined data based on identical column names or based on a condition that you defined.

In addition, when the joined tables have identical column names, the processor automatically adds a t1_ or t2_ prefix to the column names in the generated output to avoid duplicate names. Previously, you optionally configured the processor to add prefixes to column names and then defined the text to add.

These behavior changes might require you to modify Join processor pipelines.

Behavior Changes

Review Join processor pipelines

With this release, the Join processor joins data based on matching values in specific columns only. The processor can no longer join data based on a condition that you define.

If you previously configured the processor to join by condition, review the pipelines that include the processor and specify the column names with matching values used to join the data.

In addition, when the joined tables have identical column names, the Join processor now includes both columns in the generated output, automatically adding a t1_ or t2_ prefix to each column name to avoid duplicate names. You can no longer specify the text to use for the prefix. Previously, when the join columns had identical names, the processor included a single column in the generated output, and you could optionally specify the text to use as a prefix.

If you previously configured the processor to join tables based on join columns with identical names, review the pipelines that include the processor and add a Column Remover processor if you want to remove one of the joined columns in the output or add a Column Renamer processor if you want to remove the t1_ or t2_ prefix from the column names.

If you previously configured the processor to add prefixes to column names, review the pipelines that include the processor and add a Column Renamer processor if you still require those prefixes in the column names.

Enhancements

Apply Functions processor

You can use the Apply Functions processor to apply the REGEXP_REPLACE Snowflake function to a column.

Column Type Converter processor

The Column Type Converter processor can convert columns with the Varchar, Variant, or Binary data type to the Geography data type.

Fixed Issues

• If a table column name uses non-standard characters, the Slowly Changing Dimension processor fails to process the columns correctly.
• When the Apply Function processor uses the Round() and Truncate() functions and is configured to keep only two decimal digits, the processor incorrectly replaces the additional digits with zeros instead of removing them.
• The JSON Parser processor converts the original String column that contains the JSON to be parsed into a Variant column.