Release Notes

Enhancements

Search

Search includes the following enhancements:

• You can use the new Pipeline ID property to search for pipelines or fragments by the pipeline ID.
• The ID property available for pipelines and fragments has been renamed to Commit ID to clarify that you use the property to search for pipelines or fragments by the commit ID.

  Existing saved searches for pipelines or fragments that use the previous ID property name continue to work and find pipelines or fragments by the commit ID.

Environments

When you create an environment, you select the feature version to use for that environment and for all deployments created for the environment.

A feature version allows you to decide when an existing environment and all of its deployments gain access to newer features that might require additional permissions in your cloud service provider account or that might require a restart of engines belonging to the deployments.

AWS environments support a new AWS_2023_12_15 feature version which includes support for Amazon EC2 spot instances and AWS launch templates.

At this time, all other environment types support an initial feature version.

Deployments

Deployments include the following enhancements:

• Cloning deployments - Clone a deployment to quickly create another deployment with similar configurations. You can select the same engine version or a later engine version than used in the original deployment.

  You might clone a deployment as the first step in the engine upgrade process. When you edit a cloned deployment that uses a later engine version than the original deployment, you can view and compare the engine configuration differences between the original and cloned deployments.

• Amazon EC2 deployments - You can configure an Amazon EC2 deployment to provision EC2 spot instances, in addition to on-demand instances. Requires the AWS_2023_12_15 environment feature version.

  Amazon EC2 deployments that use the initial feature version provision EC2 on-demand instances only.

• Azure VM deployments - An Azure VM deployment for Data Collector 5.8.0 and later deploys the Data Collector engine as a tarball rather than a Docker image.

  Azure VM deployments for earlier Data Collector versions and for Transformer continue to deploy those engines as a Docker image.

Organization security

The primary organization administrator can designate another user assigned the Organization Administrator role to be the primary organization administrator.

Fixed Issues

• You cannot export a draft pipeline or fragment using the Export option that removes plain text credentials.

• You cannot import a draft pipeline that includes a fragment.

Enhancements

Preview

Preview includes the following enhancements:

• Keyboard shortcut - You can use Command+Return on Mac and Ctrl+Enter on Windows machines to start previewing pipelines and fragments. Control Hub uses the existing preview configuration for the preview.
• Table view - When you configure preview to display field types, the field type now shows in Table view next to the field name, as it does in list view.

Environments

When you view environment details in the Environments view, you can view the list of deployments that belong to the environment. To view details about a specific deployment, click the deployment name.

Deployments

Deployments include the following enhancements:

• All deployment types - When you view deployment details in the Deployments view, you can click the name of the parent environment to navigate to the environment details.
• Kubernetes deployments - You now configure advanced mode for a Kubernetes deployment in the main deployment wizard rather than in a separate dialog box.

Scheduled tasks

When creating a scheduled task that runs on a minute basis, you can specify which minute to start the job at. For example, if you create a scheduled task that runs every 15 minutes, you can schedule the task to start at 3 minutes after every hour, such that the task starts at 2:03, 2:18, 2:33, and 2:48.

Previously, all scheduled tasks that ran on a minute basis always started at the top of the hour. For example, all scheduled tasks that ran every 15 minutes ran at 2:00, 2:15, 2:30, and 2:45.

Organization default system limits

StreamSets has added a default system limit for the number groups that can exist in an organization. The limit is 1,000 groups.

Behavior Change

Pipeline and fragment version history

With this release, StreamSets has simplified pipeline and fragment version history. You can now access the version history of a pipeline or fragment from the pipeline canvas only.

Previously, you could access different views of the version history - one from the pipeline canvas and another from the Pipelines or Fragments view.

Fixed Issues

• When users without the Organization Administrator role try to share an object that they have read access to, the Sharing Settings dialog box displays empty user and group names.
• While editing a connection with an engine of a more recent version than the one the connection was created with, the wizard never moves to the second step for some connection types.
• When a self-managed deployment defines a percentage JVM memory strategy for engines, the following Docker engine installation types on Linux with cgroup v2 enabled incorrectly allocate a percentage of the available memory on the host machine as the Java heap size, instead of the available memory in the Docker container:
  • Data Collector
  • Transformer prebuilt with Scala 2.11

Enhancements

Search

When performing a basic string search, you can select the operator does not contain to find objects that do not contain a specified string. For example, the following basic search condition finds all pipelines that do not contain the string ADLS2 in the pipeline name:

Name does not contain ADLS2

Previously, you had to use advanced mode to define a condition using the not equal to operator (!=) to search for objects that did not contain a specified string.

Monitoring jobs

When viewing the engine log for an active job, you can select another engine that you want to view the log for.

Editing environments and deployments

When you edit an existing environment or deployment, you can easily scroll to the configuration properties that you want to change.

Previously, you had to click through each step in the environment or deployment wizard to find the properties that you wanted to change.

Kubernetes deployments

When using advanced mode to configure a Kubernetes deployment, you can download the YAML file, edit the file in a text editor, and then upload the edited file.

Fixed Issues

• If you use bulk edit mode to define runtime parameters for a job instance or job template and change the value of a runtime parameter to a number, you cannot edit the job instance or job template.
• Control Hub incorrectly retrieves the latest offset for a single instance job, causing an outdated or incorrect offset to display.
• Sorting scheduled tasks by the Next Execution Time property does not change the order in which the tasks are displayed.
• When you change the owner of a pipeline, the new owner is not granted permission on the pipeline.

Enhancements

Behavior Change

Fixed Issues

Enhancements

Using runtime parameters in pipelines

To use runtime parameters to define values for stage and pipeline properties, you can now select from a list of existing parameters or you can create and then use a new parameter.

Previously, you first had to define parameters on the pipeline Parameters tab. Then from the stage or pipeline property, you had to type the parameter name using the correct expression language syntax.

For details about using parameters, see the appropriate engine documentation:

• Data Collector
• Transformer
• Transformer for Snowflake

Restart engines for an active deployment

You can restart all engine instances belonging to an active deployment from the Deployments view.

Previously, you had to navigate to the Engines view to restart engine instances.

Mapping jobs in a topology

When editing a topology and you click Add Job > Show All Jobs, you can search for jobs that you want to map to the topology. You can perform both basic and advanced searches.

Previously, you had to scroll through all jobs to select jobs to map to the topology.

Authoring engine timeout

An organization administrator can modify the default authoring engine timeout in the organization properties. Individual users can then override the default organization value in their browser settings within the My Account window.

Previously, an organization administrator could not modify the default organization value. The organization default was always five seconds, and individual users could modify that default value in their browser settings within the My Account window.

If you previously modified the authoring engine timeout in your browser settings, the modified value overrides any changes that an organization administrator makes to the default value.

Organization default system limits

StreamSets has increased the default system limits for the following types of objects:

• Jobs - Increased from 500 to 10,000. The system limit for active jobs that can run concurrently is 1,000.
• Scheduled tasks - Increased from 100 to 5,000.

Fixed Issues

• Pipeline preview incorrectly showed multiple spaces in record values as a single space.
• When using the Stream Selector processor, connecting the last stream when there were more than 4 streams required clicking on the very edge of the icon.
• The Job Templates, Job Instances, and Draft Runs views display duplicate preset searches.
• When a pipeline is using an inaccessible engine and then you switch to an accessible engine, Control Hub incorrectly indicates that the newly selected engine is not accessible until you refresh the browser.
• When you edit a job using the context menu on the right side of the list of jobs, the Number of Instances property is always set to 0.
• When you update an external resource archive to remove a file and then restart all engine instances in the deployment, Control Hub copies the updated archive file contents to the engine instances without first removing the deleted file. As a result, the file deleted from the archive still exists in the engine instances.

Behavior Change

Update firewall outbound allowlists

The StreamSets authentication service, https://cloud.login.streamsets.com, is now hosted in another cloud service provider.

If you access the StreamSets platform from a machine that resides behind a firewall or in a system that limits access to specific DNS names and IP addresses, ensure that your firewall allows outbound connectivity to all of the following DNS names and IP addresses to ensure that you can continue to log into StreamSets:

• 3.133.68.10
• 3.134.148.19
• 3.137.15.194

Behavior Change

Enhancements

Scheduled tasks

When you view the list of scheduled tasks, the next scheduled execution time displays for each task.

External resources

You can store an external resource archive file in a private Amazon S3 or Google Cloud Storage (GCS) bucket or in a private Azure Blob Storage or Azure Data Lake Storage Gen2 container, as long as the deployment using the archive file is of the same cloud service provider type. For example, if using an Amazon EC2 deployment, you can store the archive file in a private Amazon S3 bucket. You cannot store the archive file in a private GCS bucket.

To ensure that all engine instances managed by the deployment can access the private file, grant read access to the file to the AWS instance profile, the GCE instance service account, or the Azure managed identity associated with the provisioned instances.

Previously when you stored an external resource archive file in an Amazon S3 or GCS bucket or in an Azure container, you had to share the file publicly.

Engine resource thresholds

A deployment defines the maximum thresholds for the following engine resources:

• CPU load
• Memory used
• Number of running pipelines

All engine instances belonging to the deployment inherit the same resource threshold values.

For advanced use cases, some organizations can override the resource thresholds for individual engine instances. However, when an engine restarts, overridden values are lost and the engine inherits the resource thresholds set for the deployment.

Previously, you defined resource thresholds for each engine instance. If you previously modified the default values for resource thresholds, those modified values are retained until the engine restarts.

This enhancement can result in a behavior change for existing automated testing.

Self-managed deployments

When you stop a self-managed deployment, running engine instances belonging to that deployment are also shut down. After you restart the self-managed deployment, you manually start each engine instance from the command line.

Previously when you stopped a self-managed deployment, running engine instances belonging to that deployment were deactivated and could no longer communicate with Control Hub, but continued to run. As a result, engine instances continued to use computing resources although they could no longer process data.

Google Compute Engine (GCE) deployments

When you create a GCE deployment, you specify whether to use an automatic or user managed replication policy to store GCP Secret Manager secrets. If you specify a user managed policy, then you also select one or more locations to replicate the secrets to.

If your Google Cloud organization has disabled global resource creation, then you must configure the deployment to use a user managed replication policy.

Previously, GCE deployments supported only the automatic replication policy.

Behavior Change

Fixed Issues

• When you export a draft pipeline or fragment, Control Hub does not remove plain text credentials configured directly in the pipeline or fragment.
• A Control Hub Kubernetes deployment becomes stuck in an Activating state when you configure only the http.nonProxyHosts property for the engine proxy properties.

• The Create Scheduled Task dialog box displays blank error messages.
• When a deployment defines a percentage JVM memory strategy for engines, a Docker engine installation incorrectly allocates a percentage of the available memory on the host machine as the Java heap size, instead of the available memory in the Docker container.

Enhancements

Welcome view

After you log in, Control Hub displays a Welcome view that lists your most recently edited pipelines and jobs so that you can quickly access them.

Quick Start

Click Quick Start in the top toolbar to quickly build a Data Collector or Transformer for Snowflake pipeline. A blank pipeline opens in the canvas so that you can immediately begin building the pipeline using all available stages. For a Data Collector pipeline, you must select or deploy an engine before you can preview, run, validate, or check in the pipeline.

Previously, you could use the Quick Start menu to quickly deploy a Data Collector or Transformer engine using Docker or as a shortcut to build any type of pipeline.

Edit Data Collector pipelines when engine is not accessible

When using Data Collector version 5.4.0 or later, you can edit pipelines and fragments in the pipeline canvas when the engine is not accessible. The engine must be accessible before you can preview, run, validate, or check in the pipeline or fragment.

Import pipelines and fragments

When you import a pipeline or fragment that uses connections, you can choose how to replace each connection based on whether the connection exists in the target organization:

• Connection exists - You can choose to use the existing connection, create a placeholder connection, or remove the connection.
• Connection does not exist - You can choose to create a placeholder connection or remove the connection.

Previously, if the connections existed, the imported pipeline or fragment automatically used the existing connections. If the connections did not exist, you could choose to import the pipeline or fragment without the connections or you could cancel the import.

Job templates

When you delete a job template that has attached job instances, Control Hub also deletes the attached job instances at the same time.

Previously, you could not delete a job template that had attached job instances. You first had to navigate to the Job Instances view and delete all attached job instances that were created from that template.

Control Hub-managed deployments

Control Hub-managed deployments include the following enhancements:

• For Amazon EC2, Azure VM, and GCE deployments, the Engine Instances property has been renamed to Desired Instances.
• For Amazon EC2, Azure VM, GCE, and Kubernetes deployments, you can set the Desired Instances property to 0 to temporarily prevent engine instances from running, as an alternative to stopping the deployment.

  Previously, the minimum number of instances was 1. You had to stop the deployment to temporarily prevent engine instances from running.

Groups

Groups require a unique display name in addition to a unique group ID. Previously, groups required only a unique group ID.

Alerts

When the alert text defined for a data SLA or pipeline alert exceeds 255 characters, Control Hub truncates the text to 244 characters and adds a [TRUNCATED] suffix to the text so that the triggered alert is visible when you click the Alerts icon in the top toolbar.

Previously, when the alert text exceeded 255 characters, Control Hub did not display the alert when you clicked the Alerts icon.

Fixed Issues

• If you change the values of existing pipeline fragment parameters, publish a new version of the fragment, and then update a pipeline using that fragment to use the latest version, the update incorrectly changes the default values for any existing runtime parameters in the pipeline.
• Fragment export fails when a fragment contains multiple stages of the same type that use different library versions.
• When you select a stage in the canvas while monitoring a running Data Collector or Transformer pipeline, an “Unexpected error occurred” message temporarily displays.
• When configuring a GCE deployment, you are required to select a service account even if the GCP environment has a default service account defined. Similarly, when configuring an Azure VM deployment, you are required to select a managed identity and resource group, even if the Azure environment has defaults defined for those objects.

Enhancements

Fragment icons

By default when you add a published fragment to a pipeline, the fragment displays as a single stage with a puzzle piece icon: . You can modify the icon to represent the fragment processing logic. For example, if a fragment merges two streams of data, you might configure the fragment to use the predefined Merge icon: .

You can select from a set of predefined icons, or you can upload an image to use as the icon.

Pipeline canvas toolbar

The new pipeline canvas UI displays the following icons in different colors, making it easier to locate these icons in the toolbar:

• The Stop and Force Stop icons display in red: .
• The Check In icon displays in red () when the pipeline or fragment has not passed implicit validation and cannot be checked in. The Check In icon displays in green () when the pipeline or fragment has passed implicit validation and is ready to be checked in.

Fragment export

You can export and import draft fragments.

GCE deployments

You can optionally configure a GCE deployment to provision Google Cloud VM instances without external IP addresses.

Configure the deployment to provision VM instances without external IP addresses only when your GCP project does not allow externally accessible IP addresses and your Google Cloud administrator has created a Cloud NAT gateway as a GCP environment prerequisite.

Previously, a GCE deployment always provisioned VM instances with external IP addresses.

Fixed Issue

• If the last page of results for a job search previously returned items, but no longer does, no items are shown on the page. Users must navigate to the previous page to see items.

Fixed Issue

• When a deployment uses an external resource archive and the externalResources folder contains an empty user-libs folder, engines belonging to the deployment fail to start with the following error message:

  Exception in thread "main" java.lang.IllegalArgumentException: Stage libraries directory '/<installation directory>/externalResources/user-libs' does not exist

Enhancements

First login to StreamSets

When you log in to StreamSets for the first time and you do not have access to an engine deployed by another user, a blank Data Collector pipeline opens in the canvas. You can immediately begin building the pipeline using all available stages.

When you attempt to preview, publish, validate, or start a draft run of this first pipeline, Control Hub presents a simplified process to help you quickly deploy your first engine.

Pipeline canvas

When using the new pipeline canvas UI and you select a link connecting two stages, Control Hub displays a pop-up menu that includes the following icons:

• Insert Stage () - Inserts another stage between the connected stages.
• Delete () - Deletes the selected link.

Previously, the pop-up menu included only an Insert Stage icon.

Job templates

On the Job Templates view, the Start Jobs menu option has been renamed to Create Instances to more clearly indicate that you are creating and starting job instances from a job template.

Fixed Issues

• Users and engines randomly encounter 401 authorization errors from Control Hub.
• Editing an existing job template fails with the following error:

  'len' parameter must be a valid integer equal to -1 or greater than zero. len = '0' : RESTAPI_05

• Searching for a pipeline by ID fails with the following error:

  "RSQL_00:Bean 'RPipeline' property 'id' does not have a mirrored property in 'PPipeline'"

• The pipeline canvas incorrectly enables the Draft Run menu even though the selected authoring engine is not accessible.
• When you modify or delete a parameter from a fragment being used in a pipeline, publish the fragment, and then update the pipeline to use the latest fragment version, the parameter changes are not reflected in the pipeline.
• If you use advanced mode to edit the dnsPolicy attribute in a Control Hub Kubernetes YAML file, your modifications do not take effect. The deployment always uses the Default DNS policy.
• If you specify a service account name for a Control Hub Kubernetes deployment, start the deployment, and then edit the active deployment to delete the service account name, the change does not take effect and the previous service account remains associated with the Kubernetes deployment.

Enhancements

Search

Search includes the following enhancements:

• Control Hub includes additional preset searches for fragments, pipelines, job templates, job instances, and draft runs that are starred by default.
• When searching for pipelines or fragments, you can use the Latest Version property to restrict searches to the latest version of each pipeline or fragment.

Pipeline and fragment preview

Previewing pipelines and fragments includes the following enhancements:

• When you preview a pipeline or fragment, Control Hub uses the default preview configuration and no longer displays the Preview Configuration dialog box. While running preview, you can click the Preview Configuration icon () to change the configuration and then run the preview again.
• When you preview a Transformer or Transformer for Snowflake pipeline or fragment, the data displays in table view by default.

  When you preview a Data Collector pipeline or fragment, the data continues to display in list view by default.

• Table view displays only output data by default. You can optionally choose to display both input and output data.
• Table view no longer displays colors for different types of data and changed data. List view continues to display colors.
• While running preview, you can click the Expand icon () to quickly expand the preview panel to view more preview data.

Pipeline validation

If pipeline validation fails due to a timeout error, you can increase the validation timeout value.

Job templates

When you create job instances from a job template, you can specify whether the job instances inherit the permissions assigned to the parent job template.

Pipeline export

You can export and import draft pipelines.

Customize table columns

You can resize, hide, and reorder the table columns displayed in all views.

Azure VM deployments

The tracking URL for an active Azure VM deployment opens the overview page of the Azure deployment created for your StreamSets deployment. The deployment overview page enables you to more quickly access information about the Azure resources provisioned for the StreamSets deployment.

Previously, the tracking URL opened the overview page of the Azure virtual machine scale set.

Fixed Issues

• Basic searches that use the not includes operator or advanced searches that use the =out= operator incorrectly return results that do include the specified values.
• If you view the run history of a job and the pipeline version for one of the runs has been deleted, then the run history displays an error.
• When a pipeline fails over to another Data Collector, the input and output records that display in the Summary tab and in the run summary for a selected job run accessed from the History tab do not include the records for all Data Collectors.
• Control Hub allows you to set the CPU Threshold Percentage for a Control Hub Kubernetes deployment to a value between 0 and 100, but 0 is not a valid value in the Kubernetes cluster.

New Features and Enhancements

Kubernetes integration

Control Hub provides an integration with Kubernetes. When you use Control Hub Kubernetes environments and deployments, you launch a StreamSets Kubernetes agent that runs in your Kubernetes cluster. The agent communicates with Control Hub to provision the Kubernetes resources needed to run StreamSets engines and to deploy engine instances to those resources.

Search

Search is no longer a Technology Preview functionality and is now enabled by default. Search replaces filtering and is implemented for the following object types:

• Fragments
• Pipelines
• Job templates
• Job instances
• Draft runs

StreamSets will implement search for additional object types in future releases. If needed, you can disable search and revert to the original filtering functionality.

Search includes the following additional enhancements:

• Control Hub stores your saved searches in the backend rather than in your current browser. If you saved searches in a previous release, this enhancement has a behavior change.
• You can star your most important saved searches so that you can easily find them later.
• Control Hub includes several preset saved searches for pipelines that are starred by default. You can unstar these preset searches, but cannot edit or delete them. StreamSets will implement preset searches for additional object types in future releases.
• You can search for draft runs that include a pipeline with a specified pipeline status or pipeline version.

Pipeline canvas

By default, Control Hub now displays the new pipeline canvas UI.

You can optionally switch to the classic UI while viewing a pipeline or fragment in the canvas.

Customize table columns

You can resize, hide, and reorder the table columns displayed in the Job Templates, Job Instances, and Draft Runs views.

In addition, you can reset table column widths to the default size after resizing a table.

Behavior Changes

Saved searches

With this release, Control Hub stores your saved searches in the backend. Previously, Control Hub stored saved searches in your current browser. Saved searches did not apply if you logged in using another browser, and saved searches were removed if you cleared the browser cache.

As a result of this change, you can no longer access previously saved searches.

In addition, if you previously configured Control Hub to persist the last search, Control Hub might unexpectedly display no found objects when you first access a view that supports search, such as the Pipelines view. To resolve this issue, perform another search or click the More icon () in the search section, and then click Clear All.

Job instances with no job runs

With this release, Control Hub automatically deletes inactive job instances older than 365 days that have never been run.

Previously, Control Hub indefinitely retained inactive job instances with no job runs.

Fixed Issue

• When a pipeline fails over to another Data Collector, the input and output records that display in the job history do not include the records for the original Data Collector.

Fixed Issue

• When SCIM provisioning is enabled and Microsoft Entra ID synchronizes users with expired Control Hub invitations, these users cannot log into Control Hub, even though the users have been correctly provisioned.

Enhancements

View and edit connection details from a published pipeline

When viewing a published pipeline that includes a stage using a connection, you can click the Edit Connection icon in the stage properties to view and edit the connection details.

Previously, only draft pipelines displayed the Edit Connection icon.

Customize table columns

You can resize, hide, and reorder the table columns displayed in the Connections view.

StreamSets will provide support to customize columns in additional views in future releases.

Job run history for job instances started from a job template

Control Hub now includes job instances started from a job template in the total count of job runs for the organization. As a result, Control Hub purges the run history for job instances started from a job template in the same way that it purges the run history for job and draft runs.

Previously, Control Hub indefinitely retained the run history for job instances started from a job template.

Search

The Technology Preview search functionality includes the following enhancements:

• You can search for attached job instances created from job templates.
• You can search for job instances that include a pipeline with a specified pipeline status.
• You can search for job templates that have been archived.
• In basic search, you can use auto-completion for the following search properties:
  • Label property for pipelines and fragments
  • Tag and Engine Label properties for job instances and job templates

  As you begin typing a value to search for, Control Hub displays a drop-down menu that lists values that match the entered characters.

Fixed Issues

• Preview does not work when the Run Preview Through Stage property is set to a fragment.
• The Engines view does not correctly sort the list of engines by the Memory Used column.
• When using the new pipeline canvas UI while monitoring a job, the title of the job incorrectly displays the pipeline name instead of the job name.
• If you create and publish a pipeline fragment using pipeline stages and you quickly enter a fragment prefix in the last step of the wizard, the newly created fragment is not added to the pipeline.
• When using the Technology Preview search functionality and you sort pipeline search results by the Last Modified By column, the pipelines are incorrectly sorted by name.

Enhancements

Pipeline fragments

When you create a pipeline fragment using pipeline stages and you choose to publish the new fragment, Control Hub displays the original pipeline in the canvas, automatically replacing the individual stages with the newly published fragment.

Previously, Control Hub displayed the original pipeline in the canvas with the individual stages.

Draft runs

While monitoring an engine in the Engines view, you can stop all draft runs currently running on that engine.

Search

The Technology Preview search functionality includes the following enhancements:

• You can perform both basic and advanced searches to find specific draft runs.
• You can search for job instances by the job status color.

Subscriptions

You can configure a webhook action for a subscription to use one of the following additional authentication types to connect to the receiving system:

• API Key
• Bearer Token
• OAuth 2.0

Fixed Issues

• The Control Hub REST API incorrectly returns an HTTP 400 status code instead of a 404 status code when the specified environment or deployment does not exist.
• When using the Technology Preview search functionality to find job templates or job instances last modified by a user, the search results incorrectly display the job created by that user.
• When the Legacy Kubernetes integration is enabled, users cannot create a pipeline when they are assigned the Deployment Manager role but not the Provisioning Operator role or the Organization Administrator role.

Enhancements

Deployments

In the Deployments view, you can filter the list of displayed deployments by engine type.

Pipeline design

When configuring the conditions for a Stream Selector processor in a Data Collector pipeline, you can reorder the output conditions.

Search

The Technology Preview search functionality includes the following enhancements:

• You can search by user email address to find all pipelines or fragments committed or last modified by the user or to find all job templates or job instances created or last modified by the user.
• When using basic mode to search for job templates or job instances, you can include or exclude the v prefix. For example, you can search for both v1 or 1 when defining the Pipeline Version property.

  Previously in basic mode, you had to include the prefix. For example, you had to search for v1.

Fixed Issues

• When you expand or collapse a fragment in the job monitoring UI, Control Hub does not update the metric diagrams to display statistics for individual stages in the expanded fragment or for the single fragment stage.
• When the Enable WebSocket Tunneling for UI Communication property is enabled for an organization, all users should be able to configure the Browser to Engine Communication type; however, only organization administrators can do so.
• When using the Technology Preview search functionality, searching for job instances with an INACTIVE job status excludes job instances that have never run.
• When using the Technology Preview search functionality and you sort the search results by job instance status, job instances that have never run are shown first when sorting by ascending status, and last when sorting by descending status.
• When you add a fragment that uses parameters to a pipeline and define no parameter name prefix, Control Hub silently fails to add the fragment to the pipeline.
• When you configure an Azure VM deployment, Control Hub does not display existing SSH key pairs in the Key Pair Name property because the documentation to create an Azure AD custom role for the parent environment does not include the following required permission:

  Microsoft.Compute/sshPublicKeys/read

  Important: StreamSets strongly recommends updating the custom role for all existing Azure environments, as described in the Azure environment prerequisites.

New Feature

Provisioning of user accounts from Microsoft Entra ID

After enabling SAML authentication using Microsoft Entra ID (previously known as Azure AD), you can configure the provisioning of user accounts from Entra ID to StreamSets. When a user or group is created, updated, or deleted in Entra ID, the same changes are automatically made in StreamSets.

StreamSets supports System for Cross-domain Identity Management (SCIM) 2.0 for the provisioning of user accounts from an identity provider (IdP).

StreamSets will provide SCIM provisioning support for additional IdPs in future releases.

Enhancements

Supported browsers

StreamSets Control Hub now supports the latest version of Microsoft Edge as a browser.

Search

The Technology Preview search functionality includes the following enhancements:

• For job instance and job template search, the Pipeline Commit Label property has been renamed to Pipeline Version to more clearly indicate that the property specifies the version of the pipeline included in a job instance or job template.
• Basic search for job templates no longer displays the Status property because job templates do not have a status.

Proxy server configuration for engines

When you set up a deployment that configures a Data Collector or Transformer engine to use a proxy server for outbound network requests, you can now include special characters in the proxy user and password values, except for an exclamation point (!), a backward slash (\), a leading number sign (#), or a leading or trailing space.

Fixed Issues

• The Test Connection button in the connection wizard incorrectly requires a user to have the Deployment Manager role.
• Creating a pipeline incorrectly requires a user to have the Deployment Manager role.
• The Fields to Convert property in the Field Type Converter processor does not allow an expression that includes a comma.
• If you configure an engine to use a proxy server and you include a space, single quotation mark, or double quotation mark in the defined proxy properties, then either the engine installation script fails or the proxy authentication fails.
• Topology data SLAs do not trigger alerts because they do not correctly retrieve data.

Fixed Issue

Fixed Issue

New Features and Enhancements

Search

You can perform both basic and advanced searches to find specific pipelines, fragments, job instances, or job templates. With basic search, you define search conditions by selecting the object properties, operators, and values that you want to search for.

With advanced search, you define search conditions by writing a search query using the StreamSets advanced query language (SAQL). Advanced search allows you to specify criteria that cannot be defined in basic search, such as defining complex conditions or changing the order of precedence for multiple conditions.

Search replaces filtering and is a Technology Preview functionality. To try the functionality, you must enable search in your browser settings. StreamSets will implement search for additional object types in future releases.

Copy field value in preview

You can quickly copy a field value from pipeline preview using the Copy Field Value to Clipboard icon: .

Pipeline preview displays the new icon next to the existing Copy Field Path to Clipboard icon: . The icons display for each field in the previewed records, as follows:

Previously, pipeline preview displayed only a Copy Field Path to Clipboard icon for each field.

Pipeline version history

When using the new pipeline canvas UI, the pipeline or fragment version history that opens from the canvas displays all information on the initial panel. You do not need to expand the version history to manage the versions including viewing commit messages, comparing versions, creating tags for versions, and deleting versions.

In addition, the new version history includes a link to each version and displays the minimum engine version that the pipeline or fragment version can run on, as follows:

Import pipelines and fragments without connections

When you import pipelines or fragments that use connections and those connections do not exist in the target organization, you can choose to import the objects without connections. After the import, you must edit the pipelines or fragments to define the connections.

Previously, to import pipelines or fragments that use connections, the connections had to exist in the target organization.

Subscription parameters

You can use the ERROR_MESSAGE parameter for a subscription triggered by a maximum global failover retries exhausted event.

Engine resource thresholds

The default value of the Max Memory threshold for an engine is now 100%. Previously, the default value was 80%. Existing engines retain the currently configured threshold.

Behavior Change

With this release, the sample IAM policy for credentials provided by a Control Hub AWS environment includes the following additional permission:

autoscaling:DescribeWarmPool

This permission is required when you update the number of engine instances for an active Amazon EC2 deployment belonging to an AWS environment.

If the IAM policy does not include this permission and you update the number of engine instances, the deployment transitions to an Activation Error state. When you access the tracking URL to the AWS Management Console, the Events tab displays the following error:

API: autoscaling:DescribeWarmPool User: arn:aws:sts::${ACCOUNT_ID}:assumed-role/${CROSS_ACCOUNT_ROLE}/STREAMSETS_SCH is not authorized to perform: autoscaling:DescribeWarmPool because no identity-based policy allows the autoscaling:DescribeWarmPool action

StreamSets strongly recommends updating the IAM policy for credentials for all existing StreamSets AWS environments to avoid this deployment error.

...
{
    "Sid": "0",
    "Effect": "Allow",
    "Action": [
        "ec2:DescribeImages",
        "autoscaling:DescribeScalingActivities",
        "ec2:DescribeVpcs",
         "autoscaling:DescribeAutoScalingGroups",
        "ec2:DescribeRegions",
        "autoscaling:DescribeLaunchConfigurations",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInstanceTypeOfferings",
        "ec2:DescribeSubnets",
        "ec2:DescribeKeyPairs",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeInstances",
        "autoscaling:DescribeScheduledActions”,
        "autoscaling:DescribeWarmPool"
        ],
    "Resource": "*"
},
...

You do not need to restart active StreamSets AWS environments or Amazon EC2 deployments after updating the policy. However, if an existing Amazon EC2 deployment has transitioned to an Activation Error state due to this error, then you must stop that deployment and start it again.

For more information about the required IAM policy, see Configure AWS Credentials.

Fixed Issues

• The Topologies Dashboard can display an incorrect number of engines.
• Pipelines that contain fragments with names starting with a number do not correctly display in the new pipeline canvas UI and encounter unexpected errors in the classic pipeline UI.
• Stages that were originally in a fragment could become uneditable.

Enhancements

Scroll zoom in pipeline canvas

You can now zoom in or out of the pipeline canvas using the mouse scroll wheel or using the trackpad. By default, scroll zoom is enabled. You can disable scroll zoom if needed.

Hide or show contextual help in wizards

When you create or edit an object, such as an environment, deployment, pipeline, or job, you can now hide or show the contextual help that displays on the right of the wizard.

Behavior Change

Filters for job templates, job instances, or draft runs

With this release, when you select the Keep Filter Persistent checkbox to retain the filter on the Job Templates, Job Instances, or Draft Runs view, Control Hub retains a unique filter for each view. Previously, Control Hub incorrectly used the same filter for all of these views.

As a result, when you first access the Job Templates, Job Instances, or Draft Runs view, no filter is applied, even if you previously persisted the filters.

Fixed Issues

• In rare cases when editing a deployment, the External Resource Source property can be set to null which causes a null pointer exception.
• The TRIGGERED_COUNT and TRIGGERED_ON subscription parameters do not contain the correct values.
• The Windowing Aggregator processor does not display aggregation charts.
• You can delete a job associated with a running scheduled task, resulting in a scheduled task that attempts to trigger actions on a job that doesn’t exist.
• In rare cases, the configured time zone for a scheduled task is ignored, which causes the task to start and finish at undesired times.
• Clicking Upload Offset & Start while viewing job instance details uploads the offset but doesn’t start the job.
• Because the initialization script for an Azure VM deployment does not run before engine instances start, you cannot use the script to update DNS entries. As a result, engines fail to launch when the Azure VNet uses custom DNS servers.

Enhancements

Runtime parameters for pipelines

Some pipeline and stage properties conditionally display child properties. For example, if you configure an origin to use the Delimited data format, the origin displays a set of Delimited configuration properties. If you configure that origin to use the JSON data format, it displays a different set of JSON configuration properties.

However, if you use a runtime parameter to define a parent property, all child properties now display so that you can configure valid values for all dependent properties.

For example, if you convert the Data Format drop-down menu to a text box and then call the dataformat parameter using the required syntax, the origin displays all of the Delimited, JSON, and Text configuration properties.

Job monitoring toolbar

When you monitor a job, the canvas includes a new toolbar that more clearly indicates the action that each toolbar icon completes.

You can switch between the new and classic UI while monitoring a job.

Starting multiple scheduled jobs at the same time

When you start multiple jobs at the exact same time using the scheduler, the number of pipelines running on an engine can exceed the Max Running Pipeline Count configured for the engine.

If exceeding the resource threshold is not acceptable, you can enable an organization property that synchronizes the start of multiple scheduled jobs. However, be aware that enabling the property can cause scheduled jobs to take longer to start.

Fixed Issues

Enhancement

Fixed Issues

Enhancement

View and edit connection details from stage properties

When configuring a pipeline or fragment stage that uses a connection, you can click the Edit Connection icon in the stage properties to view and edit the connection details.

Fixed Issues

• The Available Pipeline Runners histogram does not display when you monitor a job for a Data Collector multithreaded pipeline.
• The pipeline canvas cannot handle more than 100 instances of the same stage.
• Provisioning Agents are incorrectly included in the system limit for engines.

Enhancements

Compare pipeline and fragment versions

When using the new pipeline canvas UI to compare two versions of a pipeline or pipeline fragment, you can click the Open in Canvas icon () to open one of the versions in the pipeline canvas.

Engine type icon in Fragments, Pipelines, and Sample Pipelines views

The Fragments view, Pipelines view, and Sample Pipelines view include an engine type icon next to each pipeline or fragment name so that you can quickly determine the pipeline or fragment type.

The following table lists each engine type icon:

Icon	Engine Type
	Data Collector
	Transformer
	Transformer for Snowflake

Fixed Issues

• Credential fields in a pipeline state notification webhook do not properly show the value of entered passwords when authentication is used.
• On Windows 10, some file archive programs cannot extract an exported Control Hub ZIP file because of the colon (:) character in the JSON file names.

New Features and Enhancements

Draft runs

Pipeline test runs have been renamed to draft runs to more clearly indicate that a draft run is the execution of a draft pipeline.

Control Hub includes a new Run > Draft Runs view that you use to manage all draft runs that you have access to. You can view active and inactive draft runs, view the draft run history for a pipeline, and can stop or delete draft runs as needed.

Previously, you could only manage active draft runs from the pipeline canvas.

Stage selector in the pipeline canvas

The new pipeline canvas UI includes an enhanced stage selector with a horizontal layout, making it easier to filter the stages by type.

In addition, when you add a stage with an open link, the pipeline canvas displays a dotted link connected to the Add Stage icon, clearly indicating that you need to add another stage:

When you click the Add Stage icon, the stage selector opens, allowing you to search for and select the next stage to add:

Fixed Issues

• When authorizing Control Hub API calls, Control Hub does not consider the roles assigned by a user’s group.
• When you change the owner of a scheduled task, the previous owner is incorrectly listed as the user that executes the task.

Fixed Issue

New Features and Enhancements

Init scripts for cloud service provider deployments

You can define an initialization script for cloud service provider deployments, such as Amazon EC2, Azure VM, and GCE deployments. Control Hub runs the init script while provisioning a new instance in your cloud account.

Use the script to set up provisioned instances with additional software as required by your organization. For example, you might use an init script to install required certificates or packages on each provisioned instance.

Pipeline canvas toolbar

The pipeline canvas includes a new toolbar that more clearly indicates the action that each toolbar icon completes. By default, the pipeline canvas continues to display the classic toolbar.

You can switch between the new and classic UI while viewing a pipeline or fragment in the canvas.

Job status

Jobs that have not run are listed with an Inactive status. Previously, jobs that had not run were listed without a status.

Editable group ID

When you create a group, you define the group display name and can optionally edit the group ID. By default, Control Hub generates the group ID from the display name, using all lowercase characters and replacing any spaces with underscores. Users type the group ID when using credential functions in pipelines. As a result, you might want to edit the default group ID to make it easier to use with credential functions.

Previously when you created a group, you defined only the group display name. Control Hub used a randomly generated ID for the group ID which was not editable. When using credential functions, you had to type the randomly generated ID. Groups created before this release retain their existing group IDs.

Fixed Issue

Enhancements

Simplified pipeline and fragment wizards

The pipeline wizard and the fragment wizard have been simplified and no longer include the redundant Review & Open step.

Upgrading a pipeline to use a later authoring engine version

When you select a later authoring engine version for a pipeline, Control Hub now informs you that you are upgrading the pipeline so that it can no longer run on the earlier engine version. If you select a later engine version for a draft pipeline, you are given the choice to publish the draft pipeline first, and then create a new draft pipeline that is upgraded to run on the later engine version. That way, you retain a pipeline version that can run on the earlier engine version.

Previously, when you selected a later engine version for a pipeline, Control Hub automatically upgraded the pipeline without the upgrade warning.

Fixed Issues

• When you disable a deployment and then access a pipeline using an authoring engine from that deployment, you are logged out.
• Export All Published Pipelines exports only some published pipelines.
• Pressing Ctrl+Z doesn't undo text changes after you edit text in the Control Hub UI.
• Exporting jobs can generate a corrupt ZIP file.

New Feature

Transformer for Snowflake available in public preview

New and existing organizations can create and run Transformer for Snowflake pipelines.

Transformer for Snowflake is a component of the StreamSets platform that enables processing Snowflake data using Snowflake's Snowpark client libraries.

Transformer for Snowflake provides a user interface that allows designing and performing complex processing in Snowflake without having to write SQL queries or templates. It also provides easy access to Snowpark DataFrame-based processing so you can use Snowpark without having to set it up in your Snowflake account.

Important: While in public preview, do not use Transformer for Snowflake for production workloads. For help with Transformer for Snowflake, join the StreamSets Community and use the Transformer for Snowflake tag when you post your question. To provide suggestions and feedback, add a new idea at https://ibm-data-and-ai.ideas.ibm.com/?project=SSETS.

For more information about Transformer for Snowflake, see the Transformer for Snowflake documentation.

Fixed Issues

• When you stop an Azure VM deployment, the Azure key vaults created to store secrets for the provisioned VM instances are not deleted from your Azure account.

  If you added Azure tags to Azure VM deployments that were stopped before this fix, you can use the tags to find the key vaults and then delete them from your Azure account.

• Subscriptions do not trigger when permission enforcement is disabled for the organization.

Enhancements

Fixed Issues

Enhancement

Fixed Issue

Enhancements

Fixed Issues

Enhancement

Fixed Issue

Enhancements

Fixed Issues

Behavior Change

Enhancements

Behavior Changes

Fixed Issues

New Features and Enhancements

Fixed Issues

Enhancement

New Feature

New Feature

Fixed Issues

New Features and Enhancements

Fixed Issues

Fixed Issue

Enhancements

Enhancements

New Features and Enhancements

Behavior Changes

Fixed Issues

New Features and Enhancements

Fixed Issues

Fixed Issues

• When the parent AWS environment uses an AWS region other than us-west-2, an Amazon EC2 deployment fails to launch a StreamSets engine with the following error:

  Parameter validation failed: parameter value ami-<AMI_ID> for parameter name AMI does not exist

• Using the API Credentials view to generate and manage credentials for use with the Control Hub REST API is not supported at this time.

New Features and Enhancements

Enhancements

Environments

The Allow Nightly Engine Builds property for environments has been moved to an advanced option. In most cases, you don't need to modify this property.

Roles

The Auth Token Administrator and Engine Guest roles have been removed because they are not applicable for the StreamSets platform.

Fixed Issues

• If you create a Microsoft account just before signing up as a new StreamSets user with that Microsoft account, the sign in might fail.

• In rare and randomly occurring scenarios, deleting a user from an organization might only partially succeed. Users in this state are not able to create a new login session, but may still show up in the Control Hub list of users for an organization.

• A user with the Organization Administrator role requires the Metrics Reader role to view topology metrics. The Organization Administrator role should be sufficient.