Release Notes

• When you delete multiple users at the same time, a user synchronization error message displays even though the users are successfully deleted.

• When SAML authentication is enabled, only users with the Organization Administrator role can log in.

...
{
    "Sid": "0",
    "Effect": "Allow",
    "Action": [
        "ec2:DescribeImages",
        "autoscaling:DescribeScalingActivities",
        "ec2:DescribeVpcs",
         "autoscaling:DescribeAutoScalingGroups",
        "ec2:DescribeRegions",
        "autoscaling:DescribeLaunchConfigurations",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInstanceTypeOfferings",
        "ec2:DescribeSubnets",
        "ec2:DescribeKeyPairs",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeInstances",
        "autoscaling:DescribeScheduledActions”,
        "autoscaling:DescribeWarmPool"
        ],
    "Resource": "*"
},
...

• When sharing an object, you cannot search for user and group names that include the search string, only names that start with the search string.
• If you rename a job, scheduled tasks for that job do not update the job name.
• When you delete users or groups and then create users or groups with the same ID, permissions might behave unexpectedly.

New stages centered in the canvas

When you use the stage library panel to add a new stage to the pipeline canvas, the stage appears in the center of the current view of the canvas. Previously, it appeared to the right of the right-most stage in the pipeline.

• When a deployment contains multiple engine instances and you try to share the deployment with another user or group for a second time, Control Hub fails to update the permissions due to a 500 internal server error.
• A single faulty subscription prevents other suitable subscriptions from triggering.
• You cannot access a draft run of a pipeline when you view the list of running pipelines on an engine from the Engines view.

• You cannot create new users using the Control Hub REST API.

• You cannot directly log in to the SAML configuration page as an Organization Administrator if you belong to another organization with SAML authentication disabled.

Install stage libraries while creating connections

When using an authoring Data Collector 4.4.x or later, you can select from all possible connection types while creating a connection, even if the corresponding stage library is not installed on the authoring Data Collector. If you select an uninstalled connection type, you can choose to update the deployment with the missing stage library that includes the connection type.

​​SDC_ID subscription parameter displays as Engine ID in the user interface

When you create a subscription and define a simple condition for an execution engine not responding event, the SDC_ID parameter now displays as Engine ID in the UI, instead of SDC_ID. The Engine ID label indicates that the parameter can include the ID of any engine type.

Continue to use SDC_ID as the parameter name when you use the parameter to represent StreamSets engines.

Scheduled tasks

While creating a scheduled task, you can search for the job to schedule.

• Exporting multiple pipelines can generate a corrupt ZIP file.
• You cannot download a snapshot captured for a pipeline test run or job run.
• When you edit a subscription condition in the UI, the changes are not saved.
• The job History tab inconsistently displays Data Collector URLs.

Supported SAML identity providers

StreamSets supports PingFederate as a SAML identity provider.

• Control Hub allows you to test a SAML draft configuration when SP-initiated logins are disabled, even though testing SAML from Control Hub is only supported when SP-initiated logins are enabled. Testing when SP-initiated logins are disabled results in a 404 Not Found error.

Create fragments using pipeline stages

When you create a fragment from selected stages in a pipeline, you can choose to immediately publish the fragment as long as the fragment meets the validation requirements to be published. Or, you can choose to create a draft fragment and then continue designing the pipeline.

If you immediately publish the fragment, you can then replace the individual stages in the pipeline with the newly published fragment.

GCE deployments support selecting a subnet

GCP environments support VPC networks that have an auto or custom subnet creation mode. As a result, while creating a GCE deployment, you must select the subnet within the VPC network to provision the Compute Engine instances in.

Previously, GCP environments only supported auto mode VPC networks, where the VPC automatically creates one subnet from each region. You did not have to select a subnet while creating a GCE deployment because the VPC automatically selected a suitable subnet.

Contextual help sidebar

After selecting a view in the Navigation panel, click the Help icon () above the listed objects to open the contextual help sidebar. To close the contextual help, click outside of the sidebar. The contextual help sidebar has been implemented for most views. It will be implemented for additional views in future releases.

Use the contextual help to access common getting started, usage, troubleshooting, and best practice help topics and StreamSets Learning Academy videos. You can also search for topics in the StreamSets documentation.

The help links that display on each tab depend on the selected view. For example, the following image displays the getting started help topics for the Pipelines view:

Contextual help in environment, deployment, and pipeline wizards

When you create or edit an environment, deployment, or pipeline, the wizard displays contextual help on the right. The help contents depend on the step being completed in the wizard. Use the scroll bar to view the complete help contents for each step.

For example, the following image displays the contextual help provided in the first step of the pipeline wizard:

• Restarting multiple engines at the same time causes an error.
• When you select a later version of an authoring Data Collector while a pipeline is in read-only mode, stage libraries are not updated in the pipeline.
• The Global Failover Retries property does not display when creating or editing a Transformer job.
• GCP environments support only VPC networks that have automatic subnet creation enabled.
• GCP environments do not support using a shared VPC network from a host GCP project.

Encrypting SAML assertions

When you disable SP-initiated logins for SAML authentication, you can optionally configure the IdP to encrypt the SAML assertion. If you choose not to configure the IdP to encrypt SAML assertions, then you must disable the Require Encryption on Assertion property in the draft SAML configuration for your organization.

Previously, when you disabled SP-initiated logins, you had to configure the IdP to encrypt the SAML assertion.

• If you belong to multiple organizations that have SAML authentication enabled and you sign in to StreamSets using SSO SAML, you cannot choose which organization to log into.

Create connections in the pipeline canvas

While building a pipeline, you can create a new connection for a stage without leaving the pipeline canvas.

Connection details

The Connection details page has been enhanced to make the available actions more visible and to more clearly display details about all pipeline and fragment versions using the connection.

Import existing pipelines or fragments as new

When you import a pipeline or fragment that already exists in the target organization, you can choose to import the object as a new pipeline or fragment.

Contextual help in the pipeline canvas

While building a pipeline, click the Help tab in the pipeline properties panel to access common getting started, pipeline usage, troubleshooting, and best practice help topics and StreamSets Learning Academy videos. You can also search for topics in the StreamSets documentation.

The help links that display on each tab depend on the pipeline type and the stages added to the pipeline. For example, the following image displays the getting started help topics for a Data Collector pipeline:

• When viewing Transformer engines in the Engines view, clicking the More icon and then clicking Transformer Components displays the Getting Started page.
• Pipeline export fails when a pipeline contains multiple stages of the same type that use different library versions.
• You cannot download an engine support bundle.

Transformer version for new deployments

Starting with Transformer version 4.2.0 released on January 21, 2022, you can create a new deployment only for Transformer 4.2.0 or later. Earlier Transformer versions are not supported in new deployments.

Existing deployments can continue to use the earlier Transformer versions.

Duplicate connections

You can duplicate a connection to create a copy of an existing connection. You can then change the configuration of the copy.

Share pipeline or fragment while publishing

While using the Check In wizard to publish a pipeline or publish a fragment, you can share the pipeline or fragment with other users and groups.

Data Collector version for new deployments

Starting with Data Collector version 4.3.0 released on January 13, 2022, you can create a new deployment only for Data Collector 4.3.0 or later. Earlier Data Collector versions are not supported in new deployments.

Existing deployments can continue to use the earlier Data Collector versions.

• A running job becomes corrupted if you import another version of the job that uses a newer pipeline version.
• A subscription for a job status change event with a JOB_OWNER condition set to a specific email address fails to trigger because the JOB_OWNER parameter is always null.

Job templates

Job templates include the following enhancements:

• Job templates display on the Job Templates view, separate from the list of job instances that display on the Job Instances view.
• When you create job instances from a job template, the instances are attached to the parent job template by default. Attached job instances display in the parent job template’s run history and are updated when the parent job template is updated.

  You can optionally detach job instances from the parent job template when you want to use the job details and default parameter values defined in the template, but don't want subsequent changes to the template to be applied to the job instance.

• When you create a job template for a pipeline that uses runtime parameters, you define whether each parameter functions as a dynamic parameter that can be overridden in child job instances or as a static parameter that cannot be overridden.
• You can archive a job template when you do not want new job instances to be created from the template, but want existing job instances to continue to run.
• Job instances created from a job template inherit all tags added to the template.

Stage library selection for deployments

When you define the engine configuration for a deployment, you can select individual stages to install or uninstall, in addition to selecting stage libraries.

When you select individual stages, you can:

• Filter the list of available stages by type or search for stages by name.
• Select the stage library version to install, not just the latest version.
• View the complete list of stages included in each stage library.

In addition, the stage library selection window includes a summary view of the currently selected stage libraries by file name. For example, the Basic stage library is listed as streamsets-datacollector-basic-lib.

Engine installation script for self-managed deployments

When using a self-managed deployment, you can configure the engine installation script to run the engine as a foreground or background process.

Previously, a tarball installation script always ran the engine as a foreground process. A Docker image installation script always ran the engine as a background process.

Authoring engine selection for pipelines, fragments, and connections

When you create pipelines, fragments, or connections, the authoring engine selection window displays the current CPU and memory usage of each engine.

The resource usage can help you troubleshoot issues with inaccessible engines. For example, when an engine uses an excessive amount of available resources, the machine running the engine might lose connection to Control Hub or the browser.

Sticky notes for pipeline canvas

To delete a sticky note from the pipeline canvas, you click the Delete icon () instead of the X in the note header.

• When legacy Kubernetes is enabled for your organization and you have created legacy deployments only, you receive the following message when trying to create a pipeline:

  Attention: You need to set up a deployment with an engine before you can create a pipeline

• If you have an existing Data Collector deployment with an installed enterprise stage library, and you edit the deployment to upgrade to a new version of the same enterprise stage library, you cannot remove the existing version of the stage library. As a result, Data Collector fails to start with the following error:

  REST_1001 - Unable to find following stage libraries in repository list: streamsets-datacollector-<enterprise library type>-lib:<new version>, streamsets-datacollector-<enterprise library type>-lib:<existing version>

• If existing Control Hub users have an Invited or Expired status and then you enable and disable SAML authentication, disabling SAML fails with the following error:

  javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement

Supported SAML identity providers

StreamSets supports Microsoft Azure Active Directory (Azure AD) as a SAML identity provider.

SAML authentication

StreamSets DataOps Platform supports single sign-on (SSO) authentication with SAML 2.0 with selected identity providers (IdPs).

Enabling SAML authentication requires registering StreamSets as a service provider in one of the supported IdPs and configuring SAML authentication for your Control Hub organization. Once enabled, all users in the organization must log in with SAML authentication.

At this time, StreamSets supports the following SAML identity providers:

• Microsoft Active Directory Federation Services (AD FS)
• Okta

Legacy Kubernetes integration

Control Hub provides a legacy Kubernetes integration that you can use to automatically provision Data Collectors on Kubernetes. Provisioning includes deploying, registering, starting, scaling, and stopping Data Collector Docker containers in a Kubernetes cluster. Legacy Kubernetes integration requires that the Provisioning Agent use the Control Agent Docker image version 4.0.0 or later.

Legacy Kubernetes integration is enabled only for some paid accounts.

• When viewing real-time statistics and metrics for an active Transformer job, the names of stages display incorrectly.
• Jobs can transition to an inactive_error status with the following error because multiple Job Runner applications attempt to process the same job:

  JOBRUNNER_69 At least one of the execution engines <engine ID> didn’t respond to the stop command

Microsoft Azure integration

Control Hub provides an integration with your Microsoft Azure account. When you use Azure environments and Azure VM deployments, Control Hub automatically provisions Azure VM instances needed to run StreamSets engines in your Azure account, and then deploys engine instances to those VM instances.

JVM memory strategy for deployed engines

By default, engines are now configured to use 50 percent of the available memory on the host machine as the Java heap size. Previously, engines were configured to use an absolute value, 1024 MB by default.

You can edit the Java heap size percentage or set an absolute value by modifying the engine advanced configuration properties for a deployment. However, in most cases, the default percentage value is sufficient.

Quick Start

When you create self-managed deployments from the Quick Start menu, Control Hub assigns the deployments a quick-start tag and names the deployments as follows, based on the selected engine type:

• Docker Data Collector <number> (Quick Start)
• Docker Transformer <number> (Quick Start)

For example, if you create two deployments of Data Collector from the Quick Start menu, Control Hub names the deployments Docker Data Collector 1 (Quick Start) and Docker Data Collector 2 (Quick Start).

Previously, all deployments created from the Quick Start menu were named local.

Job error acknowledgement

By default when a job encounters an inactive error status, users must acknowledge the error message before the job can be restarted. You can configure a job to skip job error acknowledgements. You might want to skip job error acknowledgement for scheduled jobs so the job can automatically be restarted without requiring user intervention. However, be aware that skipping job error acknowledgement might hide errors that the job has encountered.

Upgrade to a paid subscription

If you have a Free account, you can upgrade to a paid subscription from the My Account menu.

• The pipeline canvas incorrectly allows adding a fragment to another fragment although nested fragments are not supported.
• After reloading a pipeline, parameters called from properties that display as checkboxes are not displayed.
• Control Hub does not allow the at sign (@) in the property name for any engine advanced configuration properties.

• Pipeline fragment creation is failing.

In-application chat

You can use the in-application chat to start a conversation with the StreamSets team. You can enable or disable the chat in your account settings accessible from the My Account window.

Sticky notes for pipeline canvas

You can add sticky notes to the pipeline canvas to include notes that might be useful to you or your team as you build a pipeline or pipeline fragment. For example, you might add a note as a reminder to revisit a portion of the pipeline design or to change a stage after running initial tests of the pipeline.

Updated usage limits

The usage limits for organizations with a Free account have been updated as follows:

• 2 users - For an existing Free account with more than 2 users, all users can continue to log in as a member of the organization. However, you cannot invite additional users to the organization.
• 10 published pipelines - For an existing Free account with more than 10 published pipelines, you can continue to use all published pipelines.

The usage limit of 2 active jobs remains unchanged.

My Account

All properties previously available in the User Settings window have been moved to the My Account window. As a result, you can manage all your account settings from a single window. You can also modify your display name in the My Account window.

Confirmation dialog boxes

The OK button in several confirmation dialog boxes has been renamed to reflect the action that you are taking. For example, when deleting a connection, the OK button has been renamed to Delete.

Roles and permissions required to delete an engine

A user must have the following roles and permissions to delete an engine:

Task	Roles	Permissions
Delete an engine.	Engine Administrator Deployment Manager	Write on engine

Previously, a user required the following roles and permissions to delete an engine:

Task	Roles	Permissions
Delete an engine.	Engine Administrator Organization Administrator	Execute on engine

This change does not affect role and permission assignments for existing users. A user can still delete an engine with the previous role and permission requirements.

For the complete list of required roles and permissions to complete common engine tasks, see Engine Tasks.

Deployment and engine permissions

With this release, engines always inherit the same permissions assigned to the deployment. Those inherited permissions cannot be modified at the engine level. For example, if you grant a user Execute permission on a deployment, that user also has Execute permission on all engines managed by that deployment.

Previously, you could modify permissions at the engine level. As a result, users could have different permissions on engines managed by the same deployment.

If you previously modified engine permissions, those changes are now overridden with the deployment permissions. Check that users' access levels to engines have not unexpectedly increased or decreased, and modify the deployment permissions as needed.

Engines

• Shut down engines - You can shut down engines belonging to a self-managed deployment from the Engines view. You can also restart or shut down multiple engines at the same time.
• Resource thresholds - The default values of the Max CPU Load and Max Memory thresholds for an engine are now 80%. Previously, the default values were 100%.

Connections

When creating or editing a connection, you can filter the list of available authoring Data Collectors by deployment, version, or label. The Data Collector with the most recent reported time is selected by default.

Pipelines

• Pipeline and fragment creation - While creating a new pipeline or pipeline fragment, you can choose to open the pipeline or fragment in the canvas before completing the Share Pipeline or Share Pipeline Fragment step.
• Sample pipelines - When you open a sample pipeline in the pipeline canvas, the Duplicate button has been renamed to Create a pipeline from sample.

Free and paid accounts

DataOps Platform now offers paid accounts in addition to free accounts.

With a free account, your use is subject to a number of limits. For legal related limits - including those related to warranty, support, and SLA - please see Section 17 of the Terms of Use. For usage related limits, such as organization limits, see Organization Default System Limits. A free account is not limited to internal evaluation purposes, but may be terminated by either party at any time.

With a paid account, you can increase the organization limits. For more information about paid accounts, contact your StreamSets account team or send an email to subscriptions@streamsets.com.

Users

• Session timeout - After 30 minutes of inactivity, user sessions expire and users are logged out. An organization administrator can change the inactivity period for the organization.
• Deleting users - When you attempt to delete an active user, Control Hub prompts you to deactivate the user during the deletion process.

Data Collector version for new deployments

With this release, you can create a new deployment only for Data Collector version 4.0.2 or later. Earlier Data Collector versions are not supported in new deployments.

Existing deployments can continue to use the earlier Data Collector versions.

• If you stop multiple deployments at the same time, Control Hub might fail to completely stop the deployments with the following error:

  An AppException occurred deactivating deployment tokens after stack was removed:
  Issues: [CSP_046 - Rest Api '<Control Hub URL>/security/rest/v1/organization/<organization ID>/components/delete' failed with status:'500']

• No errors display in the UI when you restart Docker engine instances for a self-managed deployment with an external resource archive that uses an incorrect file format.

Quick Start

To get started with StreamSets, click Quick Start in the top toolbar to quickly deploy a Data Collector or Transformer engine using Docker. After the engine is launched and running, click Quick Start > Create a pipeline to quickly create a pipeline.

Environments

• Environment state - The environment state and status have been combined into a single state to simplify the state concept. For example, an environment with an Enabled state and an OK status now has a single Active state.
• Activate and deactivate actions - The Enable and Disable actions for an environment have been renamed to Activate and Deactivate to align the actions with the new environment states.

Deployments

• Deployment state - The deployment state and status have been combined into a single state to simplify the state concept. For example, a deployment with a Disabled state and an OK status now has a single Deactivated state.
• Start and stop actions - The Enable action for a deployment has been renamed to Start to more clearly indicate that after you start a deployment, the deployment is ready to launch engine instances. The Disable action has been renamed to Stop to more clearly indicate that after you stop a deployment, you can no longer launch engine instances for the deployment.
• Deployment details - The Deployment details page has been enhanced to make the available actions more visible and to more clearly display details about the existing engines belonging to the deployment.
• Launched engine status for self-managed deployments - When copying the installation script to launch an engine instance for a self-managed deployment, you can choose to check the engine status in the Control Hub UI. Previously, you could view the engine status only in the command prompt.
• Engine installation type for self-managed deployments - After saving a self-managed deployment, you can change the engine installation type. For example, you can change a Tarball installation type to the Docker image installation type.

• The list of EC2 instance types available when creating an Amazon EC2 deployment does not take into account the availability zone used by the parent AWS environment. If you select an unsupported instance type, AWS CloudFormation cannot provision the EC2 instances and the following error message displays:

  Your requested instance type (<type>) is not supported in your requested Availability Zone (<zone>)

  Important: To fix this issue, StreamSets has updated the sample IAM policy used to configure AWS credentials for an AWS environment. If you encountered this issue with an existing AWS environment and EC2 deployment, then update the IAM policy to include the ec2:DescribeInstanceTypeOfferings permission. If you have not encountered this issue, there is no need to update the IAM policy.
• You can delete a pipeline included in a job when you do not have permission on the job, causing the job to become invalid.

Deployments

• Engine version - The version number of a nightly engine build includes the build number in addition to the -SNAPSHOT suffix.

• GCE deployments - When configuring the GCE autoscaling group for a deployment, you can enter the network tags to provision the VM instances with.

Jobs

• Failover retries - When you enable a Data Collector or Transformer job for pipeline failover, you can configure the global number of pipeline failover retries to attempt across all available engines. When the limit is reached, Control Hub stops the job.

• Balance jobs icon - The Engines view includes a new Balance Jobs icon: .

Subscriptions

You can configure a subscription action for a maximum global failover retries exhausted event. For example, you might create a subscription that sends an alert to a Slack channel when a job has exceeded the maximum number of pipeline failover retries across all available engines.