Release Notes

This release fixes the following issue:

• Jobs can become slow to start and stop.

This release includes several enhancements and fixed issues.

This release includes an enhancement and several behavior changes and fixed issues.

This release includes a behavior change and a fixed issue.

This release fixes the following issues:

• While monitoring a Transformer job, you cannot view and download the Spark driver log from the Control Hub UI.
• When you configure a pipeline to write aggregated statistics to Kafka using the SSL security protocol, the generated system pipeline incorrectly uses the plain text security protocol.
• If more than twenty authoring Data Collectors are available while creating a connection and you select an authoring Data Collector's radio button from the bottom of the list, the screen goes blank.

This release includes several new features and fixed issues.

This release includes new features and fixed issues.

This release includes the following new feature:

Control Hub REST API

When you use the Control Hub Job Runner REST API to return all jobs, you can now use additional parameters to search for jobs by name. You can search for jobs with names that equal the specified search text or search for jobs with names that start with the specified search text.

This release includes new features and fixed issues.

This release includes new features and fixed issues.

This release includes new features and fixed issues.

This release includes new features and fixed issues.

This release includes new features and fixed issues.

This release includes new features and fixed issues.

This release includes new features and fixed issues.

This release includes new features and fixed issues.

This release includes new features and fixed issues.

This release includes new features, deprecated features, and fixed issues.

This release includes new features and fixed issues.

This release fixes the following known issues:

• Pipelines with multiple event-creating destinations do not render the destinations correctly.
• Stopping and restarting jobs on the same instance of Data Collector causes incorrect counts of running pipelines.
• Data preview does not work for published pipelines.
• Data preview does not wrap long values.
• Changes to pipelines cannot be published.

This release fixes the following known issue:

• The statuses of Data Collector pipelines are not shown correctly.

This release includes the following new feature:

Uploading Offsets with the REST API

You can now upload a valid offset with the Control Hub REST API, which uploads the offset as JSON data.

This release fixes the following known issues:

• Uploading an initial offset file fails when an offset is already set for the job.
• When you define a second table configuration for the JDBC Multitable Consumer origin, properties in the first table configuration are overridden.
• When you start a job that contains a Transformer pipeline and the pipeline encounters a START_ERROR state, the job remains in an active state.

This release fixes the following known issue:

• Cannot preview or perform a test run of a draft pipeline when using an authoring Data Collector version earlier than 3.10.0.

This release includes the following new features and enhancements:

Transformer Integration

This release integrates StreamSets Transformer in Control Hub.

Just as Data Collector pipelines run on a Data Collector engine, Transformer pipelines run on a Transformer engine. Since the Transformer engine is built on Apache Spark, an open-source cluster-computing framework, Transformer pipelines can perform heavy processing on the entire data set in batch or streaming mode.

To use Transformer with Control Hub, install Transformer on a machine that is configured to submit Spark jobs to a cluster, such as a Hadoop edge or data node or a cloud virtual machine. Then register Transformer with Control Hub. Use Pipeline Designer to design Transformer pipelines and configure a job to run the pipeline, just as you would a Data Collector pipeline.

For comparison of Transformer with Data Collector, see Transformer for Data Collector Users.

Pipeline Design

Pipeline Designer includes the following enhancements:

• Delete a draft pipeline or fragment in Pipeline Designer - While editing a draft version of a pipeline or fragment, you can now delete that draft version to revert to the previous published version of the pipeline or fragment. Previously, you could not delete a draft pipeline or fragment that was open in Pipeline Designer. You had to view the pipeline history, and then select the draft version to delete.
• View the input and output schema for each stage - After running preview for a pipeline, you can now view the input and output schema for each stage on the Schema tab in the pipeline properties panel. The schema includes each field name and data type.

  Use the Schema tab when you configure pipeline stages that require field names. For example, let’s say you are configuring a Field Type Converter processor to convert the data type of a field by name. You can run preview, copy the field name from the Schema tab, and then paste the field name into the processor configuration.

• Bulk update pipelines to use a different fragment version - When viewing a published pipeline fragment in Pipeline Designer, you can now update multiple pipelines at once to use a different version of that fragment. For example, if you edit a fragment and then publish a new version of the fragment, you can easily update all pipelines using that fragment to use the latest version.
• Import a new version of a published pipeline in Pipeline Designer - While viewing a published pipeline in Pipeline Designer, you can import a new version of the pipeline. You can import any pipeline exported from Data Collector for use in Control Hub or any pipeline exported from Control Hub as a new version of the current pipeline.
• User-defined sample pipelines - You can now create a user-defined sample pipeline by assigning the templates pipeline label to a published pipeline. Users with read permission on the published pipeline can select the pipeline as a user-defined sample when developing a new pipeline.
• Test run of a draft pipeline - You can now perform a test run of a draft pipeline in Pipeline Designer. Perform a test run of a draft pipeline to quickly test the pipeline logic. You cannot perform a test run of a published pipeline. To run a published pipeline, you must first add the pipeline to a job and then start the job.
• Shortcut keys to undo and redo actions - You can now use the following shortcut keys to easily undo and redo actions in Pipeline Designer:
  • Press Command+Z to undo an action.
  • Press Command+Shift+Z to redo an action.

Jobs

Jobs include the following enhancements:

• Monitoring errors - When you monitor an active Data Collector job with a pipeline stage that encounters errors, you can now view details about each error record on the Errors tab in the Monitor panel.
• Export and import job templates - When you export and import a job template, the template is now imported as a job template. You can then create job instances from that template in the new organization. You cannot export and import a job instance. Previously, when you exported and imported a job template or a job instance, the imported job template or instance functioned as a regular job in the new organization.

Subscriptions

You can now configure a subscription action for a ​changed job status color. For example, you might create a subscription that sends an email when a job status changes from active green to active red.

Roles

All Data Collector roles have been renamed to Engine roles and now enable performing tasks in registered Data Collectors and registered Transformers.

For example, the Data Collector Administrator role has been renamed to the Engine Administrator role. The Engine Administrator role now allows users to perform all tasks in registered Data Collectors and registered Transformers.

Provisioned Data Collectors

Provisioned Data Collectors include the following enhancements:

• Upload the deployment YAML specification file - When you create a deployment, you can now upload the deployment YAML specification file instead of copying the contents of the file in the YAML Specification property.
• View YAML specification file for active deployments - You can now view the contents of the YAML specification file when you view the details of an active deployment.
• Configurable Kerberos principal user name - When you define a deployment YAML specification file to provision Data Collector containers enabled for Kerberos authentication, you can now optionally define the Kerberos principal user name to use for the deployment. If you do not define a Kerberos user name, the Provisioning Agent uses sdc as the user name.

This release includes the following new features and enhancements:

Pipeline Design

Pipeline Designer includes the following enhancements:

• Preview time zone - You can now select the time zone to use for the preview of date, datetime, and time data. Previously, preview always displayed data using the browser time zone.
• Compare pipeline versions - When you compare pipeline versions, you can now click the name of either pipeline version to open that version in the pipeline canvas.

  Previously, you had to return to the Navigation panel and then select the pipeline version from the Pipeline Repository to open one of the versions in the pipeline canvas.

Jobs

You can now upload an initial offset file for a job. Upload an initial offset file when you first run a pipeline in Data Collector, publish the pipeline to Control Hub, and then want to continue running the pipeline from the Control Hub job using the last-saved offset maintained by Data Collector.

SAML Authentication

When SAML authentication is enabled, users with the new Control Hub Authentication role can complete the following tasks that require users to be authenticated by Control Hub:

• Use the Data Collector command line interface.
• Log into a Data Collector running in disconnected mode.
• Use the Control Hub REST API.

Previously, only users with the Organization Administrator role could complete these tasks when SAML authentication was enabled.

Provisioned Data Collectors

The Control Agent Docker image version 3.10.0 now requires that each YAML specification file that defines a deployment use the Kubernetes API version apps/v1. Previously, the Control Agent Docker image required that each YAML specification file use the API version extensions/v1beta1 for a deployment. Kubernetes has deprecated the extensions/v1beta1 version for deployments.

If you upgrade a Provisioning Agent to use the Control Agent Docker image version 3.10.0 or later or if a Provisioning Agent uses latest as the Control Agent Docker image version, you must update all deployment YAML specification files before you redeploy the Provisioning Agent. For more information, see Update Deployments for Provisioned Data Collectors.

This release also fixes the following known issues:

• Jobs might incorrectly display metrics when you create a job for a pipeline that was downloaded from Control Hub into a registered Data Collector, and then exported from Data Collector to Control Hub.
• Job history does not track the inactive status.
• Duplicating a pipeline that contains two or more pipeline fragments displays a validation error about unconnected stages.
• An email subscription for a Pipeline Committed event does not display the pipeline commit message when the pipeline is published from Pipeline Designer.
• Importing jobs does not work reliably because Control Hub displays the rules associated with the job instead of with the pipeline.
• Cannot view logs when monitoring an active job.
• When you create a pipeline with the SFTP/FTP Client origin, the Pipeline Designer displays a validation error.
• Upgrading a job to the latest pipeline version does not upgrade the pipeline rules.
• Creating a new user requires that you directly assign the Organization Administrator or Organization User role to the user even if that user inherits those roles from an assigned group.
• Pipeline is duplicated in the Pipelines view when the pipeline is shared with a user and with a group that the user belongs to.
• Control Hub encounters a data truncation error when saving Data Collector metrics.
• During the maintenance window for a Control Hub Cloud release, all traffic is redirected to https://trust.streamsets.com, which causes Data Collectors to be inaccessible.
• When SAML authentication is enabled for an organization, all new users are required to update their password.
• Control Hub incorrectly checks for password expiration when SAML authentication is enabled.

This release includes the following new features and enhancements:

Provisioned Data Collectors

You can now create a Provisioning Agent that provisions Data Collector containers enabled for Kerberos authentication.

StreamSets recommends using Helm to create a Provisioning Agent that can provision Data Collectors enabled for Kerberos. Helm is a tool that streamlines installing and managing Kubernetes applications.

Jobs

You can now upgrade active jobs to use the latest pipeline version. When you upgrade an active job, Control Hub stops the job, updates the job to use the latest pipeline version, and then restarts the job.

You can manually upgrade jobs or you can schedule the upgrade of jobs on a regular basis. For example, you might use the Control Hub scheduler to create a scheduled task that runs every Saturday at 12:00 am to check if an active job has a later pipeline version. If a later pipeline version exists, the scheduled task stops the job, updates the job to use the latest pipeline version, and then restarts the job.

Pipeline Design

When you use the Control Hub Pipeline Designer to design pipelines, you can now call pipeline parameters for properties that display as checkboxes and drop-down menus. The parameters must evaluate to a valid option for the property.

Permissions

You can now share and grant permissions on multiple objects at the same time.

Subscriptions

When you create a subscription, you now configure the subscription in a single dialog box instead of clicking through multiple pages.

This release includes the following new features and enhancements:

Preview in Pipeline Designer

Pipeline Designer can now display preview data in table view.

Subscriptions

Subscriptions include the following enhancements:

• Pipeline status change event - You can now configure a subscription action for a ​changed pipeline status. For example, you might create a subscription that sends an email when a pipeline status changes to RUN_ERROR.
• Expression completion to filter events - You can now use expression completion to determine the functions and parameters that you can use for each subscription filter.

Scheduler

The Control Hub scheduler can now stop a job at a specified frequency. For example, you might want to run a streaming job every day of the week except for Sunday. You create one scheduled task that starts the job every Monday at 12:00 am. Then, you create another scheduled task that stops the same job every Sunday at 12:00 am.

SAML Authentication

When you map a Control Hub user account to a SAML IdP user account, the SAML User Name property now defaults to the email address associated with the Control Hub user account. Previously, the default value was the user ID associated with the Control Hub user account.

This release includes the following new features and enhancements:

Data Protector

This release supports the latest version of Data Protector, Data Protector 1.4.0.

Preview in Pipeline Designer

You can now preview multiple stages in Pipeline Designer. When you preview multiple stages, you select the first stage and the last stage in the group. The Preview panel then displays the output data of the first stage in the group and the input data of the last stage in the group.

Job Templates

When you create a job for a pipeline that uses runtime parameters, you can now enable the job to work as a job template. A job template lets you run multiple job instances with different runtime parameter values from a single job definition.

For example, you create a pipeline that uses a runtime parameter to read from different Google Cloud Storage buckets. You create a single job template for that pipeline, and then start multiple instances of the job, specifying a unique value for the bucket parameter for each job instance.

Subscribe to Unresponsive Data Collector or Data Collector Edge Events

You can now configure a subscription action for a Data Collector or Data Collector Edge not responding event. For example, you might create a subscription that sends an alert to a Slack channel when a registered Data Collector stops responding.

This release includes the following new features and enhancements:

Failover Retries for Jobs

When a job is enabled for failover, Control Hub by default retries the pipeline failover an infinite number of times. If you want the pipeline failover to stop after a given number of retries, you can now define the maximum number of retries to perform. Control Hub maintains the failover retry count for each available Data Collector.

Starting Jobs with the REST API

You can now define runtime parameter values for a job when you start the job using the Control Hub REST API.

This release includes the following new features and enhancements:

StreamSets Data Protector

You can now use StreamSets Data Protector to perform global in-stream discovery and protection of data in motion with Control Hub.

Data Protector provides StreamSets classification rules and enables creating custom classification rules to identify sensitive data. Custom protection policies provide rules-based data protection for every job that you run. You can also use Data Protector stages in pipelines for localized protection needs.

Data Protector is available as an add-on option with a StreamSets Enterprise subscription. For more information, contact us.

Pipeline Designer

Pipeline Designer includes the following enhancements:

• Expression completion - Pipeline Designer now completes expressions in stage and pipeline properties to provide a list of data types, runtime parameters, fields, and functions that you can use.
• Manage pipeline and fragment versions - When configuring a pipeline or pipeline fragment in Pipeline Designer, you can now view the following visualization of the pipeline or fragment version history:

  

  When you expand the version history, you can manage the pipeline or fragment versions including comparing versions, creating tags for versions, and deleting versions.

• Pipeline fragment expansion in pipelines - You can now expand and collapse individual pipeline fragments when used in a pipeline. Previously, expanding a fragment meant that all fragments in the pipeline were expanded.

  When a fragment is expanded, the pipeline enters read-only mode allowing no changes. Collapse all fragments to make changes to the pipeline.

• Preview and validate edge pipelines - You can now use Pipeline Designer to preview and validate edge pipelines.
• Shortcut menu for stages - When you select a stage in the canvas, a shortcut menu now displays with a set of options:
  • For a pipeline fragment stage, you can copy, expand or delete the fragment.
  • For all other stages, you can copy or delete the stage, or create a pipeline fragment using the selected stage or set of stages.

Data Collectors

You can now use an automation tool such as Ansible, Chef, or Puppet to automate the registering and unregistering of Data Collectors using the following commands:

streamsets sch register
streamsets sch unregister

This release includes the following new features and enhancements:

Pipelines and Pipeline Fragments

• Data preview enhancements:
  • Data preview support for pipeline fragments - You can now use data preview with pipeline fragments. When using Data Collector 3.4.0 for the authoring Data Collector, you can also use a test origin to provide data for the preview. This can be especially useful when the fragment does not contain an origin.
  • Edit data and stage properties - You can now edit preview data and stage properties, then run the preview with your changes. You can also revert data changes and refresh the preview to view additional data.
• Select multiple stages - When you design pipelines and pipeline fragments, you can now select multiple stages in the canvas by selecting the Shift key and clicking each stage. You can then move or delete the selected stages.
• Export enhancement - When you export a single pipeline or a single fragment, the pipeline or fragment is now saved in a zip file of the same name, as follows: <pipeline or fragment name>.zip. Exporting multiple pipelines or fragments still results in the following file name: <pipelines|fragments>.zip.
• View where fragments are used - When you view the details of a fragment, Pipeline Designer now displays the list of pipelines that use the fragment.

Jobs

• Runtime parameters enhancements - When you edit a job, you can now use the Get Default Parameters option to retrieve all parameters and their default values as defined in the pipeline. You can also use simple edit mode, in addition to bulk edit mode, to define parameter values.
• Pipeline failover enhancement - When determining which available Data Collector restarts a failed pipeline, Control Hub now prioritizes Data Collectors that have not previously failed the pipeline.

Data Collectors

• Monitor Data Collector performance - When you view registered Data Collectors version 3.4.0 from the Execute view, you can now monitor the CPU load and memory usage of each Data Collector.

Edge Data Collectors (SDC Edge)

• Monitor SDC Edge performance - When you view registered Edge Data Collectors version 3.4.0 from the Execute view, you can now monitor the CPU load and memory usage of each SDC Edge.

Data Delivery Reports

• Destination statistics - Data delivery reports for jobs and topologies now contain statistics for destinations.

Documentation

• Documentation enhancement - The online help has a new look and feel. All of the previous documentation remains exactly where you expect it, but it is now easier to view and navigate on smaller devices like your tablet or mobile phone.

This release fixes the following known issues:

• Viewing pipeline details from the Topology view causes an error to occur.
• Time series charts for jobs cannot be viewed from the Topology view even though time series analysis is enabled.
• When a Kubernetes pod is restarted, the Provisioning Agent fails to register the Data Collector containers with Control Hub.

This release includes the following new features and enhancements:

Pipeline Fragments

Control Hub now includes pipeline fragments. A pipeline fragment is a stage or set of connected stages that you can reuse in Data Collector or SDC Edge pipelines. Use pipeline fragments to easily add the same processing logic to multiple pipelines and to ensure that the logic is used as designed.

Pipeline fragments can only be created in the Control Hub Pipeline Designer. You can use any stage available in the authoring Data Collector in a fragment. Pipeline fragments cannot be designed within the Data Collector user interface.

Scheduler

Control Hub now includes a scheduler that manages long-running scheduled tasks. A scheduled task periodically triggers the execution of a job or a data delivery report at the specified frequency. For example, a scheduled task can start a job or generate a data delivery report on a weekly or monthly basis.

Before you can schedule jobs and data delivery reports, the Scheduler Operator role must be assigned to your user account.

Data Delivery Reports

Control Hub now includes data delivery reports that show how much data was processed by a job or topology over a given period of time. You can create periodic reports with the scheduler, or create an on-demand report.

Before you can manage data delivery reports, the Reporting Operator role must be assigned to your user account.

Jobs

• Edit a pipeline version directly from a job - When viewing the details of a job or monitoring a job, you can now edit the latest version of the pipeline directly from the job. Previously, you had to locate the pipeline in the Pipeline Repository view before you could edit the pipeline.
• Enable time series analysis - You can now enable time series analysis for a job. When enabled, you can view historical time series data when you monitor the job or a topology that includes the job.

  When time series analysis is disabled, you can still view the total record count and throughput for a job or topology, but you cannot view the data over a period of time. For example, you can’t view the record count for the last five minutes or for the last hour.

  By default, all existing jobs have time series analysis enabled. All new jobs have time series analysis disabled. You might want to enable time series analysis for new jobs for debugging purposes or to analyze dataflow performance.

• Pipeline force stop timeout - In some situations when you stop a job, a remote pipeline instance can remain in a Stopping state for a long time. When you configure a job, you can now configure the number of milliseconds to wait before forcing remote pipeline instances to stop. The default time to force a pipeline to stop is 2 minutes.
• View logs- While monitoring an active job, the top toolbar now includes a View Logs icon that displays the logs for any remote pipeline instance run from the job.

Subscriptions

• Email action - You can now create a subscription that listens for Control Hub events and then sends an email when those events occur. For example, you might send an email each time a job status changes.
• Pipeline committed event - You can configure an action for a pipeline committed event. For example, you might send a message when a pipeline is committed with the name of the user who committed it.
• Filter the events to subscribe to - You can now use the StreamSets expression language to create an expression that filters the events that you want to subscribe to. You can include subscription parameters and StreamSets string functions in the expression.
  For example, you might enter the following expression for a Job Status Change event so that the subscription is triggered only when the specified job ID encounters a status change:

  ${JOB_ID == '99efe399-7fb5-4383-9e27-e4c56b53db31:MyCompany'}

  If you do not filter the events, then the subscription is triggered each time an event occurs for all objects that you have at least read permission on.

• Permissions - When permission enforcement is enabled for your organization, you can now share and grant permissions on subscriptions.

Provisioned Data Collectors

When you define a deployment YAML specification file for provisioned Data Collectors, you can now optionally associate a Kubernetes Horizontal Pod Autoscaler, service, or Ingress with the deployment.

Define a deployment and Horizontal Pod Autoscaler in the specification file for a deployment of one or more execution Data Collectors that must automatically scale during times of peak performance. The Kubernetes Horizontal Pod Autoscaler automatically scales the deployment based on CPU utilization.

Define a deployment and service in the specification file for a deployment of a single development Data Collector that must be exposed outside the cluster using a Kubernetes service. Optionally associate an Ingress with the service to provide load balancing, SSL termination, and virtual hosting to the service in the Kubernetes cluster.

This release includes the following new features and enhancements:

Pipelines

Pipelines include the following enhancements:

• Duplicate pipelines - You can now select a pipeline in the Pipeline Repository view and then duplicate the pipeline. A duplicate is an exact copy of the original pipeline.
• Commit message when publishing pipelines - You can now enter commit messages when you publish pipelines from Pipeline Designer. Previously, you could only enter commit messages when you published pipelines from a registered Data Collector.

Export and Import

You can now use Control Hub to export and import the following objects:

• Jobs and topologies - You can now export and import jobs and topologies to migrate the objects from one organization to another. You can export a single job or topology or you can export a set of jobs and topologies.

  When you export and import jobs and topologies, you also export and import dependent objects. For jobs, you also export and import the pipelines included in the jobs. For topologies, you also export and import the jobs and pipelines included in the topologies.

• Sets of pipelines - You can now select multiple pipelines in the Pipeline Repository view and export the pipelines as a set to a ZIP file. You can also now import pipelines from a ZIP file containing multiple pipeline files.

Alerts

The Notifications view has now been renamed the Alerts view.

Subscriptions

You can now create a subscription that listens for Control Hub events and then completes an action when those events occur. For example, you might create a subscription that sends a message to a Slack channel each time a job status changes.

When you create a subscription, you select the Control Hub events to subscribe to - such as a changed job status or a triggered data SLA. You then configure the action to take when the events occur - such as using a webhook to send an HTTP request to an external system.

Important: By default, an organization is not enabled to send events that trigger subscriptions. Before Control Hub can trigger subscriptions for your organization, your organization administrator must enable events for the organization.

Jobs

• Scale out active jobs - When the Number of Instances property for a job is set to -1, Control Hub can now automatically scale out pipeline processing for the active job.

  When Number of Instances is set to any other value, you must synchronize the active job to start additional pipeline instances on newly available Data Collectors or Edge Data Collectors.

  For example, if Number of Instances is set to -1 and three Data Collectors have all of the specified labels for the job, Control Hub runs three pipeline instances, one on each Data Collector. If you register another Data Collector with the same labels as the active job, Control Hub automatically starts a fourth pipeline instance on that newly available Data Collector.

  Previously, you had to synchronize all active jobs - regardless of the Number of Instances value - to start additional pipeline instances on a newly registered Data Collector.

• View logs for an active job - When monitoring an active job, you can now view the logs for a remote pipeline instance from the Data Collectors tab.

This release includes the following new features and enhancements:

Product Rename

With this release, we have created a new product called StreamSets Control Hub^TM that includes a number of new cloud-based dataflow design, deployment, and scale-up features. Since this release is now our core service for controlling dataflows, we have renamed the StreamSets cloud experience from "Dataflow Performance Manager (DPM)" to "StreamSets Control Hub”.

DPM now refers to the performance management functions that reside in the cloud such as live metrics and data SLAs. Customers who have purchased the StreamSets Enterprise Edition will gain access to all Control Hub functionality and continue to have access to all DPM functionality as before.

To understand the end-to-end StreamSets Data Operations Platform and how the products fit together, visit https://streamsets.com/products/.

Pipeline Designer

You can now create and design pipelines directly in the Control Hub Pipeline Designer after you select an authoring Data Collector for Pipeline Designer to use. You select one of the following types of Data Collectors to use as the authoring Data Collector:

• System Data Collector - Use to design pipelines only - cannot be used to preview or explicitly validate pipelines. The system Data Collector is provided with Control Hub for exploration and light development. Includes the latest version of all stage libraries available with the latest version of Data Collector.
• Registered Data Collector using the HTTPS protocol - Use to design, preview, and explicitly validate pipelines. Includes the stage libraries and custom stage libraries installed in the registered Data Collector.

When you create pipelines in Pipeline Designer, you can create a blank pipeline or you can create a pipeline from a sample. Use sample pipelines to quickly design pipelines for typical use cases.

Provisioning Data Collectors

You can now automatically provision Data Collectors on a Kubernetes container orchestration framework. Provisioning includes deploying, registering, starting, scaling, and stopping Data Collector Docker containers in the Kubernetes cluster.

Use provisioning to reduce the overhead of managing a large number of Data Collector instances. Instead, you can manage a central Kubernetes cluster used to run multiple Data Collector containers.

Integration with Data Collector Edge

Control Hub now works with Data Collector Edge (SDC Edge) to execute edge pipelines. SDC Edge is a lightweight agent without a UI that runs pipelines on edge devices with limited resources. Edge pipelines read data from the edge device or receive data from another pipeline and then act on that data to control the edge device.

You install SDC Edge on edge devices, then register each SDC Edge with Control Hub. You assign labels to each SDC Edge to determine which jobs are run on that SDC Edge.

You either design edge pipelines in the Control Hub Pipeline Designer or in a development Data Collector. After designing edge pipelines, you publish the pipelines to Control Hub and then add the pipelines to jobs that run on a registered SDC Edge.

Pipeline comparison

When you compare two pipeline versions, Control Hub now highlights the differences between the versions in the pipeline canvas. Previously, you had to visually compare the two versions to discover the differences between them.

Aggregated statistics

You can now configure a pipeline to write aggregated statistics to MapR Streams.

Balancing jobs

When a job is enabled for pipeline failover, you can now balance the job to redistribute the pipeline load across available Data Collectors that are running the fewest number of pipelines. For example, let’s say that a failed pipeline restarts on another Data Collector due to the original Data Collector shutting down. When the original Data Collector restarts, you can balance the job so that Control Hub redistributes the pipeline to the restarted Data Collector not currently running any pipelines.

Roles

You can now assign provisioning roles to user accounts, which enable users to view and work with Provisioning Agents and deployments to automatically provision Data Collectors.

You must assign the appropriate provisioning roles to users before they can access the Provisioning Agents and Deployments views in the Navigation panel.

Navigation panel

The Navigation panel now groups the Data Collectors view under an Execute menu, along with the new Edge Data Collectors, Provisioning Agents, and Deployments views:

Dashboards

The default dashboard now includes the number of users in your organization when your user account has the Organization Administrator role.

This release includes the following new features and enhancements:

Jobs

• Number of pipeline instances - The default value for the number of pipeline instances for a job is now 1. This runs one pipeline instance on an available Data Collector running the fewest number of pipelines.

  Previously, the default value for the number of pipeline instances was -1, which ran one pipeline instance on each available Data Collector. For example, if three Data Collectors had all of the specified labels for the job, by default DPM ran three pipeline instances, one on each Data Collector.

• Job history - When you monitor a job, the History tab now includes the following additional information:
  • All user actions completed on the job - such as when a user starts, stops, resets the offset, or acknowledges an error for the job.
  • The progress of all Data Collectors running remote a pipeline instance for the job - such as when each Data Collector starts and stops the remote pipeline instance.
• Inactive job status when pipelines finish - When all pipelines run from an active job reach a finished state, the job now transitions to an inactive status. Previously, the job remained in the active status.

Data Collectors

• Data Collector versions - The Data Collectors view now displays the version of each registered Data Collector. You can filter the list of registered Data Collectors by version.
• Registering Data Collectors from DPM - After you generate an authentication token to register a Data Collector from DPM, you can now simply click Copy Token to copy the token from the Authentication Tokens window. Previously, you had to select the entire token string, right-click, and then select Copy to copy the token.

Roles

You can now assign the Auth Token Administrator role to user accounts, which enables users to complete the following tasks:

• Register, unregister, and deactivate Data Collectors using DPM.
• Regenerate authentication tokens and delete unregistered authentication tokens.

Previously, only users assigned the Organization Administrator role could perform these tasks. Users assigned the Organization Administrator role can still perform these tasks.

This release includes the following new features and enhancements:

SAML authentication

If your company uses a Security Assertion Markup Language (SAML) identity provider (IdP), you can use the IdP to authenticate DPM users.

SAML provides single sign on for web applications. SAML single sign on transfers the user’s identity from one place (the IdP) to another (the service provider). DPM acts as the SAML service provider that works with the SAML IdP that you specify.

To use SAML authentication, you must register DPM as a service provider with the IdP of your choice. Then within DPM, you enable SAML authentication for your organization. You also must create a DPM user account for each user that needs to access DPM or a registered Data Collector. When you create the user accounts, you map each DPM user account to an IdP user account.

Send pipeline statistics directly to DPM

You can now use Data Collector to configure a pipeline to write statistics directly to DPM. Write statistics directly to DPM when you run a job for the pipeline on a single Data Collector.

When you run a job on multiple Data Collectors, a remote pipeline instance runs on each of the Data Collectors. To view aggregated statistics for the job within DPM, you must configure the pipeline to write the statistics to a Kafka cluster, Amazon Kinesis Streams, or SDC RPC.

Jobs

• Runtime parameters - You can now specify the values to use for runtime parameters when you create or edit a job that includes a pipeline with runtime parameters.

  You configure runtime parameters for a pipeline in Data Collector. Use runtime parameters to represent any stage or pipeline property with a value that must change for each pipeline run - such as batch sizes and timeouts, directories, or URI.

  After you publish the pipeline to DPM, you can change the parameter values for each job that runs the pipeline without having to edit the pipeline.

• Use latest pipeline version - DPM now notifies you when a job includes a pipeline that has a later version by displaying the New Pipeline Version icon () next to the job. When the job is inactive, you can simply click the icon to update the job to use the latest pipeline version.
• Filter jobs by label - You can now filter jobs by label in the Jobs view.
• Create jobs for multiple pipelines - You can now use the Pipeline Repository view to select multiple pipelines and then create jobs for each of the pipelines.
• Create multiple jobs for a single pipeline - In the Add Job window, you can now choose to create multiple jobs for the selected pipeline. For example, if you use runtime parameters, you can quickly create multiple jobs for the same pipeline, defining different values for the runtime parameters for each job.
• Add to a topology during job creation - You can now add a job to an existing topology when you create the job.
• Create a topology from the Jobs view - You can now select multiple jobs in the Jobs view and create a topology that includes those jobs.

Topologies

• Manage jobs from a topology - You can now perform the following actions for jobs from a topology:
  • Acknowledge errors for a job.
  • Force stop a job.
  • Start and stop all jobs.
• Auto discover connecting systems - DPM can now automatically discover connecting systems between jobs in a topology. DPM discovers possible connecting systems and then offers you suggestions of how you might want to connect the systems, which you can accept or reject.
• Display of topology details - Topology details now display on the right side of the canvas instead of on the bottom. Double-click the canvas or click the Open Detail Pane arrow to display the topology detail pane. You can close the detail pane to view the canvas only, or you can resize the detail pane.

  The following image shows the new display of topology details:

  

Notifications

When you click the Notifications icon () in the top toolbar, you can now view the following notifications:

• Triggered alerts - Displays all triggered alerts that have not been acknowledged.
• History of error messages - Displays recent error messages that briefly displayed in the UI.

This release includes the following new features and enhancements:

Groups

You can now create groups of users to more efficiently manage user accounts. You can assign roles and permissions to individual user account or to groups.

DPM provides a default all@<organization ID> group that includes every user in the organization.

Permissions

You can now can share and grant permissions on Data Collectors, pipelines, jobs, topologies, and data SLAs. Permissions determine the access level that users and groups have on objects belonging to the organization.

To create a multitenant environment within your organization, create groups of users and then share objects with the groups to grant different levels of access.

When you create an object within DPM, you become the owner of that object and have full access to the object. You can share the object with other groups or user accounts within your organization. When you share the object, you grant others permission to the object - granting read, write, or execute access to the object. Any user with the Organization Administrator role has full access to all objects in the organization, and can grant other users and groups permission to access each object.

To perform DPM tasks, you must have the appropriate object permissions as well as the role associated with the task. For example, if you have the Pipeline Editor role, you can delete pipeline versions from the repository only when granted write permission on the pipeline.

By default, permission enforcement is not enabled for existing organizations. You can still assign permissions. However, DPM does not enforce the permissions until you enable enforcement. To enable permission enforcement, click Administration > Organizations, and then click the Organization Configurations icon. Select the Enforce permissions during object access property.

Data SLAs for Topologies

You can now configure data SLAs (service level agreements) for topologies. Data SLAs trigger an alert when a specified threshold has been reached. You configure data SLAs on the jobs included in the topology. Data SLAs enable you to monitor incoming data to ensure that it meets business requirements for availability and accuracy.

For example, you can configure a data SLA for a topology to trigger an alert when the throughput rate on a job reaches a minimum value. When the alert triggers, DPM notifies you in the top toolbar and in the new Notifications view.

The tasks you can perform for data SLAs and notifications are determined by the following new roles:

• Data SLA Editor and Data SLA User
• Notification User

By default, these new roles are not assigned to existing users. A user with the Organization Administrator role must assign these roles to other users and groups.

Job Offsets

The job History view now displays the last-saved job offset sent by each Data Collector running a remote pipeline instance for the job.

Aggregated Statistics

You can now configure a pipeline to write aggregated statistics to SDC RPC. Write statistics to SDC RPC for development purposes only. For a production environment, use a Kafka cluster or Amazon Kinesis Streams to aggregate statistics.

Register Data Collectors with DPM

If Data Collector uses file-based authentication and if you register the Data Collector from the Data Collector UI, you can now create DPM user accounts and groups during the registration process.

Organization Configuration

You can now configure the following information for your organization:

• Maximum number of minutes that a user session can remain inactive before timing out.
• Maximum number of days that a user password is valid.

There are no known issues at this time.