Release Notes

• Stages that use Amazon Web Services - Stages that use AWS can connect to endpoints and regions and can assume a role to connect to AWS on all connections. You can configure a separate region and endpoint for the stage and the assumed role.
  The following stages can access AWS:

  • Amazon S3 origin, destination, and executor

  • Amazon SQS Consumer origin

  • Databricks destination

  • Databricks Query executor

  • Encrypt and Decrypt Fields processor

  • Kinesis Consumer origin

  • Kinesis Firehose destination

  • Kinesis Producer destination

  • Teradata destination

  This change has upgrade impact.

• Kafka stages - The Kafka Multitopic Consumer origin and Kafka Producer destination now support Apache Kafka 3.8.x.

• Kafka Producer destination - You can configure the destination to propagate Data Collector record headers to Kafka message headers by enabling the Propagate Headers property. In new pipelines, this property is enabled by default. In pipelines upgraded from an earlier version of Data Collector, this property is disabled so that there is no behavior change in upgraded pipelines.

• MongoDB Atlas CDC Client origin - You can configure the action the origin takes when it is unable to retrieve the full record for updates.

• Snowflake and Teradata stages - The Connection property on the Staging tab of Snowflake and Teradata stages has been renamed to include the staging location. For example, the property name for a stage using an Amazon S3 staging location is Amazon S3 Connection.

  This change affects the following stages:

  • Snowflake Bulk origin

  • Snowflake destination

  • Teradata destination

• Start Jobs origin and processor - You can configure the stage to generate a single record per job when multiple jobs are started.

• Web Client stages - The Web Client origin, processor, and destination allow you to configure per-status actions for unknown HTTP statuses.

• This release includes the following new stage libraries:

  Stage Library	Description
  streamsets-datacollector-apache-kafka-lib	For Kafka
  streamsets-datacollector-file-transfer-lib	For SFTP/FTP/FTPS
  streamsets-datacollector-http-lib	For HTTP

• All versioned Kafka stage libraries have been removed and replaced with a single stage library, streamsets-datacollector-apache-kafka-lib, for all supported Kafka versions.
• The following stages have been removed from the basic stage library and added to the new SFTP/FTP/FTPS stage library, streamsets-datacollector-file-transfer-lib:

  • SFTP/FTP/FTPS Client origin

  • SFTP/FTP/FTPS Client destination

  • SFTP/FTP/FTPS Client executor

  This change has upgrade impact.

• The following stages have been removed from the basic stage library and added to the new HTTP stage library, streamsets-datacollector-http-lib:

  • HTTP Client origin, processor, and destination

  • HTTP Router processor

  • HTTP Server origin

  This change has upgrade impact.

With this release, the following stage supports using Control Hub connections:

• Splunk destination

• The following properties have been renamed to reflect the IBM inclusive naming policy. The functionality of these properties has not changed.

  • Aerospike destination:

    • Commit Master is now Commit Primary

  • Data Collector configuration:

    • system.stagelibs.whitelist is now system.stagelibs.allowlist

    • system.stagelibs.blacklist is now system.stagelibs.blocklist

    • user.stagelibs.whitelist is now user.stagelibs.allowlist

    • user.stagelibs.blacklist is now user.stagelibs.blocklist

  • Encrypt and Decrypt Fields processor:

    • Master Key Provider is now Primary Key Provider

  • Kudu Lookup processor and Kudu destination:

    • Kudu Masters is now Kudu Primary Nodes

• AWS Secrets Manager credential store - The credential store now supports Instance Metadata Service version 2 when using instance profile authentication for the credential store. This change has upgrade impact.

• Data Collector configuration - You can configure the number of seconds Data Collector waits between clearing expression language caches by using the static.cache.cleanup.interval.ms property.

• Hashicorp Vault credential store - The credential store has added support for Azure Key Vault version 2.

• Microsoft Azure Key Vault credential store - The credential method property must be configured to use either client key or managed identity authentication. If the property is not configured or is set to a different value, it generates an error. This change has upgrade impact.

In version 6.0.x, Data Collector uses client key authentication for an Azure Key Vault credential store if the credential method property is not configured. Starting with version 6.1.0, Data Collector generates an error if the credential method property is not configured with either client key or managed identity authentication.

After upgrading to version 6.1.x from version 6.0.x, review the configuration file for upgraded Azure Key Vault credential stores to ensure the credential method property is properly configured.

Starting with Data Collector 6.1.0, SFTP/FTP/FTPS stages have been removed from the basic stage library and added to the new SFTP/FTP/FTPS stage library.

After upgrading to version 6.1.0, if you have upgraded pipelines using an SFTP/FTP/FTPS origin, destination, or executor, you must install the streamsets-datacollector-file-transfer-lib stage library.

Starting with Data Collector 6.1.0, HTTP stages have been removed from the basic stage library and added to the new HTTP stage library.

After upgrading to version 6.1.0, install the streamsets-datacollector-http-lib stage library if you have upgraded pipelines using any of the following stages:

• HTTP Client origin, processor, and destination

• HTTP Router processor

• HTTP Server origin

Starting with Data Collector 6.1.0, AWS Secrets Manager credential stores support Instance Metadata Service version 2. For version 2, Amazon recommends setting the hop limit to 2 in container environments to avoid delays.

If Data Collector runs in a container environment on an Amazon EC2 instance with Instance Metadata Service version 2 and uses instance profile authentication for an AWS Secrets Manager credential store, after upgrading to Data Collector 6.1.0, set the Instance Metadata Service hop limit to 2. For more information, see the Amazon EC2 documentation.

Starting with Data Collector 6.1.0, you can configure a region and endpoint for the Assume Role property in stages using AWS. In previous versions, the assumed role uses the same regional endpoint configured for the stage. Upgraded stages configured to assume a role get the region and endpoint for the assumed role from the AWS service it connects to. If those are not available, the assumed role uses the global endpoint for assumed roles.

After upgrading to version 6.1.0, review upgraded stages using AWS to ensure all stages configured to assume a role are configured with the appropriate region and endpoint.

• Azure Blob Storage origin - The default spooling period is increased to 30 seconds. This change has an upgrade impact.
• MongoDB Atlas CDC origin - When the origin is configured to read from a MongoDB change stream, the origin supports the REPLACE CRUD operation type.

• Web Client origin, processor, and destination:

  • You can configure Web Client stages to log HTTP requests and responses. These properties are disabled by default.

  • The option to specify a status failure action for a per-status action only appears for retry actions.

• Snowflake destination - The Replicate Decimal Columns property has been removed. The destination now always creates Decimal columns for decimal data.

• This release includes the following new stage library:

  Stage Library	Description
  streamsets-datacollector-cdp_7_1_9-lib	For Cloudera CDP 7.1.9

• The Pulsar stage library is renamed from streamsets-datacollector-apache-pulsar_2-lib to streamsets-datacollector-apache-pulsar-lib.

• With this release, Data Collector no longer supports Java 8. This change has an upgrade impact.
• With this release, the standalone Data Collector UI has been removed. After installation or upgrade, all Data Collectors must be registered with Control Hub. This change has an upgrade impact.
• The following stages, which were deprecated in earlier versions, have been removed:

  • ADLS Gen1 File Metadata executor

  • Aerospike destination

  • Azure Data Lake Storage (Legacy) destination

  • Azure Data Lake Storage Gen1 origin and destination

  • Dev SDC RPC with Buffering origin

  • Flume destination

  • Google Big Query (Legacy) destination

  • Hadoop FS origin

  • Hive Streaming destination

  • Kafka Consumer origin

  • KineticDB destination

  • MapR FS origin

  • NiFi HTTP Server origin

  • Omniture origin

  • Start Pipelines origin

  • Start Pipelines processor

  • SDC RPC origin and destination

  • SQL Server 2019 BDC Bulk Loader destination

  • SQL Server 2019 BDC Multitable Consumer origin

  • Spark Evaluator processor

  • Value Replacer processor

  • Wait For Pipelines processor

  This change has an upgrade impact.

• The following features, which were deprecated in earlier versions, have been removed:

  • Aggregated pipeline statistics

  • Cluster pipelines

  • SDC RPC pipelines

  This change has an upgrade impact.

• Antenna Doctor has been removed. This change has an upgrade impact.
• HashiCorp Vault credential store - The authMethod property must be set to one of the following values:

  • appId

  • appRole

  • azure

  This change has an upgrade impact.

A number of stages and features that were deprecated in earlier versions of Data Collector have been removed with version 6.0. Functionality has been removed because it is not commonly used, has been replaced, is no longer maintained, or connects to a system with an end-of-service date.

In previous versions of Data Collector, you could configure Antenna Doctor to suggest potential fixes and workarounds to common pipeline issues. Starting with version 6.0, Antenna Doctor is not included in Data Collector.

After upgrading to version 6.0, Antenna Doctor does not send pipeline messages.

In previous versions of Data Collector, the default spooling period for Azure Blob Storage origins was 5 seconds. Starting with version 6.0, the default spooling period is 30 seconds, and upgraded origins are given the new default value of 30 seconds.

After upgrading to version 6.0, review upgraded Azure Blob Storage origins to ensure they are configured with an appropriate spooling period.

In previous versions of Data Collector, the HashiCorp Vault credential store authMethod property could be left empty. Starting with version 6.0, the authentication method must be set to one of the following values:

• appId

• appRole

• azure

After upgrading to version 6.0, verify that the authentication method for upgraded credential stores is set to a valid value.

• JDBC Lookup processor - You can configure the processor to enforce query validation, which improves memory management.

• ML Evaluator processor - This stage, which was deprecated in an earlier version, has been removed.

  This change has an upgrade impact.

• Oracle Multitable Consumer origin and Oracle destination - The Oracle JDBC driver is no longer included in the streamsets-datacollector-jdbc-branded-oracle-lib stage library. Before using these stages, you must install the driver as an external library into the stage library.

  This change has an upgrade impact.

The Databricks ML Evaluator processor, which was deprecated in an earlier version, has been removed from Data Collector with version 5.12.0. As an alternative, you can use StreamSets Transformer. For more information, see the Transformer documentation.

After upgrading to version 5.12.0, verify that no pipelines are using the Databricks ML Evaluator processor.

Starting with version 5.12.0, the Oracle JDBC driver is no longer included in the streamsets-datacollector-jdbc-branded-oracle-lib stage library. You must manually install the driver into the stage library before using an Oracle Multitable Consumer origin or Oracle destination.

After upgrading to version 5.12.0, if you have upgraded pipelines using an Oracle Multitable Consumer origin or Oracle destination, install the Oracle JDBC driver as an external library for the streamsets-datacollector-jdbc-branded-oracle-lib stage library.

• Jira origin - The new origin reads data from a Jira instance. This origin is a Technology Preview feature. It is not meant for use in production.

• Jira destination - The new destination writes data to a Jira instance. This destination is a Technology Preview feature. It is not meant for use in production.

• Oracle Multitable Consumer origin - The new origin reads data from multiple Oracle database tables.

• Databricks Query executor - The Spark SQL Query property has been renamed SQL Queries to better indicate that you can define multiple queries in the property. The functionality of the property has not changed.

• Databricks Query, JDBC Query, and Snowflake executors:

  • You can configure the maximum number of attempts the executor tries to make a successful connection and the amount of time the executor waits between each attempt.

  • The Max Attempts property has been renamed Max Connection Attempts to better indicate the action being attempted. The functionality of the property has not changed.

• Directory origin - You can configure the origin to retrieve files based on file system events instead of polling for new files.

• HDFS File Metadata executor - The Hadoop FS Configuration property supports expression language and parameters in the Name field. It also supports credentials, expressions language, and parameters in the Value field.

• Kafka stages - The Kafka Consumer origin, Kafka Multitopic Consumer origin, and Kafka Producer destination now support Apache Kafka 3.7.x.

• Kafka Producer destination - The Key Serializer property has been removed. Data Collector automatically uses the appropriate key serializer for the configured message key format.

• MongoDB Atlas stages - The MongoDB Atlas origin, MongoDB Atlas Lookup processor, and MongoDB Atlas destination now support MongoDB Atlas 7.x.

• Oracle CDC Client origin - Session integrity verification has been improved.

• Oracle destination - The destination is no longer a technology preview feature and can be used in production environments.

• Snowflake Bulk origin - You can configure the maximum number of downloaded files in the origin’s processing queue.

• Snowflake stages - The Snowflake Bulk origin, Snowflake Query executor, Snowflake destination, and Snowflake File Uploader destination support the TIMESTAMP_LTZ data type.

• Snowflake Bulk origin, Snowflake executor, and Snowflake destination - The Override Database, Override Schema, and Override Warehouse properties have been moved to the Snowflake Connection Info tab.

• Snowflake executor - You can configure whether the stage treats queries as case-sensitive. By default, queries are not case-sensitive.

• Snowflake File Uploader destination - The Database, Schema, and Warehouse properties have been removed. This change has an upgrade impact.

• Start Jobs processor - The following properties have been renamed to better reflect that these values are Control Hub credential ID and token:

  • Auth ID is now Cred ID

  • Password is now Cred Token

  The functionality of these properties has not changed.

• Teradata destination - The destination is no longer a technology preview feature and can be used in production environments.

• Web Client stages:

  • The Offset in Header pagination option has been renamed Offset to better reflect what the option does. The functionality of the option has not changed.

  • You can configure the stage to continue pagination even after a page returns no data.

  • You can configure per-status actions for status code groups as well as individual status codes. The following properties for specific status code groups have been removed:

    • Accept Informational Statuses (1xx)

    • Accept Successful Statuses (2xx)

    • Accept Redirection Statuses (3xx)

  • The Request Endpoint property supports regular expressions.

  • All Web Client stages evaluate common headers.

  • You can configure retry behavior for the following transient errors:

    • Connection Error

    • No Route To Host

    • Unknown Host

    • Unreachable Port

  • You can configure all Web Client stages to log requests and responses.

• Web Client origin - The Polling Interval property has been removed.

• Web Socket Client origin - You can configure the maximum message size for the origin, up to 10 MB.

• Default Java version - Data Collector Docker images are now bundled with OpenJDK 17 by default.

  Previously, Data Collector Docker images were bundled with OpenJDK 8.0.372.

• Email notifications - The format of pipeline notification emails has changed to include a new error code format. This enhancement has an upgrade impact.

• Hashicorp Vault - Improved authentication support for Azure. This enhancement is currently supported only for KV1 type secret engines.

Starting with version 5.11.0, you cannot configure characters to represent null values or newline characters for Google BigQuery or Snowflake destinations when writing JSON data. Upgraded destinations do not change null values or newline characters.

After you upgrade to version 5.11.0, review pipelines using Google BigQuery or Snowflake destinations writing JSON data to ensure the destination does not receive any null values or newline characters from the pipeline that should not be passed to the external system.

Starting with version 5.11.0, the Database and Schema properties have been removed from the Snowflake File Uploader destination. In Data Collector 5.10.0, if the Stage Database and Stage Schema properties were not configured, the destination used the database and schema values configured for the destination or Control Hub connection instead.

When upgrading from version 5.10.0 to 5.11.0, Snowflake File Uploader destinations that do not use a Control Hub connection and do not have values configured for the Stage Database or Stage Schema properties are assigned a staging database value equal to the configured database value and a staging schema value equal to the configured schema value. Snowflake File Uploader destinations that use a Control Hub connection and do not have values configured for the Stage Database or Stage Schema properties are not assigned any values for these properties and must have them configured after upgrading.

After you upgrade to version 5.11.0 from version 5.10.0, review Snowflake File Uploader destinations that use Control Hub connections and make sure the Stage Database and Stage Schema properties are configured.

Starting with version 5.11.0, the format of pipeline notification emails has changed to include a new error code format.

After you upgrade to version 5.11.0, review notification email configurations for upgraded pipelines to ensure they behave as expected.

Starting with version 5.11.0, the origin correctly interprets the batch wait time value as seconds. In earlier releases, the origin incorrectly interpreted the value as milliseconds.

After you upgrade to version 5.11.0, review Directory origins to ensure they are configured with an appropriate batch wait time.

• Azure Blob Storage destination - The new destination writes data to Azure Blob Storage.
• Oracle destination - The new destination writes data to one or more tables in an Oracle database. This destination is a Technology Preview feature. It is not meant for use in production.
• Web Client origin - The new origin reads data from an HTTP endpoint. This origin is a Technology Preview feature. It is not meant for use in production.
• Web Client processor - The new processor sends requests to an HTTP endpoint and writes responses to records, with enhanced security options and the ability to execute concurrent requests. This processor is a Technology Preview feature. It is not meant for use in production.
• Web Client destination - The new destination writes data to an HTTP endpoint. This destination is a Technology Preview feature. It is not meant for use in production.

Starting with version 5.10.0, you can configure the maximum size of the record cache for an Oracle CDC Client origin using the Records Cache Size property. Upgraded pipelines are given the default value of -2, which represents two times the batch size.

After upgrading to Data Collector 5.10.0, verify that Oracle CDC Client origins are configured with the appropriate record cache size.

Starting with version 5.10.0, Start Jobs stages have updated search mode options. Pipelines upgraded from version 5.2.x or earlier that were configured with the Contain search mode option are updated to use the new Contains Unique search mode option.

After upgrading to Data Collector 5.10.0 from Data Collector 5.2.x or earlier, verify that Start Jobs pipelines are using the appropriate search mode.

• Teradata destination - The new destination writes data to one or more tables in a Teradata database. This destination is a Technology Preview feature. It is not meant for use in production.

Starting with version 5.9.0, the Oracle CDC origin has a Max Batch Vault Size property that allows you to configure the maximum number of batches the origin pre-generates while the pipeline is processing other batches. In upgraded pipelines, the origin uses the default maximum batch vault size of 64.

After you upgrade to version 5.9.0, review Oracle CDC origin pipelines. If the maximum batch vault size is not appropriate, update the pipelines accordingly.

• Amazon S3

• Amazon SQS Consumer

• Azure Blob Storage

• Azure Data Lake Storage Gen2

• Data Parser

• JMS Consumer

• Pulsar Consumer

• Pulsar Consumer (Legacy)

When configured to read tables that contain duplicate column names, the origin treats the tables as invalid and generates an error.

After you upgrade to version 5.9.0, review pipelines that use these origins. If any pipelines require the ability to read tables containing multiple columns with the same name, configure the origins to ignore column headers.

• Couchbase origin - The new origin reads JSON data from Couchbase Server.
• MongoDB Atlas CDC origin - The new origin reads data from MongoDB Atlas or MongoDB Enterprise Server. The origin can read changes from Mongo Change Streams or directly from the Oplog.

• Azure Synapse SQL destination, Databricks Delta Lake destination, Google Big Query destination, and Snowflake origin and destination - The Purge Stage File After Ingesting property is selected by default. Existing pipelines are not affected by this change.
• Couchbase destination:
  • You can choose which stage library the destination uses on the General tab.
  • You can configure the collection and scope for the destination to use on the Couchbase tab.
  • For Couchbase 6.5 and later, you can configure the destination to use durable writes on the Document Handling tab.
• Google Cloud Storage origin and destination - You can configure the maximum number of times that the stage retries a failed connection and the amount of time between each retry.

• Oracle Bulkload origin:

  • Validation for empty tables has been removed, so the origin does not stop the pipeline when it encounters and empty table. This change has an upgrade impact.

  • You can configure the maximum number of seconds the origin waits before timing out.

• Oracle CDC origin:
  • The origin now supports Blob and Clob large data types. LOB_WRITE, LOB_TRIM and LOB_ERASE operations are supported for Blob and Clob columns.
  • Data Collector publishes data lineage for the origin.

• Oracle CDC Client origin - You can configure the origin to raise an error when LogMiner encounters unsupported operations.

• PostgreSQL stages - The PostgreSQL CDC Client origin and PostgreSQL Metatdata processor now support the following data types:
  • cidr
  • inet
  • interval
  • JSON
  • macaddr8 - Supported with PostgreSQL version 10.0.0 and later.
• SFTP/FTP/FTPS Client origin - The origin now supports binary data format.
• Snowflake Bulk origin:
  • The origin now supports Geography and Geometry data types. The origin receives Geography data in Well-Known Text (WKT) format using the GEOGRAPHY_OUTPUT_FORMAT session parameter. Because pipelines handle Geography data as a string, functions related to Geography, such as area calculation, perimeter, and distance, are not available.
  • The origin validates that configured tables exist and returns an error if it cannot find a configured table.
• SQL Server CDC Client origin - The new Automatically Reset Offset After Cleanup property lets you configure the origin to automatically reset the offset after the CDC Cleanup process has been executed. By default, the property is disabled.

• Recommended Java versions - Data Collector Docker images are now bundled with OpenJDK 8.0.372.

  For a tarball installation of Data Collector, all Java 8 versions are still supported; however StreamSets now recommends using Oracle Java JDK 8u361 or later, or OpenJDK 8.0.372 or later.

• Parquet data format:
  • Parquet data format is no longer considered a Technology Preview feature and is approved for use in production.
  • You can select Parquet as a data format for the following origins:
    • Amazon S3
    • Azure Blob Storage
    • Azure Data Lake Storage Gen2
    • Directory
    • Google Cloud Storage
  • The default schema location is now Infer From Records.

Cluster pipelines have been deprecated for several releases. The Cluster Mesos Streaming execution mode has been removed from Data Collector with this release.

grant select on all_tab_cols to <user name>;

For CDB databases, run the command from the root container, cdb$root. Then run it again from the pluggable database. For non-CDB databases, run the command from the primary database.

Starting with version 5.7.0, Amazon S3, Azure Blob Storage, and Azure Data Lake Storage Gen2 origins have a File Processing Delay property that allows you to configure the minimum number of milliseconds that must pass from the time a file is created before it is processed. In upgraded pipelines, these origins receive the default file processing delay of 10,000 milliseconds.

After you upgrade to version 5.7.0, review pipelines that include the Amazon S3, Azure Blob Storage, and Azure Data Lake Storage Gen2 origins. If the 10,000 millisecond delay is not appropriate, update the pipelines accordingly.

After upgrading to version 5.7.0, review the batch wait time for ALDS Gen2 (Legacy), Directory, and Hadoop FS Standalone origins in upgraded pipelines, and update the value if necessary.

• Aerospike Client destination - The new destination writes data to Aerospike.
• Kaitai Struct Parser processor - The new processor parses binary data using a Kaitai Struct format description.
• Snowflake Bulk origin - The new origin reads the available data from multiple Snowflake tables or views and then stops the pipeline. The origin can use multiple threads to perform parallel processing.

• Amazon S3, Azure Blob Storage, Directory, and Google Cloud Storage origins - These origins now support the binary data type.
• Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, Hadoop FS, Local FS, and MapR FS destinations - In the Compression Codec property, the Snappy option has been changed to Snappy (Airlift) to indicate that the destination uses the Airlift version of Snappy rather than the standard Snappy version.
• Azure Synapse SQL destination - On the Azure Synapse SQL tab, you can define property names and values in Additional Connection Properties to specify standard driver and Hikari properties.
• Databricks Delta Lake destination and Databricks Query executor - These stages are no longer part of an enterprise stage library. These stages are now available in the Databricks stage library, streamsets-datacollector-sdc-databricks-lib. The Databricks stage library requires the scheme jdbc:databricks rather than jdbc:spark in the URL or connection string. This change impacts upgrades.
• Databricks Delta Lake destination:
  • When processing CDC data, the destination can now use the primary key information from record header attributes. In the new Primary Key Location property, you configure where the destination finds the primary key, either in the header attributes or in the stage configuration for each table.
  • The Key Columns property has been renamed Table Key Columns. The property is available if you set the Primary Key Location property to Specify for each table. For keys specified in the stage, the Key Columns property has been renamed Table Key Columns.
  • With relaxed requirements for table names, the destination supports writing data to Delta Lake tables managed by Unity Catalog.
• HTTP Client processor:
  • Records that are not processed before the batch wait time expires are sent for error processing rather than discarded. If One Request per Batch is enabled and the records are not processed before the batch wait time expires, all the records in the batch are sent for error processing.
  • In the new Compression Library property, you can specify the compression library used to decompress files before reading. By default, the processor independently detects the compression library for each file. For the processor to read files compressed with the Airlift version of Snappy, you must select Snappy (Airlift Snappy) in the Compression Library property. For the processor to read files compressed with the standard version of Snappy, select the default option to detect the compression library automatically.
• MongoDB Atlas origin - The Initial Offset property no longer has a default value.
• Oracle Bulkload origin - The origin is no longer part of an Enterprise stage library. The origin is included in the JDBC Oracle stage library, streamsets-datacollector-jdbc-oracle-lib. This change impacts upgrades.
• Oracle CDC and Oracle CDC Client origins - The origins contain a new property, Fetch Strategy, that sets the method for staging LogMiner results. Staging to a disk-based queue can alleviate memory issues.
• Orchestration stages - You can configure the Start Jobs origin, Control Hub API processor, Start Jobs processor, or Wait for Jobs processor to use a Control Hub connection.
• Origins with a Compression Format property - For compressed formats, you can specify the compression library that the origin uses to decompress files in the new Compression Library property. By default, the origin independently detects the compression library for each file. For origins to read files compressed with the Airlift version of Snappy, including files from destinations, you must select Snappy (Airlift Snappy) in the Compression Library property. For origins to read files compressed with the standard version of Snappy, select the default option to detect the compression library automatically. This change impacts upgrades.
• Snowflake stages - All Snowflake stages, including the Snowflake destination, the Snowflake File Uploader destination, and the Snowflake executor, have improved connection and authentication options:
  • To connect to a virtual private Snowflake installation, you have two options. You can configure the stages to compute the virtual private URL automatically from the values in the Account property and either the Snowflake Region or Organization property. Alternatively, you can enter a custom JDBC URL.
  • For authentication, the stages support OAuth and key pairs as alternatives to user credentials. To use other authentication methods, you can enter custom connection properties.
• Snowflake destination:
  • The destination supports arrays. On the Data Advanced tab, you can use the ARRAY Default property to configure the default value for missing or incorrect array values.
  • For Amazon S3 external stages, you can configure the S3 Tags property to specify a list of tags to add to created objects.
  • On the Snowflake tab, the Error Behavior, Skip File On Error, Max Error Records, and Max Error Record Percentage properties have been moved to the bottom of the tab.
• SQL Parser processor - For tables with a primary key, the processor includes the following new record header attributes to track changes in the primary key:
  • jdbc.primaryKeySpecification - Includes a JSON-formatted string that lists the columns that form the primary key in the table and the metadata for those columns.
  • jdbc.primaryKey.before.<primary key column> - Includes the previous value for the specified primary key column.
  • jdbc.primaryKey.after.<primary key column> - Includes the new value for the specified primary key column.
• SQL Server CDC Client origin - The Combine Update Records property has been replaced by the Record Format attribute. You can choose between three options to configure how the origin generates records:
  • Basic - Generates two records for updates, one with the old data and one with the changed data. This option produces the same result as if Combine Update Records were set to false.
  • Basic discarding ‘Before Update’ records - Generates one record for updates, containing the changed data.
  • Rich - Generates one record for updates with data written to the Data field, OldData field, or both. This option produces the same result as if Combine Update Records were set to true.

  The origin generates a record header property named record_format, which indicates the format of the generated record: 1 indicates basic format, 2 indicates basic discarding “before update” records, and 3 indicates rich.

• Aerospike connection - You can use this new Control Hub connection with the Aerospike Client destination.
• Databricks Delta Lake connection - The connection requires the Databricks stage library, streamsets-datacollector-sdc-databricks-lib. The Databricks stage library requires the scheme jdbc:databricks rather than jdbc:spark in the URL. This change impacts upgrades.
• Orchestrator connection - You can use this new Control Hub connection with the Start Jobs origin, Control Hub API processor, Start Jobs processor, or Wait for Jobs processor.
• Snowflake connection:
  • You can use the Snowflake connection with the new Snowflake Bulk origin.
  • The connection has improved connection and authentication options, described above for the Snowflake stages under “Stage enhancements.”
  • Tests of the Snowflake connection properly use every configuration set under Connection Properties.

• Microsoft Azure Key Vault credential store - Data Collector can use managed identities in addition to client keys to authenticate with Azure Key Vault.
• The sdc.properties file contains a new property. When the property is enabled, Data Collector provides an HTTP Strict-Transport-Security (HSTS) response header. To enable the new property, you must configure the https.port property.
• Time functions - The StreamSets expression language includes the following new time functions:
  • time:nowNumber - Creates a Date object set to the current date and time with milliseconds precision.
  • time:nowNanoInstant - Creates a LocalDateTime object set to the current date and time with nanoseconds precision.
  • time:nowNanoZonedInstant - Creates a ZonedDateTime object set to the current date and time with nanoseconds precision.
  • time:nowNanoTimestampNumber - Creates a Number object set to the current time, specified as epoch time, with nanoseconds precision.
  • time:nowNanoTimestampString - Creates a String object set to the current time, specified as epoch time, with nanoseconds precision.
• Timestamp functions - The StreamSets expression language introduces a new type of function that offers improved performance over time functions. You can replace a time function with the corresponding timestamp function for better performance. StreamSets recommends using timestamp functions for all new development. The expression language includes the following timestamp functions:
  • timestamp:nowDate - Creates a Date object set to the current date and time with millisecond precision. Use an alternative to the time:now function.
  • timestamp:nowLocal - Creates a LocalDateTime object set to the current date and time with nanosecond precision. Use as an alternative to the time:now function.
  • timestamp:nowMilliseconds - Creates a Long object set to the current date and time with millisecond precision. Use as an alternative to the time:nowNumber function.
  • timestamp:nowMillisecondsString - Creates a String object set to the current date and time with millisecond precision. Use as an alternative to the time:nowNanoTimestampString function.
  • timestamp:nowNanoseconds - Creates a Double object set to the current date and time with nanosecond precision. Use as an alternative to the time:nowNanoTimestampNumber function.
  • timestamp:nowNanosecondsString - Creates a String object set to the current date and time with nanosecond precision. Use as an alternative to the time:nowNanoTimestampString function.
  • timestamp:nowZoned - Creates a ZonedDateTime object set to the current date and time with nanosecond precision. Use as an alternative to the time:nowNanoZonedInstant function.
  • timestamp:extractStringFromDate - Converts a Date object into a String object, based on a specified date-time format, with nanosecond precision. Use as an alternative to the time:extractStringFromDate function.
  • timestamp:extractStringFromDateAndZone - Converts a Date object into a String object, based on a specified date-time format and a time zone, with nanosecond precision. Use as an alternative to the time:extractStringFromDateTZ function.
  • timestamp:extractLongFromDate - Converts a Date object into a Long object, based on a specified date-time format, with nanosecond precision. Use as an alternative to the time:extractLongFromDate function.
  • timestamp:createDateFromString - Converts a String object into a Date object, based on a specified date-time format, with nanosecond precision. Use as an alternative to the time:extractDateFromString function.
  • timestamp:createDateFromStringAndZone - Converts a String object into a Date object based on a specified date-time format and a time zone, with nanosecond precision. Use as an alternative to the time:createDateFromStringTZ function.
• Data lineage publication - For each job run, Data Collector can publish data lineage information in a JSON file. You can export and use the file in your data governance solution. To enable data lineage publication, you must add the following lines to the sdc.properties file:

  lineage.publishers=json
  lineage.publisher.json.def=streamsets-datacollector-basic-lib::com_streamsets_pipeline_lib_lineage_JSONLineagePublisher
  lineage.publisher.json.config.outputDir=<directory for files>
  

  You can optionally include the following line:

  lineage.publisher.json.config.saveInterval=<time in milliseconds>
  

  Data Collector publishes the data lineage information at the specified interval. By default, the interval is set to 60,000 milliseconds. If you specify a negative interval, Data Collector only publishes the information when the pipeline finishes.

  Data Collector publishes data lineage information for the following stages:

  • Databricks Job Launcher executor
  • Dev Data Generator origin
  • Google Big Query origin and destination
  • Google Cloud Storage origin and destination
  • Google Pub Sub Publisher destination
  • HBase destination
  • HTTP Client origin
  • JDBC Multitable Consumer origin
  • JDBC Query Consumer origin
  • Kinesis Consumer origin
  • Kinesis Producer destination
  The following stages include field-level data:

  • JDBC Multitable Consumer origin
  • JDBC Query Consumer origin
  • Dev Data Generator origin
  • HBase destination

• Bucket property for the Amazon S3 origin
• Bucket and path property for the Amazon S3 destination and executor

• Amazon S3
• Azure Blob Storage
• Azure Data Lake Storage Gen1
• Azure Data Lake Storage Gen2 (Legacy)
• Azure IoT/Event Hub Consumer
• CoAP Server
• Directory
• File Tail
• Hadoop FS Standalone
• Google Cloud Storage
• Google Pub/Sub Subscriber
• gRPC Client
• HTTP Client
• HTTP Server
• Kafka Multitopic Consumer
• MQTT Subscriber
• REST Service
• SFTP/FTP/FTPS Client
• TCP Server
• WebSocket Client
• WebSocket Server

• Azure Data Lake Storage Gen2 origin - The new Azure Data Lake Storage Gen2 origin reads data from Microsoft Azure Data Lake Storage Gen2. The new origin connects to Azure Data Lake Storage Gen2 through the API, which results in improved performance.

  The existing Azure Data Lake Storage Gen2 origin has been renamed Azure Data Lake Storage Gen2 (Legacy). Use the new origin for all new development.

• Azure Blob Storage origin - The new Azure Blob Storage origin reads data from Microsoft Azure Blob Storage.
• SingleStore destination - The new SingleStore destination writes data to a SingleStore database table.

• Azure Data Lake Storage Gen2 stages - These stages support Azure Managed Identity authentication.
• Azure Synapse SQL destination:
  • The stage is no longer part of an Enterprise stage library. The stage is part of the Azure stage library, streamsets-datacollector-azure-lib. This change has an upgrade impact.
  • With the new Propagate Numeric Precision and Scale property enabled, the destination creates new numeric columns with the precision and scale specified in JDBC record header attributes. With the property disabled or when JDBC header attributes are not available, the stage creates new numeric columns with the default numeric definition.
• Google Big Query destination - The destination can stage data in JSON files.
• Hive Metadata processor, Hive Metastore destination, and Hive Query executor - In these stages, the Additional Hadoop Configuration property supports credential values.
• MongoDB Atlas origin and destination - For pipelines with a MongoDB Atlas origin or destination, validation passes if at least one host specified in the connection string is reachable. Validation fails if all of the hosts specified in the connection string are unreachable.
• MongoDB Atlas destination - The destination can update or upsert a map, list, or list-map as a nested or non-nested MongoDB document, depending on the number of unique keys.
• OPC UA Client origin:
  • The new Max Recursion Depth property defines the maximum depth to browse for recursive processing.
  • The Channel Config tab has been renamed Encoding Limits.
  • The Max Array Length and Max String Length properties have been removed because they are redundant. The existing Max Message Size property limits the size of the message. This change has an upgrade impact.
• PostgreSQL CDC Client and Aurora PostgreSQL CDC Client origins - The origins interpret the values in the Poll Interval property as milliseconds.
• Snowflake File Uploader destination - For the file-closure event, the destination writes the file name and path information to two fields: Filename and Filepath.
• SQL Server CDC Client origin - The origin generates the primary key record header attributes regardless of whether the Combine Update Records property is enabled.

• Azure Blob Storage - This new Control Hub connection is included for use with the new Azure Blob Storage origin.
• Azure Data Lake Storage Gen2:
  • You can use the connection with the new Azure Data Lake Storage Gen 2 origin.
  • The connection now supports Azure Managed Identity authentication.
• Azure Synapse - Because the Azure Synapse SQL destination is no longer part of an Enterprise stage library, you must install the Azure stage library, streamsets-datacollector-azure-lib, to configure or use Azure Synapse connections. This change has an upgrade impact.

• Cloudera Manager - To support Data Collector installation with Cloudera Manager, StreamSets now provides a Cloudera parcel for RHEL 8.

• Runtime properties - The runtime.conf.location property supports both a relative and absolute path. When configuring a separate runtime properties file, specify a relative path for a file inside the Data Collector installation directory or specify an absolute path for a file outside the Data Collector installation directory.
• Error information - A new Error Information Level property sets the amount of error information included in email. You configure the property for notifications of changes in pipeline state and for notifications triggered by rules.
• Health Inspector - The Health Inspector page includes information about the operating system and version, process uptime, engine time zone, external IP address, and ping and traceroute attempts to Control Hub.
• Time functions - The StreamSets expression language includes two new time functions to operate on dates:
  • time:dateAddition - Adds an interval to a date.
  • time:dateDifference - Determines the interval between two dates.

• Amazon stages - Amazon stages include updated regions in the AWS Region property.
• Azure Data Lake Storage Gen2 stages - You can configure the new Endpoint URL property when using OAuth with Service Principal authentication with the Azure Data Lake Storage Gen2 origin and destination, and the ADLS Gen2 File Metadata executor.
• JDBC Multitable Consumer origin - You can configure the new Maximum Number of Tables property to limit the number of tables to prefetch.
• Google BigQuery destination - The destination allows enabling the Create Table property only when it is configured to handle data drift.

  This is a reversion of a change in 5.3.0, which allows creating tables when the destination is not configured to handle data drift.

• JDBC Tee processor and JDBC Producer - You can use a new useLegacyZonedDatetime JDBC configuration property to help with MySQL driver upgrade issues.

  MySQL driver versions 8.0.23 and later return zoned datetimes in a different format. If you upgrade from an older MySQL driver to 8.0.23 and later, you can add useLegacyZonedDatetime as an Additional JDBC Configuration property and set it to true to have the stages provide zoned datetimes in the previous format.

• MongoDB Atlas stages - You can configure the UUID Interpretation Mode advanced property to specify how a MongoDB Atlas stage handles UUID fields.
• Orchestration stages - Orchestration stages that connect to Control Hub include new Max Number Of Tries and Retry Interval properties that determine how the stages try to connect to Control Hub after encountering communication errors.
• Salesforce stages - All Salesforce stages now use version 57.0.0 of the Salesforce API by default.
• Snowflake stages: The following enhancements apply to the Snowflake and Snowflake File Uploader destinations, and the Snowflake executor:
  • Snowflake stages are no longer part of an Enterprise stage library. The stages are now available in the Snowflake stage library, streamsets-datacollector-sdc-snowflake-lib. This change has upgrade impact.
  • When specifying an organization on the Snowflake Connection Info tab, you no longer need to specify a Snowflake region.
  • Snowflake stages can access all of the latest regions for AWS, GCP, and Azure.
• Snowflake destination:
  • The destination can automatically create tables when configured to handle data drift or use Snowpipe to load data.

    Previously, it did not create tables when using Snowpipe to load data.

  • When you configure the Snowflake destination to use an Amazon S3 staging location, you no longer specify the S3 region. Data Collector now queries Snowflake for that information.
• Snowflake executor - You can use the new Warehouse property to define the warehouse to connect to.
• SQL Parser processor - You can configure the Parsing Thread Pool Size property to enable the processor to use multiple threads when processing data.

• PostgreSQL 15.x support - You can use the PostgreSQL CDC Client origin and JDBC stages with PostgreSQL 15.x.
• Red Hat Enterprise Linux support - You can install Data Collector on Red Hat Enterprise Linux version 9.x, in addition to 6.x - 8.x.
• New stage library:

  Stage Library	Description
  streamsets-datacollector-apache-kafka_3_3-lib	For Kafka version 3.3.x.
  streamsets-datacollector-sdc-snowflake-lib	For Snowflake.

• New function - A new pipeline:email() function returns the email address of the user who started the pipeline.
• Data governance tools - Data governance tools can now publish metadata about the Kinesis Consumer origin.
• Java Security Manager - Data Collector can use the Java Security Manager only when using Java 8. Oracle has deprecated and marked Java Security Manager for removal. As a result, when using Java 9 or later, Data Collector cannot use the security manager.

  Previous releases enabled the Java Security Manager by default for all Java versions, which in some cases caused known issues when using Java 9 or later. To avoid those issues, you had to disable the security manager by setting the SDC_SECURITY_MANAGER_ENABLED environment variable to false.

Starting with version 5.4.0, using Snowflake stages and Snowflake connections requires installing the Snowflake stage library. In previous releases, Snowflake stages and connections were available with the Snowflake Enterprise stage library.

• Amazon S3 executor - The executor supports server-side encryption.
• Dev Data Generator origin - This development origin can generate field attributes to enable easily testing field attribute functionality in pipelines.
• Directory and HDFS Standalone origins - The Directory and HDFS Standalone origins include a new Ignore Temporary Files property that enables the origins to skip processing Data Collector temporary files with a _tmp_ prefix.
• Field Flattener processor - The Output Type property allows you to choose the root field type for flattened records: Map or List-Map.
• Field Mapper processor - The Create New Paths property allows the processor to create new paths when changing record structures.
• Field Replacer processor - The Field Does Not Exist property includes the following new options:
  • Add New Field - Adds the fields defined on the Replace tab to records if they do not exist.
  • Ignore New Field - Ignores any fields defined on the Replace tab if they do not exist.

  These new options replace the Include without Processing option. Upgraded pipelines are set to Add New Field. This can have upgrade impact.

• Google BigQuery destination and executor - These stages are no longer part of an Enterprise stage library. This includes the following updates:
  • The stages, previously known as the Google BigQuery (Enterprise) destination and the Google BigQuery (Enterprise) executor, are renamed to the Google BigQuery destination and Google BigQuery executor.
  • The stages are now available in the Google Cloud stage library, streamsets-datacollector-google-cloud-lib, and are available to install like any other Data Collector stage. This change has upgrade impact.
• Google BigQuery destination:
  • The destination can write nested Avro data.
  • The destination can use BigQuery to generate schemas.
  • The destination now allows enabling the Create Table property when the destination is not configured to handle data drift.
• JDBC Multitable Consumer origin - You can no longer set the Minimum Idle Connections property higher than the Number of Threads property.

  Upgraded pipelines have the Minimum Idle Connections property set to the same value as the Number of Threads property. This can have upgrade impact.

• Kafka message header support - The following functionality is available when using a Kafka Java client version 0.11 or later:
  • The Kafka Multitopic Consumer origin includes Kafka message headers as record header attributes.

  • The Kafka Producer destination includes all user-defined record header attributes as Kafka message headers when writing to Kafka.

• MongoDB Atlas destination - The destination can update nested documents.
• OPC UA Client origin - The origin includes a new Override Host property which overrides the host name returned from the OPC UA server with the host name configured in the resource URL.
• Oracle CDC Client origin - The minimum value for the following advanced properties have changed from 1 millisecond to 0 milliseconds:
  • Time between Session Windows
  • Time after Session Window Start
• Start Job origin and processor - The Search Mode property enables the origin and processor to search for the Control Hub job to start.

• Google BigQuery update - Due to the BigQuery change from enterprise, you must install the Google Cloud stage library, streamsets-datacollector-google-cloud-lib, to configure or use Google BigQuery connections. This change has upgrade impact.

• Data governance tools - Data governance tools support publishing metadata about the following additional stages:
  • HTTP Client origin
  • JDBC Multitable Consumer origin
• Advanced data format properties - Optional properties for data formats have become advanced options. You must now view advanced options to configure them.
• Delimited data format enhancements:
  • When writing delimited data using a destination or the HTTP Client processor, you can define properties to enable writing multicharacter delimited data.
  • When writing delimited data using a custom delimiter format, you can configure the Record Separator String property to define a custom record separator.
• Pipeline start events - Pipeline start event records include a new email field that contains the email address of the user who started the pipeline.

• Add New Field - Adds the fields defined on the Replace tab to records if they do not exist.
• Ignore New Field - Ignores any fields defined on the Replace tab if they do not exist.

This can adversely affect communicating with Snowflake when Snowflake connection information specified in a Snowflake stage or connection, such as a URL, includes underscores. When needed, you can bypass this behavior by setting the allowUnderscoresInHost driver property to true. For more information and alternate solutions, see this Snowflake community article.

• MongoDB Atlas origin and destination - You can use the new MongoDB Atlas origin and destination to read from and write to MongoDB Atlas and MongoDB Enterprise Server.

• Groovy stages - The Groovy Scripting origin and the Groovy Evaluator processor now support Groovy 4.0.
• JDBC Multitable Consumer origin - The origin now provides the jdbc.primaryKeySpecification record header attribute for records from tables with a primary key and the jdbc.vendor record header attribute for all records.
• JDBC Tee processor and JDBC Producer destination - These stages can manage primary key values updates using the jdbc.primaryKey.before.columnName record header attribute for the old value and the jdbc.primaryKey.before.columnName record header attribute for the new value.
• MQTT stages - The MQTT Subscriber origin and the MQTT Publisher destination now support entering a list of brokers from high availability MQTT clusters without a load balancer.
• Oracle CDC Client origin:
  • Conditional Blob and Clob support - When the origin buffers changes locally, you can configure the origin to process Blob and Clob data using the following advanced properties:
    • Enable Blob and Clob Columns Processing property - Enable this property to process Blob and Clob columns.
    • Maximum LOB Size property - Optional property to define the maximum LOB size. When specified, overflow data is discarded.
  • LogMiner Query Timeout property - This property defines how long the origin waits for a LogMiner query to complete.
  • Time between Session Windows property - This advanced property sets the time to wait after a LogMiner session has been completely ingested. This ensures a minimum LogMiner window size.
  • Time after Session Window Start property - This advanced property sets the time to wait after a LogMiner session starts. This allows Oracle to finish setting up before processing begins.
• Pipeline Finisher executor - The executor includes new React to Events and Event Type properties that enables the executor to stop a pipeline only upon receiving the specified event record type.

  For example, you can now configure the executor to stop the pipeline only after receiving a no-more-data event record, and to ignore all other records that it might receive. Previously, you might have used a precondition or a Filter processor to ensure that the executor received only no-more-data events.

• Snowflake stages - Snowflake stages have been updated to support all Snowflake regions.
• SQL Server CDC Client origin - The origin can be configured to combine the two update records that SQL Server creates into a single record and generate the records differently.

  With this property enabled, the origin generates record header attributes about the primary key.

• SQL Server Change Tracking origin - The origin generates record header attributes about the primary key.

• MongoDB Atlas origin
• MongoDB Atlas destination
• SQL Parser processor

This release includes the following new stage libraries:

• streamsets-datacollector-groovy_4_0-lib

• streamsets-datacollector-mongodb_atlas-lib

• New Data Collector configuration property - To cache Domain Name Service (DNS) lookups, you can use the new networkaddress.cache.ttl property in the $SDC_DIST/etc/sdc-java-security.properties file.

  With this change, the java.security.networkaddress.cache.ttl Data Collector property has been deprecated.

• Help options - The Local Help option has been removed from the Help configuration option in Help > Settings. When you view Data Collector help, you will always view it on the StreamSets documentation website: https://docs.streamsets.com.

With 5.2.0, the MySQL Binary Log origin converts MySQL Enum and Set fields to String fields.

In previous releases, when reading from a database where the binlog_row_metadata MySQL database property is set to MINIMAL, Enum fields are converted to Long, and Set fields are converted to Integer.

In 5.2.0 as well as previous releases, when the binlog_row_metadata MySQL database property is set to FULL, Enum and Set fields are converted to String.

With 5.2.0, the Oracle CDC Client origin has new advanced properties that enable processing Blob and Clob columns. You can use these properties when the origin buffers changes locally. They are disabled by default.

In previous releases, the origin does not process Blob or Clob columns. However, when the Unsupported Fields to Records property is enabled, the origin includes Blob and Clob field names and raw string values in records.

• Pulsar Consumer origin - The new Pulsar Consumer origin can use multiple threads to read from Pulsar. The origin supports schema validation and Pulsar namespaces configured to enforce schema validation. You can specify the schema used to determine compatibility between the origin and a Pulsar topic. You can also use JWT authentication with the new origin.

  With this new origin, the existing Pulsar Consumer has been renamed Pulsar Consumer (Legacy). Use this new Pulsar origin for all new development.

• Aurora PostgreSQL CDC Client and PostgreSQL CDC Client origins - Both origins can now generate a record for each individual operation.

  Previously, the origins could only generate a record for each transaction.

• Aurora PostgreSQL CDC Client, PostgreSQL CDC Client, and MySQL Binary Log origins - These origins include the following new record header attributes when a table includes a primary key:
  • jdbc.primaryKeySpecification - Includes a JSON-formatted string that lists the columns that form the primary key in the table and the metadata for those columns.
  • jdbc.primaryKey.before.<primary key column> - Includes the previous value for the specified primary key column.
  • jdbc.primaryKey.after.<primary key column> - Includes the new value for the specified primary key column.

• Kafka Multitopic Consumer origin and Kafka Producer destination - The stages include a new Custom Authentication security option that enables specifying custom properties that contain the information required by a security protocol, rather than using predefined properties associated with other security options.
• OPC UA Server origin - The origin now supports using a user name and password to authenticate with the OPC UA server, in addition to an anonymous log in.
• Oracle CDC Client origin:
  • The origin now supports reading from Oracle 21c databases.
  • The field order of generated records now matches the column order in database tables. Previously, the field order was not guaranteed.
  • When you configure the origin to use local buffers and write to disk, you can specify an existing directory to use.
  • A new Data Collector configuration property affects the origin and can have upgrade impact. For details, see Upgrade Impact.
• Pulsar Consumer (Legacy) origin:
  • The origin, formerly named Pulsar Consumer, has been renamed with this release.

    This change has no upgrade impact. However, we recommend using the new Pulsar Consumer origin, which supports multithreaded processing, to read from Pulsar.

  • You can specify the schema used to determine compatibility between the origin and a Pulsar topic.
  • You can also use JWT authentication with the origin.
• Salesforce Bulk API 2.0 stages:
  • All Salesforce Bulk API 2.0 stages include a new Salesforce Query Timeout property which defines the number of seconds that the stage waits for a response to a query.
  • The Salesforce Bulk API 2.0 origin and Salesforce Bulk API 2.0 Lookup processor both include a new Maximum Query Columns property that limits the number of columns that can be returned by a query.
• Scripting stages - The Groovy, JavaScript, and Jython Evaluator origins and processors now generate metrics for script execution and locking details that you can view when monitoring the pipeline.
• Field Remover - The processor now includes the On Record Error property on the General tab.
• Field Type Converter processor - When converting a Date, Datetime, or Time field, the Date Format property now offers explicit options to specify that the field contains a Unix timestamp in milliseconds or seconds.

  If the field contains a Unix timestamp and you select an alternate date format, then the behavior is unchanged: the processor assumes the timestamps are in milliseconds.

• SQL Parser processor:
  • The processor adds the fields from the SQL statement in the same order as the corresponding columns in the database tables.
  • The processor now includes field attributes for columns converted to the Decimal or Datetime data types in Data Collector. The attributes provide additional information for each field.
• Pulsar Producer destination - You can now use JWT authentication with the destination.

• Kafka connection - The connection includes a new Custom Authentication security option that enables specifying custom properties that contain the information required by a security protocol, rather than using predefined properties associated with other security options.
• OPC UA connection - The connection now supports using a user name and password to authenticate with the OPC UA server in addition to an anonymous log in to the server.
• Pulsar connection - The connection can now use JWT authentication to connect to Pulsar.
• Snowflake connection - The new Connection Properties property enables you to specify additional connection properties for Snowflake connections.

• Azure Synapse 1.3.0
• Databricks 1.7.0
• Snowflake 1.12.0

• Data Collector Docker image - The Docker image for Data Collector 5.1.0, streamsets/datacollector:5.1.0, uses Ubuntu 20.04 LTS (Focal Fossa) as a parent image. This change can have upgrade impact.
• Microsoft JDBC Driver for SQL Server - Data Collector uses version 10.2.1 of the driver to connect to Microsoft SQL Server. Due to changes in the driver, this can have upgrade impact.
• Data Collector configuration properties - You can define the following new configuration properties:
  • stage.conf_com.streamsets.pipeline.lib.jdbc.disableSSL - When this configuration property is set to true, Data Collector attempts to disable SSL for all JDBC connections. This property is commented out by default.
  • stage.conf_com.streamsets.pipeline.stage.origin.jdbc.cdc.oracle.monitorbuffersize - When this configuration property is set to true, Data Collector reports memory consumption when the Oracle CDC Client origin uses local buffers.

    This property is set to false by default. In previous releases, the origin reported this information by default, so this enhancement has upgrade impact.

• streamsets-datacollector-apache-kafka_3_0-lib - For Apache Kafka 3.0.
• streamsets-datacollector-apache-kafka_3_1-lib - For Apache Kafka 3.1.
• streamsets-datacollector-apache-kafka_3_2-lib - For Apache Kafka 3.2.

• sdc-log4j2.properties
• log4j2.component.properties

• Red Hat Enterprise Linux 8.x - Data Collector now supports installation on RHEL 8.x, in addition to 6.x and 7.x.

• InfluxDB 2.x destination - Use the destination to write to InfluxDB 2.x databases.

• Couchbase Lookup processor property name updates - For clarity, the following property names have been changed:
  • Property Name is now Sub-Document Path.
  • Return Properties is now Return Sub-Documents.
  • SDC Field is now Output Field.
  • When performing a key value lookup and configuring multiple return properties, the Property Mappings property is now Sub-Document Mappings.
  • When performing an N1QL lookup and configuring multiple return properties, the Property Mappings property is now Sub-N1QL Mappings.
• Einstein Analytics destination enhancements:
  • The Einstein Analytics destination has been renamed the Tableau CRM destination to match the Salesforce rebranding.
  • The new Tableau CRM destination can perform automatic recovery.
• HTTP Client stage statistics - HTTP Client stages provide additional metrics when you monitor the pipeline.
• PostgreSQL CDC Client origin - You can specify the SSL mode to use on the new Encryption tab of the origin.
• Salesforce destination - The destination supports performing hard deletes when using the Salesforce Bulk API. Hard deletes permanently delete records, bypassing the Salesforce Recycle Bin.
• Salesforce stages - Salesforce stages now use version 53.0.0 of the Salesforce API by default.
• SFTP/FTP/FTPS stages - All SFTP/FTP/FTPS Client stages now support HTTP and Socks proxies.

• When Data Collector version 6.1.0 is registered with Control Hub cloud or with Control Hub on-premises version 3.19.x or later, the following stage supports using Control Hub connections:
  • Cassandra destination
• SFTP/FTP/FTPS enhancement - The SFTP/FTP/FTPS connection allows configuring the new SFTP/FTP/FTPS proxy properties.

• Enabling HTTPS for Data Collector - You can now store the keystore file in the Data Collector resources directory, $SDC_RESOURCES, and then enter a path relative to that directory when you define the keystore location. This can have upgrade impact.
• Google Secret Manager enhancement - You can configure a new enforceEntryGroup Google Secret Manager credential store property to validate a user’s group against a comma-separated list of groups allowed to access each secret.

• Google Cloud Storage executor - You can use this executor to create new objects, copy or move objects, or add metadata to new or existing objects.

• Amazon stages - When you configure the Region property, you can select from several additional regions.
• Kudu stages - The default value for the Maximum Number of Worker Threads property is now 2. Previously, the default was 0, which used the Kudu default.

  Existing pipelines are not affected by this change.

• Orchestration stages - You can use an expression when you configure the Control Hub URL property in orchestration stages.
• Salesforce stages - All Salesforce stages now support using version 52.2.0 of the Salesforce API.
• Scripting processors - In the Groovy Evaluator, JavaScript Evaluator, and Jython Evaluator processors, you can select the Script Error as Record Error property to have the stage handle script errors based on how the On Record Error property is configured for the stage.

• Google Cloud Storage origin - You can configure post processing actions to take on objects that the origin reads.
• MySQL Binary Log origin - The origin now recovers automatically from the following issues:
  • Lost, damaged, or unestablished connections.
  • Exceptions raised from MySQL Binary Log being out-of-sync in some cluster nodes, or from being unable to communicate with the MySQL Binary Log origin.
• Oracle CDC Client origin:
  • The origin includes a Batch Wait Time property that determines how long the origin waits for data before sending an empty or partial batch through the pipeline.
  • The origin provides additional LogMiner metrics when you monitor a pipeline.
• RabbitMQ Consumer origin - You can configure the origin to read from quorum queues by adding x-queue-type as a Declaration Property in the Queue tab and setting it to quorum.

• SQL Parser processor - You can configure the processor to use the Oracle PEG parser instead of the default parser.

• Google BigQuery (Legacy) destination - The destination now supports writing Decimal data to Google BigQuery Decimal columns.
• MongoDB destination - You can use the Improve Type Conversion property to improve how the destination handles date and decimal data.
• RabbitMQ Producer destination - You can configure the destination to write to quorum queues by adding x-queue-type as a Declaration Property in the Queue tab and setting it to quorum.
• Splunk destination - You can use the Additional HTTP Headers property to define additional key-value pairs of HTTP headers to data written to Splunk.

• New Google Secret Manager support - You can use Google Secret Manager as a credential store for Data Collector.
• CyberArk enhancement - You can configure the credentialStore.cyberark.config.ws.proxyURI property to allow defining the URI for the proxy that should be used to reach the CyberArk services.

• Google

• When Data Collector version 6.1.0 is registered with Control Hub cloud or with Control Hub on-premises version 3.19.x or later, the following stages support using Control Hub connections:
  • MongoDB stages
  • RabbitMQ stages
  • Redis stages

• Salesforce enhancement - The Salesforce connection includes the following role properties:
  • Use Snowflake Role
  • Snowflake Role Name

• Excel data format enhancement - Stages that support reading the Excel data format include an Include Cells With Empty Value property to include empty cells in records.

• Control Hub orchestration stages - Orchestration stages use API credentials to connect to Control Hub in the StreamSets platform. This affects the following stages:
  • Start Jobs origin
  • Control Hub API processor
  • Start Jobs processor
  • Wait for Jobs processor
• Kafka stages - Kafka stages include an Override Stage Configurations property that enables custom Kafka properties defined in the stage to override other stage properties.

  This can impact existing pipelines.

• MapR Streams stages - MapR Streams stages also include an Override Stage Configurations property that enables the additional MapR or Kafka properties defined in the stage to override other stage properties.

  This can impact existing pipelines.

• Salesforce stages - The Salesforce origin, processor, destination, and the Tableau CRM destination include the following new timeout properties:
  • Connection Handshake Timeout
  • Subscribe Timeout
• Oracle CDC Client origin:
  • You can now remove Oracle pseudocolumns from parsed redo logs and place the information in record header attributes.
  • The origin includes an oracle.cdc.oracle.pseudocolumn.<pseudocolumn name> attribute for each pseudocolumn in the original statement.
  • Starting with version 4.0.1, the origin includes a Batch Wait Time property.
• Field Type Converter processor - The Source Field is Empty property enables to you specify the action to take when an input field is an empty string.
• HTTP Client processor:
  • Two Pass Records properties allow you to pass a record through the pipeline when all retries fail for per-status actions and for timeouts.
  • The following record header attributes are populated when you use one of the Pass Records properties:
    • httpClientError
    • httpClientStatus
    • httpClientLastAction
    • httpClientTimeoutType
    • httpClientRetries
• SQL Parser processor:
  • You can now remove Oracle pseudocolumns from parsed redo logs and place the information in record header attributes.
  • The processor includes an oracle.cdc.oracle.pseudocolumn.<pseudocolumn name> attribute for each pseudocolumn in the original statement.

• Oracle CDC Client origin
• SQL Server CDC Client origin
• SQL Server Change Tracking Client origin

• SDC_EXTERNAL_RESOURCES environment variable - An optional root directory for external resources, such as custom stage libraries, external libraries, and runtime resources.

  The default location is $SDC_DIST/externalResources.

• Support Bundle - Support bundles now include the System Messages log file when you include log files in the bundle.