Release Notes

• Aerospike Client destination - The new destination writes data to Aerospike.
• Kaitai Struct Parser processor - The new processor parses binary data using a Kaitai Struct format description.
• Snowflake Bulk origin - The new origin reads the available data from multiple Snowflake tables or views and then stops the pipeline. The origin can use multiple threads to perform parallel processing.

• Amazon S3, Azure Blob Storage, Directory, and Google Cloud Storage origins - These origins now support the binary data type.
• Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, Hadoop FS, Local FS, and MapR FS destinations - In the Compression Codec property, the Snappy option has been changed to Snappy (Airlift) to indicate that the destination uses the Airlift version of Snappy rather than the standard Snappy version.
• Azure Synapse SQL destination - On the Azure Synapse SQL tab, you can define property names and values in Additional Connection Properties to specify standard driver and Hikari properties.
• Databricks Delta Lake destination and Databricks Query executor - These stages are no longer part of an enterprise stage library. These stages are now available in the Databricks stage library, streamsets-datacollector-sdc-databricks-lib. The Databricks stage library requires the scheme jdbc:databricks rather than jdbc:spark in the URL or connection string. This change impacts upgrades.
• Databricks Delta Lake destination:
  • When processing CDC data, the destination can now use the primary key information from record header attributes. In the new Primary Key Location property, you configure where the destination finds the primary key, either in the header attributes or in the stage configuration for each table.
  • The Key Columns property has been renamed Table Key Columns. The property is available if you set the Primary Key Location property to Specify for each table. For keys specified in the stage, the Key Columns property has been renamed Table Key Columns.
  • With relaxed requirements for table names, the destination supports writing data to Delta Lake tables managed by Unity Catalog.
• HTTP Client processor:
  • Records that are not processed before the batch wait time expires are sent for error processing rather than discarded. If One Request per Batch is enabled and the records are not processed before the batch wait time expires, all the records in the batch are sent for error processing.
  • In the new Compression Library property, you can specify the compression library used to decompress files before reading. By default, the processor independently detects the compression library for each file. For the processor to read files compressed with the Airlift version of Snappy, you must select Snappy (Airlift Snappy) in the Compression Library property. For the processor to read files compressed with the standard version of Snappy, select the default option to detect the compression library automatically.
• MongoDB Atlas origin - The Initial Offset property no longer has a default value.
• Oracle Bulkload origin - The origin is no longer part of an Enterprise stage library. The origin is included in the JDBC Oracle stage library, streamsets-datacollector-jdbc-oracle-lib. This change impacts upgrades.
• Oracle CDC and Oracle CDC Client origins - The origins contain a new property, Fetch Strategy, that sets the method for staging LogMiner results. Staging to a disk-based queue can alleviate memory issues.
• Orchestration stages - You can configure the Start Jobs origin, Control Hub API processor, Start Jobs processor, or Wait for Jobs processor to use a Control Hub connection.
• Origins with a Compression Format property - For compressed formats, you can specify the compression library that the origin uses to decompress files in the new Compression Library property. By default, the origin independently detects the compression library for each file. For origins to read files compressed with the Airlift version of Snappy, including files from destinations, you must select Snappy (Airlift Snappy) in the Compression Library property. For origins to read files compressed with the standard version of Snappy, select the default option to detect the compression library automatically. This change impacts upgrades.
• Snowflake stages - All Snowflake stages, including the Snowflake destination, the Snowflake File Uploader destination, and the Snowflake executor, have improved connection and authentication options:
  • To connect to a virtual private Snowflake installation, you have two options. You can configure the stages to compute the virtual private URL automatically from the values in the Account property and either the Snowflake Region or Organization property. Alternatively, you can enter a custom JDBC URL.
  • For authentication, the stages support OAuth and key pairs as alternatives to user credentials. To use other authentication methods, you can enter custom connection properties.
• Snowflake destination:
  • The destination supports arrays. On the Data Advanced tab, you can use the ARRAY Default property to configure the default value for missing or incorrect array values.
  • For Amazon S3 external stages, you can configure the S3 Tags property to specify a list of tags to add to created objects.
  • On the Snowflake tab, the Error Behavior, Skip File On Error, Max Error Records, and Max Error Record Percentage properties have been moved to the bottom of the tab.
• SQL Parser processor - For tables with a primary key, the processor includes the following new record header attributes to track changes in the primary key:
  • jdbc.primaryKeySpecification - Includes a JSON-formatted string that lists the columns that form the primary key in the table and the metadata for those columns.
  • jdbc.primaryKey.before.<primary key column> - Includes the previous value for the specified primary key column.
  • jdbc.primaryKey.after.<primary key column> - Includes the new value for the specified primary key column.
• SQL Server CDC Client origin - The Combine Update Records property has been replaced by the Record Format attribute. You can choose between three options to configure how the origin generates records:
  • Basic - Generates two records for updates, one with the old data and one with the changed data. This option produces the same result as if Combine Update Records were set to false.
  • Basic discarding ‘Before Update’ records - Generates one record for updates, containing the changed data.
  • Rich - Generates one record for updates with data written to the Data field, OldData field, or both. This option produces the same result as if Combine Update Records were set to true.

  The origin generates a record header property named record_format, which indicates the format of the generated record: 1 indicates basic format, 2 indicates basic discarding “before update” records, and 3 indicates rich.

• Aerospike connection - You can use this new Control Hub connection with the Aerospike Client destination.
• Databricks Delta Lake connection - The connection requires the Databricks stage library, streamsets-datacollector-sdc-databricks-lib. The Databricks stage library requires the scheme jdbc:databricks rather than jdbc:spark in the URL. This change impacts upgrades.
• Orchestrator connection - You can use this new Control Hub connection with the Start Jobs origin, Control Hub API processor, Start Jobs processor, or Wait for Jobs processor.
• Snowflake connection:
  • You can use the Snowflake connection with the new Snowflake Bulk origin.
  • The connection has improved connection and authentication options, described above for the Snowflake stages under “Stage enhancements.”
  • Tests of the Snowflake connection properly use every configuration set under Connection Properties.

• Microsoft Azure Key Vault credential store - Data Collector can use managed identities in addition to client keys to authenticate with Azure Key Vault.
• The sdc.properties file contains a new property. When the property is enabled, Data Collector provides an HTTP Strict-Transport-Security (HSTS) response header. To enable the new property, you must configure the https.port property.
• Time functions - The StreamSets expression language includes the following new time functions:
  • time:nowNumber - Creates a Date object set to the current date and time with milliseconds precision.
  • time:nowNanoInstant - Creates a LocalDateTime object set to the current date and time with nanoseconds precision.
  • time:nowNanoZonedInstant - Creates a ZonedDateTime object set to the current date and time with nanoseconds precision.
  • time:nowNanoTimestampNumber - Creates a Number object set to the current time, specified as epoch time, with nanoseconds precision.
  • time:nowNanoTimestampString - Creates a String object set to the current time, specified as epoch time, with nanoseconds precision.
• Timestamp functions - The StreamSets expression language introduces a new type of function that offers improved performance over time functions. You can replace a time function with the corresponding timestamp function for better performance. StreamSets recommends using timestamp functions for all new development. The expression language includes the following timestamp functions:
  • timestamp:nowDate - Creates a Date object set to the current date and time with millisecond precision. Use an alternative to the time:now function.
  • timestamp:nowLocal - Creates a LocalDateTime object set to the current date and time with nanosecond precision. Use as an alternative to the time:now function.
  • timestamp:nowMilliseconds - Creates a Long object set to the current date and time with millisecond precision. Use as an alternative to the time:nowNumber function.
  • timestamp:nowMillisecondsString - Creates a String object set to the current date and time with millisecond precision. Use as an alternative to the time:nowNanoTimestampString function.
  • timestamp:nowNanoseconds - Creates a Double object set to the current date and time with nanosecond precision. Use as an alternative to the time:nowNanoTimestampNumber function.
  • timestamp:nowNanosecondsString - Creates a String object set to the current date and time with nanosecond precision. Use as an alternative to the time:nowNanoTimestampString function.
  • timestamp:nowZoned - Creates a ZonedDateTime object set to the current date and time with nanosecond precision. Use as an alternative to the time:nowNanoZonedInstant function.
  • timestamp:extractStringFromDate - Converts a Date object into a String object, based on a specified date-time format, with nanosecond precision. Use as an alternative to the time:extractStringFromDate function.
  • timestamp:extractStringFromDateAndZone - Converts a Date object into a String object, based on a specified date-time format and a time zone, with nanosecond precision. Use as an alternative to the time:extractStringFromDateTZ function.
  • timestamp:extractLongFromDate - Converts a Date object into a Long object, based on a specified date-time format, with nanosecond precision. Use as an alternative to the time:extractLongFromDate function.
  • timestamp:createDateFromString - Converts a String object into a Date object, based on a specified date-time format, with nanosecond precision. Use as an alternative to the time:extractDateFromString function.
  • timestamp:createDateFromStringAndZone - Converts a String object into a Date object based on a specified date-time format and a time zone, with nanosecond precision. Use as an alternative to the time:createDateFromStringTZ function.
• Data lineage publication - For each job run, Data Collector can publish data lineage information in a JSON file. You can export and use the file in your data governance solution. To enable data lineage publication, you must add the following lines to the sdc.properties file:

  lineage.publishers=json
  lineage.publisher.json.def=streamsets-datacollector-basic-lib::com_streamsets_pipeline_lib_lineage_JSONLineagePublisher
  lineage.publisher.json.config.outputDir=<directory for files>
  

  You can optionally include the following line:

  lineage.publisher.json.config.saveInterval=<time in milliseconds>
  

  Data Collector publishes the data lineage information at the specified interval. By default, the interval is set to 60,000 milliseconds. If you specify a negative interval, Data Collector only publishes the information when the pipeline finishes.

  Data Collector publishes data lineage information for the following stages:

  • Databricks Job Launcher executor
  • Dev Data Generator origin
  • Google Big Query origin and destination
  • Google Cloud Storage origin and destination
  • Google Pub Sub Publisher destination
  • HBase destination
  • HTTP Client origin
  • JDBC Multitable Consumer origin
  • JDBC Query Consumer origin
  • Kinesis Consumer origin
  • Kinesis Producer destination
  The following stages include field-level data:

  • JDBC Multitable Consumer origin
  • JDBC Query Consumer origin
  • Dev Data Generator origin
  • HBase destination

• Bucket property for the Amazon S3 origin
• Bucket and path property for the Amazon S3 destination and executor

• Amazon S3
• Azure Blob Storage
• Azure Data Lake Storage Gen1
• Azure Data Lake Storage Gen2 (Legacy)
• Azure IoT/Event Hub Consumer
• CoAP Server
• Directory
• File Tail
• Hadoop FS Standalone
• Google Cloud Storage
• Google Pub/Sub Subscriber
• gRPC Client
• HTTP Client
• HTTP Server
• Kafka Multitopic Consumer
• MQTT Subscriber
• REST Service
• SFTP/FTP/FTPS Client
• TCP Server
• WebSocket Client
• WebSocket Server

• Azure Data Lake Storage Gen2 origin - The new Azure Data Lake Storage Gen2 origin reads data from Microsoft Azure Data Lake Storage Gen2. The new origin connects to Azure Data Lake Storage Gen2 through the API, which results in improved performance.

  The existing Azure Data Lake Storage Gen2 origin has been renamed Azure Data Lake Storage Gen2 (Legacy). Use the new origin for all new development.

• Azure Blob Storage origin - The new Azure Blob Storage origin reads data from Microsoft Azure Blob Storage.
• SingleStore destination - The new SingleStore destination writes data to a SingleStore database table.

• Azure Data Lake Storage Gen2 stages - These stages support Azure Managed Identity authentication.
• Azure Synapse SQL destination:
  • The stage is no longer part of an Enterprise stage library. The stage is part of the Azure stage library, streamsets-datacollector-azure-lib. This change has an upgrade impact.
  • With the new Propagate Numeric Precision and Scale property enabled, the destination creates new numeric columns with the precision and scale specified in JDBC record header attributes. With the property disabled or when JDBC header attributes are not available, the stage creates new numeric columns with the default numeric definition.
• Google Big Query destination - The destination can stage data in JSON files.
• Hive Metadata processor, Hive Metastore destination, and Hive Query executor - In these stages, the Additional Hadoop Configuration property supports credential values.
• MongoDB Atlas origin and destination - For pipelines with a MongoDB Atlas origin or destination, validation passes if at least one host specified in the connection string is reachable. Validation fails if all of the hosts specified in the connection string are unreachable.
• MongoDB Atlas destination - The destination can update or upsert a map, list, or list-map as a nested or non-nested MongoDB document, depending on the number of unique keys.
• OPC UA Client origin:
  • The new Max Recursion Depth property defines the maximum depth to browse for recursive processing.
  • The Channel Config tab has been renamed Encoding Limits.
  • The Max Array Length and Max String Length properties have been removed because they are redundant. The existing Max Message Size property limits the size of the message. This change has an upgrade impact.
• PostgreSQL CDC Client and Aurora PostgreSQL CDC Client origins - The origins interpret the values in the Poll Interval property as milliseconds.
• Snowflake File Uploader destination - For the file-closure event, the destination writes the file name and path information to two fields: Filename and Filepath.
• SQL Server CDC Client origin - The origin generates the primary key record header attributes regardless of whether the Combine Update Records property is enabled.

• Azure Blob Storage - This new Control Hub connection is included for use with the new Azure Blob Storage origin.
• Azure Data Lake Storage Gen2:
  • You can use the connection with the new Azure Data Lake Storage Gen 2 origin.
  • The connection now supports Azure Managed Identity authentication.
• Azure Synapse - Because the Azure Synapse SQL destination is no longer part of an Enterprise stage library, you must install the Azure stage library, streamsets-datacollector-azure-lib, to configure or use Azure Synapse connections. This change has an upgrade impact.

• Cloudera Manager - To support Data Collector installation with Cloudera Manager, StreamSets now provides a Cloudera parcel for RHEL 8.

• Runtime properties - The runtime.conf.location property supports both a relative and absolute path. When configuring a separate runtime properties file, specify a relative path for a file inside the Data Collector installation directory or specify an absolute path for a file outside the Data Collector installation directory.
• Error information - A new Error Information Level property sets the amount of error information included in email. You configure the property for notifications of changes in pipeline state and for notifications triggered by rules.
• Health Inspector - The Health Inspector page includes information about the operating system and version, process uptime, engine time zone, external IP address, and ping and traceroute attempts to Control Hub.
• Time functions - The StreamSets expression language includes two new time functions to operate on dates:
  • time:dateAddition - Adds an interval to a date.
  • time:dateDifference - Determines the interval between two dates.

• Amazon stages - Amazon stages include updated regions in the AWS Region property.
• Azure Data Lake Storage Gen2 stages - You can configure the new Endpoint URL property when using OAuth with Service Principal authentication with the Azure Data Lake Storage Gen2 origin and destination, and the ADLS Gen2 File Metadata executor.
• JDBC Multitable Consumer origin - You can configure the new Maximum Number of Tables property to limit the number of tables to prefetch.
• Google BigQuery destination - The destination allows enabling the Create Table property only when it is configured to handle data drift.

  This is a reversion of a change in 5.3.0, which allows creating tables when the destination is not configured to handle data drift.

• JDBC Tee processor and JDBC Producer - You can use a new useLegacyZonedDatetime JDBC configuration property to help with MySQL driver upgrade issues.

  MySQL driver versions 8.0.23 and later return zoned datetimes in a different format. If you upgrade from an older MySQL driver to 8.0.23 and later, you can add useLegacyZonedDatetime as an Additional JDBC Configuration property and set it to true to have the stages provide zoned datetimes in the previous format.

• MongoDB Atlas stages - You can configure the UUID Interpretation Mode advanced property to specify how a MongoDB Atlas stage handles UUID fields.
• Orchestration stages - Orchestration stages that connect to Control Hub include new Max Number Of Tries and Retry Interval properties that determine how the stages try to connect to Control Hub after encountering communication errors.
• Salesforce stages - All Salesforce stages now use version 57.0.0 of the Salesforce API by default.
• Snowflake stages: The following enhancements apply to the Snowflake and Snowflake File Uploader destinations, and the Snowflake executor:
  • Snowflake stages are no longer part of an Enterprise stage library. The stages are now available in the Snowflake stage library, streamsets-datacollector-sdc-snowflake-lib. This change has upgrade impact.
  • When specifying an organization on the Snowflake Connection Info tab, you no longer need to specify a Snowflake region.
  • Snowflake stages can access all of the latest regions for AWS, GCP, and Azure.
• Snowflake destination:
  • The destination can automatically create tables when configured to handle data drift or use Snowpipe to load data.

    Previously, it did not create tables when using Snowpipe to load data.

  • When you configure the Snowflake destination to use an Amazon S3 staging location, you no longer specify the S3 region. Data Collector now queries Snowflake for that information.
• Snowflake executor - You can use the new Warehouse property to define the warehouse to connect to.
• SQL Parser processor - You can configure the Parsing Thread Pool Size property to enable the processor to use multiple threads when processing data.

• PostgreSQL 15.x support - You can use the PostgreSQL CDC Client origin and JDBC stages with PostgreSQL 15.x.
• Red Hat Enterprise Linux support - You can install Data Collector on Red Hat Enterprise Linux version 9.x, in addition to 6.x - 8.x.
• New stage library:

  Stage Library	Description
  streamsets-datacollector-apache-kafka_3_3-lib	For Kafka version 3.3.x.
  streamsets-datacollector-sdc-snowflake-lib	For Snowflake.

• New function - A new pipeline:email() function returns the email address of the user who started the pipeline.
• Data governance tools - Data governance tools can now publish metadata about the Kinesis Consumer origin.
• Java Security Manager - Data Collector can use the Java Security Manager only when using Java 8. Oracle has deprecated and marked Java Security Manager for removal. As a result, when using Java 9 or later, Data Collector cannot use the security manager.

  Previous releases enabled the Java Security Manager by default for all Java versions, which in some cases caused known issues when using Java 9 or later. To avoid those issues, you had to disable the security manager by setting the SDC_SECURITY_MANAGER_ENABLED environment variable to false.

Starting with version 5.4.0, using Snowflake stages and Snowflake connections requires installing the Snowflake stage library. In previous releases, Snowflake stages and connections were available with the Snowflake Enterprise stage library.

• Amazon S3 executor - The executor supports server-side encryption.
• Dev Data Generator origin - This development origin can generate field attributes to enable easily testing field attribute functionality in pipelines.
• Directory and HDFS Standalone origins - The Directory and HDFS Standalone origins include a new Ignore Temporary Files property that enables the origins to skip processing Data Collector temporary files with a _tmp_ prefix.
• Field Flattener processor - The Output Type property allows you to choose the root field type for flattened records: Map or List-Map.
• Field Mapper processor - The Create New Paths property allows the processor to create new paths when changing record structures.
• Field Replacer processor - The Field Does Not Exist property includes the following new options:
  • Add New Field - Adds the fields defined on the Replace tab to records if they do not exist.
  • Ignore New Field - Ignores any fields defined on the Replace tab if they do not exist.

  These new options replace the Include without Processing option. Upgraded pipelines are set to Add New Field. This can have upgrade impact.

• Google BigQuery destination and executor - These stages are no longer part of an Enterprise stage library. This includes the following updates:
  • The stages, previously known as the Google BigQuery (Enterprise) destination and the Google BigQuery (Enterprise) executor, are renamed to the Google BigQuery destination and Google BigQuery executor.
  • The stages are now available in the Google Cloud stage library, streamsets-datacollector-google-cloud-lib, and are available to install like any other Data Collector stage. This change has upgrade impact.
• Google BigQuery destination:
  • The destination can write nested Avro data.
  • The destination can use BigQuery to generate schemas.
  • The destination now allows enabling the Create Table property when the destination is not configured to handle data drift.
• JDBC Multitable Consumer origin - You can no longer set the Minimum Idle Connections property higher than the Number of Threads property.

  Upgraded pipelines have the Minimum Idle Connections property set to the same value as the Number of Threads property. This can have upgrade impact.

• Kafka message header support - The following functionality is available when using a Kafka Java client version 0.11 or later:
  • The Kafka Multitopic Consumer origin includes Kafka message headers as record header attributes.

  • The Kafka Producer destination includes all user-defined record header attributes as Kafka message headers when writing to Kafka.

• MongoDB Atlas destination - The destination can update nested documents.
• OPC UA Client origin - The origin includes a new Override Host property which overrides the host name returned from the OPC UA server with the host name configured in the resource URL.
• Oracle CDC Client origin - The minimum value for the following advanced properties have changed from 1 millisecond to 0 milliseconds:
  • Time between Session Windows
  • Time after Session Window Start
• Start Job origin and processor - The Search Mode property enables the origin and processor to search for the Control Hub job to start.

• Google BigQuery update - Due to the BigQuery change from enterprise, you must install the Google Cloud stage library, streamsets-datacollector-google-cloud-lib, to configure or use Google BigQuery connections. This change has upgrade impact.

• Data governance tools - Data governance tools support publishing metadata about the following additional stages:
  • HTTP Client origin
  • JDBC Multitable Consumer origin
• Advanced data format properties - Optional properties for data formats have become advanced options. You must now view advanced options to configure them.
• Delimited data format enhancements:
  • When writing delimited data using a destination or the HTTP Client processor, you can define properties to enable writing multicharacter delimited data.
  • When writing delimited data using a custom delimiter format, you can configure the Record Separator String property to define a custom record separator.
• Pipeline start events - Pipeline start event records include a new email field that contains the email address of the user who started the pipeline.

• Add New Field - Adds the fields defined on the Replace tab to records if they do not exist.
• Ignore New Field - Ignores any fields defined on the Replace tab if they do not exist.

• MongoDB Atlas origin and destination - You can use the new MongoDB Atlas origin and destination to read from and write to MongoDB Atlas and MongoDB Enterprise Server.

• Groovy stages - The Groovy Scripting origin and the Groovy Evaluator processor now support Groovy 4.0.
• JDBC Multitable Consumer origin - The origin now provides the jdbc.primaryKeySpecification record header attribute for records from tables with a primary key and the jdbc.vendor record header attribute for all records.
• JDBC Tee processor and JDBC Producer destination - These stages can manage primary key values updates using the jdbc.primaryKey.before.columnName record header attribute for the old value and the jdbc.primaryKey.before.columnName record header attribute for the new value.
• MQTT stages - The MQTT Subscriber origin and the MQTT Publisher destination now support entering a list of brokers from high availability MQTT clusters without a load balancer.
• Oracle CDC Client origin:
  • Conditional Blob and Clob support - When the origin buffers changes locally, you can configure the origin to process Blob and Clob data using the following advanced properties:
    • Enable Blob and Clob Columns Processing property - Enable this property to process Blob and Clob columns.
    • Maximum LOB Size property - Optional property to define the maximum LOB size. When specified, overflow data is discarded.
  • LogMiner Query Timeout property - This property defines how long the origin waits for a LogMiner query to complete.
  • Time between Session Windows property - This advanced property sets the time to wait after a LogMiner session has been completely ingested. This ensures a minimum LogMiner window size.
  • Time after Session Window Start property - This advanced property sets the time to wait after a LogMiner session starts. This allows Oracle to finish setting up before processing begins.
• Pipeline Finisher executor - The executor includes new React to Events and Event Type properties that enables the executor to stop a pipeline only upon receiving the specified event record type.

  For example, you can now configure the executor to stop the pipeline only after receiving a no-more-data event record, and to ignore all other records that it might receive. Previously, you might have used a precondition or a Filter processor to ensure that the executor received only no-more-data events.

• Snowflake stages - Snowflake stages have been updated to support all Snowflake regions.
• SQL Server CDC Client origin - The origin can be configured to combine the two update records that SQL Server creates into a single record and generate the records differently.

  With this property enabled, the origin generates record header attributes about the primary key.

• SQL Server Change Tracking origin - The origin generates record header attributes about the primary key.

• MongoDB Atlas origin
• MongoDB Atlas destination
• SQL Parser processor

This release includes the following new stage libraries:

• streamsets-datacollector-groovy_4_0-lib

• streamsets-datacollector-mongodb_atlas-lib

• New Data Collector configuration property - To cache Domain Name Service (DNS) lookups, you can use the new networkaddress.cache.ttl property in the $SDC_DIST/etc/sdc-java-security.properties file.

  With this change, the java.security.networkaddress.cache.ttl Data Collector property has been deprecated.

• Help options - The Local Help option has been removed from the Help configuration option in Help > Settings. When you view Data Collector help, you will always view it on the StreamSets documentation website: https://docs.streamsets.com.

With 5.2.0, the MySQL Binary Log origin converts MySQL Enum and Set fields to String fields.

In previous releases, when reading from a database where the binlog_row_metadata MySQL database property is set to MINIMAL, Enum fields are converted to Long, and Set fields are converted to Integer.

In 5.2.0 as well as previous releases, when the binlog_row_metadata MySQL database property is set to FULL, Enum and Set fields are converted to String.

With 5.2.0, the Oracle CDC Client origin has new advanced properties that enable processing Blob and Clob columns. You can use these properties when the origin buffers changes locally. They are disabled by default.

In previous releases, the origin does not process Blob or Clob columns. However, when the Unsupported Fields to Records property is enabled, the origin includes Blob and Clob field names and raw string values in records.

• Pulsar Consumer origin - The new Pulsar Consumer origin can use multiple threads to read from Pulsar. The origin supports schema validation and Pulsar namespaces configured to enforce schema validation. You can specify the schema used to determine compatibility between the origin and a Pulsar topic. You can also use JWT authentication with the new origin.

  With this new origin, the existing Pulsar Consumer has been renamed Pulsar Consumer (Legacy). Use this new Pulsar origin for all new development.

• Aurora PostgreSQL CDC Client and PostgreSQL CDC Client origins - Both origins can now generate a record for each individual operation.

  Previously, the origins could only generate a record for each transaction.

• Aurora PostgreSQL CDC Client, PostgreSQL CDC Client, and MySQL Binary Log origins - These origins include the following new record header attributes when a table includes a primary key:
  • jdbc.primaryKeySpecification - Includes a JSON-formatted string that lists the columns that form the primary key in the table and the metadata for those columns.
  • jdbc.primaryKey.before.<primary key column> - Includes the previous value for the specified primary key column.
  • jdbc.primaryKey.after.<primary key column> - Includes the new value for the specified primary key column.

• Kafka Multitopic Consumer origin and Kafka Producer destination - The stages include a new Custom Authentication security option that enables specifying custom properties that contain the information required by a security protocol, rather than using predefined properties associated with other security options.
• OPC UA Server origin - The origin now supports using a user name and password to authenticate with the OPC UA server, in addition to an anonymous log in.
• Oracle CDC Client origin:
  • The origin now supports reading from Oracle 21c databases.
  • The field order of generated records now matches the column order in database tables. Previously, the field order was not guaranteed.
  • When you configure the origin to use local buffers and write to disk, you can specify an existing directory to use.
  • A new Data Collector configuration property affects the origin and can have upgrade impact. For details, see Upgrade Impact.
• Pulsar Consumer (Legacy) origin:
  • The origin, formerly named Pulsar Consumer, has been renamed with this release.

    This change has no upgrade impact. However, we recommend using the new Pulsar Consumer origin, which supports multithreaded processing, to read from Pulsar.

  • You can specify the schema used to determine compatibility between the origin and a Pulsar topic.
  • You can also use JWT authentication with the origin.
• Salesforce Bulk API 2.0 stages:
  • All Salesforce Bulk API 2.0 stages include a new Salesforce Query Timeout property which defines the number of seconds that the stage waits for a response to a query.
  • The Salesforce Bulk API 2.0 origin and Salesforce Bulk API 2.0 Lookup processor both include a new Maximum Query Columns property that limits the number of columns that can be returned by a query.
• Scripting stages - The Groovy, JavaScript, and Jython Evaluator origins and processors now generate metrics for script execution and locking details that you can view when monitoring the pipeline.
• Field Remover - The processor now includes the On Record Error property on the General tab.
• Field Type Converter processor - When converting a Date, Datetime, or Time field, the Date Format property now offers explicit options to specify that the field contains a Unix timestamp in milliseconds or seconds.

  If the field contains a Unix timestamp and you select an alternate date format, then the behavior is unchanged: the processor assumes the timestamps are in milliseconds.

• SQL Parser processor:
  • The processor adds the fields from the SQL statement in the same order as the corresponding columns in the database tables.
  • The processor now includes field attributes for columns converted to the Decimal or Datetime data types in Data Collector. The attributes provide additional information for each field.
• Pulsar Producer destination - You can now use JWT authentication with the destination.

• Kafka connection - The connection includes a new Custom Authentication security option that enables specifying custom properties that contain the information required by a security protocol, rather than using predefined properties associated with other security options.
• OPC UA connection - The connection now supports using a user name and password to authenticate with the OPC UA server in addition to an anonymous log in to the server.
• Pulsar connection - The connection can now use JWT authentication to connect to Pulsar.
• Snowflake connection - The new Connection Properties property enables you to specify additional connection properties for Snowflake connections.

• Azure Synapse 1.3.0
• Databricks 1.7.0
• Snowflake 1.12.0

• Data Collector Docker image - The Docker image for Data Collector 5.1.0, streamsets/datacollector:5.1.0, uses Ubuntu 20.04 LTS (Focal Fossa) as a parent image. This change can have upgrade impact.
• Microsoft JDBC Driver for SQL Server - Data Collector uses version 10.2.1 of the driver to connect to Microsoft SQL Server. Due to changes in the driver, this can have upgrade impact.
• Data Collector configuration properties - You can define the following new configuration properties:
  • stage.conf_com.streamsets.pipeline.lib.jdbc.disableSSL - When this configuration property is set to true, Data Collector attempts to disable SSL for all JDBC connections. This property is commented out by default.
  • stage.conf_com.streamsets.pipeline.stage.origin.jdbc.cdc.oracle.monitorbuffersize - When this configuration property is set to true, Data Collector reports memory consumption when the Oracle CDC Client origin uses local buffers.

    This property is set to false by default. In previous releases, the origin reported this information by default, so this enhancement has upgrade impact.

• streamsets-datacollector-apache-kafka_3_0-lib - For Apache Kafka 3.0.
• streamsets-datacollector-apache-kafka_3_1-lib - For Apache Kafka 3.1.
• streamsets-datacollector-apache-kafka_3_2-lib - For Apache Kafka 3.2.

• sdc-log4j2.properties
• log4j2.component.properties

• Red Hat Enterprise Linux 8.x - Data Collector now supports installation on RHEL 8.x, in addition to 6.x and 7.x.

• InfluxDB 2.x destination - Use the destination to write to InfluxDB 2.x databases.

• Couchbase Lookup processor property name updates - For clarity, the following property names have been changed:
  • Property Name is now Sub-Document Path.
  • Return Properties is now Return Sub-Documents.
  • SDC Field is now Output Field.
  • When performing a key value lookup and configuring multiple return properties, the Property Mappings property is now Sub-Document Mappings.
  • When performing an N1QL lookup and configuring multiple return properties, the Property Mappings property is now Sub-N1QL Mappings.
• Einstein Analytics destination enhancements:
  • The Einstein Analytics destination has been renamed the Tableau CRM destination to match the Salesforce rebranding.
  • The new Tableau CRM destination can perform automatic recovery.
• HTTP Client stage statistics - HTTP Client stages provide additional metrics when you monitor the pipeline.
• PostgreSQL CDC Client origin - You can specify the SSL mode to use on the new Encryption tab of the origin.
• Salesforce destination - The destination supports performing hard deletes when using the Salesforce Bulk API. Hard deletes permanently delete records, bypassing the Salesforce Recycle Bin.
• Salesforce stages - Salesforce stages now use version 53.0.0 of the Salesforce API by default.
• SFTP/FTP/FTPS stages - All SFTP/FTP/FTPS Client stages now support HTTP and Socks proxies.

• When Data Collector version 5.6.0 is registered with Control Hub cloud or with Control Hub on-premises version 3.19.x or later, the following stage supports using Control Hub connections:
  • Cassandra destination
• SFTP/FTP/FTPS enhancement - The SFTP/FTP/FTPS connection allows configuring the new SFTP/FTP/FTPS proxy properties.

• Enabling HTTPS for Data Collector - You can now store the keystore file in the Data Collector resources directory, $SDC_RESOURCES, and then enter a path relative to that directory when you define the keystore location. This can have upgrade impact.
• Google Secret Manager enhancement - You can configure a new enforceEntryGroup Google Secret Manager credential store property to validate a user’s group against a comma-separated list of groups allowed to access each secret.

• Google Cloud Storage executor - You can use this executor to create new objects, copy or move objects, or add metadata to new or existing objects.

• Amazon stages - When you configure the Region property, you can select from several additional regions.
• Kudu stages - The default value for the Maximum Number of Worker Threads property is now 2. Previously, the default was 0, which used the Kudu default.

  Existing pipelines are not affected by this change.

• Orchestration stages - You can use an expression when you configure the Control Hub URL property in orchestration stages.
• Salesforce stages - All Salesforce stages now support using version 52.2.0 of the Salesforce API.
• Scripting processors - In the Groovy Evaluator, JavaScript Evaluator, and Jython Evaluator processors, you can select the Script Error as Record Error property to have the stage handle script errors based on how the On Record Error property is configured for the stage.

• Google Cloud Storage origin - You can configure post processing actions to take on objects that the origin reads.
• MySQL Binary Log origin - The origin now recovers automatically from the following issues:
  • Lost, damaged, or unestablished connections.
  • Exceptions raised from MySQL Binary Log being out-of-sync in some cluster nodes, or from being unable to communicate with the MySQL Binary Log origin.
• Oracle CDC Client origin:
  • The origin includes a Batch Wait Time property that determines how long the origin waits for data before sending an empty or partial batch through the pipeline.
  • The origin provides additional LogMiner metrics when you monitor a pipeline.
• RabbitMQ Consumer origin - You can configure the origin to read from quorum queues by adding x-queue-type as an Additional Client Configuration property and setting it to quorum.

• SQL Parser processor - You can configure the processor to use the Oracle PEG parser instead of the default parser.

• Google BigQuery (Legacy) destination - The destination now supports writing Decimal data to Google BigQuery Decimal columns.
• MongoDB destination - You can use the Improve Type Conversion property to improve how the destination handles date and decimal data.
• Splunk destination - You can use the Additional HTTP Headers property to define additional key-value pairs of HTTP headers to data written to Splunk.

• New Google Secret Manager support - You can use Google Secret Manager as a credential store for Data Collector.
• CyberArk enhancement - You can configure the credentialStore.cyberark.config.ws.proxyURI property to allow defining the URI for the proxy that should be used to reach the CyberArk services.

• Google

• When Data Collector version 5.6.0 is registered with Control Hub cloud or with Control Hub on-premises version 3.19.x or later, the following stages support using Control Hub connections:
  • MongoDB stages
  • RabbitMQ stages
  • Redis stages

• Salesforce enhancement - The Salesforce connection includes the following role properties:
  • Use Snowflake Role
  • Snowflake Role Name

• Excel data format enhancement - Stages that support reading the Excel data format include an Include Cells With Empty Value property to include empty cells in records.

• Control Hub orchestration stages - Orchestration stages use API credentials to connect to Control Hub in the StreamSets platform. This affects the following stages:
  • Start Jobs origin
  • Control Hub API processor
  • Start Jobs processor
  • Wait for Jobs processor
• Kafka stages - Kafka stages include an Override Stage Configurations property that enables custom Kafka properties defined in the stage to override other stage properties.

  This can impact existing pipelines.

• MapR Streams stages - MapR Streams stages also include an Override Stage Configurations property that enables the additional MapR or Kafka properties defined in the stage to override other stage properties.

  This can impact existing pipelines.

• Salesforce stages - The Salesforce origin, processor, destination, and the Tableau CRM destination include the following new timeout properties:
  • Connection Handshake Timeout
  • Subscribe Timeout
• Oracle CDC Client origin:
  • You can now remove Oracle pseudocolumns from parsed redo logs and place the information in record header attributes.
  • The origin includes an oracle.cdc.oracle.pseudocolumn.<pseudocolumn name> attribute for each pseudocolumn in the original statement.
  • Starting with version 4.0.1, the origin includes a Batch Wait Time property.
• Field Type Converter processor - The Source Field is Empty property enables to you specify the action to take when an input field is an empty string.
• HTTP Client processor:
  • Two Pass Records properties allow you to pass a record through the pipeline when all retries fail for per-status actions and for timeouts.
  • The following record header attributes are populated when you use one of the Pass Records properties:
    • httpClientError
    • httpClientStatus
    • httpClientLastAction
    • httpClientTimeoutType
    • httpClientRetries
• SQL Parser processor:
  • You can now remove Oracle pseudocolumns from parsed redo logs and place the information in record header attributes.
  • The processor includes an oracle.cdc.oracle.pseudocolumn.<pseudocolumn name> attribute for each pseudocolumn in the original statement.

• Oracle CDC Client origin
• SQL Server CDC Client origin
• SQL Server Change Tracking Client origin

• SDC_EXTERNAL_RESOURCES environment variable - An optional root directory for external resources, such as custom stage libraries, external libraries, and runtime resources.

  The default location is $SDC_DIST/externalResources.

• Support Bundle - Support bundles now include the System Messages log file when you include log files in the bundle.