Supported Systems and Versions

Data Collector supports working with a wide range of external systems. StreamSets tests to verify that Data Collector performs without issues when working with those systems.

The following tables list the systems that Data Collector supports and tests, and the stages that work with those systems.

Cloud Native

Data Collector supports the cloud native providers listed in the following table. StreamSets tests the listed stages on the specified environments.

Customers with an enterprise account can receive help with the listed stages on the tested environment.
Note: Some of the following supported and tested versions have been deprecated. For a list, see Deprecated Functionality.
Supported Cloud Provider Stages Tested Environment Java Versions
Amazon Origins:
  • Amazon S3
  • Amazon SQS Consumer
  • Kinesis Consumer
Destinations:
  • Amazon S3
  • Kinesis Firehose
  • Kinesis Producer
Executor:
  • Amazon S3

Credential Store:

  • Amazon Secrets Manager

 AWS All supported versions
Databricks (Runtime 6.x, 7.x, 8.x, 11.x, 12.x) Databricks Delta Lake destination

Databricks Job Launcher executor

Databricks Query executor

 Databricks Delta Lake Runtime 6.x, 7.x, 8.x, 11.x, 12.x All supported versions
Google Cloud Storage Origins:
  • Google BigQuery
  • Google Cloud Storage
  • Google Pub/Sub Subscriber
Destinations:
  • Google BigQuery (Legacy)
  • Google BigQuery
  • Google Cloud Storage
  • Google Pub/Sub Subscriber
Executor:
  • Google BigQuery
  • Google Cloud Storage
Credential Store:
  • Google Secret Manager
Google Cloud Storage All supported versions
Microsoft Azure Origins:
  • Azure Blob Storage
  • Azure Data Lake Storage Gen1
  • Azure Data Lake Storage Gen2
  • Azure Data Lake Storage Gen2 (Legacy)
  • Azure IoT/Event Hub Consumer
Destinations:
  • Azure Blob Storage
  • Azure Data Lake Storage Gen1
  • Azure Data Lake Storage Gen2
  • Azure Event Hub Producer
  • Azure IoT Hub Producer
  • Azure Synapse SQL
Executors:
  • ADLS Gen1 File Metadata
  • ADLS Gen2 File Metadata
Credential Store:
  • Azure Key Vault
 Microsoft Azure All supported versions
MongoDB Atlas MongoDB Atlas origin

MongoDB Atlas CDC origin

MongoDB Atlas Lookup processor

MongoDB Atlas destination

 MongoDB Atlas All supported versions
Salesforce Origins:
  • Salesforce
  • Salesforce Bulk API 2.0
Processors:
  • Salesforce Lookup
  • Salesforce Bulk API 2.0 Lookup
Destinations:
  • Salesforce
  • Salesforce Bulk API 2.0
  • Tableau CRM
Salesforce All supported versions
Snowflake Snowflake Bulk origin

Snowflake destination

Snowflake File Uploader destination

Snowflake executor

 Amazon S3

Google Cloud Storage

Microsoft Azure

 All supported versions

Protocols

Data Collector supports the protocols listed in the following table. StreamSets tests the listed stages on the specified environments.

Customers with an enterprise account can receive help with the following protocols unless the implementation proves below standard for the protocol.

Private extensions for the protocols are not supported unless specified in the table.
Supported Protocol Stages Tested Environment Java Versions
CoAP CoAP Server origin

CoAP Client destination

 Eclipse Californium 1.0.4 All supported versions
HTTP Origins:
  • HTTP Client
  • HTTP Server
  • NiFi HTTP Server
Processors:
  • HTTP Client
  • HTTP Router
Destination:
  • HTTP Client
 Apache HTTP from Centos 6.8 All supported versions
HTTP Origin:
  • Web Client
Processor:
  • Web Client
Destination:
  • Web Client
 Flask

FastAPI

 Java 17
JMS JMS Consumer origin

JMS Producer destination

 ActiveMq 5.14.3 All supported versions
MQTT MQTT Subscriber origin

MQTT Publisher destination

 Mosquitto All supported versions
OPC UA OPC UA Client origin OPC Unified Architecture 1.03 All supported versions
SFTP/ FTP / FTPS SFTP/FTP/FTPS Client origin

SFTP/FTP/FTPS Client destination

SFTP/FTP/FTPS Client executor

 vsftpd 3.0 All supported versions
Syslog Syslog destination Full testing not performed at this time All supported versions
TCP TCP Server origin Java TCP Stack All supported versions
UDP UDP Multithreaded Source origin

UDP Source origin

 Java UDP Stack All supported versions
Websocket Origins:
  • WebSocket Client
  • WebSocket Server
Destination:
  • WebSocket Client
 Java HTTP Stack All supported versions

Versioned Systems

Versioned systems are external systems with multiple versions. When Data Collector supports multiple versions of an external system, you might need to install a specific stage library to work with a particular version, depending on your Data Collector installation. For details on individual stage libraries and the stages that they include, see Available Stage Libraries.

The following table lists the system versions that are supported and tested for Data Collector.

The supported versions column lists the system versions that customers with an enterprise account can receive help with. The tested versions column lists the subset of the supported versions that have been fully tested.

Note: Some of the following supported and tested versions have been deprecated. For a list, see Deprecated Functionality.
System Stages Supported Versions Tested Versions Java Versions
Aerospike Aerospike destination Aerospike 3.15.x Full testing not performed at this time All supported versions
Aerospike Client destination Aerospike 6.x Aerospike 6.3 EE All supported versions
Cassandra Cassandra destination Cassandra 1.2, 2.x, 3.x Cassandra 3.11 All supported versions
Couchbase Server Couchbase origin

Couchbase Lookup processor

Couchbase destination

 Couchbase Server 7.x Couchbase Server 7.2.2 All supported versions
Elasticsearch Elasticsearch origin

Elasticsearch destination
  • Elasticsearch 5.x - 8.x
  • Amazon OpenSearch Service that runs Elasticsearch 5.x - 7.10
 Elasticsearch 5.20, 6.8.12, 7.9.0, 8.1.1 All supported versions
Flume Flume destination
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
 All supported versions
Greenplum GPSS Producer destination Greenplum 5.x Greenplum 5.12.0 All supported versions
Hadoop Distributed File System (HDFS):

Data Collector cluster mode
Origin:
  • Hadoop FS
Destination:
  • Hadoop FS
Executors:
  • HDFS File Metadata
  • MapReduce
  • Amazon EMR 5.14.x with Hadoop 2.8.3.
 Full testing not performed at this time All supported versions
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
 Java 8
Hadoop Distributed File System (HDFS):

Data Collector standalone mode
Origin:
  • Hadoop FS Standalone
Destination:
  • Hadoop FS
Executors:
  • HDFS File Metadata
  • MapReduce
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
 Java 8
Hashicorp Vault Hashicorp Vault credential store General support Full testing not performed at this time All supported versions
HBase HBase Lookup processor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
 Java 8
Hive Hive Metadata processor

Hive Metastore destination

Hive Query executor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.x distribution of Hive 2.1
  • HDP 2.6.x distribution of Hive 1.x.
  • HDP 3.1.x
  • MapR 6.0.0 with EEP 4.x
  • MapR 6.0.1 with EEP 5.x
  • MapR 6.1.x with EEP 6.x
  • HPE Ezmeral Data Fabric 7.0.x with EEP 8.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
 Java 8
Hive Streaming Hive Streaming destination

Hive Query executor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • MapR 6.0.0 with EEP 4.x
  • MapR 6.0.1 with EEP 5.x
  • MapR 6.1.x with EEP 6.x
  • HPE Ezmeral Data Fabric 7.0.x with EEP 8.x
 Full testing not performed at this time Java 8
InfluxDB InfluxDB destination InfluxDB 0.9 - 1.x InfluxDB 0.13, 1.7.10 All supported versions
InfluxDB 2.x destination InfluxDB 2.x InfluxDB 2.0.8 All supported versions
Java Keystore Java Keystore credential store Java Virtual Machine Java Virtual Machine All supported versions
Jira Jira origin

Jira destination

 Jira REST API version 2 Jira REST API version 2 Java 17
Kafka:

Data Collector cluster mode

 Kafka Consumer origin
  • CDH 6.0.x - 6.3.x
  • CDH Kafka 3.1.x, 4.1.x with:
    • CDS powered by Spark 2.2 release 1
    • CDS powered by Spark 2.3 release 2, 3, 4
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.0
  • CDP Private Cloud Base 7.1.x
 Java 8
Kafka:

Data Collector standalone mode

 Origins:
  • Kafka Consumer
  • Kafka Multitopic Consumer
Destination:
  • Kafka Producer
  • Apache Kafka 1.0.x, 1.1.x, 2.0.x - 2.8.x, 3.0.x - 3.7.x
  • Apache Kafka 1.0.x, 1.1.x, 2.0.x - 2.8.x, 3.0.x - 3.7.x
 All supported versions
  • CDH 6.0.x - 6.3.x
  • CDH Kafka 3.1.x, 4.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.0
  • CDH Kafka 2.1.0, 3.0.0, 3.1.0, 4.1.0
  • CDP Private Cloud Base 7.1.x

  • HDP 3.1.0

 Java 8
KineticaDB Kinetica destination
  • KineticaDB 6.0.x - 6.2.x
  • KineticaDB 7.0.x
 Full testing not performed at this time All supported versions
Kudu Kudu Lookup processor

Kudu destination
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
 Java 8
MariaDB Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 MariaDB 10.2 - 10.7 as limited drop-in replacements for MySQL 5.7.

For more information, see the MariaDB documentation.

Also supported:

  • Hosted systems, such as SkySQL, that run supported versions of MariaDB

  • Derived systems that run supported versions of MariaDB
 MariaDB 10.7 All supported versions
MapR DB Origin:
  • MapR DB
Destinations:
  • MapR DB
  • MapR DB JSON
  • MapR 6.0.0 with optional EEP 4.x
  • MapR 6.0.1 with optional EEP 5.x
  • MapR 6.1.x with optional EEP 6.x
  • MapR 6.0.0 with EEP 4
  • MapR 6.0.1 with EEP 5
  • MapR 6.1.x with EEP 6
 Java 8
  • HPE Ezmeral Data Fabric 7.0.x with optional EEP 8.x
  • HPE Ezmeral Data Fabric 7.0 with EEP 8
 Java 11, 17
MapR FS:

Data Collector cluster mode

 MapR FS origin

MapR FS destination
  • MapR 6.0.0 with EEP 4.x
  • MapR 6.0.1 with EEP 5.x
  • MapR 6.1.x with EEP 6.x
  • MapR 6.0.0 with EEP 4
  • MapR 6.0.1 with EEP 5
  • MapR 6.1.x with EEP 6
 Java 8
  • HPE Ezmeral Data Fabric 7.0.x with EEP 8.x
 Full testing not performed at this time Java 11, 17
MapR FS:

Data Collector standalone mode

 MapR FS Standalone origin

MapR FS destination

MapReduce executor
  • MapR 6.0.0 with optional EEP 4.x
  • MapR 6.0.1 with optional EEP 5.x
  • MapR 6.1.x with optional EEP 6.x
  • MapR 6.0.0 with EEP 4
  • MapR 6.0.1 with EEP 5
  • MapR 6.1.x with EEP 6
 Java 8
MapR Streams Origins:
  • MapR Multitopic Streams Consumer
  • MapR Streams Consumer
  • MapR DB CDC
Destination:
  • MapR Streams Producer
  • MapR 6.1.x with optional EEP 6.x
  • MapR 6.1.x with EEP 6
 Java 8
  • HPE Ezmeral Data Fabric 7.0.x with optional EEP 8.x
  • HPE Ezmeral Data Fabric 7.0 with EEP 8
 Java 11, 17
MemSQL MemSQL Fast Loader destination MemSQL 6.8 and later MemSQL 6.8.15 with the MySQL Connector/J 8.0.12 driver All supported versions
Microsoft SQL Server SQL Server 2019 BDC origin SQL Server 2019 Big Data Cluster SQL Server 2019 Big Data Cluster All supported versions
SQL Server CDC Client origin

SQL Server Change Tracking origin
  • SQL Server 2017
  • SQL Server 2019

Hosted systems and derived systems are not supported.
  • SQL Server 2017
  • SQL Server 2019
 All supported versions
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 SQL Server 2017 and later
Also supported:
  • Hosted systems, such as Azure SQL, that run supported versions of SQL Server
  • Derived systems that run supported versions of SQL Server
  • SQL Server 2017
  • SQL Server 2019
 All supported versions
MongoDB Origins:
  • MongoDB
  • MongoDB Oplog
Processor:
  • MongoDB Lookup
Destination:
  • MongoDB
 MongoDB 3.x, 4.x MongoDB 3.6, 4.0 All supported versions
Origin:
  • MongoDB Atlas
Processor:
  • MongoDB Atlas Lookup
Destination:
  • MongoDB Atlas
 MongoDB 3.x, 4.x, 5.x, 6.x, 7.x MongoDB 3.6, 4.0, 6.0.6, 7.1 All supported versions
MySQL Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 MySQL 5.7 and later
Also supported:
  • Hosted systems, such as Amazon Aurora, that run supported versions of MySQL
  • Derived systems, such as MySQL Cluster, that run supported versions of MySQL
  • MySQL 5.7 with the MySQL Connector/J 8.0.12 driver
  • MySQL 8.0 with the MySQL Connector/J 8.0.12 driver
 All supported versions
MySQL Binary Log MySQL 5.7 and later

Also supported:

  • Hosted systems, such as Amazon Aurora, that run supported versions of MySQL
  • MySQL 5.7 with the MySQL Connector/J 8.0.12 driver
  • MySQL 8.0 with the MySQL Connector/J 8.0.12 driver
 All supported versions
NiFi NiFi HTTP Server origin General support Full testing not performed at this time All supported versions
Omniture Omniture origin General support Full testing not performed at this time All supported versions
Oracle Origin:
  • Oracle Multitable Consumer
Destination:
  • Oracle
 Multitenant and non-CDB databases of the following versions:
  • Oracle 11g Release 2, 12c, 18c, 19c, 21c
  • Oracle Real Application Clusters (RAC) 11g Release 2, 12c, 18c, 19c, 21c
Also supported:
  • Hosted systems, such as Amazon RDS, that run supported versions of Oracle RAC
  • Derived systems, such as Oracle Exadata, that run supported versions of Oracle RAC

  • Oracle 11g Release 2, 19c, 21c

  • Oracle RAC 12c

All testing is performed with the Oracle 21.8.0.0 JDBC driver.

 All supported versions
Oracle Bulkload origin Multitenant and non-CDB databases of the following versions:
  • Oracle 11g Release 2, 12c, 18c, 19c

Hosted systems and derived systems are not supported.
  • Oracle 11g Release 2, 19c with the Oracle 21.8.0.0 JDBC driver version
 All supported versions
Oracle CDC origin Multitenant and non-CDB databases of the following versions:
  • Oracle 12c, 18c, 19c, 21c
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c, 21c
  • Oracle Exadata appliances that run supported versions of Oracle RAC
Also supported:
  • Standby databases on supported versions

Hosted systems and derived systems are not supported unless listed by name above.

  • Oracle 19c, 21c
  • Oracle RAC 12c
  • Primary and standby databases on Oracle 19c

All testing is performed with the Oracle 21.8.0.0 JDBC driver.

 All supported versions
Oracle CDC Client origin Multitenant and non-CDB databases of the following versions:
  • Oracle 11g Release 2, 12c, 18c, 19c, 21c
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c, 21c
  • Oracle Exadata appliances that run supported versions of Oracle RAC

Hosted systems and derived systems are not supported unless listed by name above.

  • Oracle 11g Release 2, 19c, 21c
  • Oracle RAC 12c, 19c

All testing is performed with the Oracle 21.8.0.0 JDBC driver.

 All supported versions
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
  • Oracle
Executor:
  • JDBC Query
 Multitenant and non-CDB databases of the following versions:
  • Oracle 11g Release 2, 12c, 18c, 19c, 21c
  • Oracle Real Application Clusters (RAC) 11g Release 2, 12c, 18c, 19c, 21c
Also supported:
  • Hosted systems, such as Amazon RDS, that run supported versions of Oracle RAC
  • Derived systems, such as Oracle Exadata, that run supported versions of Oracle RAC

  • Oracle 11g Release 2, 19c, 21c

  • Oracle RAC 12c

All testing is performed with the Oracle 21.8.0.0 JDBC driver.

 All supported versions
PMML PMML Evaluator processor General support Full testing not performed at this time All supported versions
PostgreSQL Aurora PostgreSQL CDC Client origin
  • Aurora PostgreSQL 2.2.0 (with PostgreSQL 10.6 - 10.17) and later
  • Aurora PostgreSQL 3 (with PostgreSQL 11.0 - 11.12)
  • Aurora PostgreSQL 4 (with PostgreSQL 12.0 - 12.7)
 Aurora PostgreSQL 4 (with PostgreSQL 12.7) All supported versions
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 PostgreSQL 9.x and later
Also supported:
  • Hosted systems, such as Amazon Aurora, that run supported versions of PostgreSQL
  • Derived systems that run supported versions of PostgreSQL
  • PostgreSQL 9.6.9
  • PostgreSQL 10.4
  • PostgreSQL 11.7
  • PostgreSQL 12.2
  • PostgreSQL 13.0
  • PostgreSQL 14.0
  • PostgreSQL 15.0
 All supported versions
PostgreSQL CDC Client origin
  • PostgreSQL 9.4 or later 9.x
  • PostgreSQL 10.x -13.x
  • PostgreSQL 9.6.9
  • PostgreSQL 10.4
  • PostgreSQL 11.7
  • PostgreSQL 12.2
  • PostgreSQL 13.0
  • PostgreSQL 14.0
  • PostgreSQL 15.0
 All supported versions
Pulsar Origins:
  • Pulsar Consumer
  • Pulsar Consumer (Legacy)
Destination:
  • Pulsar Producer
Pulsar 2.x
  • Pulsar 2.1.0
  • Pulsar 2.2.1
  • Pulsar 2.3.2
  • Pulsar 2.4.2
  • Pulsar 2.5.1
  • Pulsar 2.6.2
 All supported versions
RabbitMQ RabbitMQ Consumer origin

RabbitMQ Producer destination

 RabbitMQ 3.5.x and later RabbitMQ 3.5.6, 3.8.0 All supported versions
Redis Redis Consumer origin

Redis destination

 Redis 2.x - 4.x Redis 4.0.1 All supported versions
SAP HANA SAP HANA Query Consumer origin SAP HANA 2.4.x SAP HANA 2.0 with the SAP HANA JDBC driver version 2.4.76 All supported versions
SingleStore SingleStore destination SingleStore 1.1.4 SingleStore 1.1.4 All supported versions
Solr Solr destination
  • Apache Solr 6.x
Full testing not performed at this time All supported versions
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
 Java 8
Spark

Spark Evaluator processor

Spark executor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDH Spark 2.1.x Release 1
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
 Java 8
Splunk Splunk destination General support Full testing not performed at this time All supported versions
TensorFlow TensorFlow Evaluator processor TensorFlow 1.x Full testing not performed at this time All supported versions
Teradata Teradata destination Teradata VantageCloud Teradata VantageCloud with the Teradata JDBC driver version 20.00.00.15 All supported versions
Teradata Teradata Consumer origin Teradata 16.x and later Teradata Database release 16.20 with the Teradata JDBC driver version 16.20.00.08 All supported versions
Thycotic Secret Server Thycotic Secret Server credential store Full testing not performed at this time Full testing not performed at this time All supported versions