Supported Systems and Versions

Data Collector supports working with a wide range of external systems. StreamSets tests to verify that Data Collector performs without issues when working with those systems.

The following tables list the systems that Data Collector supports and tests, and the stages that work with those systems.

Cloud Native

Data Collector supports the cloud native providers listed in the following table. StreamSets tests the listed stages on the specified environments.

Customers with an enterprise account can receive help with the listed stages on the tested environment.
Note: Some of the following supported and tested versions have been deprecated. For a list, see Deprecated Functionality.
Supported Cloud Provider Stages Tested Environment
Amazon Origins:
  • Amazon S3
  • Amazon SQS Consumer
  • Kinesis Consumer
Destinations:
  • Amazon S3
  • Kinesis Firehose
  • Kinesis Producer
Executor:
  • Amazon S3

Credential Store:

  • Amazon Secrets Manager

 AWS
Databricks (Runtime 6.x, 7.x, or 8.x) Databricks Delta Lake destination

Databricks Job Launcher executor

Databricks Query executor

 Databricks Delta Lake Runtime 6.x, 7.x, or 8.x
Google Cloud Storage Origins:
  • Google BigQuery
  • Google Cloud Storage
  • Google Pub/Sub Subscriber
Destinations:
  • Google BigQuery (Legacy)
  • Google BigQuery
  • Google Cloud Storage
  • Google Pub/Sub Subscriber
Executor:
  • Google BigQuery
  • Google Cloud Storage
Credential Store:
  • Google Secret Manager
Google Cloud Storage
Microsoft Azure Origins:
  • Azure Blob Storage
  • Azure Data Lake Storage Gen1
  • Azure Data Lake Storage Gen2
  • Azure Data Lake Storage Gen2 (Legacy)
  • Azure IoT/Event Hub Consumer
Destinations:
  • Azure Data Lake Storage Gen1
  • Azure Data Lake Storage Gen2
  • Azure Event Hub Producer
  • Azure IoT Hub Producer
  • Azure Synapse SQL
Executors:
  • ADLS Gen1 File Metadata
  • ADLS Gen2 File Metadata
Credential Store:
  • Azure Key Vault
 Microsoft Azure
MongoDB Atlas MongoDB Atlas origin

MongoDB Atlas destination

 MongoDB Atlas
Salesforce Origins:
  • Salesforce
  • Salesforce Bulk API 2.0
Processors:
  • Salesforce Lookup
  • Salesforce Bulk API 2.0 Lookup
Destinations:
  • Salesforce
  • Salesforce Bulk API 2.0
  • Tableau CRM
Salesforce
Snowflake Snowflake Bulk origin

Snowflake destination

Snowflake File Uploader destination

Snowflake executor

 Amazon S3

Google Cloud Storage

Microsoft Azure

Protocols

Data Collector supports the protocols listed in the following table. StreamSets tests the listed stages on the specified environments.

Customers with an enterprise account can receive help with the following protocols unless the implementation proves below standard for the protocol.

Private extensions for the protocols are not supported unless specified in the table.
Supported Protocol Stages Tested Environment
CoAP CoAP Server origin

CoAP Client destination

 Eclipse Californium 1.0.4
HTTP Origins:
  • HTTP Client
  • HTTP Server
  • NiFi HTTP Server
Processors:
  • HTTP Client
  • HTTP Router
Destinations:
  • HTTP Client
 Apache HTTP from Centos 6.8
JMS JMS Consumer origin

JMS Producer destination

 ActiveMq 5.14.3
MQTT MQTT Subscriber origin

MQTT Publisher destination

 Mosquitto
OPC UA OPC UA Client origin Full testing not performed at this time
SFTP/ FTP / FTPS SFTP/FTP/FTPS Client origin

SFTP/FTP/FTPS Client destination

SFTP/FTP/FTPS Client executor

 vsftpd 3.0
Syslog Syslog destination Full testing not performed at this time
TCP TCP Server origin Java TCP Stack
UDP UDP Multithreaded Source origin

UDP Source origin

 Java UDP Stack
Websocket Origins:
  • WebSocket Client
  • WebSocket Server
Destination:
  • WebSocket Client
 Java HTTP Stack

Versioned Systems

Versioned systems are external systems with multiple versions. When Data Collector supports multiple versions of an external system, you might need to install a specific stage library to work with a particular version, depending on your Data Collector installation. For details on individual stage libraries and the stages that they include, see Available Stage Libraries.

The following table lists the system versions that are supported and tested for Data Collector.

The supported versions column lists the system versions that customers with an enterprise account can receive help with. The tested versions column lists the subset of the supported versions that have been fully tested.

Note: Some of the following supported and tested versions have been deprecated. For a list, see Deprecated Functionality.
System Stages Supported Versions Tested Versions
Aerospike Aerospike destination Aerospike 3.15.x Full testing not performed at this time
Aerospike Client destination Aerospike 6.x Aerospike 6.3 EE
Cassandra Cassandra destination Cassandra 1.2, 2.x, 3.x Cassandra 3.11
Couchbase Server Couchbase destination Couchbase Server 5.x Couchbase Server 5.1.1
Elasticsearch Elasticsearch origin

Elasticsearch destination

 Elasticsearch 5.x - 8.x Elasticsearch 5.20, 6.8.12, 7.9.0, 8.1.1
Flume Flume destination
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
Greenplum GPSS Producer destination Greenplum 5.x Greenplum 5.12.0
Hadoop Distributed File System (HDFS):

Data Collector cluster mode
Origin:
  • Hadoop FS
Destination:
  • Hadoop FS
Executors:
  • HDFS File Metadata
  • MapReduce
  • Amazon EMR 5.14.x with Hadoop 2.8.3.
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Hadoop Distributed File System (HDFS):

Data Collector standalone mode
Origin:
  • Hadoop FS Standalone
Destination:
  • Hadoop FS
Executors:
  • HDFS File Metadata
  • MapReduce
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Hashicorp Vault Hashicorp Vault credential store General support Full testing not performed at this time
HBase HBase Lookup processor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Hive Hive Metadata processor

Hive Metastore destination

Hive Query executor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.x distribution of Hive 2.1
  • HDP 2.6.x distribution of Hive 1.x.
  • HDP 3.1.x
  • MapR 6.0.0 with EEP 4.x
  • MapR 6.0.1 with EEP 5.x
  • MapR 6.1.x with EEP 6.x
  • HPE Ezmeral Data Fabric 7.0.x with EEP 8.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Hive Streaming Hive Streaming destination

Hive Query executor
  • Hive 0.13 and later
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • MapR 6.0.0 with EEP 4.x
  • MapR 6.0.1 with EEP 5.x
  • MapR 6.1.x with EEP 6.x
  • HPE Ezmeral Data Fabric 7.0.x with EEP 8.x
 Full testing not performed at this time
InfluxDB InfluxDB destination InfluxDB 0.9 - 1.x InfluxDB 0.13, 1.7.10
InfluxDB 2.x destination InfluxDB 2.x InfluxDB 2.0.8
Java Keystore Java Keystore credential store Java Virtual Machine Java Virtual Machine
Kafka:

Data Collector cluster mode

 Kafka Consumer origin
  • CDH 6.0.x - 6.3.x
  • CDH Kafka 3.1.x, 4.1.x with:
    • CDS powered by Spark 2.2 release 1
    • CDS powered by Spark 2.3 release 2, 3, 4
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.0
  • CDP Private Cloud Base 7.1.x
Kafka:

Data Collector standalone mode

 Origins:
  • Kafka Consumer
  • Kafka Multitopic Consumer
Destination:
  • Kafka Producer
  • Apache Kafka 1.0.x, 1.1.x, 2.0.x - 2.8.x, 3.0.x - 3.4.x
  • CDH 6.0.x - 6.3.x
  • CDH Kafka 3.1.x, 4.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.0
  • Apache Kafka 1.0.x, 1.1.x, 2.0.x - 2.8.x, 3.0.x - 3.4.x
  • CDH Kafka 2.1.0, 3.0.0, 3.1.0, 4.1.0
  • CDP Private Cloud Base 7.1.x

  • HDP 3.1.0
KineticaDB Kinetica destination
  • KineticaDB 6.0.x - 6.2.x
  • KineticaDB 7.0.x
 Full testing not performed at this time
Kudu Kudu Lookup processor

Kudu destination
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
MariaDB Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 MariaDB 10.2 - 10.7 as limited drop-in replacements for MySQL 5.7.

For more information, see the MariaDB documentation.

Also supported:

  • Hosted systems, such as SkySQL, that run supported versions of MariaDB

  • Derived systems that run supported versions of MariaDB
 MariaDB 10.7
MapR DB Origin:
  • MapR DB
Destinations:
  • MapR DB
  • MapR DB JSON
  • MapR 6.0.0 with optional EEP 4.x
  • MapR 6.0.1 with optional EEP 5.x
  • MapR 6.1.x with optional EEP 6.x
  • HPE Ezmeral Data Fabric 7.0.x with optional EEP 8.x
  • MapR 6.0.0 with EEP 4
  • MapR 6.0.1 with EEP 5
  • MapR 6.1.x with EEP 6
  • HPE Ezmeral Data Fabric 7.0 with EEP 8
MapR FS:

Data Collector cluster mode

 MapR FS origin

MapR FS destination
  • MapR 6.0.0 with EEP 4.x
  • MapR 6.0.1 with EEP 5.x
  • MapR 6.1.x with EEP 6.x
  • HPE Ezmeral Data Fabric 7.0.x with EEP 8.x
  • MapR 6.0.0 with EEP 4
  • MapR 6.0.1 with EEP 5
  • MapR 6.1.x with EEP 6
  • HPE Ezmeral Data Fabric 7.0 with EEP 8
MapR FS:

Data Collector standalone mode

 MapR FS Standalone origin

MapR FS destination

MapReduce executor
  • MapR 6.0.0 with optional EEP 4.x
  • MapR 6.0.1 with optional EEP 5.x
  • MapR 6.1.x with optional EEP 6.x
  • HPE Ezmeral Data Fabric 7.0.x with optional EEP 8.x
  • MapR 6.0.0 with EEP 4
  • MapR 6.0.1 with EEP 5
  • MapR 6.1.x with EEP 6
  • HPE Ezmeral Data Fabric 7.0 with EEP 8
MapR Streams Origins:
  • MapR Multitopic Streams Consumer
  • MapR Streams Consumer
  • MapR DB CDC
Destination:
  • MapR Streams Producer
  • MapR 6.1.x with optional EEP 6.x
  • HPE Ezmeral Data Fabric 7.0.x with optional EEP 8.x
  • MapR 6.1.x with EEP 6
  • HPE Ezmeral Data Fabric 7.0 with EEP 8
MemSQL MemSQL Fast Loader destination MemSQL 6.8 and later MemSQL 6.8.15 with the MySQL Connector/J 8.0.12 driver
Microsoft SQL Server SQL Server 2019 BDC origin SQL Server 2019 Big Data Cluster SQL Server 2019 Big Data Cluster
SQL Server CDC Client origin

SQL Server Change Tracking origin
  • SQL Server 2017
  • SQL Server 2019

Hosted systems and derived systems are not supported.
  • SQL Server 2017
  • SQL Server 2019
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 SQL Server 2017 and later
Also supported:
  • Hosted systems, such as Azure SQL, that run supported versions of SQL Server
  • Derived systems that run supported versions of SQL Server
  • SQL Server 2017
  • SQL Server 2019
MongoDB Origins:
  • MongoDB
  • MongoDB Oplog
Processor:
  • MongoDB Lookup
Destination:
  • MongoDB
 MongoDB 3.x, 4.x MongoDB 3.6, 4.0
Origin:
  • MongoDB Atlas
Destination:
  • MongoDB Atlas
 MongoDB 3.x, 4.x, 5.x, 6.x MongoDB 3.6, 4.0, 6.0.6
MySQL Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 MySQL 5.7 and later
Also supported:
  • Hosted systems, such as Amazon Aurora, that run supported versions of MySQL
  • Derived systems, such as MySQL Cluster, that run supported versions of MySQL
  • MySQL 5.7 with the MySQL Connector/J 8.0.12 driver
  • MySQL 8.0 with the MySQL Connector/J 8.0.12 driver
MySQL Binary Log MySQL 5.7 and later

Also supported:

  • Hosted systems, such as Amazon Aurora, that run supported versions of MySQL
  • MySQL 5.7 with the MySQL Connector/J 8.0.12 driver
  • MySQL 8.0 with the MySQL Connector/J 8.0.12 driver
NiFi NiFi HTTP Server origin General support Full testing not performed at this time
Omniture Omniture origin General support Full testing not performed at this time
Oracle Oracle Bulkload origin
  • Oracle 11g Release 2, 12c, 18c, 19c

Hosted systems and derived systems are not supported.
  • Oracle 11g Release 2, 19c with the Oracle 21.8.0.0 JDBC driver version
Oracle CDC origin
  • Oracle 12c, 18c, 19c, 21c
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c, 21c
  • Oracle Exadata appliances that run supported versions of Oracle RAC
Also supported:
  • Standby databases on supported versions

Hosted systems and derived systems are not supported unless listed by name above.

  • Oracle 19c, 21c
  • Oracle RAC 12c
  • Primary and standby databases on Oracle 19c

All testing is performed with the Oracle 21.8.0.0 JDBC driver.
Oracle CDC Client origin
  • Oracle 11g Release 2, 12c, 18c, 19c, 21c
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c, 21c
  • Oracle Exadata appliances that run supported versions of Oracle RAC

Hosted systems and derived systems are not supported unless listed by name above.

  • Oracle 11g Release 2, 19c, 21c
  • Oracle RAC 12c, 19c

All testing is performed with the Oracle 21.8.0.0 JDBC driver.
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
  • Oracle 11g Release 2, 12c, 18c, 19c, 21c
  • Oracle Real Application Clusters (RAC) 11g Release 2, 12c, 18c, 19c, 21c
Also supported:
  • Hosted systems, such as Amazon RDS, that run supported versions of Oracle RAC
  • Derived systems, such as Oracle Exadata, that run supported versions of Oracle RAC

  • Oracle 11g Release 2, 19c, 21c

  • Oracle RAC 12c

All testing is performed with the Oracle 21.8.0.0 JDBC driver.
PMML PMML Evaluator processor General support Full testing not performed at this time
PostgreSQL Aurora PostgreSQL CDC Client origin
  • Aurora PostgreSQL 2.2.0 (with PostgreSQL 10.6 - 10.17) and later
  • Aurora PostgreSQL 3 (with PostgreSQL 11.0 - 11.12)
  • Aurora PostgreSQL 4 (with PostgreSQL 12.0 - 12.7)
 Aurora PostgreSQL 4 (with PostgreSQL 12.7)
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 PostgreSQL 9.x and later
Also supported:
  • Hosted systems, such as Amazon Aurora, that run supported versions of PostgreSQL
  • Derived systems that run supported versions of PostgreSQL
  • PostgreSQL 9.6.9
  • PostgreSQL 10.4
  • PostgreSQL 11.7
  • PostgreSQL 12.2
  • PostgreSQL 13.0
  • PostgreSQL 14.0
  • PostgreSQL 15.0
PostgreSQL CDC Client origin
  • PostgreSQL 9.4 or later 9.x
  • PostgreSQL 10.x -13.x
  • PostgreSQL 9.6.9
  • PostgreSQL 10.4
  • PostgreSQL 11.7
  • PostgreSQL 12.2
  • PostgreSQL 13.0
  • PostgreSQL 14.0
  • PostgreSQL 15.0
Pulsar Origins:
  • Pulsar Consumer
  • Pulsar Consumer (Legacy)
Destination:
  • Pulsar Producer
Pulsar 2.x
  • Pulsar 2.1.0
  • Pulsar 2.2.1
  • Pulsar 2.3.2
  • Pulsar 2.4.2
  • Pulsar 2.5.1
  • Pulsar 2.6.2
RabbitMQ RabbitMQ Consumer origin

RabbitMQ Producer destination

 RabbitMQ 3.5.x and later RabbitMQ 3.5.6, 3.8.0
Redis Redis Consumer origin

Redis destination

 Redis 2.x - 4.x Redis 4.0.1
SAP HANA SAP HANA Query Consumer origin SAP HANA 2.4.x SAP HANA 2.0 with the SAP HANA JDBC driver version 2.4.76
SingleStore SingleStore destination SingleStore 1.1.4 SingleStore 1.1.4
Solr Solr destination
  • Apache Solr 6.x
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Spark

Spark Evaluator processor

Spark executor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDH Spark 2.1.x Release 1
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Splunk Splunk destination General support Full testing not performed at this time
TensorFlow TensorFlow Evaluator processor TensorFlow 1.x Full testing not performed at this time
Teradata Teradata Consumer origin Teradata 16.x and later Teradata Database release 16.20 with the Teradata JDBC driver version 16.20.00.08
Thycotic Secret Server Thycotic Secret Server credential store Full testing not performed at this time Full testing not performed at this time