Supported Systems and Versions

Data Collector supports working with a wide range of external systems. StreamSets tests to verify that Data Collector performs without issues when working with those systems.

The following tables list the systems that Data Collector supports and tests, and the stages that work with those systems.

Cloud Native

Data Collector supports the cloud native providers listed in the following table. StreamSets tests the listed stages on the specified environments.

Customers with an enterprise account can receive help with the listed stages on the tested environment.
Note: Some of the following supported and tested versions have been deprecated. For a list, see Deprecated Functionality.
Supported Cloud Provider Stages Tested Environment
Amazon Origins:
  • Amazon S3
  • Amazon SQS Consumer
  • Kinesis Consumer
Destinations:
  • Amazon S3
  • Kinesis Firehose
  • Kinesis Producer
Executor:
  • Amazon S3

Credential Store:

  • Amazon Secrets Manager

AWS
Databricks (Runtime 6.x, 7.x, or 8.x) Databricks Delta Lake destination

Databricks Job Launcher executor

Databricks Query executor

Databricks Delta Lake Runtime 6.x, 7.x, or 8.x
Google Cloud Storage Origins:
  • Google BigQuery
  • Google Cloud Storage
  • Google Pub/Sub Subscriber
Destinations:
  • Google BigQuery (Enterprise)
  • Google Cloud Storage
  • Google Pub/Sub Subscriber
Executor:
  • Google BigQuery (Enterprise)
  • Google Cloud Storage
Credential Store:
  • Google Secret Manager
Google Cloud Storage
Microsoft Azure Origins:
  • Azure Data Lake Storage Gen2
  • Azure IoT/Event Hub Consumer
Destinations:
  • Azure Data Lake Storage Gen2
  • Azure Event Hub Producer
  • Azure IoT Hub Producer
  • Azure Synapse SQL
Executors:
  • ADLS Gen2 File Metadata
Credential Store:
  • Azure Key Vault
Microsoft Azure
MongoDB Atlas MongoDB Atlas origin

MongoDB Atlas destination

MongoDB Atlas
Salesforce Origins:
  • Salesforce
  • Salesforce Bulk API 2.0
Processors:
  • Salesforce Lookup
  • Salesforce Bulk API 2.0 Lookup
Destinations:
  • Salesforce
  • Salesforce Bulk API 2.0
  • Tableau CRM
Salesforce
Snowflake Snowflake destination

Snowflake File Uploader destination

Snowflake executor

Amazon S3

Microsoft Azure

Protocols

Data Collector supports the protocols listed in the following table. StreamSets tests the listed stages on the specified environments.

Customers with an enterprise account can receive help with the following protocols unless the implementation proves below standard for the protocol.

Private extensions for the protocols are not supported unless specified in the table.
Supported Protocol Stages Tested Environment
CoAP CoAP Server origin

CoAP Client destination

Eclipse Californium 1.0.4
HTTP Origins:
  • HTTP Client
  • HTTP Server
Processors:
  • HTTP Client
  • HTTP Router
Destinations:
  • HTTP Client
Apache HTTP from Centos 6.8
JMS JMS Consumer origin

JMS Producer destination

ActiveMq 5.14.3
MQTT MQTT Subscriber origin

MQTT Publisher destination

Mosquitto
OPC UA OPC UA Client origin Full testing not performed at this time
SFTP/ FTP / FTPS SFTP/FTP/FTPS Client origin

SFTP/FTP/FTPS Client destination

SFTP/FTP/FTPS Client executor

vsftpd 3.0
Syslog Syslog destination Full testing not performed at this time
TCP TCP Server origin Java TCP Stack
UDP UDP Multithreaded Source origin

UDP Source origin

Java UDP Stack
Websocket Origins:
  • WebSocket Client
  • WebSocket Server
Destination:
  • WebSocket Client
Java HTTP Stack

Versioned Systems

Versioned systems are external systems with multiple versions. When Data Collector supports multiple versions of an external system, you might need to install a specific stage library to work with a particular version. For details on individual stage libraries and the stages that they include, see Common Stage Libraries.

The following table lists the system versions that are supported and tested for Data Collector.

The supported versions column lists the system versions that customers with an enterprise account can receive help with. The tested versions column lists the subset of the supported versions that have been fully tested.

System Stages Supported Versions Tested Versions
Cassandra Cassandra destination Cassandra 1.2, 2.x, 3.x Cassandra 3.11
Couchbase Server Couchbase destination Couchbase Server 5.x Couchbase Server 5.1.1
Elasticsearch Elasticsearch origin

Elasticsearch destination

Elasticsearch 5.x - 8.x Elasticsearch 5.20, 6.8.12, 7.9.0, 8.1.1
Hadoop Distributed File System (HDFS)
Origin:
  • Hadoop FS Standalone
Destination:
  • Hadoop FS
Executors:
  • HDFS File Metadata
  • MapReduce
  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
Hashicorp Vault Hashicorp Vault credential store General support Full testing not performed at this time
HBase HBase Lookup processor
  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
Hive Hive Metadata processor

Hive Metastore destination

Hive Query executor

  • CDP Private Cloud Base 7.1.x
  • MapR 6.1.x with MEP 6.x
  • CDP Private Cloud Base 7.1.x
Hive Streaming

Hive Query executor

  • Hive 0.13 and later
  • CDP Private Cloud Base 7.1.x
  • MapR 6.1.x with MEP 6.x
Full testing not performed at this time
InfluxDB InfluxDB destination InfluxDB 0.9 - 1.x InfluxDB 0.13, 1.7.10
InfluxDB 2.x destination InfluxDB 2.x InfluxDB 2.0.8
Java Keystore Java Keystore credential store Java Virtual Machine Java Virtual Machine
Kafka Origins:
  • Kafka Multitopic Consumer
Destination:
  • Kafka Producer
  • Apache Kafka 1.0.x, 1.1.x, 2.0.x - 2.8.x, 3.0.x - 3.2.x
  • CDP Private Cloud Base 7.1.x
  • Apache Kafka 1.0.x, 1.1.x, 2.0.x - 2.8.x, 3.0.x - 3.2.x
  • CDP Private Cloud Base 7.1.x
Kudu Kudu Lookup processor

Kudu destination

  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
MariaDB Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
MariaDB 10.2 - 10.7 as limited drop-in replacements for MySQL 5.7.

For more information, see the MariaDB documentation.

MariaDB 10.7
MapR DB Origin:
  • MapR DB
Destinations:
  • MapR DB
  • MapR DB JSON
  • MapR 6.1.x with optional MEP 6.x
  • MapR 6.1.x with MEP 6
MapR FS MapR FS Standalone origin

MapR FS destination

MapReduce executor

  • MapR 6.1.x with optional MEP 6.x
  • MapR 6.1.x with MEP 6
MapR Streams Origins:
  • MapR Multitopic Streams Consumer
  • MapR Streams Consumer
  • MapR DB CDC
Destination:
  • MapR Streams Producer
MapR 6.1.x with optional MEP 6.x MapR 6.1.x with MEP 6
Microsoft SQL Server SQL Server 2019 BDC origin SQL Server 2019 Big Data Cluster SQL Server 2019 Big Data Cluster
SQL Server CDC Client origin

SQL Server Change Tracking origin

  • SQL Server 2017
  • SQL Server 2019
  • SQL Server 2017
  • SQL Server 2019
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
SQL Server 2017 and later
  • SQL Server 2017
  • SQL Server 2019
MongoDB Origins:
  • MongoDB
  • MongoDB Atlas
Processor:
  • MongoDB Lookup
Destinations:
  • MongoDB
  • MongoDB Atlas
MongoDB 3.x, 4.x MongoDB 3.6, 4.0
MongoDB Oplog origin MongoDB 3.x, 4.x MongoDB 3.6, 4.0
MySQL Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
MySQL 5.7 and later
  • MySQL 5.7 with the MySQL Connector/J 8.0.12 driver
  • MySQL 8.0 with the MySQL Connector/J 8.0.12 driver
MySQL Binary Log MySQL 5.7 and later
  • MySQL 5.7 with the MySQL Connector/J 8.0.12 driver
  • MySQL 8.0 with the MySQL Connector/J 8.0.12 driver
Oracle Oracle Bulkload origin
  • Oracle 11g, 12c, 18c, 19c

Hosted systems and derived systems are not supported.

  • Oracle 11g, 19c with the Oracle 12.2.0.1.0 JDBC driver version
Oracle CDC Client origin
  • Oracle 11g, 12c, 18c, 19c, 21c
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c, 21c
  • Oracle Exadata appliances that run supported versions of Oracle RAC

Hosted systems and derived systems are not supported unless listed by name above.

  • Oracle 12c, 19c, 21c with the Oracle 12.2.0.1.0 JDBC driver version
  • Oracle RAC 12c, 19c with the Oracle 12.2.0.1.0 JDBC driver version
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
  • Oracle 11g, 12c, 18c, 19c, and later
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c, and later
Also supported:
  • Hosted systems, such as AWS RDS, that run supported versions of Oracle RAC
  • Derived systems, such as Oracle Exadata, that run supported versions of Oracle RAC
  • Oracle 11g, 19c with the Oracle 12.2.0.1.0 JDBC driver version

  • Oracle RAC 12c, 19c with the Oracle 12.2.0.1.0 JDBC driver version

PMML PMML Evaluator processor General support Full testing not performed at this time
PostgreSQL Aurora PostgreSQL CDC Client origin
  • Aurora PostgreSQL 2.2.0 (with PostgreSQL 10.6 - 10.17) and later
  • Aurora PostgreSQL 3 (with PostgreSQL 11.0 - 11.12)
  • Aurora PostgreSQL 4 (with PostgreSQL 12.0 - 12.7)
Aurora PostgreSQL 4 (with PostgreSQL 12.7)
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
PostgreSQL 9.x and later
  • PostgreSQL 9.6.9
  • PostgreSQL 10.4
  • PostgreSQL 11.7
  • PostgreSQL 12.2
  • PostgreSQL 13.0
  • PostgreSQL 14.0
PostgreSQL CDC Client origin
  • PostgreSQL 9.4 or later 9.x
  • PostgreSQL 10.x -13.x
  • PostgreSQL 9.6.9
  • PostgreSQL 10.4
  • PostgreSQL 11.7
  • PostgreSQL 12.2
  • PostgreSQL 13.0
  • PostgreSQL 14.0
Pulsar Origins:
  • Pulsar Consumer
  • Pulsar Consumer (Legacy)
Destination:
  • Pulsar Producer
Pulsar 2.x
  • Pulsar 2.1.0
  • Pulsar 2.2.1
  • Pulsar 2.3.2
  • Pulsar 2.4.2
  • Pulsar 2.5.1
  • Pulsar 2.6.2
RabbitMQ RabbitMQ Consumer origin

RabbitMQ Producer destination

RabbitMQ 3.5.x and later RabbitMQ 3.5.6, 3.8.0
Redis Redis Consumer origin

Redis destination

Redis 2.x - 4.x Redis 4.0.1
SAP HANA SAP HANA Query Consumer origin SAP HANA 2.4.x SAP HANA 2.0 with the SAP HANA JDBC driver version 2.4.76
Solr Solr destination
  • Apache Solr 6.x
  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
Spark

Spark executor

  • CDP Private Cloud Base 7.1.x
  • CDP Private Cloud Base 7.1.x
Splunk Splunk destination General support Full testing not performed at this time
TensorFlow TensorFlow Evaluator processor TensorFlow 1.x Full testing not performed at this time
Thycotic Secret Server Thycotic Secret Server credential store Full testing not performed at this time Full testing not performed at this time