Available Stage Libraries
A full Data Collector installation includes all of the following stage libraries. A core installation includes only some of the following stage libraries and typically requires you to install additional stage libraries. A common installation includes commonly-used stage libraries.
You can install additional stage libraries into either a core or common installation.
Stage Library Name | Included Stages |
---|---|
streamsets-datacollector-aerospike-lib | For Aerospike version 3.15.x. Includes the Aerospike destination. |
streamsets-datacollector-apache-kafka_1_0-lib | For Kafka version 1.0.x. Includes:
|
streamsets-datacollector-apache-kafka_1_1-lib | For Kafka version 1.1.x. Includes:
|
streamsets-datacollector-apache-kafka_2_0-lib | For Kafka version 2.0.x. Includes:
|
streamsets-datacollector-apache-kafka_2_1-lib | For Kafka version 2.1.x. Includes:
|
streamsets-datacollector-apache-kafka_2_2-lib | For Kafka version 2.2.x. Includes:
|
streamsets-datacollector-apache-kafka_2_3-lib | For Kafka version 2.3.x. Includes:
|
streamsets-datacollector-apache-kafka_2_4-lib | For Kafka version 2.4.x. Includes:
|
streamsets-datacollector-apache-kafka_2_5-lib | For Kafka version 2.5.x. Includes:
|
streamsets-datacollector-apache-kafka_2_6-lib | For Kafka version 2.6.x. Includes:
|
streamsets-datacollector-apache-kafka_2_7-lib | For Kafka version 2.7.x. Includes:
|
streamsets-datacollector-apache-kafka_2_8-lib | For Kafka version 2.8.x. Includes:
|
streamsets-datacollector-apache-kafka_3_0-lib | For Kafka version 3.0.x. Includes:
|
streamsets-datacollector-apache-kafka_3_1-lib | For Kafka version 3.1.x. Includes:
|
streamsets-datacollector-apache-kafka_3_2-lib | For Kafka version 3.2.x. Includes:
|
streamsets-datacollector-apache-kafka_3_3-lib | For Kafka version 3.3.x. Includes:
|
streamsets-datacollector-apache-kudu_1_3-lib | For Kudu version 1.3.x. Includes the Kudu Lookup processor and Kudu destination. |
streamsets-datacollector-apache-kudu_1_4-lib | For Kudu version 1.4.x. Includes the Kudu Lookup processor and Kudu destination. |
streamsets-datacollector-apache-kudu_1_5-lib | For Kudu version 1.5.x. Includes the Kudu Lookup processor and Kudu destination. |
streamsets-datacollector-apache-kudu_1_6-lib | For Kudu version 1.6.x. Includes the Kudu Lookup processor and Kudu destination. |
streamsets-datacollector-apache-kudu_1_7-lib | For Kudu version 1.7.x. Includes the Kudu Lookup processor and Kudu destination. |
streamsets-datacollector-apache-pulsar_2-lib | For Apache Pulsar version 2.x. Includes:
|
streamsets-datacollector-apache-solr_6_1_0-lib | For Apache Solr version 6.1. Includes the Solr destination. |
streamsets-datacollector-aws-lib | For Amazon Web Services 1.11.x. Includes:
|
streamsets-datacollector-aws-secrets-manager-credentialstore-lib | For the AWS Secrets Manager credential store. |
streamsets-datacollector-azure-keyvault-credentialstore-lib | For the Microsoft Azure Key Vault credential store. |
streamsets-datacollector-azure-lib | For Microsoft Azure. Includes:
|
streamsets-datacollector-basic-lib |
Includes the following origins:
Includes the following processors:
Includes the following destinations:
Includes the following executors:
|
streamsets-datacollector-bigtable-lib | For Google Cloud Bigtable. Includes the Google Bigtable destination. |
streamsets-datacollector-cassandra_3-lib | For Cassandra 1.2, 2.x, and 3.x. Includes the Cassandra destination. |
streamsets-datacollector-cdh_5_14-lib | For the Cloudera CDH version 5.14.x distribution of Apache
Hadoop. Includes:
|
streamsets-datacollector-cdh_5_15-lib | For the Cloudera CDH version 5.15.x distribution of Apache
Hadoop. Includes:
|
streamsets-datacollector-cdh_5_16-lib | For the Cloudera CDH version 5.16.x distribution of Apache
Hadoop. Includes:
|
streamsets-datacollector-cdh_6_0-lib | For the Cloudera CDH version 6.0.x distribution of Apache Hadoop.
Includes:
|
streamsets-datacollector-cdh_6_1-lib | For the Cloudera CDH version 6.1.x distribution of Apache Hadoop.
Includes:
|
streamsets-datacollector-cdh_6_2-lib | For the Cloudera CDH version 6.2.x distribution of Apache Hadoop.
Includes:
|
streamsets-datacollector-cdh_6_3-lib | For the Cloudera CDH version 6.3.x distribution of Apache Hadoop.
Includes:
|
streamsets-datacollector-cdh_kafka_3_1-lib | For the Cloudera distribution of Apache Kafka - CDK 3.1.0 (based
on Apache Kafka version 1.0.1). Includes:
|
streamsets-datacollector-cdh_kafka_4_1-lib | For the Cloudera distribution of Apache Kafka - CDK 4.1.0 (based
on Apache Kafka version 2.2.1). Includes:
|
streamsets-datacollector-cdh_spark_2_1_r1-lib | For the Cloudera distribution of Spark 2.1 release 1.
Includes:
|
streamsets-datacollector-cdh_spark_2_2-lib | For the Cloudera CDH cluster Kafka with CDS powered by Spark 2.2
release 1. Includes the Kafka Consumer origin for cluster mode pipelines. |
streamsets-datacollector-cdh_spark_2_3-lib | For the Cloudera CDH cluster Kafka with CDS powered by Spark 2.3
release 2. Includes the Kafka Consumer origin for cluster mode pipelines. |
streamsets-datacollector-cdh_spark_2_3_r3-lib | For the Cloudera CDH cluster Kafka with CDS powered by Spark 2.3
release 3. Includes the Kafka Consumer origin for cluster mode pipelines. |
streamsets-datacollector-cdh_spark_2_3_r4-lib | For the Cloudera CDH cluster Kafka with CDS powered by Spark 2.3
release 4. Includes the Kafka Consumer origin for cluster mode pipelines. |
streamsets-datacollector-cdp_7_1-lib | For Cloudera CDP 7.1.x. Includes:
|
streamsets-datacollector-connx-lib | For CONNX. Includes:
|
streamsets-datacollector-couchbase_5-lib | For Couchbase. Includes:
|
streamsets-datacollector-crypto-lib | For cryptography stages. Includes the Encrypt and Decrypt Fields processor. |
streamsets-datacollector-cyberark-credentialstore-lib | For the CyberArk credential store. |
streamsets-datacollector-databricks-ml_2-lib | For Databricks ML. Includes the Databricks ML Evaluator processor. |
streamsets-datacollector-dataformats-lib |
Contains parsers and generators for the data formats supported by Data Collector. |
streamsets-datacollector-dev-lib | For developing and testing pipelines. Includes:
Note: Do not use these stages in production pipelines.
|
streamsets-datacollector-elasticsearch_5-lib | For Elasticsearch 1.x, 2.x, and 5.x. Includes the Elasticsearch origin and destination. |
streamsets-datacollector-elasticsearch_6-lib | For Elasticsearch 6.x. Includes the Elasticsearch origin and destination. |
streamsets-datacollector-elasticsearch_7-lib | For Elasticsearch 7.x. Includes the Elasticsearch origin and destination. |
streamsets-datacollector-elasticsearch_8-lib | For Elasticsearch 8.x. Includes the Elasticsearch origin and destination. |
streamsets-datacollector-emr_hadoop_2_8_3-lib | For Amazon EMR 5.14.x with Hadoop 2.8.3. Includes the Hadoop FS origin for cluster mode pipelines. |
streamsets-datacollector-google-cloud-lib | For Google Cloud. Includes:
|
streamsets-datacollector-google-secret-manager-credentialstore-lib | For the Google Secret Manager credential store. |
streamsets-datacollector-groovy_2_4-lib | For Groovy version 2.4. Includes:
|
streamsets-datacollector-groovy_4_0-lib | For Groovy version 4.0. Includes:
|
streamsets-datacollector-hdp_3_1-lib | For Hortonworks version 3.1. Includes:
|
streamsets-datacollector-influxdb_0_9-lib | For InfluxDB version 0.9 - 1.x. Includes the InfluxDB destination. |
streamsets-datacollector-influxdb_2_0-lib | For InfluxDB version 2.x. Includes the InfluxDB 2.x destination. |
streamsets-datacollector-jdbc-lib | For JDBC access to databases. Includes:
|
streamsets-datacollector-jdbc-oracle-lib | For Oracle. Includes the Oracle CDC origin. |
streamsets-datacollector-jdbc-sap-hana-lib | For JDBC access to SAP HANA databases. Includes the SAP HANA Query Consumer origin. |
streamsets-datacollector-jks-credentialstore-lib | For the Java keystore credential store. |
streamsets-datacollector-jms-lib | For Java Messaging Services (JMS). Includes the JMS Consumer origin and JMS Producer destination. |
streamsets-datacollector-jython_2_7-lib | For Jython version 2.7.x. Includes:
|
streamsets-datacollector-kinesis-lib | For Amazon Kinesis. Includes:
|
streamsets-datacollector-kinetica_6_0-lib | For Kinetica 6.0. Includes the KineticaDB destination. |
streamsets-datacollector-kinetica_6_1-lib | For Kinetica 6.1. Includes the KineticaDB destination. |
streamsets-datacollector-kinetica_6_2-lib | For Kinetica 6.2. Includes the KineticaDB destination. |
streamsets-datacollector-kinetica_7_0-lib | For Kinetica 7.0. Includes the KineticaDB destination. |
streamsets-datacollector-mapr_6_0-lib | For MapR version 6.0.0 and 6.0.1. Includes:
|
streamsets-datacollector-mapr_6_1-lib | For MapR version 6.1.0. Includes:
|
streamsets-datacollector-mapr_6_0-mep4-lib | For MapR 6.0.0 with EEP 4.x. Includes:
|
streamsets-datacollector-mapr_6_0-mep5-lib | For MapR 6.0.1 with EEP 5.x. Includes:
|
streamsets-datacollector-mapr_6_1-mep6-lib | For MapR 6.1.0 with EEP 6.x. Includes:
|
streamsets-datacollector-mapr_7_0-lib | For HPE Ezmeral Data Fabric 7.0.x. Includes:
|
streamsets-datacollector-mapr_7_0-mep8-lib | For HPE Ezmeral Data Fabric 7.0.x with EEP 8.x. Includes:
|
streamsets-datacollector-mleap-lib | For MLeap. Includes the MLeap Evaluator processor. |
streamsets-datacollector-mongodb_3-lib | For MongoDB 3.0 with Java driver 3.5.0. Includes:
|
streamsets-datacollector-mongodb_4-lib | For MongoDB 4.0 with Java driver 3.12.0. Includes:
|
streamsets-datacollector-mongodb-atlas-lib | For MongoDB Atlas and MongoDB Enterprise Server. Includes:
|
streamsets-datacollector-mysql-binlog-lib | For MySQL binary logs. Includes the MySQL Binary Log origin. |
streamsets-datacollector-omniture-lib | For Omniture. Includes the Omniture origin. |
streamsets-datacollector-orchestrator-lib | For the orchestration stages. Includes:
|
streamsets-datacollector-postgres-aurora-lib | For Amazon Aurora PostgreSQL versions 1 through 4. Includes the Aurora PostgreSQL CDC Client origin. |
streamsets-datacollector-rabbitmq-lib | For RabbitMQ version 3.5.6. Includes the RabbitMQ Consumer origin and RabbitMQ Producer destination. |
streamsets-datacollector-redis-lib | For Redis versions 2.8 and 3.0. Includes:
|
streamsets-datacollector-salesforce-lib |
For Salesforce. Includes:
|
streamsets-datacollector-sdc-snowflake-lib | For Snowflake. Includes:
|
streamsets-datacollector-singlestore-lib | For SingleStore. Includes the SingleStore destination. |
streamsets-datacollector-stats-lib |
StreamSets Control Hub requires that the statistics stage library be installed on each registered Data Collector. |
streamsets-datacollector-tensorflow-lib | For TensorFlow. Includes the TensorFlow Evaluator processor. |
streamsets-datacollector-thycotic-credentialstore-lib | For the Thycotic Secret Server credential store. |
streamsets-datacollector-vault-credentialstore-lib | For the Hashicorp Vault credential store. |
streamsets-datacollector-wholefile-transformer-lib | Includes the Whole File Transformer processor. |
streamsets-datacollector-windows-lib |
For Windows. Includes the Windows Event Log origin. |