Stage Libraries
A Control Hub deployment defines the stage libraries that are installed on all engine instances managed by the deployment. When you configure any deployment type, you select the stage libraries to install on the engine.
Common Stage Libraries
Common stage libraries include stages that are the most commonly used.
Stage Library Name | Included Stages |
---|---|
streamsets-datacollector-apache-kafka_1_0-lib | For Kafka version 1.0.x. Includes:
|
streamsets-datacollector-apache-kafka_1_1-lib | For Kafka version 1.1.x. Includes:
|
streamsets-datacollector-apache-kafka_2_0-lib | For Kafka version 2.0.x. Includes:
|
streamsets-datacollector-apache-kafka_2_1-lib | For Kafka version 2.1.x. Includes:
|
streamsets-datacollector-apache-kafka_2_2-lib | For Kafka version 2.2.x. Includes:
|
streamsets-datacollector-apache-kafka_2_3-lib | For Kafka version 2.3.x. Includes:
|
streamsets-datacollector-apache-kafka_2_4-lib | For Kafka version 2.4.x. Includes:
|
streamsets-datacollector-apache-kafka_2_5-lib | For Kafka version 2.5.x. Includes:
|
streamsets-datacollector-apache-kafka_2_6-lib | For Kafka version 2.6.x. Includes:
|
streamsets-datacollector-apache-kafka_2_7-lib | For Kafka version 2.7.x. Includes:
|
streamsets-datacollector-apache-kafka_2_8-lib | For Kafka version 2.8.x. Includes:
|
streamsets-datacollector-apache-kafka_3_0-lib | For Kafka version 3.0.x. Includes:
|
streamsets-datacollector-apache-kafka_3_1-lib | For Kafka version 3.1.x. Includes:
|
streamsets-datacollector-apache-kafka_3_2-lib | For Kafka version 3.2.x. Includes:
|
streamsets-datacollector-apache-pulsar_2-lib | For Apache Pulsar version 2.x. Includes:
|
streamsets-datacollector-apache-solr_6_1_0-lib | For Apache Solr version 6.1. Includes the Solr destination. |
streamsets-datacollector-aws-lib | For Amazon Web Services 1.11.x. Includes:
|
streamsets-datacollector-aws-secrets-manager-credentialstore-lib | For the AWS Secrets Manager credential store. |
streamsets-datacollector-azure-keyvault-credentialstore-lib | For the Microsoft Azure Key Vault credential store. |
streamsets-datacollector-azure-lib | For Microsoft Azure. Includes:
|
streamsets-datacollector-basic-lib |
Includes the following origins:
Includes the following processors:
Includes the following destinations:
Includes the following executors:
|
streamsets-datacollector-bigtable-lib | For Google Cloud Bigtable. Includes the Google Bigtable destination. |
streamsets-datacollector-cassandra_3-lib | For Cassandra 1.2, 2.x, and 3.x. Includes the Cassandra destination. |
streamsets-datacollector-cdp_7_1-lib | For Cloudera CDP 7.1.x. Includes:
|
streamsets-datacollector-couchbase_5-lib | For Couchbase. Includes:
|
streamsets-datacollector-crypto-lib | For cryptography stages. Includes the Encrypt and Decrypt Fields processor. |
streamsets-datacollector-cyberark-credentialstore-lib | For the CyberArk credential store. |
streamsets-datacollector-dataformats-lib |
Contains parsers and generators for the data formats supported by Data Collector. |
streamsets-datacollector-dev-lib | For developing and testing pipelines. Includes:
Note: Do not use these stages in production pipelines.
|
streamsets-datacollector-elasticsearch_5-lib | For Elasticsearch 1.x, 2.x, and 5.x. Includes the Elasticsearch origin and destination. |
streamsets-datacollector-elasticsearch_6-lib | For Elasticsearch 6.x. Includes the Elasticsearch origin and destination. |
streamsets-datacollector-elasticsearch_7-lib | For Elasticsearch 7.x. Includes the Elasticsearch origin and destination. |
streamsets-datacollector-elasticsearch_8-lib | For Elasticsearch 8.x. Includes the Elasticsearch origin and destination. |
streamsets-datacollector-google-cloud-lib | For Google Cloud. Includes:
|
streamsets-datacollector-google-secret-manager-credentialstore-lib | For the Google Secret Manager credential store. |
streamsets-datacollector-groovy_2_4-lib | For Groovy version 2.4. Includes:
|
streamsets-datacollector-groovy_4_0-lib | For Groovy version 4.0. Includes:
|
streamsets-datacollector-influxdb_0_9-lib | For InfluxDB version 0.9 - 1.x. Includes the InfluxDB destination. |
streamsets-datacollector-influxdb_2_0-lib | For InfluxDB version 2.x. Includes the InfluxDB 2.x destination. |
streamsets-datacollector-jdbc-lib | For JDBC access to databases. Includes:
|
streamsets-datacollector-jdbc-sap-hana-lib | For JDBC access to SAP HANA databases. Includes the SAP HANA Query Consumer origin. |
streamsets-datacollector-jks-credentialstore-lib | For the Java keystore credential store. |
streamsets-datacollector-jms-lib | For Java Messaging Services (JMS). Includes the JMS Consumer origin and JMS Producer destination. |
streamsets-datacollector-jython_2_7-lib | For Jython version 2.7.x. Includes:
|
streamsets-datacollector-kinesis-lib | For Amazon Kinesis. Includes:
|
streamsets-datacollector-mapr_6_1-lib | For MapR version 6.1.0. Includes:
|
streamsets-datacollector-mapr_6_1-mep6-lib | For MapR 6.1.0 with MEP 6.x. Includes:
|
streamsets-datacollector-mapr_7_0-lib | For MapR 7.0.x. Includes:
|
streamsets-datacollector-mapr_7_0-mep8-lib | For MapR 7.0.x with MEP 8.x. Includes:
|
streamsets-datacollector-mleap-lib | For MLeap. Includes the MLeap Evaluator processor. |
streamsets-datacollector-mongodb_3-lib | For MongoDB 3.0 with Java driver 3.5.0. Includes:
|
streamsets-datacollector-mongodb_4-lib | For MongoDB 4.0 with Java driver 3.12.0. Includes:
|
streamsets-datacollector-mongodb-atlas-lib | For MongoDB Atlas and Mongo Enterprise Server. Includes:
|
streamsets-datacollector-mysql-binlog-lib | For MySQL binary logs. Includes the MySQL Binary Log origin. |
streamsets-datacollector-orchestrator-lib | For the orchestration stages. Includes:
|
streamsets-datacollector-postgres-aurora-lib | For Amazon Aurora PostgreSQL versions 1 through 4. Includes the Aurora PostgreSQL CDC Client origin. |
streamsets-datacollector-rabbitmq-lib | For RabbitMQ version 3.5.6. Includes the RabbitMQ Consumer origin and RabbitMQ Producer destination. |
streamsets-datacollector-redis-lib | For Redis versions 2.8 and 3.0. Includes:
|
streamsets-datacollector-salesforce-lib |
For Salesforce. Includes:
|
streamsets-datacollector-stats-lib |
StreamSets Control Hub requires that the statistics stage library be installed on each Data Collector. |
streamsets-datacollector-tensorflow-lib | For TensorFlow. Includes the TensorFlow Evaluator processor. |
streamsets-datacollector-thycotic-credentialstore-lib | For the Thycotic Secret Server credential store. |
streamsets-datacollector-vault-credentialstore-lib | For the Hashicorp Vault credential store. |
streamsets-datacollector-wholefile-transformer-lib | Includes the Whole File Transformer processor. |
streamsets-datacollector-windows-lib |
For Windows. Includes the Windows Event Log origin. |
Enterprise Stage Libraries
Enterprise stage libraries provide stages that connect to advanced external systems. Releases of Enterprise stage libraries occur separately from Data Collector releases.
The release notes for Enterprise stage libraries are available on the StreamSets Documentation page.
Stage Library | Stage Library Name | Description |
---|---|---|
Azure Synapse | streamsets-datacollector-azure-synapse-lib | For Azure Synapse. Includes the Azure Synapse SQL destination. |
Databricks | streamsets-datacollector-databricks-lib | For Databricks. Includes the Databricks Delta Lake destination and the Databricks Query executor. |
Oracle | streamsets-datacollector-oracle-lib | For bulk loading from Oracle tables. Includes the Oracle Bulkload origin. |
Protector | streamsets-datacollector-protector-lib | For protecting sensitive data. Includes a set of Protector stages. For a full list, see the Protector release notes. |
Snowflake | streamsets-datacollector-snowflake-lib | For Snowflake. Includes the Snowflake destination, the Snowflake File Uploader destination, and the Snowflake executor. |
Microsoft SQL Server 2019 Big Data Cluster | streamsets-datacollector-sql-server-bdc-lib | For SQL Server 2019 Big Data Cluster. Includes the SQL Server 2019 BDC Multitable Consumer origin and the SQL Server 2019 BDC Bulk Loader destination. |