Stage Libraries
A deployment defines the stage libraries that are installed on the engine. The installed stage libraries determine the stages that you can use when designing pipelines and the credential stores that pipelines can retrieve secrets from.
- Group of related stages
- Most stage libraries include a group of related stages. For example, the Data Collector Amazon Web Services stage library includes the Amazon S3 origin, destination, and executor and the Amazon SQS Consumer origin. For a description of the stages included in each Data Collector stage library, see the Data Collector documentation.
- Supported credential store
- Some stage libraries include support for connecting to a credential store system to retrieve pipeline secrets. For example, the AWS Secrets Manager Credential Store stage library allows you to configure an engine to retrieve secrets from the AWS Secrets Manager credential store system. For more information about credential stores, see the following engine documentation:
By default, a deployment includes some of the common stage libraries. You can remove the default stage libraries and add additional libraries. The available stage libraries depend on the engine type and version.
You can update stage libraries when you create or edit a deployment. In the Configure Engine step, click <number> stage libraries selected next to the Stage Libraries property.
In the Select Stage Libraries window, you can select stage libraries to add or remove by individual stages or credential stores or by stage library name and version. You can also view a summary of your selections before saving your changes.
For detailed instructions on updating stage libraries for an existing deployment, see Updating Stage Libraries.
Select from Stages
The Select from Stages tab lists all stages and credential stores. Use this tab when you want to explore which stages are included in the stage libraries.
- Available items
- Lists stages and credential stores that have not been added to the
deployment. Click the Add icon (
) to add a stage or credential store to the deployment. - Selected items
- Lists stages and credential stores that have been added to the deployment.
Click the Remove icon (
) to remove a stage or credential store from the
deployment.
You can filter the lists by type, such as by origins, processors, or credential stores. Or you can search for stages and credential stores by name. For example, the following image filters the lists by origins and searches for origins that contain google in the name:
When you select a stage, the Select Stage Library Versions window displays all stage library versions that include the stage. Expand a stage library to view the complete list of stages included in the library. If the selected stage is included in multiple stage libraries, you can select the specific stage library to add or remove.
For example, if you add the Kafka Multitopic Consumer origin, the Select Stage Library Versions window displays all stage library versions that include the origin. You can select one or more stage libraries to add. When you expand the Apache Kafka 2.8.0 stage library, you can see that the stage library includes the Kafka Multitopic Consumer origin and the Kafka Producer destination:
Select from Stage Libraries
The Select from Stage Libraries tab lists all stage libraries by name and version. It does not list the individual stages included in the libraries. Use this tab when you want to quickly select a stage library to add or remove and you already know the stages included in that library.
For Transformer engines, all stage libraries display under Common Stage Libraries.
- Common Stage Libraries - Lists most stage libraries, including those for common stages and credential stores.
- Enterprise Stage Libraries - Lists enterprise stage libraries, such as Snowflake and Microsoft SQL Server 2019 Big Data Cluster.
Select a stage library to add it to the deployment. Clear a stage library to remove it from the deployment.
For example, in the following image, the Databricks Enterprise stage library is selected to add to the deployment:
View the Summary
The Summary tab lists the currently selected stage libraries by file name. For
example, the Basic stage library is listed as
streamsets-datacollector-basic-lib.
View the summary to confirm your selections before saving your changes.
For example, the following Summary tab displays the list of default Data Collector stage libraries:
