External Libraries
You can install a driver or other library as an external library to make it available to a Transformer stage.
- Some stages, such as the Oracle JDBC Table origin and the MySQL JDBC Table origins, require installing a driver as an external library.
- Some stages, such as the JDBC origins, lookup, and destination, include several drivers, but require installing a driver to access certain databases.
- Some stages provide the required libraries, but you can install custom libraries to access custom functionality. For example, you might install a custom Java or Scala library for the Scala processor.
When installing an external library, you install it into the stage library that includes the stage. For example, to use a custom Scala library with Scala processors, you install the Scala library as an external library for the Basic stage library.
To use an external library with multiple stage libraries, install the external library into each stage library associated with the stages. For example, if you want to use an Oracle JDBC driver with the Scala processor and the Oracle JDBC Table origin, you install the driver as an external library for the Basic stage library and for the JDBC stage library.
To install an external library, add the external library to an external resource archive file for the deployment.
When needed, you can update or remove an existing external library. For more information, see Managing External Libraries.
Managing External Libraries
When you run a pipeline that uses a stage library with related external libraries, Transformer uploads those libraries to the cluster as needed.
- EMR, EMR Serverless, Databricks, and Dataproc clusters
- Transformer automatically updates the staging directories for these clusters. You do not need to manually manage external libraries in these cluster staging directories.
- Other supported clusters
- For all other cluster types, you must manually manage external library updates for both Transformer and cluster staging directories.