Install External Libraries

Install external libraries to make them available to Data Collector stages.

You can install external libraries for the following stages:
  • Before you use the following stages, install JDBC drivers for the implementation that you want to use:
    • JDBC Multitable Consumer origin
    • JDBC Query Consumer origin
    • MySQL Binary Log origin
    • Oracle Bulkload origin
    • Oracle CDC origin
    • Oracle CDC Client origin
    • SAP HANA Query Consumer origin
    • JDBC Lookup processor
    • JDBC Tee processor
    • SQL Parser processor, when using the database to resolve the schema
    • JDBC Producer destination
    • JDBC Query executor

    For example, to use the JDBC Query Consumer origin or the JDBC Producer destination with Oracle, install the Oracle JDBC drivers.

  • You can install external Java libraries to call external Java code from the scripting processors: Groovy, Java, and Jython Evaluator.
  • You can call external Python modules from the Jython Evaluator processor.
  • You can install the DataStax Enterprise (DSE) Java driver to configure the Cassandra destination to use DSE username and password authentication or Kerberos authentication.
  • Before you use the Google Bigtable destination, install the BoringSSL library.
  • Before you use the JMS Consumer origin or the JMS Producer destination, install the JMS drivers for the implementation that you are using.
  • You can install the Impala JDBC driver for use with the Hive Query executor. For more information, see Installing the Impala Driver.

When installing an external library, you install it into the stage library that includes the stage. For example, to use an external Java library with the Groovy Evaluator processor, you install the Java library as an external library for the Groovy stage library, streamsets-datacollector-groovy_4_0-lib.

To use an external library with multiple stage libraries, install the external library into each stage library associated with the stages. For example, if you want to use a MySQL JDBC driver with the JDBC Lookup processor and with the MySQL Binary Log origin, you install the driver as an external library for the JDBC stage library, streamsets-datacollector-jdbc-lib, and for the MySQL Binary Log stage library, streamsets-datacollector-mysql-binlog-lib.

By default, external libraries are installed to the $SDC_EXTERNAL_RESOURCES/streamsets-libs-extras directory. StreamSets recommends configuring Data Collector to use an external directory to enable use of the libraries after Data Collector upgrades.

You can install external libraries any of the following ways: