Custom Stage Libraries

If you develop custom stages, store the stage libraries in a local directory external to the Data Collector installation directory. Use an external directory to enable use of the custom stage libraries after Data Collector upgrades.

Use the required procedure for your installation type.

We have tutorials for creating a custom origin, processor, and destination on our StreamSets tutorials page. To check them out, go to: https://github.com/streamsets/tutorials.

Storing Custom Libraries for Tarball or RPM

To store custom stage libraries for a tarball or RPM installation, perform the following steps:

Create a local directory external to the Data Collector installation directory for the custom stage libraries. Use an external directory to enable use of the libraries after Data Collector upgrades.
For example, if you installed Data Collector in the following directory:
```
/opt/sdc/
```
you might create the custom stage library directory at:
```
/opt/sdc-user-libs
```
Copy the custom stage libraries to the directory, using a unique subfolder for each custom stage.
For example to store libraries for a custom stage named customstage1, you would copy them to the following directory:
```
/opt/sdc-user-libs/customstage1
```
Add the USER_LIBRARIES_DIR environment variable to the appropriate file and point it to the custom stage library directory.
Modify environment variables using the method required by your installation type.

Set the environment variable as follows:
```
export USER_LIBRARIES_DIR="<custom stage library directory>"
```
For example:
```
export USER_LIBRARIES_DIR="/opt/sdc-user-libs/"
```
When using the Java Security Manager, which is enabled by default, update the Data Collector security policy to include the custom stage library directory as follows:
1. In the Data Collector configuration directory, open the security policy file:
```
$SDC_CONF/sdc-security.policy
```
2. Add the following lines to the file:
```
// custom stage library directory
grant codebase "file://<custom stage library directory>/-" {
   permission java.security.AllPermission;
};
```
  For example:
```
// custom stage library directory
grant codebase "file:///opt/sdc-user-libs/-" {
   permission java.security.AllPermission;
};
```
Restart Data Collector.
The Data Collector stage library displays the custom stages.

Storing Custom Libraries for Cloudera Manager

To store custom stage libraries for a Cloudera Manager installation, perform the following steps:

On every node that runs Data Collector, create a local directory for the custom stage libraries.
For example, you might create the custom stage library directory at:
```
/opt/sdc-user-libs
```
On every node that runs Data Collector, copy the custom stage libraries to the directory defined for the USER_LIBRARIES_DIR environment variable. Use a unique subfolder for each custom stage.
For example to store libraries for a custom stage named customstage1, you would copy them to the following directory:
```
/opt/sdc-user-libs/customstage1
```
In Cloudera Manager, select the StreamSets service and then click Configuration.
On the Configuration page, in the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-env.sh field, add the USER_LIBRARIES_DIR environment variable and point it to the custom stage library directory, as follows:
```
export USER_LIBRARIES_DIR="<custom stage library directory>"
```
For example:
```
export USER_LIBRARIES_DIR="/opt/sdc-user-libs/"
```
When using the Java Security Manager, which is enabled by default, update the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-security.policy property to include the custom stage library directory as follows:
```
// custom stage library directory
grant codebase "file://<custom stage library directory>/-" {
   permission java.security.AllPermission;
};
```
For example:
```
// custom stage library directory
grant codebase "file:///opt/sdc-user-libs/-" {
   permission java.security.AllPermission;
};
```
Restart Data Collector.