Data Collector Environment Configuration

Modifying Environment Variables

The method that you use to modify environment variables depends on the Data Collector installation type:

Tarball installation started manually from the command line

When you start Data Collector manually from the command line on any operating system, edit the $SDC_DIST/libexec/sdc-env.sh file to modify environment variables.

Use a text editor to edit the sdc-env.sh file. Some of the environment variables in the file are commented out and do not reflect the default values. Be sure to uncomment the line when you change a variable value.

After you edit the file, restart Data Collector from the command prompt to enable the changes.

Note: Do not restart Data Collector from the user interface after modifying environment variables.

Tarball or RPM installation started as a service on operating systems that use the SysV init system

When you start Data Collector as a service on CentOS 6, Oracle Linux 6, Red Hat Enterprise Linux 6, or Ubuntu 14.04 LTS, edit the $SDC_DIST/libexec/sdcd-env.sh file to modify environment variables.

Use a text editor to edit the sdcd-env.sh file.

After you edit the file, restart Data Collector to enable the changes.

Tarball or RPM installation started as a service on operating systems that use the systemd init system

When you start Data Collector as a service on CentOS 7, Oracle Linux 7, Red Hat Enterprise Linux 7, or Ubuntu 16.04 LTS, edit the sdc.service file to modify environment variables.

Note: When you install Data Collector through a cloud service provider marketplace, Data Collector is deployed as an RPM package on an operating system that uses the systemd init system.

The location of the sdc.service file depends on how you installed Data Collector:

From the RPM package - /usr/lib/systemd/system/sdc.service
From the tarball - /etc/systemd/system/sdc.service

Override the default values in the sdc.service file using the same procedure that you use to override unit configuration files on a systemd init system. For an example, see "Example 2. Overriding vendor settings" in this systemd.unit manpage.

After overriding the default values, use the following command to reload the systemd manager configuration:

systemctl daemon-reload

Then restart Data Collector to enable the changes.

Cloudera Manager installation

When you install Data Collector through Cloudera Manager, modify environment variables by configuring the StreamSets service through Cloudera Manager.

Data Collector Directories

Data Collector includes environment variables that define the directories used to store files used by Data Collector, such as configuration files, log files, and runtime resources.

The SDC_DIST environment variable defines the Data Collector runtime directory. The runtime directory is the base Data Collector directory that stores the executables and related files. This environment variable is set during installation.

When you start Data Collector manually, the default values of the remaining directory variables are relative to the $SDC_DIST runtime directory. When you start Data Collector as a service, the default values of the remaining directory variables are absolute paths that are outside of the $SDC_DIST runtime directory.

Modify environment variables using the method required by your installation type.

Note: StreamSets does not recommend using NFS or NAS to store Data Collector files.

You can configure the following environment variables that define directories:


Environment Variable	Description
SDC_CONF	Defines the configuration directory for the Data Collector configuration file, `sdc.properties`, and related realm properties files and keystore files. Also includes the logj4 properties file. Default directories: Manual start: `$SDC_DIST/etc` Service start: `/etc/sdc`
SDC_DATA	Defines the data directory for pipeline configuration and run details. Default directories: Manual start: `$SDC_DIST/data` Service start: `/var/lib/sdc`
SDC_LOG	Defines the log directory. Default directories: Manual start: `$SDC_DIST/log` Service start: `/var/log/sdc`
SDC_EXTERNAL_RESOURCES	Defines an optional external resources directory. By default, this directory contains the following directories: SDC_RESOURCES STREAMSETS_LIBRARIES_EXTRA_DIR USER_LIBRARIES_DIR To define this directory, you must add the environment variable to the appropriate file. Set the variable to a directory outside of the `$SDC_DIST` runtime directory. Default directory: `$SDC_DIST/externalResources`
SDC_RESOURCES	Defines the directory for runtime resource files. To configure this environment variable, you must uncomment the variable in the appropriate file. Set the variable to a directory outside of the `$SDC_DIST` runtime directory. Default directories: Manual start for tarball installations: `$SDC_EXTERNAL_RESOURCES/resources` This resolves to the following directory unless you define the SDC_EXTERNAL_RESOURCES environment variable: `$SDC_DIST/externalResources/resources` Manual start for other installations: `$SDC_DIST/resources` Service start: `/var/lib/sdc-resources`
STREAMSETS_LIBRARIES_EXTRA_DIR	Defines the directory for external libraries. To configure this environment variable, you must add the environment variable to the appropriate file. Set the variable to a directory outside of the `$SDC_DIST` runtime directory. Default directory: `$SDC_EXTERNAL_RESOURCES/streamsets-libs-extras` This resolves to the following directory unless you define the SDC_EXTERNAL_RESOURCES environment variable: `$SDC_DIST/externalResources/streamsets-libs-extras`
USER_LIBRARIES_DIR	Defines the directory for custom stage libraries. To configure this environment variable, you must add it to the appropriate file. Set the variable to a directory outside of the `$SDC_DIST` runtime directory. Default directory: `$SDC_EXTERNAL_RESOURCES/user-libs` This resolves to the following directory unless you define the SDC_EXTERNAL_RESOURCES environment variable: `$SDC_DIST/externalResources/user-libs`

User and Group for Service Start

When you run Data Collector as a service, Data Collector runs as the system user account and group defined in environment variables. The default system user and group are named sdc.

You can modify the values of the environment variables to point to another system user or group.

Modify environment variables using the method required by your installation type.

If you change the system user, you must make the new system user the owner of all Data Collector directories.

For example, if you change the system user and group to myuser, use the following command to change the owner of the configuration directory, $SDC_CONF, and all files in the directory to myuser:myuser:

chown -R myuser:myuser /etc/sdc

Note: When you run Data Collector manually, Data Collector runs as the system user account logged into the command prompt when the launch command is run. To run as another user account, see Full Installation and Launch (Manual Start).

Java Configuration Options

You define Java configuration options used by Data Collector in environment variables.

For a tarball or RPM installation, define Java configuration options in the following environment variables:

SDC_JAVA_OPTS - Includes configuration options for Java.
SDC_JAVA8_OPTS - Includes configuration options specific to Java 8.

Data Collector loads the value of the version-specific environment variable and adds it to the SDC_JAVA_OPTS environment variable.

When defining Java configuration options, avoid defining duplicate options. If you do define duplicates, the last option passed to the JVM usually takes precedence.

For a Cloudera Manager installation, define Java configuration options by configuring the StreamSets service through Cloudera Manager.

Note: To modify Java configuration options for the Data Collector command line interface, use the SDC_CLI_JAVA_OPTS environment variable. For more information, see Java Configuration Options for the Cli Command.

Java Heap Size

Modify the Data Collector Java heap size as necessary, based on the resources available on the host machine. By default, the Java heap size is 1024 MB.

The Java heap size determines the heap size allocated to Data Collector and affects the amount of memory Data Collector can use when it runs a pipeline. Running a pipeline can use up to 65% of the allocated heap size. For example, with a heap size of 2048 MB, you can configure a pipeline to use up to 65% - that's 1331 MB of memory.

Use the following Java options to define the Java heap size:

Xmx - Defines the maximum heap size.
Xms - Defines the minimum heap size.

Tip: To avoid constant recalculation of the allocated heap size, set both the minimum and maximum properties to the same value. To define the unit of measure, use m for MB and g for GB.

Note: By default, Java 8 and Java 11 enable the UseCompressedOops option, which allows a maximum of 32 GB of heap size regardless of the configured size. To allocate more than 32 GB, disable the option by adding the following Java option : -XX:-UseCompressedOops

Define the heap size based on your installation:

Tarball or RPM installation

Define the heap size in the SDC_JAVA_OPTS environment variable.

For example, to double the heap size, increase the Xmx and Xms settings as follows:

export SDC_JAVA_OPTS="${SDC_JAVA_OPTS} -Xmx2048m -Xms2048m -server"

Modify environment variables using the method required by your installation type.

Cloud service provider installation

Define the heap size percentage in the SDC_HEAP_SIZE_PERCENTAGE environment variable. Default is 50% of the available memory on the virtual machine.

To define a specific heap size instead of a percentage, comment out the SDC_HEAP_SIZE_PERCENTAGE environment variable and then define the heap size in the SDC_JAVA_OPTS environment variable as described above for an RPM installation.

Modify environment variables using the method required by your installation type.

Cloudera Manager installation

Define the heap size in the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-env.sh field for the StreamSets service in Cloudera Manager.

For example, to double the heap size, add the following to the sdc-env.sh safety valve:

export SDC_JAVA_OPTS="-Xmx2048m -Xms2048m"

Remote Debugging

You can enable remote debugging to debug a Data Collector instance running on a remote machine.

Enable remote debugging based on your installation:

Tarball or RPM installation

Define debugging options in the SDC_JAVA_OPTS environment variable.

Add the following debugging options to the environment variable, where port_number is an open port number on the remote machine running Data Collector:

-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=<port_number>,suspend=n

For example, to debug Data Collector on a remote machine using port number 2005, define SDC_JAVA_OPTS as follows:

export SDC_JAVA_OPTS="${SDC_JAVA_OPTS} -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=2005,suspend=n"

Modify environment variables using the method required by your installation type.

Cloudera Manager installation

Define the debugging options in the Java Options property for the StreamSets service in Cloudera Manager.

For example, to debug Data Collector on a remote machine using port number 2005, add the debugging options to the property as follows:

-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=2005,suspend=n

Garbage Collector

You can define the Java garbage collector that Data Collector uses. The default garbage collector depends on the Java version installed on the Data Collector machine:

Java 8 - Default is the Concurrent Mark Sweep (CMS) garbage collector.
Java 11 or later - Default is the G1 garbage collector.

If you define another garbage collector, test and evaluate Data Collector performance before making the same change in a production environment. Garbage collector performance depends on each particular use case.

Define the garbage collector based on your installation:

Tarball or RPM installation

Define the garbage collector in the SDC_JAVA8_OPTS environment variable.

For example, the default garbage collector is defined as follows:

export SDC_JAVA8_OPTS=${SDC_JAVA8_OPTS:-"-XX:+UseConcMarkSweepGC -XX:+UseParNewGC"}

To use the G1 garbage collector, set the option as follows:

export SDC_JAVA8_OPTS=${SDC_JAVA8_OPTS:-"-XX:+UseG1GC"}

Modify environment variables using the method required by your installation type.

Cloudera Manager installation

Define the garbage collector in the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-env.sh field for the StreamSets service in Cloudera Manager.

For example, to use the G1 garbage collector, add the following to the sdc-env.sh safety valve:

 export SDC_JAVA8_OPTS="-XX:+UseG1GC"

Logging

Data Collector enables garbage collector logging by default to facilitate troubleshooting. Log files are written to $SDC_LOG/gc.log. You can disable logging.

Disable garbage collector logging based on your installation:

Tarball or RPM installation

Set the SDC_GC_LOGGING environment variable to false. For example:

export SDC_GC_LOGGING=false

Modify environment variables using the method required by your installation type.

Cloudera Manager installation

Set the SDC_GC_LOGGING environment variable to false in the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-env.sh field for the StreamSets service in Cloudera Manager.

For example, add the following to the sdc-env.sh safety valve:

export SDC_GC_LOGGING=false

Security Manager for Java 8

When using Java 8, Data Collector includes a Java Security Manager that is enabled by default. For enhanced security, you can enable the Data Collector Security Manager which prevents stages from accessing files in protected Data Collector directories.

Important: Oracle has deprecated and marked Java Security Manager for removal. As a result, when using Java 9 or later, Data Collector cannot use either security manager.

When using Java 8, Data Collector can use one of the following security managers:

Java Security Manager

By default, Data Collector uses the Java Security Manager. The Java Security Manager restricts the runtime permissions of user libraries. This allows administrators to control user libraries actions on production systems. For example, by default, user libraries cannot call out to network resources and potentially cause denial-of-service (DDoS) attacks.

The security policy is defined in the $SDC_CONF/sdc-security.policy file. The file syntax is java standard.

Data Collector Security Manager

For enhanced security, enable the Data Collector Security Manager. The Data Collector Security Manager prevents stages from accessing files in protected Data Collector directories, regardless of how you define the $SDC_CONF/sdc-security.policy file.

To enable the Data Collector Security Manager, uncomment the security_manager.sdc_manager.enable property in the Data Collector configuration file, $SDC_CONF/sdc.properties.

Disabling Java Security Manager

If needed, you can disable Java Security Manager for Data Collector in the Data Collector configuration properties.

Note: Disabling Java Security Manager also disables Data Collector Security Manager if it is enabled.

To disable Java Security Manager, perform the following steps:

Use a text editor to open sdc.properties.
Uncomment the security_manager.sdc_manager.enable property, and set it to false.
To enable the changes, restart Data Collector.

Protected Directories

When using Java 8 and the Data Collector Security Manager is enabled, the following Data Collector directories are protected directories:

$SDC_CONF - Stages cannot access files in the configuration directory.
$SDC_DATA - Stages cannot access files in the data directory.
$SDC_EXTERNAL_RESOURCES - Stages can read files in the resources directory, but cannot write to files in the directory.
$SDC_RESOURCES - Stages can read files in the resources directory, but cannot write to files in the directory.

If needed, you can allow stages to access specific files in these protected directories by modifying Data Collector Security Manager exception properties in the $SDC_CONF/sdc-security.policy file. However, use caution when configuring exceptions to these protected directories.

You can configure exceptions to protected directories as follows:

Exceptions for all stage libraries

To allow all stage libraries access to files in protected directories, modify the security_manager.sdc_dirs.exceptions property to define files that can be accessed.

Exceptions for specific stage libraries

To allow a specific stage library access to files in protected directories, add the following property and then define the files that the stage library can access:

security_manager.sdc_dirs.exceptions.<stage_library_name>=<file_path>

For example, the default Data Collector configuration file includes an exception for the Java keystore credential store stage library defined as follows:

security_manager.sdc_dirs.exceptions.lib.streamsets-datacollector-jks-credentialstore-lib=$SDC_CONF/jks-credentialStore.pkcs12

When you configure a Security Manager exception property, use the appropriate directory environment variable in the file path: $SDC_CONF, $SDC_DATA, or $SDC_RESOURCES. You can enter multiple file paths separated by commas.

Root Classloader

You can edit the SDC_ROOT_CLASSPATH environment variable to define the path to JAR files to be added to the Data Collector root classloader.

Use the variable for components that must be in the root classloader, such as Snappy. Default is $SDC_DIST/root-lib/'*'.

Modify environment variables using the method required by your installation type.

Heap Dump Creation

By default, when Data Collector encounters an out of memory error (OOME), it creates a heap dump.

By default, heap dump files are written to the file defined in the SDC_LOG environment variable and use a naming convention that allows generating multiple heap dump files, as follows: $SDC_LOG/sdc_heapdump_${timestamp}.hprof.

You can change the name of the heap dump files, but we recommend using the ${timestamp} or similar variable to ensure that the heap dump name is unique.

Note that Java Virtual Machine, and therefore Data Collector, does not overwrite existing heap dump files. For example, if you use $SDC_LOG/sdc_heapdump.hprof as the file name, after Data Collector creates the first heap dump file, it will not create another until you remove the existing file.

Note: Depending on the number and size of the generated heap dump files, you might want to increase the Data Collector Java heap size.

You can configure the following heap dump environment variables:


Heap Dump Environment Variable	Description
SDC_HEAPDUMP_ON_OOM	Specifies whether Data Collector generates a heap dump upon encountering an out of memory error. Default is true.
SDC_HEAPDUMP_PATH	Specifies the file name and location to use for heap dump files. By default, heap dumps are written to `$SDC_LOG/sdc_heapdump_${timestamp}.hprof`. To specify a different file name or location, uncomment the property and enter the location and file name to use. Tip: To write multiple heap dump files to a directory, use a function or variable to ensure that the file name is unique. If a file of the same name exists in the directory, Data Collector does not create a new heap dump file.

Modify environment variables using the method required by your installation type.