Self-Managed Deployment Requirements

When your organization uses deployed Transformer for Snowflake engines, you can create self-managed deployments to take full control of procuring the resources needed to run a Transformer for Snowflake engine.

Before launching a Transformer for Snowflake engine for a self-managed deployment, set up a machine with the following requirements, as appropriate.

Minimum System Requirements

For self-managed deployments, install Transformer for Snowflake on machines that meet the following minimum requirements:


Component	Minimum System Requirement
Cores	2
Disk space	6 GB StreamSets does not recommend using NFS or NAS to store Transformer for Snowflake files.
Operating system	One of the following operating systems and versions: CentOS 6.x or 7.x Oracle Linux 6.x - 8.x Red Hat Enterprise Linux 6.x - 8.x Ubuntu 14.04 LTS or 16.04 LTS
RAM	1 GB

Tarball Prerequisites

For a tarball installation of Transformer for Snowflake, the machine must meet the following additional requirements:


Component	Minimum Requirement
File descriptors	32768
Java	Oracle Java 11 or OpenJDK 11

Configuring the Open File Limit

Transformer for Snowflake requires a large number of file descriptors to work correctly with all stages. Most operating systems provide a configuration to limit the number of files a process or a user can open. The default values are usually less than the Transformer for Snowflake requirement of 32768 file descriptors.

Use the following command to verify the configured limit for the current user:

ulimit -n

Most operating systems use two ways of configuring the maximum number of open files - the soft limit and the hard limit. The hard limit is set by the system administrator. The soft limit can be set by the user, but only up to the hard limit.

Increasing the open file limit differs for each operating system. Consult your operating system documentation for the preferred method.

Increase the Limit on Linux

To increase the open file limit on Linux, see the following solution: How to set ulimit values.

This solution should work on Red Hat Enterprise Linux, Oracle Linux, CentOS, and Ubuntu. However, refer to the administrator documentation for your operating system for the preferred method.

Increase the Limit on MacOS

The method you use to increase the limit on MacOS can differ with each version. Refer to the documentation for your operating system version for the preferred method.

To increase the limit for the computer - so that the limits are retained after relaunching the terminal and restarting the computer - create a property list file. The following steps are valid for recent MacOS versions:

Use the following command to create a property list file named limit.maxfiles.plist:
```
sudo vim /Library/LaunchDaemons/limit.maxfiles.plist
```

Add the following contents to the file, modifying the maxfiles attribute as needed.

The maxfiles attribute defines the open file limit. The first value in the file is the soft limit. The second value is the hard limit.

For example, in the following limit.maxfiles.plist file, both the soft and hard limit are set to 32,768:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>limit.maxfiles</string>
    <key>ProgramArguments</key>
    <array>
      <string>launchctl</string>
      <string>limit</string>
      <string>maxfiles</string>
      <string>32768</string>
      <string>32768</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>ServiceIPC</key>
    <false/>
  </dict>
</plist>

Use the following commands to load the new settings:

sudo launchctl unload -w /Library/LaunchDaemons/limit.maxfiles.plist
sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist

Use the following command to check that the system limits were modified:
```
launchctl limit maxfiles
```
Use the following command to set the session limit:
```
ulimit -n 32768
```

Docker Image Requirement

For a Docker image deployment of Transformer for Snowflake, the machine must also have Docker installed.