Hive

Available when using an authoring Data Collector version 5.0.0 or later.

To create a Hive connection, one of the following stage libraries must be installed on the selected authoring Data Collector:
  • Cloudera CDP, streamsets-datacollector-cdp_<version>-lib
  • MapR with MEP, streamsets-datacollector-mapr_<version>-mep<version>-lib

For a description of the Hive connection properties, see Hive Connection Properties.

After you create a Hive connection, you can use the connection in the following stages:

Engine Stages
Data Collector 5.0.0 or later
  • Hive Metadata processor
  • Hive Metastore destination
  • Hive Query executor

Hive Connection Properties

When creating a Hive connection, configure the following properties on the Hive tab:
Hive Property Description
JDBC URL

JDBC URL for Hive. For details about specifying the URL, see our StreamSets Community post.

You can optionally include the user name and password in the JDBC URL. If you include a password with special characters, you must URL-encode (also called percent-encoding) the special characters. Otherwise errors will occur when validating or running your pipeline. For example, if your JDBC URL looks like this:
jdbc:hive2://sunnyvale:12345/default;user=admin;password=a#b!c$e
URL-encode your password so that your JDBC URL looks like this:
jdbc:hive2://sunnyvale:12345/default;user=admin;password=a%23b%21c%24e
Tip: To secure sensitive information, you can use credential stores or runtime resources.
JDBC Driver Name The fully-qualified JDBC driver name.

Before using an Impala JDBC driver for the Hive Query executor, install the driver as an external library for the stage library used by the executor. For more information, see Installing the Impala Driver.

Use Credentials Enables entering credentials in properties. Use when you do not include credentials in the JDBC URL.
Note: To impersonate the current user in connections to Hive, you can edit the Data Collector configuration file, $SDC_CONF/sdc.properties to configure Data Collector to automatically impersonate the user without specifying credentials in the pipeline. See Configuring Data Collector in the Data Collector documentation.
Username User name for the JDBC connection.

The user account must have the correct permissions or privileges in the database.

Password Password for the JDBC user name.
Tip: To secure sensitive information, you can use credential stores or runtime resources.
Additional JDBC Configuration Properties Additional JDBC configuration properties to pass to the JDBC driver.

Using simple or bulk edit mode, click Add to add additional properties and define the property name and value. Use the property names and values as expected by the JDBC driver.

Hadoop Configuration Directory

Absolute path to the directory containing the following Hive and Hadoop configuration files:

  • core-site.xml
  • hdfs-site.xml
  • hive-site.xml

For a Cloudera Manager installation, enter hive-conf.

Note: Properties in the configuration files are overridden by individual properties defined in the Additional Hadoop Configuration property.
Additional Hadoop Configuration

Additional properties to use.

Using simple or bulk edit mode, click Add to add additional properties and define the property name and value. Use the property names and values as expected by HDFS and Hive.