MapR DB

The MapR DB destination writes data to MapR DB binary tables. The destination can write data to MapR DB as text, binary data, or JSON strings. You can define the data format for each column written to MapR DB.

MapR is now HPE Ezmeral Data Fabric. At times, this documentation uses "MapR" to refer to both MapR and HPE Ezmeral Data Fabric. For information about supported versions, see Supported Systems and Versions in the Data Collector documentation.
Note: To write JSON documents to MapR DB JSON tables, use the MapR DB JSON destination. For more information, see MapR DB JSON.

When you configure the MapR DB destination, you specify the MapR DB configuration properties, including the table name. You specify the row key for the table, and then map fields from the pipeline to MapR DB columns.

When necessary, you can enable Kerberos authentication and specify an HBase user. You can also use HDFS configuration files and add other HDFS configuration properties as needed.

Before you use any MapR stage in a pipeline, you must perform additional steps to enable Data Collector to process MapR data. For more information, see MapR Prerequisites in the Data Collector documentation.

Note: The MapR DB destination currently supports the HBase APIs only.

Field Mappings

When you configure the MapR DB destination, you map fields from records to MapR DB columns.

You can map fields to columns in the following ways:

Explicit field mappings
By default, the MapR DB destination uses explicit field mappings. You select the fields from records to map to MapR DB columns. Specify the MapR DB columns using the following format: <column-family>:<qualifier>. You then define the storage type for the column in MapR DB.
When you use explicit field mappings, you can configure the destination to ignore missing field paths. If the destination encounters a mapped field path that doesn’t exist in the record, the destination ignores the missing field path and writes the remaining fields in the record to MapR DB.
Implicit field mappings
When you configure the MapR DB destination to use implicit field mappings, the destination writes data based on the matching field names. You can use implicit field mappings when the field paths use the following format:
<column-family>:<qualifier>
For example, if a field path is "cf:a", the destination can implicitly map the field to the MapR DB table with the column family "cf" and the qualifier "a".
When you use implicit field mappings, you can configure the destination to ignore invalid columns. If the destination encounters a field path that cannot be mapped to a valid MapR DB column, the destination ignores the invalid column and writes the remaining fields in the record to MapR DB.
Both implicit and explicit field mappings
You can configure the destination to use implicit field mappings and then you can override the mappings by defining explicit mappings for specific fields.
For example, a record might contain some field paths that use the <column-family>:<qualifier> format and other field paths that don’t use the required format. You can add explicit field mappings for the field paths that do not use the required format. Or, you can use explicit field mappings for fields that use the required format, but need to be written to a different column.

Time Basis

The time basis determines the timestamp value added for each column written to MapR DB.

You can use the following times as the time basis:

Processing Time
When you use processing time as the time basis, the destination uses the Data Collector processing time as the timestamp value. The processing time is calculated once per batch.
To use the processing time as the time basis, use the following expression: ${time:now()}.
Record Time
When you use the time associated with a record as the time basis, you specify a Date or Datetime field in the record. The destination uses the field value as the timestamp value.
To use a time associated with the record, use an expression that calls a field and resolves to a date or datetime value, such as ${record:value("/Timestamp")}.
System Time
When you leave the Time Basis field empty, the destination uses the timestamp value automatically generated by MapR when the column is written to MapR DB.
This is the default time basis.

Kerberos Authentication

You can use Kerberos authentication to connect to MapR DB. When you use Kerberos authentication, Data Collector uses the Kerberos principal and keytab to connect to MapR DB. By default, Data Collector uses the user account who started it to connect.

The Kerberos principal and keytab are defined in the Data Collector configuration file, $SDC_CONF/sdc.properties. To use Kerberos authentication, configure all Kerberos properties in the Data Collector configuration file.

For more information about enabling Kerberos authentication for Data Collector, see Kerberos Authentication in the Data Collector documentation.

Using an HBase User

Data Collector can either use the currently logged in Data Collector user or a user configured in the destination to write to MapR DB.

A Data Collector configuration property can be set that requires using the currently logged in Data Collector user. When this property is not set, you can specify a user in the origin. For more information about Hadoop impersonation and the Data Collector property, see Hadoop Impersonation Mode in the Data Collector documentation.

Note that the destination uses a different user account to connect.By default, Data Collector uses the user account who started it to connect to external systems. When using Kerberos, Data Collector uses the Kerberos principal.

To configure a user in the destination to write to MapR DB, perform the following tasks:
  1. On MapR, configure the user as a proxy user and authorize the user to impersonate the HBase user.

    For more information, see the HBase documentation.

  2. In the MapR DB destination, enter the HBase user name.

HDFS Properties and Configuration File

You can configure the MapR DB destination to use individual HDFS properties or HDFS configuration files:

HBase configuration file
You can use the following HDFS configuration file with the MapR DB destination:
  • hbase-site.xml
To use HDFS configuration files:
  1. Store the files or a symlink to the files in the Data Collector resources directory.
  2. In the MapR DB destination, specify the location of the files.
Individual properties
You can configure individual HBase properties in the MapR DB destination. To add an HBase property, you specify the exact property name and the value. The MapR DB destination does not validate the property names or values.
Note: Individual properties override properties defined in the HBase configuration file.

Configuring a MapR DB Destination

Configure a MapR DB destination to write data as text, binary data, or JSON strings to MapR DB binary tables.

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    Stage Library Library version that you want to use.
    Required Fields Fields that must include data for the record to be passed into the stage.
    Tip: You might include fields that the stage uses.

    Records that do not include all required fields are processed based on the error handling configured for the pipeline.
    Preconditions Conditions that must evaluate to TRUE to allow a record to enter the stage for processing. Click Add to create additional preconditions.

    Records that do not meet all preconditions are processed based on the error handling configured for the stage.
    On Record Error Error record handling for the stage:
    • Discard - Discards the record.
    • Send to Error - Sends the record to the pipeline for error handling.
    • Stop Pipeline - Stops the pipeline.
  2. On the HBase tab, configure the following properties:
    HBase Property Description
    Table Name Name of the MapR DB binary table to use. Enter a table name or a namespace and table name as follows: <namespace>.<tablename>.

    If you do not enter a table name, MapR uses the default namespace.

    Row Key Row key for the table.
    Storage Type Storage type of the row key.
    Fields Explicitly map fields from records to MapR DB columns, and then define the storage type for the column in MapR DB.

    Using simple or bulk edit mode, click the Add icon to create additional explicit field mappings.

    Ignore Missing Field Path Ignores missing field paths. Used when you define explicit field mappings.

    If selected and the destination encounters a mapped field path that doesn’t exist in the record, the destination ignores the missing field path and writes the remaining fields in the record to MapR DB. If cleared and the destination encounters a mapped field path that doesn't exist in the record, the record is sent to the stage for error handling.
    Implicit Field Mapping Uses implicit field mappings so that the destination writes data to MapR DB columns based on the matching field names. The field paths must use the following format:
    <column-family>:<qualifier>
    Ignore Invalid Column Ignores invalid columns. Used when you configure implicit field mappings.

    If selected and the destination encounters a field path that cannot be mapped to a valid MapR DB column, the destination ignores the invalid column and writes the remaining fields in the record to MapR DB. If cleared and the destination encounters an invalid column, the record is sent to the stage for error handling.

    Kerberos Authentication Uses Kerberos credentials to connect to MapR DB.

    When selected, uses the Kerberos principal and keytab defined in the Data Collector configuration file, $SDC_CONF/sdc.properties.

    HBase User The HBase user to use to write to MapR DB. When using this property, make sure MapR DB is configured appropriately.

    When not configured, the pipeline uses the currently logged in Data Collector user.

    Not configurable when Data Collector is configured to use the currently logged in Data Collector user. For more information, see Hadoop Impersonation Mode in the Data Collector documentation.
    Time Basis Time basis to use for the timestamp value added to each column written to MapR DB. Use one of the following expressions:
    • ${time:now()} - Uses the Data Collector processing time as the time basis.
    • ${record:value(<date field path>)} - Uses the time associated with the record as the time basis.

    Or, leave empty to use the system time automatically generated by MapR as the time basis.

    HBase Configuration Directory Location of the HDFS configuration files. Use a directory or symlink within the Data Collector resources directory.
    You can use the following file with the MapR DB destination:
    • hbase-site.xml
    Note: Properties in the configuration files are overridden by individual properties defined in the stage.
    HBase Configuration

    Additional HBase configuration properties to use.

    To add properties, click Add and define the property name and value. Use the property names and values as expected by MapR DB.