Hive Metadata

metadata record

When encountering compatible metadata changes, the Hive Metadata processor generates a metadata record. The metadata record passes the following information to the Hive Metastore destination:

• The expected table structure for compatible changes, based on the record.

  Compatible changes include new tables and partitions, and the addition or removal of fields in the record. Changes in data type are not compatible.

• Any user-defined record header attributes configured in the stage.

When the Hive Metastore destination receives the metadata record, the destination performs a final check against Hive metadata and creates or alters tables as needed.

data record header attributes

The Hive Metadata processor adds the following attributes to the record header for data records:

• targetDirectory - The location where each record should be written.

  The processor generates the directory based on the database, table, and partition information for each record and writes it to the targetDirectory header attribute.

  When processing Parquet data, the Hive Metadata processor adds .avro to the target directory that it generates for each record. This allows the data-processing destination to write the Avro files to a directory that Hive ignores as a temporary directory.

  As a result, the destination writes files to the following directories: <generated directory>/.avro.

  Note: You can configure the MapReduce executor to write the Parquet files to the parent generated directory and to delete the Avro files after processing them. You can also delete the temporary directories after the files are processed, as needed.

  To use this header attribute, configure the Hadoop FS or MapR FS destination to write records using the directory in the record header.

• avroSchema - The Avro schema for the record.

  The processor writes the Avro schema in the avroSchema header attribute for each record. When the processor notes a compatible change in the Avro schema, it generates a new Avro schema. This attribute is used for both Avro and Parquet data.

  To use this header attribute, configure the Hadoop FS or MapR FS destination to write records using the Avro schema in the record header.

• roll - An indicator to roll the file associated with the record.

  The processor generates a roll indicator only when the Avro schema changes in a compatible way. Records with incompatible changes are sent to the stage for error handling.

  To use this header attribute, configure the Hadoop FS or MapR FS destination to roll records when encountering the roll attribute in the record header. And then, use the default "roll" as the name of the header attribute.

For more information about using destinations to process these attributes, see Record Header Attributes for Record-Based Writes. For general information about record header attributes, see Record Header Attributes.

Database and table expressions

The database expression represents the database where Hadoop FS or MapR FS destination should write the record. If you omit the database expression, the processor uses the default Hive database.

The table expression represents the table to use. If the table doesn't exist, the processor generates a metadata record to create the table.

The database and table expressions are also incorporated into the targetDirectory to allow record-based writes to the database.

Tips for configuring the database and table expressions:

• If all records are to be written to a single database or table, you can enter the database or table name instead of an expression.
• If the database or table name can be extrapolated from record data or header attributes, you can enter an expression that evaluates to the database or table name.
• When necessary, you can use an Expression Evaluator earlier in the pipeline to perform calculations and write the results to a new field or a header attribute, to be used by the Hive Metadata processor.

Partition configuration information

You can optionally configure partition properties to write to partitions. When you configure partition information, you state the Hive partition column name, an expression that evaluates to the partition name, and the data format of the partition data. You can use the Int, Bigint, and String data formats for partition data.

Like with database and table expressions, you can configure the partition expression as needed:

• If all records are to be written to a single partition, you can enter the partition name for the expression.
• If the partition depends on information in the record, you can enter an expression that evaluates to the partition.
• When necessary, you might use an Expression Evaluator earlier in the pipeline to generate the partition name and write it to the record as a new field or the record header as a header attribute.

You can use datetime variables such as ${YYYY()} or ${DD()} to create datetime-based partitions. When creating datetime-based partitions, consider the time basis that you want to use. By default, the processor uses the time of processing as the time basis, but you can use the time associated with a record as well.

For details about datetime variables, see Datetime Variables.

${record:attribute(str:concat(str:concat('jdbc.', field:field()), '.precision'))}
${record:attribute(str:concat(str:concat('jdbc.', field:field()), '.scale'))}

${record:attribute(jdbc.<fieldname>.precision)}
${record:attribute(jdbc.<fieldname>.scale)}

processing time

When you use processing time as the time basis, the processor uses the processing time and the partition value expression to determine the partition value to use in the metadata record and the partition portion of the targetDirectory header attribute.

For example, say a partition value expression creates a new partition every day and the time basis is the time of processing. Then, the processor generates a daily metadata record that the Hive Metastore destination uses to create the daily partition. And the processor adds the daily partition value to the targetDirectory path.

To use the processing time as the time basis, use the following expression: ${time:now()}. This is the default time basis.

record-based time

When you use the time associated with a record as the time basis, you specify a Date field in the record as part of the partition value expression. The processor uses the datetimes associated with the records and the partition value expression to determine the partition value to use in the metadata record and the partition portion of the targetDirectory header attribute.

For example, say a partition value expression creates directories every hour and the time basis is based on the record. Then, for every hour associated with a record, the processor generates a metadata record so the Hive Metastore destination can create hourly partitions as needed. And the processor adds the hourly partition value to the targetDirectory path.

To use a time associated with the record, use an expression that calls a field and resolves to a datetime value, such as ${record:value("/Timestamp")}.

• Database and table to be written to
• Hive table properties
• Column names and data types in the table
• Avro schema
• Partition values