Hadoop FS Standalone

/hr/employees/CA/SFO
/hr/employees/CA/SJC
/hr/employees/CA/LAX
/hr/employees/WA/SEA

Last Modified Timestamp

The Hadoop FS Standalone origin can read files in ascending order based on the last modified timestamp associated with the file. When the origin reads from a secondary location - not the directory where the files are created and written - the last-modified timestamp should be when the file is moved to the directory to be processed.

Tip: Avoid moving files using commands that preserve the existing timestamp, such as cp -p. Preserving the existing timestamp can be problematic in some cases, such as moving files across time zones.

When ordering based on timestamp, any files with the same timestamp are read in lexicographically ascending order based on the file names.

For example, when reading files with the log*.json file name pattern, the origin reads the following files in the following order:

File Name	Last Modified
log-1.json	APR 24 2016 14:03:35
log-903.json	APR 24 2016 14:05:03
log-2.json	APR 24 2016 14:45:11
log-3.json	APR 24 2016 14:45:11

Notice, the log-2.json and log-3.json files have identical timestamps, and so are processed in lexicographically ascending order based on their file names.

Lexicographically Ascending File Names

The Hadoop FS Standalone origin can read files in lexicographically ascending order based on file names. Lexicographically ascending order reads the numbers 1 through 11 as follows:

1, 10, 11, 2, 3, 4... 9

For example, when reading files with the web*.log file name pattern, the Hadoop FS Standalone origin reads the following files in the following order:

web-1.log
web-10.log
web-11.log
web-2.log
web-3.log
web-4.log
web-5.log
web-6.log
web-7.log
web-8.log
web-9.log

To read these files in logical and lexicographically ascending order, you might add leading zeros to the file naming convention as follows:

web-0001.log
web-0002.log
web-0003.log
...
web-0009.log
web-0010.log
web-0011.log

1. On the General tab of the Hadoop FS Standalone origin, set the Stage Library property to the HDP stage library version 2.4 or later.
2. Configure Azure credentials in one of the following ways:
   • If the Azure credentials are defined in the HDFS configuration file core-site.xml, configure the origin to access the file.
     1. On the Connection tab, configure the Configuration Files Directory property to point to the directory that includes the file.
   • If the credentials are not defined in the core-site.xml file, configure the origin to pass an additional configuration property that contains the Azure credentials:
     1. On the Connection tab, configure the Additional Configuration property to pass the Azure credentials.

        You can use simple or bulk edit mode to add configuration properties.

     2. If necessary, click the Add icon to add a new additional configuration property.
     3. In the Name property, enter the Azure storage account name as follows:

        fs.azure.account.key.<storage account name>.blob.core.windows.net

        For example, if the storage account name is sdchd, then enter the following name for the property:

        fs.azure.account.key.sdchd.blob.core.windows.net

        Tip: You can find the Azure storage account name on the Access Keys page in the Microsoft Azure portal. To view the page in the Microsoft Azure portal, click All Resources > Storage Account > Access Keys. A page like the following appears, with the storage account name and access keys:

        

     4. In the Value property, enter an access key value for the Azure storage account. You can use any valid key.
        Tip: The account key value also displays on the Access Keys page. For example, on the image above, you could use either the key1 or key2 value.
3. On the Connection tab, configure the File System URI property as follows:

   <wasb[s]>://<container name>@<storage account name>.blob.core.windows.net/<path to files>

   In the URI, <container name> is the Azure container name, and <storage account name> is the Azure storage account name.

   For example, for a sdc-hd container in a storage account named sdchd, with all files in a files directory, you would define the file system URI as follows:

   wasbs://sdc-hd@sdchd.blob.core.windows.net/files

   Tip: You can find the container name and storage account name on the Essentials page in the Microsoft Azure portal. For a standard storage account, in the Microsoft Azure portal, click All Resources > Storage Account > Overview > Blobs. For a blob storage account, click All Resources > Storage Account > Overview.

   The page shows the container name and storage account name:

   

   Though the host name for the file system URI is <storage account name>.blob.core.windows.net, you can alternatively use the host name of the Azure blob service endpoint as the host name for the file system URI.

• avroSchema - When processing Avro data, provides the Avro schema.
• baseDir - Base directory containing the file where the record originated.

• filename - Provides the name of the file where the record originated.
• file - Provides the file path and file name where the record originated.
• mtime - Provides the last-modified time for the file.
• offset - Provides the file offset in bytes. The file offset is the location in the file where the record originated.

• atime - Provides the last accessed time.
• isDirectory - Indicates if the file is a directory.
• isSymbolicLink - Indicates if the file is a symbolic link.
• size - Provides the file size.
• owner - Provides the file owner.
• group - Provides the group associated with the file.
• blocksize - Provides the block size of the file.
• replication - Provides the replication of the file.
• isEncrypted - Indicates if the file is encrypted.

• With the Pipeline Finisher executor to stop the pipeline and transition the pipeline to a Finished state when the origin completes processing available data.

  When you restart a pipeline stopped by the Pipeline Finisher executor, the origin continues processing from the last-saved offset unless you reset the origin.

  For an example, see Stopping a Pipeline After Processing All Available Data.

• With the Email executor to send a custom email after receiving an event.

  For an example, see Sending Email During Pipeline Processing.

• With a destination to store event information.

  For an example, see Preserving an Audit Trail of Events.

Discard

The origin discards the record and all remaining records in the file, and then continues processing the next file.

Send to Error

With a buffer limit error, the origin cannot send the record to the pipeline for error handling because it is unable to fully process the record.

Instead, the origin creates a message stating that a buffer overrun error occurred. The message includes the file and offset where the buffer overrun error occurred. The information displays in the pipeline history.

If an error directory is configured for the stage, the origin moves the file to the error directory and continues processing the next file.

Stop Pipeline

The origin stops the pipeline and creates a message stating that a buffer overrun error occurred. The message includes the file and offset where the buffer overrun error occurred. The information displays in the pipeline history.

Note: You can also check the Data Collector log file for error details.

HDFS configuration files

You can use the following HDFS configuration files with the Hadoop FS Standalone origin:

• core-site.xml
• hdfs-site.xml

To use HDFS configuration files:

1. Store the files or a symlink to the files in the Data Collector resources directory.
2. In the Hadoop FS Standalone origin, configure the Configuration Files Directory property to specify the location of the files.

Note: For a Cloudera Manager installation, Data Collector automatically creates a symlink to the files named hadoop-conf. Enter hadoop-conf for the location of the files in the Hadoop FS Standalone origin.

Individual properties

You can configure individual HDFS properties in the origin. To add an HDFS property, you specify the exact property name and the value. The Hadoop FS Standalone origin does not validate the property names or values.

Note: Individual properties override properties defined in HDFS configuration files.

1. On Hadoop, configure the user as a proxy user and authorize the user to impersonate a Hadoop user.

   For more information, see the Hadoop documentation.

2. In the Hadoop FS Standalone origin, on the Connection tab, configure the Impersonation User property.

Avro

Generates a record for every Avro record. The origin includes the Avro schema in the avroSchema record header attribute. It also includes a precision and scale field attribute for each Decimal field.

The origin expects each file to contain the Avro schema and uses the schema to process the Avro data.

The origin reads files compressed by Avro-supported compression codecs without requiring additional configuration.

Delimited

Generates a record for each delimited line.

The CSV parser that you choose determines the delimiter properties that you configure and how the stage handles parsing errors. You can specify if the data includes a header line and whether to use it. You can define the number of lines to skip before reading, the character set of the data, and the root field type to use for the generated record.

You can also configure the stage to replace a string constant with null values and to ignore control characters.

For more information about reading delimited data, see Reading Delimited Data.

Excel

Generates a record for every row in the file. Can process .xls or .xlsx files.

You can configure the origin to read from all sheets in a workbook or from particular sheets in a workbook. You can specify whether files include a header row and whether to ignore the header row. You can also configure the origin to skip cells that do not have a corresponding header value. A header row must be the first row of a file. Vertical header columns are not recognized.

The origin cannot process Excel files with large numbers of rows. You can save such files as CSV files in Excel, and then use the origin to process with the delimited data format.

JSON

Generates a record for each JSON object. You can process JSON files that include multiple JSON objects or a single JSON array.

When an object exceeds the maximum object length defined for the origin, the origin cannot continue processing data in the file. Records already processed from the file are passed to the pipeline. The behavior of the origin is then based on the error handling configured for the stage:

• Discard - The origin continues processing with the next file, leaving the partially-processed file in the directory.
• To Error - The origin continues processing with the next file. If a post-processing error directory is configured for the stage, the origin moves the partially-processed file to the error directory. Otherwise, it leaves the file in the directory.
• Stop Pipeline - The origin stops the pipeline.

Log

Generates a record for every log line.

When a line exceeds the user-defined maximum line length, the origin truncates longer lines.

You can include the processed log line as a field in the record. If the log line is truncated, and you request the log line in the record, the origin includes the truncated line.

You can define the log format or type to be read.

Protobuf

Generates a record for every protobuf message.

Protobuf messages must match the specified message type and be described in the descriptor file.

When the data for a record exceeds 1 MB, the origin cannot continue processing data in the file. The origin handles the file based on file error handling properties and continues reading the next file.

For information about generating the descriptor file, see Protobuf Data Format Prerequisites.

SDC Record

Generates a record for every record. Use to process records generated by a Data Collector pipeline using the SDC Record data format.

For error records, the origin provides the original record as read from the origin in the original pipeline, as well as error information that you can use to correct the record.

When processing error records, the origin expects the error file names and contents as generated by the original pipeline.

Text

Generates a record for each line of text or for each section of text based on a custom delimiter.

When a line or section exceeds the maximum line length defined for the origin, the origin truncates it. The origin adds a boolean field named Truncated to indicate if the line was truncated.

For more information about processing text with a custom delimiter, see Text Data Format with Custom Delimiters.

Whole File

Streams whole files from the origin system to the destination system. You can specify a transfer rate or use all available resources to perform the transfer.

The origin generates two fields: one for a file reference and one for file information. For more information, see Whole File Data Format.

XML

Generates records based on a user-defined delimiter element. Use an XML element directly under the root element or define a simplified XPath expression. If you do not define a delimiter element, the origin treats the XML file as a single record.

Generated records include XML attributes and namespace declarations as fields in the record by default. You can configure the stage to include them in the record as field attributes.

You can include XPath information for each parsed XML element and XML attribute in field attributes. This also places each namespace in an xmlns record header attribute.

When a record exceeds the user-defined maximum record length, the origin cannot continue processing data in the file. Records already processed from the file are passed to the pipeline. The behavior of the origin is then based on the error handling configured for the stage:

• Discard - The origin continues processing with the next file, leaving the partially-processed file in the directory.
• To Error - The origin continues processing with the next file. If a post-processing error directory is configured for the stage, the origin moves the partially-processed file to the error directory. Otherwise, it leaves the file in the directory.
• Stop Pipeline - The origin stops the pipeline.

Use the XML data format to process valid XML documents. For more information about XML processing, see Reading and Processing XML Data.

Tip: If you want to process invalid XML documents, you can try using the text data format with custom delimiters. For more information, see Processing XML Data with Custom Delimiters.