Azure Data Lake Storage Gen2 (Legacy)

Complete the following prerequisites before you configure the Azure Data Lake Storage Gen2 (Legacy) origin:

1. If necessary, create a new Azure Active Directory application for Data Collector.

   For information about creating a new application, see the Azure documentation.

2. Ensure that the Azure Active Directory Data Collector application has the appropriate access control to perform the necessary tasks.

   The Data Collector application requires Read and Execute permissions to read data in Azure. If also writing to Azure, the application requires Write permission as well.

   For information about configuring Gen2 access control, see the Azure documentation.

3. Retrieve information from Azure to configure the origin.

For example, suppose you have the following folders:

/hr/employees/CA/SFO
/hr/employees/CA/SJC
/hr/employees/CA/LAX
/hr/employees/WA/SEA

When defining the directory, you can include wildcards to have the origin read files from different combinations of folders. The following table shows some possibilities:

Last Modified Timestamp

The Azure Data Lake Storage Gen2 (Legacy) origin can read files in ascending order based on the last modified timestamp associated with the file. When the origin reads from a secondary location - not the directory where the files are created and written - the last-modified timestamp should be when the file is moved to the directory to be processed.

Tip: Avoid moving files using commands that preserve the existing timestamp, such as cp -p. Preserving the existing timestamp can be problematic in some cases, such as moving files across time zones.

When ordering based on timestamp, any files with the same timestamp are read in lexicographically ascending order based on the file names.

For example, when reading files with the log*.json file name pattern, the origin reads the following files in the following order:

File Name	Last Modified
log-1.json	APR 24 2016 14:03:35
log-903.json	APR 24 2016 14:05:03
log-2.json	APR 24 2016 14:45:11
log-3.json	APR 24 2016 14:45:11

Notice, the log-2.json and log-3.json files have identical timestamps, and so are processed in lexicographically ascending order based on their file names.

Lexicographically Ascending File Names

The Azure Data Lake Storage Gen2 (Legacy) origin can read files in lexicographically ascending order based on file names. Lexicographically ascending order reads the numbers 1 through 11 as follows:

1, 10, 11, 2, 3, 4... 9

For example, when reading files with the web*.log file name pattern, the Azure Data Lake Storage Gen2 (Legacy) origin reads the following files in the following order:

web-1.log
web-10.log
web-11.log
web-2.log
web-3.log
web-4.log
web-5.log
web-6.log
web-7.log
web-8.log
web-9.log

To read these files in logical and lexicographically ascending order, you might add leading zeros to the file naming convention as follows:

web-0001.log
web-0002.log
web-0003.log
...
web-0009.log
web-0010.log
web-0011.log

For example, say you configure the Azure Data Lake Storage Gen2 (Legacy) origin to read from the /logs/ file directory, select the Last Modified Timestamp read order, and enable reading from subdirectories. The Azure Data Lake Storage Gen2 (Legacy) origin reads the following files in the following order based on timestamp, even though the files are written to different subdirectories.

The Azure Data Lake Storage Gen2 (Legacy) origin creates the following record header attributes:

• avroSchema - When processing Avro data, provides the Avro schema.
• baseDir - Base directory containing the file where the record originated.

• filename - Provides the name of the file where the record originated.
• file - Provides the file path and file name where the record originated.
• mtime - Provides the last-modified time for the file.
• offset - Provides the file offset in bytes. The file offset is the location in the file where the record originated.

• atime - Provides the last accessed time.
• isDirectory - Indicates if the file is a directory.
• isSymbolicLink - Indicates if the file is a symbolic link.
• size - Provides the file size.
• owner - Provides the file owner.
• group - Provides the group associated with the file.
• blocksize - Provides the block size of the file.
• replication - Provides the replication of the file.
• isEncrypted - Indicates if the file is encrypted.

Events generated by the Azure Data Lake Storage Gen2 (Legacy) origin can be used in any logical way. For example:

• With the Pipeline Finisher executor to stop the pipeline and transition the pipeline to a Finished state when the origin completes processing available data.

  When you restart a pipeline stopped by the Pipeline Finisher executor, the origin continues processing from the last-saved offset unless you reset the origin.

  For an example, see Stopping a Pipeline After Processing All Available Data.

• With the Email executor to send a custom email after receiving an event.

  For an example, see Sending Email During Pipeline Processing.

• With a destination to store event information.

  For an example, see Preserving an Audit Trail of Events.

When a record is larger than the specified limit, the Azure Data Lake Storage Gen2 (Legacy) origin processes the source file based on the stage error handling:

Discard

The origin discards the record and all remaining records in the file, and then continues processing the next file.

Send to Error

With a buffer limit error, the origin cannot send the record to the pipeline for error handling because it is unable to fully process the record.

Instead, the origin creates a message stating that a buffer overrun error occurred. The message includes the file and offset where the buffer overrun error occurred. The information displays in the pipeline history.

If an error directory is configured for the stage, the origin moves the file to the error directory and continues processing the next file.

Stop Pipeline

The origin stops the pipeline and creates a message stating that a buffer overrun error occurred. The message includes the file and offset where the buffer overrun error occurred. The information displays in the pipeline history.

Note: You can also check the Data Collector log file for error details.

The Azure Data Lake Storage Gen2 (Legacy) origin processes data differently based on the data format. The origin processes the following types of data:

Avro

Generates a record for every Avro record. The origin includes the Avro schema in the avroSchema record header attribute. It also includes a precision and scale field attribute for each Decimal field.

The origin expects each file to contain the Avro schema and uses the schema to process the Avro data.

The origin reads files compressed by Avro-supported compression codecs without requiring additional configuration.

Delimited

Generates a record for each delimited line.

The CSV parser that you choose determines the delimiter properties that you configure and how the stage handles parsing errors. You can specify if the data includes a header line and whether to use it. You can define the number of lines to skip before reading, the character set of the data, and the root field type to use for the generated record.

You can also configure the stage to replace a string constant with null values and to ignore control characters.

For more information about reading delimited data, see Reading Delimited Data.

Excel

Generates a record for every row in the file. Can process .xls or .xlsx files.

You can configure the origin to read from all sheets in a workbook or from particular sheets in a workbook. You can specify whether files include a header row and whether to ignore the header row. You can also configure the origin to skip cells that do not have a corresponding header value. A header row must be the first row of a file. Vertical header columns are not recognized.

The origin cannot process Excel files with large numbers of rows. You can save such files as CSV files in Excel, and then use the origin to process with the delimited data format.

JSON

Generates a record for each JSON object. You can process JSON files that include multiple JSON objects or a single JSON array.

When an object exceeds the maximum object length defined for the origin, the origin cannot continue processing data in the file. Records already processed from the file are passed to the pipeline. The behavior of the origin is then based on the error handling configured for the stage:

• Discard - The origin continues processing with the next file, leaving the partially-processed file in the directory.
• To Error - The origin continues processing with the next file. If a post-processing error directory is configured for the stage, the origin moves the partially-processed file to the error directory. Otherwise, it leaves the file in the directory.
• Stop Pipeline - The origin stops the pipeline.

Log

Generates a record for every log line.

When a line exceeds the user-defined maximum line length, the origin truncates longer lines.

You can include the processed log line as a field in the record. If the log line is truncated, and you request the log line in the record, the origin includes the truncated line.

You can define the log format or type to be read.

Protobuf

Generates a record for every protobuf message.

Protobuf messages must match the specified message type and be described in the descriptor file.

When the data for a record exceeds 1 MB, the origin cannot continue processing data in the file. The origin handles the file based on file error handling properties and continues reading the next file.

For information about generating the descriptor file, see Protobuf Data Format Prerequisites.

SDC Record

Generates a record for every record. Use to process records generated by a Data Collector pipeline using the SDC Record data format.

For error records, the origin provides the original record as read from the origin in the original pipeline, as well as error information that you can use to correct the record.

When processing error records, the origin expects the error file names and contents as generated by the original pipeline.

Text

Generates a record for each line of text or for each section of text based on a custom delimiter.

When a line or section exceeds the maximum line length defined for the origin, the origin truncates it. The origin adds a boolean field named Truncated to indicate if the line was truncated.

For more information about processing text with a custom delimiter, see Text Data Format with Custom Delimiters.

Whole File

Streams whole files from the origin system to the destination system. You can specify a transfer rate or use all available resources to perform the transfer.

The origin generates two fields: one for a file reference and one for file information. For more information, see Whole File Data Format.

XML

Generates records based on a user-defined delimiter element. Use an XML element directly under the root element or define a simplified XPath expression. If you do not define a delimiter element, the origin treats the XML file as a single record.

Generated records include XML attributes and namespace declarations as fields in the record by default. You can configure the stage to include them in the record as field attributes.

You can include XPath information for each parsed XML element and XML attribute in field attributes. This also places each namespace in an xmlns record header attribute.

When a record exceeds the user-defined maximum record length, the origin cannot continue processing data in the file. Records already processed from the file are passed to the pipeline. The behavior of the origin is then based on the error handling configured for the stage:

• Discard - The origin continues processing with the next file, leaving the partially-processed file in the directory.
• To Error - The origin continues processing with the next file. If a post-processing error directory is configured for the stage, the origin moves the partially-processed file to the error directory. Otherwise, it leaves the file in the directory.
• Stop Pipeline - The origin stops the pipeline.

Use the XML data format to process valid XML documents. For more information about XML processing, see Reading and Processing XML Data.

Tip: If you want to process invalid XML documents, you can try using the text data format with custom delimiters. For more information, see Processing XML Data with Custom Delimiters.