Azure Data Lake Storage (Legacy) (deprecated)

1. In Active Directory, create a Data Collector web application.
2. Retrieve information from Azure to configure the destination.
3. Grant execute permission to the Data Collector web application.

 /outputfiles/${record:valueOrDefault("/State", "unknown")}/${YY()}-${MM()}-${DD()}-${hh()}

Constants

You can use any constant, such as output.

Datetime Variables

You can use datetime variables, such as ${YYYY()} or ${DD()}. The destination creates directories as needed, based on the smallest datetime variable that you use. For example, if the smallest variable is hours, then the directories are created for every hour of the day that receives output records.

When you use datetime variables in an expression, use all of the datetime variables between one of the year variables and the smallest variable that you want to use. Do not skip a variable within the progression. For example, to create directories on a daily basis, use a year variable, a month variable, and then a day variable. You might use one of the following datetime variable progressions:

${YYYY()}-${MM()}-${DD()}
${YY()}_${MM()}_${DD()}

For details about datetime variables, see Datetime Variables.

every function

You can use the every function in a directory template to create directories at regular intervals based on hours, minutes, or seconds, beginning on the hour. The intervals should be a submultiple or integer factor of 60. For example, you can create directories every 15 minutes or 30 seconds.

Use the every function to replace the smallest datetime variable used in the template.

For example, the following directory template creates directories every 5 minutes, starting on the hour:

/HDFS_output/${YYYY()}-${MM()}-${DD()}-${hh()}-${every(5,mm())}

For details about the every function, see Miscellaneous Functions.

record:valueOrDefault function

You can use the record:valueOrDefault function in a directory template to create directories with the value of a field or the specified default value if the field does not exist or if the field is null:

${record:valueOrDefault(<field path>, <default value>)}

For example, the following directory template creates a directory based on the Product field every day, and if the Product field is empty or null, uses Misc in the directory path:

/${record:valueOrDefault("/Product", "Misc")}/${YY()}-${MM()}-${DD()}

This template might create the following paths:

/Shirts/2015-07-31 
/Misc/2015-07-31

Processing Time

When you use processing time as the time basis, the destination creates directories based on the processing time and the directory template, and writes records to the directories based on when they are processed.

For example, say a directory template creates directories every minute and the time basis is the time of processing. Then, directories are created for every minute that the destination writes output records. And the output records are written to the directory for that minute of processing.

To use the processing time as the time basis, use the following expression: ${time:now()}. This is the default time basis.

Record Time

When you use the time associated with a record as the time basis, you specify a Date field in the record. The destination creates directories based on the datetimes associated with the records and writes the records to the appropriate directories.

For example, say a directory template creates directories every hour and the time basis is based on the record. Then, directories are created for every hour associated with output records and the destination writes the records to the related output directory.

To use a time associated with the record, use an expression that calls a field and resolves to a datetime value, such as ${record:value("/Timestamp")}.

• You configured the maximum number of records to be written to output files or the maximum size of output files, but records have stopped arriving. An output file that has not reached the maximum number of records or the maximum file size stays open until more records arrive.
• You configured a date field in the record as the time basis and have configured a late record time limit, but records arrive in chronological order. When a new directory is created, the output file in the previous directory remains open for the configured late record time limit. However, no records are ever written to the open file in the previous directory.

  For example, when a record with a datetime of 03:00 arrives, the destination creates a new file in a new 03 directory. The previous file in the 02 directory is kept open for the late record time limit, which is an hour by default. However, when records arrive in chronological order, no records that belong in the 02 directory arrive after the 03 directory is created.

• With the Email executor to send a custom email after receiving an event.

  For an example, see Sending Email During Pipeline Processing.

• With a destination to store event information.

  For an example, see Preserving an Audit Trail of Events.

Avro

The destination writes records based on the Avro schema. You can use one of the following methods to specify the location of the Avro schema definition:

• In Pipeline Configuration - Use the schema that you provide in the stage configuration.
• In Record Header - Use the schema included in the avroSchema record header attribute.
• Confluent Schema Registry - Retrieve the schema from Confluent Schema Registry. Confluent Schema Registry is a distributed storage layer for Avro schemas. You can configure the destination to look up the schema in Confluent Schema Registry by the schema ID or subject.

  If using the Avro schema in the stage or in the record header attribute, you can optionally configure the destination to register the Avro schema with Confluent Schema Registry.

The destination includes the schema definition in each file.

You can compress data with an Avro-supported compression codec. When using Avro compression, avoid using other compression properties in the destination.

Binary

The stage writes binary data to a single field in the record.

Delimited

The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.

You can use the following delimited format types:

• Default CSV - File that includes comma-separated values. Ignores empty lines in the file.
• RFC4180 CSV - Comma-separated file that strictly follows RFC4180 guidelines.
• MS Excel CSV - Microsoft Excel comma-separated file.
• MySQL CSV - MySQL comma-separated file.
• Tab-Separated Values - File that includes tab-separated values.
• PostgreSQL CSV - PostgreSQL comma-separated file.
• PostgreSQL Text - PostgreSQL text file.
• Custom - File that uses user-defined delimiter, escape, and quote characters.
• Multi Character Delimited - File that uses multiple user-defined characters to delimit fields and lines, and single user-defined escape and quote characters.

JSON

The destination writes records as JSON data. You can use one of the following formats:

• Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
• Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.

Protobuf

Writes a batch of messages in each file.

Uses the user-defined message type and the definition of the message type in the descriptor file to generate the messages in the file.

For information about generating the descriptor file, see Protobuf Data Format Prerequisites.

Text

The destination writes data from a single text field to the destination system. When you configure the stage, you select the field to use.

You can configure the characters to use as record separators. By default, the destination uses a UNIX-style line ending (\n) to separate records.

When a record does not contain the selected text field, the destination can report the missing field as an error or to ignore the missing field. By default, the destination reports an error.

When configured to ignore a missing text field, the destination can discard the record or write the record separator characters to create an empty line for the record. By default, the destination discards the record.

Whole File

Streams whole files to the destination system. The destination writes the data to the file and location defined in the stage. If a file of the same name already exists, you can configure the destination to overwrite the existing file or send the current file to error.

By default, written files use the default access permissions for the destination system. You can specify an expression that defines access permissions.

For more information about the whole file data format, see Whole File Data Format.