Azure Data Lake Storage Gen2

The Azure Data Lake Storage Gen2 origin reads data from Microsoft Azure Data Lake Storage Gen2. The origin can create multiple threads to enable parallel processing in a multithreaded pipeline. To read data from Microsoft Azure Data Lake Storage Gen1, use the Azure Data Lake Storage Gen1 origin. For information about supported versions, see Supported Systems and VersionsSupported Systems and Versions in the Data Collector documentation.

The files to be processed must all share a file name pattern and be fully written. Use the origin only in pipelines configured for standalone execution mode.

When you configure the Azure Data Lake Storage Gen2 origin, you define the directory to use, the read order, the file name pattern, the file name pattern mode, and the first file to process. You can use glob patterns or regular expressions to define the file name pattern that you want to use.

You can configure the origin to read from subdirectories when the origin reads files by last modified timestamp. To use multiple threads for processing, specify the number of threads to use.

You can also enable reading compressed files. After processing a file, the Azure Data Lake Storage Gen2 origin can keep, archive, or delete the file.

When the pipeline stops, the Azure Data Lake Storage Gen2 origin notes where it stops reading. When the pipeline starts again, the origin continues processing from where it stopped by default. You can reset the origin to process all requested files.

The origin generates record header attributes that enable you to use the origins of a record in pipeline processing.

You can also use a connectionconnection to configure the origin.

The origin can generate events for an event stream. For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.