Whole File Transformer

The Write Avro pipeline reads data from the origin, performs any required processing, and uses a file-based Avro destination such as Local FS to write Avro files to a file system.

The destination file system acts as a staging area. As the Write Avro pipeline closes each Avro file, the Parquet Conversion pipeline can begin processing the file and convert it to Parquet.

Since each Avro file is ultimately transformed to a corresponding Parquet file, the Avro files should be of substantial size to take advantage of Parquet capabilities. 2 GB is recommended.

The Write Avro pipeline includes any origin, any number of processors, and an Avro-enabled file-based destination, as follows:

For a list of destinations that support writing Avro, see Data Format Support.

The Parquet Conversion pipeline processes the closed Avro files as whole files, leveraging the efficiency of the whole file data format. This pipeline converts the Avro files to Parquet using the Whole File Transformer processor.

The Parquet Conversion pipeline must use a file-based origin enabled for whole files, such as the Directory origin, to process the large Avro files.

The Whole File Transformer processor converts each Avro file to Parquet, writing each Parquet file temporarily to a user-defined Data Collector directory. Then a whole file destination, such as Google Cloud Storage, moves the Parquet file to the destination system.

Whole files can only be processed by the Whole File Transformer, so this pipeline performs no additional processing.

As a result, the Parquet Conversion pipeline looks like this:

For a step-by-step review of creating these pipelines, see the implementation example.

For a full list of whole file origins and destinations, see Data Format Support.

The Write Avro pipeline can be pretty simple - it just needs to write large Avro files. You can make it complicated by performing additional processing, but at its most basic, all you need is an origin and an Avro-enabled file-based destination to write the Avro files.

The Write Avro pipeline uses the Amazon S3 origin to read the multitude of small S3 objects. Then, it writes Avro files to a local file system using the Local FS destination, as follows:

Note the following configuration details:

The Parquet Conversion pipeline is simple as well. It uses the whole file data format, so no processing of file data can occur except the conversion of the Avro files to Parquet.

We use the Directory origin and whole file data format to read the fully written Avro files. Then comes the Whole File Transformer processor for the Parquet conversion, and the Amazon S3 destination to write the Parquet files back to S3 as whole files.

The Parquet Conversion pipeline looks like this:

Note the following configuration details:

We start both pipelines at the same time.

The Write Avro pipeline processes Amazon S3 objects and writes 3 GB Avro files to the /avro2parquet/ directory. The output files are named avro_<file name>.

After the first file is written, the Parquet Conversion pipeline picks up the file. It streams the file from the Directory origin to the Whole File Transformer processor. The Whole File Transformer converts the Avro file to Parquet in memory, and writes it to the specified temporary directory, /tmp/out/.parquet. The temporary file then streams from the Whole File Transformer to the Amazon S3 location defined in the destination.

In the end, the multitude of Avro files are converted to large columnar Parquet files and streamed back to Amazon S3 for easy analysis. Two simple pipelines, one Whole File Transformer, and you're done!