File

The File destination writes files to Hadoop Distributed File System (HDFS) or a local file system. The File destination writes data based on the specified data format and creates a separate file for every partition.

The File destination writes to HDFS using connection information stored in a Hadoop configuration file.

When you configure the File destination, you specify the output directory and write mode to use. When overwriting related partitions, first complete the overwrite partition requirement.

You select the data format to write and configure related properties. You can specify fields to use for partitioning files. You can also drop unrelated master records when using the destination as part of a slowly changing dimension pipeline.

You can also specify HDFS configuration properties for a HDFS-compatible system. Any specified properties override those defined in the Hadoop configuration file.