Field Flattener

The Field Flattener processor flattens map fields. Use the processor to flatten all map fields in records or to flatten specified map fields at the top level of the record. The processor does not flatten list fields. When you configure a Field Flattener processor, you configure whether to flatten all fields or specified fields in the record. You also configure the name separator to use for flattened field names.

When the Field Flattener flattens the entire record, it flattens all map fields in records. When flattening a specified field, the processor flattens map fields at the top level of the record.

Flatten all Fields

The Field Flattener processor can flatten map fields that contain additional map fields. When the Field Flattener flattens the entire record, it flattens all nested map fields of the record until the record is flat.

For example, say you have the following record with nested map fields:

{
  "contact": {
     "name": "Jane Smith",
     "id": "557",
     "address": {
       "home": {
         "street": "101 3rd St",
         "city": "Huntsville",
         "state": "NC",
         "zipcode": "27023"
          },
       "work": {
         "street": "15 Main St",
         "city": "Jonestown",
         "state": "NC",
         "zipcode": "27011"
       }
      }
  }
}

If you configure the processor to flatten all fields with the name separator as a period, the processor generates the following record:

{
   "contact.name": "Jane Smith",
   "contact.id": "557",
   "contact.address.home.street": "101 3rd St",
   "contact.address.home.city": "Huntsville",
   "contact.address.home.state": "NC",
   "contact.address.home.zipcode": "27023",
   "contact.address.work.street": "15 Main St",
   "contact.address.work.city": "Jonestown",
   "contact.address.work.state": "NC",
   "contact.address.work.zipcode": "27011"
}

Flatten Specific Fields

When you configure the Field Flattener to flatten specified fields, it flattens the specified map fields at the top level of the record. The processor cannot flatten a specified map field nested within another field. For example, with the sample data above, you cannot configure the processor to flatten only the home map field nested in the address map field.

When flattening a field, the processor flattens the specified map field. It does not flatten any additional fields nested within the map field.

For example, say you have the following record with two map fields at the top level:

{
        "Dear Life":{
          "ISBN": "0307743721",
          "Author": "Alice Munro",
          "PublicationDate": "July 30 2013"
        }
      },
      {
        "The Buried Giant": {
          "ISBN": "0307455793",
          "Author": "Kazuo Ishiguro",
          "Publication_Date":"January 5 2016" 
         }
       }

If you configure Field Flattener to flatten the Dear Life map field using an underscore as the name separator, the processor generates the following record, flattening only the Dear Life field and not the

The Buried
        Giant

field:

{
      {
      "Dear Life_ISBN": "10307743721",
      "Dear Life_Author": "Alice Munro",
      "Dear Life_PublicationDate": "July 30 2013",   
         }
       {
       "The Buried Giant":{
         "ISBN": "0307455793",
         "Author": "Kazuo Ishiguro",
         "Publication_Date":"January 5 2016" 
         }
}

Configuring a Field Flattener Processor

Configure a Field Flattener processor to flatten map fields.

In the Properties panel, on the General tab, configure the following properties:


General Property	Description
Name	Stage name.
Description	Optional description.
Cache Data	Caches data processed for a batch so the data can be reused for multiple downstream stages. Use to improve performance when the stage passes data to multiple stages. Caching can limit pushdown optimization when the pipeline runs in ludicrous mode.

On the Flatten tab, configure the following properties:


Flatten Property	Description
Flatten Type	Flattening method to use: Flatten All Fields - Flattens all map fields in the record. Flatten Specific Fields - Flattens specified map fields at the top level of the record.
Fields	When flattening specific fields, specify one or more top-level map fields.
Name Separator	One or more characters to use to create a flattened field name. For example, if you use the underscore character, a `location` field nested in a `store` field is flattened as follows: `store_location`.