Field Flattener
Supported pipeline types:
|
Data CollectorUse the Field Flattener processor when you have nested fields that you need to flatten. For example, the Drift Synchronization Solution for Hive cannot process records with nested fields, so you can use the Field Flattener processor to flatten records before passing them to the Hive Metadata processor.
When you configure a Field Flattener processor, you configure whether to flatten the entire record or specific fields in the record. When flattening specific fields, you configure whether to flatten the fields in place or into another field. You also configure the name separator to use for flattened field names and specify whether to output the flattened fields to a list-map or list root field.
Flatten the Entire Record
When the Field Flattener flattens the entire record, it flattens all nested structures in the record until the record is flat.
{
"store": {
"id": "10342",
"location": {
"street": "34 2nd St",
"city": "Wilma",
"state": "OH",
"zipcode": "33333"
},
"ip": "234.56.7890"
}
}| store.id | store.location.street | store.location.city | store.location.state | store.location.zipcode | store.ip |
|---|---|---|---|---|---|
| 10342 | 34 2nd St | Wilma | OH | 33333 | 234.56.7890 |
Flatten Specific Fields
The Field Flattener processor can flatten specified fields, either list or map fields, that contain additional nested list or map fields. When flattening a list or map field, the processor flattens all nested structures in the field until the field is flat. The processor can flatten fields in place (at their current position in the record), or the processor can flatten fields into another list or map field in the record, such as into the root field.
For example, say you have the following record with nested map fields:
{
"contact": {
"name": "Jane Smith",
"id": "557",
"address": {
"home": {
"street": "101 3rd St",
"city": "Huntsville",
"state": "NC",
"zipcode": "27023"
},
"work": {
"street": "15 Main St",
"city": "Jonestown",
"state": "NC",
"zipcode": "27011"
}
}
}
}address map field in
place using a period as the name separator, the processor generates the following record:
{
"contact": {
"name": "Jane Smith",
"id": "557",
"address": {
"home.street": "101 3rd St",
"home.city": "Huntsville",
"home.state": "NC",
"home.zipcode": "27023",
"work.street": "15 Main St",
"work.city": "Jonestown",
"work.state": "NC",
"work.zipcode": "27011"
}
}
}address map field into the target field contact using a
period as the name separator, the processor generates the following record:
{
"contact": {
"name": "Jane Smith",
"id": "557",
"address": {
"home": {
"street": "101 3rd St",
"city": "Huntsville",
"state": "NC",
"zipcode": "27023"
},
"work": {
"street": "15 Main St",
"city": "Jonestown",
"state": "NC",
"zipcode": "27011"
"home.street": "101 3rd St",
"home.city": "Huntsville",
"home.state": "NC",
"home.zipcode": "27023",
"work.street": "15 Main St",
"work.city": "Jonestown",
"work.state": "NC",
"work.zipcode": "27011"
}
}address field into the
contact field and remove the flattened field, the processor generates the
following record:
{
"contact": {
"name": "Jane Smith",
"id": "557",
"home.street": "101 3rd St",
"home.city": "Huntsville",
"home.state": "NC",
"home.zipcode": "27023",
"work.street": "15 Main St",
"work.city": "Jonestown",
"work.state": "NC",
"work.zipcode": "27011"
}
}Configuring a Field Flattener Processor
Configure a Field Flattener processor to flatten fields.
-
In the Properties panel, on the General tab, configure the
following properties:
General Property Description Name Stage name. Description Optional description. Required Fields Fields that must include data for the record to be passed into the stage. Tip: You might include fields that the stage uses.Records that do not include all required fields are processed based on the error handling configured for the pipeline.
Preconditions Conditions that must evaluate to TRUE to allow a record to enter the stage for processing. Click Add to create additional preconditions. Records that do not meet all preconditions are processed based on the error handling configured for the stage.
On Record Error Error record handling for the stage: - Discard - Discards the record.
- Send to Error - Sends the record to the pipeline for error handling.
- Stop Pipeline - Stops the pipeline. Not valid for cluster pipelines.
-
On the Flatten tab, configure the following
properties:
Flatten Property Description Flatten Select whether to flatten the entire record or to flatten specific fields. Fields When flattening specific fields, the fields to flatten. You can flatten list and map fields that contain additional nested list or map fields. Specify the path to the field, for example: /contact/address. To specify fields, you can:- Type field paths separated by commas.
- Click the Fields text box and select each field path from the list of available field paths.
- Click Select Fields Using Preview Data to open the Field Selector dialog box, and select fields from the preview data.
Flatten in Place When flattening specific fields, enables the processor to flatten the fields at their current position in the record. Target Field When not flattening in place, the map or list-map field to flatten the fields into. The field must already exist in the record. Collision Field Action When not flattening in place, the action the processor takes when the target field already contains a field with the same name as the flattened field. Possible actions: - Send record for error handling
- Override with new value
- Discard new value
Remove Flattened Field When not flattening in place, enables the processor to remove original fields that contained nonflattened fields from the record after successfully flattening fields. Name Separator One or more characters to use between nested field names to create a flattened field name. For example, if you use the underscore character, a
locationfield nested in astorefield is flattened as follows:store_location.Output Type Root field type to output the flattened fields to: list-map or list.