Field Order

The Field Order processor moves the specified fields to the listed order. It also drops all other fields from all records. As a result, the Field Order processor creates a record that includes only the listed fields, in the specified order. Use this processor with care.

When you configure the Field Order processor, you specify field names in the order that you want them to appear in the record, from left to right.

The Field Order processor is not case sensitive. When generating a record, the processor uses the exact field names that you specify in the processor.

Example

You have data from different sources that provide different sets of data, but all of the data contains a subset of fields that you want to use. The field names are the same, but there are differences in capitalization.

You configure the Fields to Order property in the processor to list the subset of fields that you want to use, in the order you want them to appear, and with the capitalization that you want to standardize on, as follows:
  • UserID
  • Transaction
  • Date

When the pipeline runs, records with the following field names pass to the Field Order processor:

  • Record 1: date, transaction, payment_type, total, userid, storeid
  • Record 2: StoreId, Userid, Transaction, Date, Total
  • Record 3: date, storeID, userId, transaction
The Field Order processor alters the three records, creating records with the following field names:
  • Record 1: UserID, Transaction, Date
  • Record 2: UserID, Transaction, Date
  • Record 3: UserID, Transaction, Date

Note that the order and capitalization of the field names now match the processor configuration. Fields that are not specified in the processor are dropped from all records.

Configuring a Field Order Processor

Configure a Field Order processor to order fields in records, and to drop all other fields from records.

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    Cache Data Caches data processed for a batch so the data can be reused for multiple downstream stages. Use to improve performance when the stage passes data to multiple stages.

    Caching can limit pushdown optimization when the pipeline runs in ludicrous mode.
  2. On the Field Order tab, configure the Fields to Order property. Click the Add icon to specify additional fields.
    Specify fields in the order that you want them to appear in the record, from left to right.
    You can use simple or bulk edit mode to configure the property. Use bulk edit mode to easily change the field order.
    The processor uses the specified field names for output records and drops all fields from the records that are not listed.