Field Remover

The Field Remover processor removes fields from records. Use the processor to discard field data that you do not need in the pipeline and to retain field data for downstream processing.

The processor can keep or remove fields based on match criteria that you define. For example, you can use the Field Remover processor to remove employee names while keeping employee IDs in the pipeline.

When you configure the processor, you specify whether to keep or remove selected fields. You also define the criteria to evaluate those fields.

Match Criteria

The Field Remover can keep or remove listed fields using one of the following methods:

Field name
Keeps or removes a field based on the field name. When you specify a field name, the processor keeps or removes a field with the specified field name. Use this method to keep or remove a single field.
Use dot notation (.) to specify a nested field. For example, to specify the zip_code field nested in the address map field, use address.zip_code.
Field name prefix
Keeps or removes a field based on the prefix of the field name. When you specify a prefix, the processor keeps or removes all fields with the field names that start with the specified prefix. For example, use this method to keep or remove all fields with the sales_ prefix.
Field name suffix
Keeps or removes a field based on the suffix of the field name. When you specify a suffix, the processor keeps or removes all fields with the field names that end with the specified suffix. For example, use this method to keep or remove all fields with the _sales suffix.
Field name contains text
Keeps or removes all fields with field names that contain the matching text. When you specify a set of characters, the processor keeps or removes all fields with names that contain the specified text. For example, use this method to keep or remove any fields containing storeID, regardless of where it appears in the field name.
Field name matches regex
Keeps or removes all fields with field names that match the Java-based regular expression. When you specify a regular expression, the processor keeps or removes all fields with names that match the specified expression. For example, you can use the regular expression employee(-|_)?.* to keep or remove the following employee ID field names: employeeID, employee-ID, employee_ID. For details on using Regular Expressions, see the Data Collector documentation.

Configuring a Field Remover Processor

Configure a Field Remover processor to keep or remove fields from records.

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    Cache Data Caches data processed for a batch so the data can be reused for multiple downstream stages. Use to improve performance when the stage passes data to multiple stages.

    Caching can limit pushdown optimization when the pipeline runs in ludicrous mode.
  2. On the Remove/Keep tab, click the Add icon and then configure the following properties:
    Field Remover Property Description
    Action Action to complete:
    • Keep listed fields
    • Remove listed fields
    Match Criteria Criteria to use to find matching fields:
    • Field name
    • Field name prefix
    • Field name suffix
    • Field name contains text
    • Field name matches regex
    Fields

    Enter a field name, characters, or regular expression to use the match criteria. In regular expressions, use a backslash ( \ ) to escape characters.
  3. To remove additional fields, click the Add icon and specify another configuration for the selected match criteria.