Static Lookup

Data Collector

The Static Lookup processor performs lookups of key-value pairs that are stored in local memory and passes the lookup values to fields. Use the Static Lookup to store String values in memory that the pipeline can look up at runtime to enrich records with additional data.

When you configure the Static Lookup processor, you specify whether the processor performs a bulk lookup of all keys in a batch, or performs an individual lookup of each key in a record. You use an expression to define the key to look up in local memory, and specify the String output field to write the lookup values to. You also define the key-value pairs to store in local memory.

Example

Let's assume that you need to develop a pipeline that reads log files created by multiple hosts. Each log file includes the name of the host. You want to process and write the log data to different local directories, based on the host that created the log file.

On the Lookup tab for the Static Lookup processor, you enter an expression that defines the hostname field as the key to look up. You specify a new directory field to write the lookup values to, as follows:

On the Static Store tab for the Static Lookup processor, you define each host name and associated directory as the key-value pairs to store in local memory:

In the Local FS destination, you include the value of the directory field in the expression for the directory template as follows:

${record:value('/directory')}/tmp/out/${YYYY()}-${MM()}-${DD()}-${hh()}

When you run the pipeline, the Static Lookup processor stores the static key-value pairs in memory. The processor looks up the value of the hostname field and passes the corresponding value to the directory output field. When the Local FS destination writes the data to output files, the destination writes the files to the appropriate directory for each host.