Redis Lookup

The Redis Lookup processor performs key-value lookups in Redis and passes the lookup values to fields. For information about supported versions, see Supported Systems and Versions.

Use the Redis Lookup to enrich records with additional data. For example, you can configure the processor to use a city field as the key to look up zip code values in Redis, and pass the values to a new zip_code output field.

When you configure the Redis Lookup processor, you specify whether the processor performs a bulk lookup of all keys in a batch, or performs an individual lookup of each key in a record. You use an expression to define the key to look up in Redis, and specify the output field to write the lookup values to. You also specify the data type of the Redis value.

You can configure the processor to locally cache the key-value pairs to improve performance.

You specify properties to connect to Redis, including the URI of the Redis server. You can also use a connection to configure the processor.

Data Types

When you define the key to look up in Redis, you specify the data type of the Redis value. The Redis Lookup processor converts the Redis data type to a Data Collector data type.

The following table lists the supported Redis data types and the corresponding Data Collector data type:
Redis Data Type Data Collector Data Type
String String
List List.

Converts the data to an indexed list of values, for example:

0: <value>
1: <value>
2: <value>
Hash Map.

Converts the data to a map of the Redis field names and values, for example:

/<first field name>: <value>
/<second field name>: <value>
/<third field name>: <value>
Set List.

Converts the data to an indexed list of values, for example:

0: <value>
1: <value>
2: <value>
Note: The remaining Redis data types are not supported.

Lookup Cache

To improve pipeline performance, you can configure the Redis Lookup processor to locally cache the key-value pairs returned from Redis.

The processor caches key-value pairs until the cache reaches the maximum size or the expiration time. When the first limit is reached, the processor evicts key-value pairs from the cache.

You can configure the following ways to evict key-value pairs from the cache:
Size-based eviction
Configure the maximum number of key-value pairs that the processor caches. When the maximum number is reached, the processor evicts the oldest key-value pairs from the cache.
Time-based eviction
Configure the amount of time that a key-value pair can remain in the cache without being written to or accessed. When the expiration time is reached, the processor evicts the key from the cache. The eviction policy determines whether the processor measures the expiration time since the last write of the value or since the last access of the value.
For example, you set the eviction policy to expire after the last access and set the expiration time to 60 seconds. After the processor does not access a key-value pair for 60 seconds, the processor evicts the key-value pair from the cache.

When you stop the pipeline, the processor clears the cache.

Configuring a Redis Lookup Processor

Configure a Redis Lookup processor to perform key-value lookups in Redis.

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    Required Fields Fields that must include data for the record to be passed into the stage.
    Tip: You might include fields that the stage uses.

    Records that do not include all required fields are processed based on the error handling configured for the pipeline.
    Preconditions Conditions that must evaluate to TRUE to allow a record to enter the stage for processing. Click Add to create additional preconditions.

    Records that do not meet all preconditions are processed based on the error handling configured for the stage.
    On Record Error Error record handling for the stage:
    • Discard - Discards the record.
    • Send to Error - Sends the record to the pipeline for error handling.
    • Stop Pipeline - Stops the pipeline. Not valid for cluster pipelines.
  2. On the Lookup tab, configure the following properties:
    Lookup Property Description
    Connection Connection that defines the information required to connect to an external system.

    To connect to an external system, you can select a connection that contains the details, or you can directly enter the details in the pipeline. When you select a connection, Control Hub hides other properties so that you cannot directly enter connection details in the pipeline.

    To create a new connection, click the Add New Connection icon: . To view and edit the details of the selected connection, click the Edit Connection icon: .
    URI URI of the Redis server. Use the following format:
    redis://<host name>:<port number>/<database>

    You can omit the database if the server uses the default database.

    You can optionally include your password to log in to the Redis server. For example:
    redis://:<password>@<host name>:<port number>/<database>
    Key Expression Key to look up in Redis. Enter a key name or enter an expression that defines the key.
    For example, enter the following expression to use the data in the city field as the lookup key:
    ${record:value('/city')}
    Output Field Name of the field in the record to pass the lookup value. You can specify an existing field or a new field. If the field does not exist, Redis Lookup creates the field.
    Data Type Data type of the value in Redis.

    Default is String.
    Mode Mode used to perform the lookups:
    • Per Batch - Performs a bulk lookup of all keys in a batch. The processor performs a single lookup for each batch.
    • Per Key in Each Record - Performs individual lookups of each key in each record. If you configure multiple key expressions, the processor performs multiple lookups for each record.

    Default is Per Batch.
    Enable Local Caching Specifies whether to locally cache the returned key-value pairs.
    Maximum Entries to Cache Maximum number of key-value pairs to cache. When the maximum number is reached, the processor evicts the oldest key-value pairs from the cache.

    Default is -1, which means unlimited.
    Eviction Policy Type Policy used to evict key-value pairs from the local cache when the expiration time has passed:
    • Expire After Last Access - Measures the expiration time since the key-value pair was last accessed by a read or a write.
    • Expire After Last Write - Measures the expiration time since the key-value pair was created, or since the value was last replaced.
    Expiration Time Amount of time that a key-value pair can remain in the local cache without being accessed or written to.

    Default is 1 second.
    Time Unit Unit of time for the expiration time.

    Default is seconds.
  3. On the Redis tab, configure the following properties:
    Redis Property Description
    Connection Timeout (sec) Maximum time in seconds to wait for a connection.

    Default is 60 seconds.