Couchbase Lookup

The Couchbase Lookup processor looks up documents in Couchbase Server and returns values to fields in the record. For information about supported versions, see Supported Systems and Versions in the Data Collector documentation.

Use the Couchbase Lookup processor to enrich records with additional data. For example, suppose Couchbase Server has multiple department documents, each listing the employees in a department. You can configure a pipeline to store the Couchbase document key, which uniquely identifies each department, in a record header attribute. Then, include a Couchbase Lookup processor to find the matching document and return the values to a new department_employees field in the record.

The Couchbase Lookup processor can look up documents using the document key or the Couchbase Server query language, N1QL. For key lookups, the processor can return the data from the entire document to a specified map field. Alternatively, for both key and N1QL lookups, the processor can return data from sub-documents to specified map fields. For N1QL lookups, when a lookup results in multiple matched documents, the Couchbase Lookup processor can return values from the first matching document or return values from all matching documents in separate records.

When you configure the Couchbase Lookup processor, you enter connection information, such as the nodes and bucket to connect to, as well as timeout properties for the connection. You can also use a connection to configure the origin. Optionally, you can enable TLS for the connection. You also enter information to authenticate with Couchbase Server.

Record Header Attributes

For key/value lookups, the Couchbase Lookup processor creates a record header attribute, couchbase.cas, that stores a value that represents the state of the looked-up document.

When configured to use CAS (compare and swap), the Couchbase destination uses this attribute value to prevent conflicts with other processes.

Configuring a Couchbase Lookup Processor

Configure a Couchbase Lookup processor to look up data in Couchbase Server.

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    Stage Library Library version that you want to use.

    The Couchbase Lookup processor supports only the Couchbase 2.x stage library. Do not choose a different library.
    Required Fields Fields that must include data for the record to be passed into the stage.
    Tip: You might include fields that the stage uses.

    Records that do not include all required fields are processed based on the error handling configured for the pipeline.
    Preconditions Conditions that must evaluate to TRUE to allow a record to enter the stage for processing. Click Add to create additional preconditions.

    Records that do not meet all preconditions are processed based on the error handling configured for the stage.
    On Record Error Error record handling for the stage:
    • Discard - Discards the record.
    • Send to Error - Sends the record to the pipeline for error handling.
    • Stop Pipeline - Stops the pipeline.
  2. On the Couchbase tab, configure the following properties:
    Couchbase Property Description
    Connection Connection that defines the information required to connect to an external system.

    To connect to an external system, you can select a connection that contains the details, or you can directly enter the details in the pipeline. When you select a connection, Control Hub hides other properties so that you cannot directly enter connection details in the pipeline.

    Node List Comma-separated list of one or more nodes in a Couchbase cluster.
    Bucket Name of an existing Couchbase bucket to connect to.
    Scope Not used at this time. Do not change from the default.

    Default is _default.
    Collection Not used at this time. Do not change from the default.

    Default is _default.
    Key-Value Timeout (ms) Maximum number of milliseconds allowed to execute each key-value operation.
    Connect Timeout (ms) Maximum number of milliseconds allowed to connect to Couchbase Server.
    Disconnect Timeout (ms) Maximum number of milliseconds allowed to gracefully close a connection.
    Advanced Environment Settings Client settings for connections with Couchbase Server. For available settings, see the Couchbase Java SDK documentation.
  3. On the Credentials tab, configure the following properties:
    Credentials Property Description
    Authentication Mode Authentication to use to connect to Couchbase Server.
    Choose one of the following authentication modes based on the version of Couchbase you are using:
    • Bucket Authentication - Use for Couchbase 4.x.
    • User Authentication - Use for Couchbase 5.x and later.
    Bucket Password Couchbase bucket password.

    Required when Authentication Mode is set to Bucket Authentication.
    User Name Couchbase user name.

    Required when Authentication Mode is set to User Authentication.
    Password Couchbase password.

    Required when Authentication Mode is set to User Authentication.
    Use TLS Enables the use of TLS.
    Use Remote Keystore Enables loading the contents of the keystore from a remote credential store or from values entered in the stage properties. For more information, see Remote Keystore and Truststore.
    Private Key Private key used in the remote keystore. Enter a credential function that returns the key or enter the contents of the key.
    Certificate Chain Each PEM certificate used in the remote keystore. Enter a credential function that returns the certificate or enter the contents of the certificate.

    Using simple or bulk edit mode, click the Add icon to add additional certificates.
    Keystore File

    Path to the local keystore file. Enter an absolute path to the file or enter the following expression to define the file stored in the Data Collector resources directory:

    ${runtime:resourcesDirPath()}/keystore.jks

    By default, no keystore is used.

    Keystore Type Type of keystore to use. Use one of the following types:
    • Java Keystore File (JKS)
    • PKCS #12 (p12 file)

    Default is Java Keystore File (JKS).

    Keystore Password

    Password to the keystore file. A password is optional, but recommended.

    Tip: To secure sensitive information such as passwords, you can use runtime resources or credential stores. For more information about credential stores, see Credential Stores in the Data Collector documentation.
    Keystore Key Algorithm

    Algorithm to manage the keystore.

    Default is SunX509.
    Use Remote Truststore Enables loading the contents of the truststore from a remote credential store or from values entered in the stage properties. For more information, see Remote Keystore and Truststore.
    Trusted Certificates Each PEM certificate used in the remote truststore. Enter a credential function that returns the certificate or enter the contents of the certificate.

    Using simple or bulk edit mode, click the Add icon to add additional certificates.
    Truststore File

    Path to the local truststore file. Enter an absolute path to the file or enter the following expression to define the file stored in the Data Collector resources directory:

    ${runtime:resourcesDirPath()}/truststore.jks

    By default, no truststore is used.

    Truststore Type
    Type of truststore to use. Use one of the following types:
    • Java Keystore File (JKS)
    • PKCS #12 (p12 file)

    Default is Java Keystore File (JKS).

    Truststore Password

    Password to the truststore file. A password is optional, but recommended.

    Tip: To secure sensitive information such as passwords, you can use runtime resources or credential stores. For more information about credential stores, see Credential Stores in the Data Collector documentation.
    Truststore Trust Algorithm

    Algorithm to manage the truststore.

    Default is SunX509.
  4. On the Lookup tab, configure the following properties:
    Lookup Property Description
    Lookup Type Method to specify lookup:
    • Key/Value - Uses the document key to find a document.
    • N1QL - Uses the Couchbase Server query language to find a document.
    Document Key Unique ID or key for the document that the processor looks up. For example, you might specify an expression that resolves to the document key.

    Available for key/value lookups.
    Return Properties Returns specific sub-documents rather than the full document.

    Available for key/value lookups.
    SDC Field Name of the map field in the record where the processor returns the data from the lookup. You can specify an existing field or a new field. If the field does not exist, the Couchbase Lookup processor creates the field.

    Available for key/value lookups when not returning sub-documents.
    N1QL Query Query that returns a document. Specify in N1QL, the Couchbase Server query language. For more information, see the Couchbase documentation.

    Available for N1QL lookups.
    Property Mappings List that maps returned sub-documents to fields in the record.
    Enter the following:
    • Property Name - Sub-document returned. Use dot notation syntax to separate components in the document hierarchy. For more information, see the Couchbase documentation.
    • SDC Field - Name of the map field in the record where the processor returns the sub-document.

    Available for key/value lookups when returning sub-documents and for N1QL lookups.

    Submit as Prepared Statement Submits the query to Couchbase as a prepared statement.

    Available for N1QL lookups.
    Query Timeout (ms) Maximum number of milliseconds allowed for Couchbase Server to complete the query.

    Available for N1QL lookups.
    Multiple Value Behavior Action to take when the lookup finds multiple documents:
    • First value only - Returns values from the first document.
    • Split into multiple records - Returns values from each document to a separate record.

    Available for N1QL lookups.
    Missing Value Behavior Action to take when the lookup returns no document:
    • Send to error - Sends the record to error.
    • Pass the record along the pipeline unchanged - Passes the record without a lookup return value.