Define the CRUD Operation

The Elasticsearch destination can create, update, delete, or index data. The destination writes the records based on the CRUD operation defined in a CRUD operation header attribute or in operation-related stage properties.

You define the CRUD operation in the following ways:

CRUD record header attribute

You can define the CRUD operation in a CRUD operation record header attribute. The destination looks for the CRUD operation to use in the sdc.operation.type record header attribute.

The attribute can contain one of the following numeric values:

• 1 for CREATE, the equivalent of INSERT
• 2 for DELETE
• 3 for UPDATE
• 4 for INDEX, the equivalent of UPSERT
• 8 for UPDATE with doc_as_upsert, the equivalent of MERGE

If your pipeline includes a CRUD-enabled origin that processes changed data, the destination simply reads the operation type from the sdc.operation.type header attribute that the origin generates. If your pipeline uses a non-CDC origin, you can use the Expression Evaluator or a scripting processor to define the record header attribute. For more information about Data Collector changed data processing and a list of CDC-enabled origins, see Processing Changed Data.

Operation stage properties

You define a default operation in the destination properties. The destination uses the default operation when the sdc.operation.type record header attribute is not set.

You can also define how to handle records with unsupported operations defined in the sdc.operation.type header attribute. The destination can discard them, send them to error, or use the default operation.