Error Handling
When you define a custom schema, you specify how the origin handles parsing errors. Parsing errors can occur when the data in a field is not compatible with the data type specified in the schema.
You can use one of the following error handling modes:
- Permissive - When the origin encounters a problem
parsing any field in the record, it creates a record with the field names
defined in the schema, but with null values in every field. The origin
can optionally write the data from the original record to a specified field in
the record.
When writing the original record to a field, you must add the field to the custom schema as a String field. That is, to have the origin write the original record to a field named
originalData, include anoriginalDataString field in the custom schema. - Drop Malformed - When the origin encounters a problem parsing any field in the record, it drops the entire record from the pipeline.
- Fail Fast - When the origin encounters a problem parsing any field in the record, it stops the pipeline.