Schema Generator

Supported pipeline types:

  • Data Collector
The Schema Generator processor generates a schema based on the structure of a record and writes the schema into a record header attribute. The Schema Generator processor generates Avro schemas at this time.

Use the Schema Generator processor to generate a basic schema when the schema is unknown. For example, you might use the processor in a pipeline to generate the latest version of the Avro schema before writing records to destination systems.

Note: The Schema Generator processor offers limited customization. If you need more customization than the processor offers, consider writing your own schema generator.

When you configure a Schema Generator processor, you can specify the namespace and description for the Avro schema. You can specify whether schema fields should allow nulls and whether schema fields should default to null. You can specify default values for most Avro primitive types, and you can allow the processor to use a larger data types for Avro types without a direct equivalent.

You can specify the names for precision and scale attributes for decimal values. And you can configure a default precision and scale for any decimal fields without that information or with an invalid precision or scale.

When appropriate, you can configure the Schema Generator to cache a number of schemas, and to apply the schemas to records based on the expression defined in the Cache Key Expression property.