Data Types

Transformer processes data using Spark data types.

Certain data types might not be valid in specific situations. For example, text-based data formats such as Delimited, JSON, and Text, do not support processing binary data. Also, the Delimited and Text data formats do not support processing complex types such as lists or maps.

To change data from one type to another within a pipeline, you can use the Type Converter processor.

For details about Spark data types, see the Spark documentation.

Data Types in Preview

When you preview data, the preview panel displays generic data types, such as Boolean, String, and List. These data types represent the Spark data types that are being used.

For example, in preview, List represents the Array Spark data type, and Map can represent either the Map or Struct Spark data types.