Data Formats by Stage

Data Format Support

This appendix lists the data formats supported by origin, processor, and destination stages.

Origins

The following table lists the data formats supported by each origin.

Origin Avro Binary Datagram Delimited Excel JSON Log Protobuf SDC Record Text Whole File XML
Amazon S3    
Amazon SQS Consumer      
Aurora PostgreSQL CDC Client * * * Not Applicable * * *
Azure Data Lake Storage Gen2    
Azure IoT/Event Hub Consumer                
CoAP Server      
CONNX * * * Not Applicable * * *
CONNX CDC * * * Not Applicable * * *
Cron Scheduler * * * Not Applicable * * *
Directory    
Elasticsearch * * * Not Applicable * * *
File Tail                  
Google BigQuery * * * Not Applicable * * *
Google Cloud Storage        
Google Pub/Sub Subscriber        
Groovy Scripting * * * Not Applicable * * *
Hadoop FS Standalone    
HTTP Client          
HTTP Server        
JavaScript Scripting * * * Not Applicable * * *
JDBC Multitable Consumer * * * Not Applicable * * *
JDBC Query Consumer * * * Not Applicable * * *
JMS Consumer      
Jython Scripting * * * Not Applicable * * *
Kafka Multitopic Consumer    
Kinesis Consumer      
MapR DB CDC * * * Not Applicable * * *
MapR DB JSON * * * Not Applicable * * *
MapR FS Standalone    
MapR Multitopic Streams Consumer      
MapR Streams Consumer        
MongoDB * * * Not Applicable * * *
MongoDB Atlas * * * Not Applicable * * *
MongoDB Oplog * * * Not Applicable * * *
MQTT Subscriber      
MySQL Binary Log * * * Not Applicable * * *
OPC UA Client * * * Not Applicable * * *
Oracle Bulkload * * * Not Applicable * * *
Oracle CDC * * * Not Applicable * * *
Oracle CDC Client * * * Not Applicable * * *
PostgreSQL CDC Client * * * Not Applicable * * *
Pulsar Consumer    
Pulsar Consumer (Legacy)    
RabbitMQ Consumer      
Redis Consumer      
REST Service        
Salesforce * * * Not Applicable * * *
Salesforce Bulk API 2.0 * * * Not Applicable * * *
SAP HANA Query Consumer * * * Not Applicable * * *
SFTP/FTP/FTPS Client    
SQL Server 2019 BDC Multitable Consumer * * * Not Applicable * * *
SQL Server CDC Client * * * Not Applicable * * *
SQL Server Change Tracking * * * Not Applicable * * *
Start Jobs * * * Not Applicable * * *
TCP Server      
UDP Multithreaded Source * * * Not Applicable * * *
UDP Source * * * Not Applicable * * *
WebSocket Client      
WebSocket Server      

Processors

The following table lists the processors that read data of the listed format:
Processor Avro Binary Datagram Delimited JSON Log Netflow Protobuf SDC Record Syslog Text XML
Data Parser        
HTTP Client    
JSON Parser                      
Log Parser                      
XML Parser                      

The following table lists the processors that write data of the specified data format to a field:

Processor Avro Binary Delimited JSON Log Protobuf SDC Record Text XML
Data Generator  
JSON Generator                

Destinations

The following table lists the data formats supported by each destination.

Destination Avro Binary Delimited JSON Protobuf SDC Record Text Whole File XML
Amazon S3  
Azure Data Lake Storage Gen2  
Azure Event Hub Producer        
Azure IoT Hub Producer          
Azure Synapse SQL * * * Not Applicable * * *
Cassandra * * * Not Applicable * * *
CoAP Client          
Couchbase    
Databricks Delta Lake * * * Not Applicable * * *
Elasticsearch * * * Not Applicable * * *
Google BigQuery * * * Not Applicable * * *
Google Bigtable * * * Not Applicable * * *
Google Cloud Storage    
Google Pub/Sub Publisher  
Hadoop FS  
HBase * * * Not Applicable * * *
Hive Metastore                
HTTP Client    
InfluxDB * * * Not Applicable * * *
InfluxDB 2.x * * * Not Applicable * * *
JDBC Producer * * * Not Applicable * * *
JMS Producer  
Kafka Producer  
Kinesis Firehose              
Kinesis Producer    
Kudu * * * Not Applicable * * *
Local FS    
MapR DB * * * Not Applicable * * *
MapR DB JSON * * * Not Applicable * * *
MapR FS    
MapR Streams Producer    
MongoDB * * * Not Applicable * * *
MongoDB * * * Not Applicable * * *
MQTT Publisher          
Named Pipe          
Pulsar Producer  
RabbitMQ Producer    
Redis    
Salesforce * * * Not Applicable * * *
Salesforce Bulk API 2.0 * * * Not Applicable * * *
Send Response to Origin                
SFTP/FTP/FTPS Client                
Snowflake * * * Not Applicable * * *
Snowflake File Uploader                
Solr * * * Not Applicable * * *
Splunk * * * Not Applicable * * *
SQL Server 2019 BDC Bulk Loader * * * Not Applicable * * *
Syslog  
Tableau CRM * * * Not Applicable * * *
To Error * * * Not Applicable * * *
Trash * * * Not Applicable * * *
WebSocket Client