Google BigQuery (Legacy) (deprecated)
Supported pipeline types:
|
When you configure the destination, you define the existing BigQuery dataset and table to stream data into. The destination streams each record into a row in a BigQuery table. You can optionally define an expression to specify the insert ID to insert or update. The insert ID is a unique ID for each row. If you do not specify an insert ID, the destination inserts each record into a new row.
The destination maps fields from records to BigQuery columns based on matching names and compatible data types. You can configure the destination to ignore invalid columns when the destination cannot map a field to an existing BigQuery column. You can also configure the table cache size for the destination.
You define the project ID and credentials to use when connecting to Google BigQuery.
You can also use a connection to configure the destination.
For more information about streaming data into Google BigQuery, see the Google BigQuery documentation.
BigQuery Data Types
The Google BigQuery (Legacy) destination maps fields from records to BigQuery columns in existing tables based on matching names and compatible data types. If needed, the destination converts Data Collector data types to BigQuery data types.
The following table lists the Data Collector data types and the BigQuery data types that the destination converts them to:
Data Collector Data Type | BigQuery Data Type |
---|---|
Boolean | Boolean |
Byte Array | Bytes |
Date | Date |
Datetime | Datetime or Timestamp |
Decimal | Decimal |
Double | Float |
Float | Float |
Integer | Integer |
List | Array |
List-Map | Record with repeated fields |
Long | Integer |
Map | Record |
Short | Integer |
String | String |
Time | Time |
- Byte
- Character
Credentials
When the Google BigQuery (Legacy) destination streams data into Google BigQuery, the destination must pass credentials to BigQuery.
- Google Cloud default credentials
- Credentials in a file
- Credentials in a stage property
For details on how to configure each option, see Security in Google Cloud Stages.
Configuring a Google BigQuery (Legacy) Destination
Configure a Google BigQuery (Legacy) destination to stream data into Google BigQuery.