Time Basis and Data Time Zone for Time-Based Buckets and Partition Prefixes

The time basis and the data time zone comprises the time used by the Amazon S3 destination to write records to a time-based bucket or partition prefix. When the configured bucket or partition prefix does not include time-based functions, you can ignore the time basis property.

A bucket or partition prefix has a time component when it includes datetime variables, such as ${YYYY()} or ${DD()}, or when it includes an expression that evaluates to a datetime value, such as ${record:valueOrDefault("/Timestamp")}.

For details about datetime variables, see Datetime Variables.

You can use the following times as the time basis:

Processing Time

When you use processing time as the time basis, the destination performs writes based on the processing time and the configured bucket and partition prefix. The processing time is the time associated with the Data Collector running the pipeline, by default. You can specify a different time zone by configuring the Data Time Zone property. To use the processing time as the time basis, use the following expression:

${time:now()}

This is the default time basis.

Record Time

When you use the time associated with a record as the time basis, you specify a date field in the record. The destination writes data based on the datetimes associated with the records, adjusting for the value specified for the Data Time Zone property.

To use a time associated with the record, use an expression that calls a field and resolves to a datetime value, such as ${record:value("/Timestamp")}.

For example, say you define the Partition Prefix property using the following datetime variables:

logs-${YYYY()}-${MM()}-${DD()}

If you use the time of processing as the time basis, the destination writes records to partitions based on when it processes each record. If you use the time associated with the data, such as a transaction timestamp, then the destination writes records to the partitions based on that timestamp. If a partition does not exist, the destination creates the needed partition.

Or, say you define the Bucket property as follows:

${YYYY()}-${MM()}

If you use the time of processing as the time basis, the destination writes records to buckets based on when it processes each record. If you use the time associated with the data, such as a transaction timestamp, then the destination writes records to the buckets based on that timestamp. If a bucket does not exist, the destination handles the record based on the error record handling configured for the stage.