Late Records and Late Record Handling

When you use a record time as the time basis, you can define a time limit for records to be written to their associated output file. When the destination creates a new output file in a new directory, the previous output file is kept open for the specified late record time limit. When records that belong in that file arrive within the time limit, the destination writes the records to the open output file. When the late record time limit is reached, the output file is closed and any record that arrives past this limit is considered late.

Tip: The late records properties are not applicable if you use processing time as the time basis. If you use processing time, set the late record time limit to one second.

You can send late records to a late records file or to the stage for error handling. When you send records to a late records file, you define a late records directory template.

For example, you use a record time as the time basis, configure a one-hour late record time limit, configure late records to be sent to the stage for error handling, and use the default directory template value:

/tmp/out/${YYYY()}-${MM()}-${DD()}-${hh()}

The first records that arrive have a datetime between the hours of 02:00 and 02:59, and so are written to an output file in the 02 directory. When records with a datetime between the hours of 03:00 and 03:59 arrive, the destination creates a new file in an 03 directory. The destination keeps the file in the 02 directory open for another hour.

If a record with a datetime between the hours of 02:00 and 02:59 arrives before the hour time limit, the destination writes the record to the open file in the 02 directory. After one hour, the destination closes the output file in the 02 directory. Any records with a datetime between the hours of 02:00 and 02:59 that arrive after the one-hour time limit are considered late. The late records are sent to the stage for error handling.