File Tail

The File Tail origin reads lines of data as they are written to an active file after reading related archived files in the same directory. File Tail generates a record for each line of data.

When you configure File Tail, you define the full path to the active file, the naming convention for archived files, and the first archived file to process. These properties determine the set of files that are read and the order of processing.

For example, a File Tail is configured to tail a log called server.log, starting with server.log.1, expecting archived files named in reverse counter order. When the pipeline runs, it starts reading lines from the beginning of server.log.1, ignoring older files. When it reaches the end of server.log.1, it continues processing data in server.log. When it completes reading existing data in server.log, it tails incoming data for the file.

When you stop a pipeline, File Tail notes where it stops reading. When you restart the pipeline, File Tail continues processing from the line where it left off, even if it has been archived to a different file.

You can reset the origin for File Tail. When you reset the origin, the File Tail treats all properties as if you are starting the pipeline for the first time - processing all requested files before continuing to the active file.

File Tail generates record header attributes that enable you to use the origins of a record in pipeline processing.

The origin can generate events for an event stream. For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.