Windows Event Log

Supported pipeline types:

  • Data Collector Edge
The Windows Event Log origin reads data from a Microsoft Windows event log located on a Windows machine. The origin generates a record for each event in the log.

Use the Windows Event Log origin only in pipelines configured for edge execution mode. Run the pipeline on StreamSets Data Collector Edge (SDC Edge) installed on the Windows machine.

For example, you might use the Windows Event Log origin in an edge pipeline to read logs from a web or application farm of Windows servers. You install SDC Edge on each Windows machine that you want to read the logs from, and run the edge pipeline on each SDC Edge installation. You design the edge pipeline to pass the log data to a Data Collector receiving pipeline that runs on StreamSets Data Collector. The Data Collector receiving pipeline performs more complex processing on the data, and then writes the data to a big data system such as Hadoop. You can then analyze the data to detect security violations such as insider threats or illegal access to the Windows machines.

When you configure the Windows Event Log origin, you specify the Windows event log to read from. You also specify whether the origin reads all events in the log or whether it reads only new events that occur after the pipeline starts.

You configure the origin to use either the Event Logging API or the Windows Event Log API to read from the log.

When the pipeline stops, the Windows Event Log origin notes where it stops reading. When the pipeline starts again, the origin continues processing from the last-saved offset. You can reset the origin to process all events in the log.

For more information about installing SDC Edge, designing edge pipelines, and running and maintaining edge pipelines, see Meet StreamSets Data Collector Edge.

Reader API Type

The origin can use one of the following APIs to read data from a Microsoft Windows event log:
Event Logging API
The Event Logging API was designed for applications that run on the Windows Server 2003, Windows XP, or Windows 2000 operating system.
When using the Event Logging API, the origin uses record numbers to manage the offset when you stop and restart the pipeline.
Windows Event Log API
The Windows Event Log API supersedes the Event Logging API beginning with the Windows Vista operating system. Microsoft recommends using the newer Windows Event Log API for applications that run on Windows Vista or later operating systems.
When using the Windows Event Log API, the origin uses bookmarks provided by the Windows Event Log API to manage the offset when you stop and restart the pipeline. You configure whether the origin uses the push or pull subscription mode to read events from the log.
With the Windows Event Log API, the origin reads the raw XML for each event. It then passes the system metadata and the log message for the event to System and Message fields in the record. The origin also passes the raw XML to a rawEventXML field, based on how you configure the origin to populate the XML:
  • On error - The origin includes the raw XML in the record only when the origin encounters an error generating the System or Message field.
  • Always - The origin always includes the raw XML in the record.

Configuring a Windows Event Log Origin

Configure a Windows Event Log origin to read data from a Windows event log.

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    On Record Error Error record handling for the stage:
    • Discard - Discards the record.
    • Send to Error - Sends the record to the pipeline for error handling.
    • Stop Pipeline - Stops the pipeline.
  2. On the Windows tab, configure the following properties:
    Windows Property Description
    Windows Log to Read From Name of the Windows event log to read from:
    • Application
    • System
    • Security
    • Custom
    Custom Log Name Name of the custom Windows event log to read from. Enter the event log name or enter an expression that evaluates to the name.
    Read Mode Determines how the origin reads the log:
    • All - Read all events in the log.
    • New - Read only new events in the log that occur after the pipeline starts.
    Reader API Type API type used to read from the log:
    • Event Logging
    • Windows Event Log

    Microsoft recommends using the newer Windows Event Log API for applications that run on Windows Vista or later operating systems.
    Buffer Size Maximum buffer size. The buffer size determines the size of the record that can be processed.

    The buffer limit helps prevent out of memory errors. Decrease when memory on the SDC Edge machine is limited. Increase to process larger records when memory is available.

    With a buffer limit error, the origin logs a message indicating that a buffer overrun error has occurred.

    Default is -1, which means no limit is set and the origin uses all available memory.
    Subscription Mode When using the Windows Event Log API, the mode to subscribe to events:
    • Push Mode
    • Pull Mode
    Populate Raw Event XML When using the Windows Event Log API, determines when the origin includes the raw XML for each event in the record:
    • On Error
    • Always
    Maximum Wait Time (secs) When using the Windows Event Log API, maximum time in seconds that the origin waits to receive events before it generates a batch.