Log Data Format

When you use an origin to read log data, you define the format of the log files to be read.

You can read log files that use the following log formats:

Common Log Format
A standardized text format used by web servers to generate log files. Also known as the NCSA (National Center for Supercomputing Applications) Common Log format.
Combined Log Format
A standardized text format based on the common log format that includes additional information. Also known as the Apache/NCSA Combined Log Format.
Apache Error Log Format
The standardized error log format generated by the Apache HTTP Server 2.2.
Apache Access Log Custom Format
A customizable access log generated by the Apache HTTP Server 2.2. Use the Apache HTTP Server version 2.2 syntax to define the format of the log file.
Regular Expression
Use a regular expression to define the structure of log data, and then assign the field or fields represented by each group.
Use any valid regular expression.
Grok Pattern
Use a grok pattern to define the structure of log data. You can use the grok patterns supported by Data Collector. You can also define a custom grok pattern and then use it as part of the log format.
For more information about supported grok patterns, see Defining Grok Patterns.
log4j
A customizable format generated by the Apache Log4j 1.2 logging utility. You can use the default format or specify a custom format. Use the Apache Log4j version 1.2 syntax to define the format of the log file.
You can also specify the action to take when the origin encounters an error when parsing a line. You can skip the line and optionally log an error. If you know that the unparsable information is part of a stack trace, you can have the origin include the unparsable information as a stack trace to the previous parsable line.
Common Event Format (CEF)
A customizable event format used by security devices to generate log events. CEF is the native format for HP ArcSight.
Log Event Extended Format (LEEF)
A customizable event format used by security devices to generate log events. LEEF is the native format for IBM Security QRadar.

For a full list of origins that support this data format, see Origins in the "Data Formats by Stage" appendix.