Acquire / file · All operations

Parse log files

A variant of Read another file format

Read web server, system and event logs into records, often with named patterns.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache NiFi (5)
  • CEFReaderread 2026-09-27

    Read Common Event Format (CEF) events into records

  • GrokReaderread 2026-09-27

    Read unstructured text such as log files into records with Grok patterns

  • ParseEvtxread 2026-09-27

    Parse a Windows Event Log (evtx) file and write it out as XML

  • Syslog5424Readerread 2026-09-27

    Read RFC 5424 syslog data into records

  • SyslogReaderread 2026-09-27

    Read RFC 5424 or RFC 3164 syslog messages into records

AWS Glue (1)
  • grokLog (read)read 2026-09-27

    Read grokLog data from Amazon S3 and other supported connection types

KNIME Analytics Platform (1)
Talend (1)
  • tApacheLogInputread 2026-09-27

    Read the access log file of an Apache HTTP server into rows

VisiData (1)

Background

In Logstash, the grok filter parses unstructured log data into something structured and queryable.[1] Syslog is a standard for message logging, in which each message is labelled with a facility code and assigned a severity level.[2]

When to use it

In Logstash, the dissect filter extracts fields using delimiters rather than regular expressions, is faster than grok, and works well when data is reliably repeated.[1] In Logstash, grok is a better choice than dissect when the structure of the text varies from line to line.[1]

NiFi's GrokReader can append a line that does not match to the previous message, skip it, or keep it in a _raw field with its other fields null.[3] Keep the lines that do not parse, with their line numbers, rather than dropping them: they may be the errors you came to find. (Meridian's view)

In The BSD syslog Protocol, Request for Comments (RFC) 3164, the TIMESTAMP field is the local time in the format Mmm dd hh:mm:ss.[4] Some administrators archiving syslog messages run a script to add the year to each stored record, which does not address associating a correct timezone with each record.[4] When reading syslog lines in the RFC 3164 format, supply the year and the timezone yourself, from the file's date or the collector's settings. (Meridian's view)

The Common Log Format is a text file format used by web servers when generating server log files.[5] In the Common Log Format, a field set to a dash indicates missing data.[5] Turn those dashes into NULL before casting a field such as the response size to a number. (Meridian's view)

The method

A grok pattern has the syntax %{SYNTAX:SEMANTIC}, where SYNTAX is the name of the pattern that matches the text and SEMANTIC is the identifier given to the matched text.[1] Logstash ships with about 120 grok patterns by default.[1] Grok sits on top of regular expressions, using the Oniguruma library, so any regular expression is valid in grok as well.[1] By default grok saves every semantic as a string, and the only supported conversions are to int and float.[1]

Syslog was developed in the 1980s by Eric Allman as part of the Sendmail project.[2] Syslog originally functioned as a de facto standard; the Internet Engineering Task Force documented it in RFC 3164 in August 2001, and it was standardised by RFC 5424 in March 2009.[2] The Combined Log Format extends the Common Log Format with referer and user-agent fields.[5]

Sources

  1. [1]Grok filter plugin | Logstash Pluginsdocumentation
  2. [2]Syslog - Wikipediasecondary source
  3. [3]GrokReader - Apache NiFidocumentation
  4. [4]RFC 3164: The BSD Syslog Protocolstandard
  5. [5]Common Log Format - Wikipediasecondary source