Understanding Logstash Parsing Configurations and options

In This Tutorial we will learn to customize Logstash to parse any type of Log Files. Logstash helps us to  process logs and other event data from a variety of systems. It also Supports variable injection into elasticsearch and has 200+ plugins.

Logstash Configuration is divided into three sections:

input {
# input config options

}
filter{
# parsing options
}
output {
  # output options
}

In Input Section we configure how we input log files for ingestion, most popular options are lumberjack, file, elasticsearch, graphite

In Ouput Section we configure on what happens to parsed lines in filter section.

In FIlter Section we parse the events. Sample Logstash FIlter Config to ingest syslog events.

filter{
    grok {
match => { "message" => "%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
}

Analysis on Config:

syslog event:  May 18 11:24:30 Jagadeesh-PC /usr/lib/gdm3/gdm-x-session[8693]: Successfully activated service 'org.gnome.Terminal'

this event will be parsed as

syslog_timestamp ===>  May 18 11:24:30
syslog_hostname  ===>  Jagadeesh-PC
syslog_program ===> usr/lib/gdm3/gdm-x-session 
syslog_pid            ===> 8693
syslog_message  ===>  Successfully activated service 'org.gnome.Terminal

TIMESTAMP_ISO8601, SYSLOGHOST, POSINT, GREEDYDATA, DATA are all pattern matchers available in grok.

You are using match to match the log event, you can use add_field, add_tag to pass extra information while storing or you can use this snippet to overwite whole message and store.

  grok {
    match => { "message" => "%{SYSLOGBASE} %{DATA:message}" }
    overwrite => [ "message" ]
  }

You can use other patterns of grok like IPORHOST, HTTPDATE, USERNAME, INT ..etc.,  to parse apache/nginx files. If pattern is not available with grok, you can build your own custom pattern matchers or processing.

Many other plugins like Json, csv, kv, metrics ..etc., are available for parsing Logstash events.

 

Posted On 22 January 2016 By MicroPyramid


Need any Help in your Project?Let's Talk

Latest Comments
Understand Deep Learning and how it works with a simple example

Deep Learning is a part of Machine Learning, it mimics the way our brains fundamentally work. We all know that our brain consists of billions …

Continue Reading...
Letsencrypt wildcard - Setup wildcard subdomain using letsencrypt and certbot

To get wildcard supported certificates, we need to pass the challenge which requires adding TXT records in your dns records.
To get certificates for single domains, …

Continue Reading...
Deploying Wordpress blog with Django using Docker Containers.

Containers run services in an isolated environment, if we deploy our applications using containers then the host system on which these containers are running will …

Continue Reading...
How to setup http password authentication with nginx

HTTP Authentication is used to allow access limit to a site or particular directories by validating the username and password. HTTP basic authentication can also …

Continue Reading...
Gitlab and Docker - Continuous Integration, Deployment and Continuous Delivery

In this blog post, we will learn how to setup and use gitlab container registry to implement, test build and deploy your code automatically with …

Continue Reading...

Subscribe To our news letter

Subscribe and Stay Updated about our Webinars, news and articles on Django, Python, Machine Learning, Amazon Web Services, DevOps, Salesforce, ReactJS, AngularJS, React Native.
* We don't provide your email contact details to any third parties