ELK Stack for Parsing your Logs

Reading Time : ~ .

In This Tutorial we will look into parsing your Syslog files and store and display it in an intractable website. We will be using ELK Stack for this purpose before Jumping Into ELK Stack Let's see what each means

E - Elasticsearch: open source search and analytics engine, which stores data and relies on Apache Lucene for searching.

L - LogStash: Logstash is an open source data collection engine with real-time pipelining capabilities. Logstash can dynamically unify data from different sources and normalize the data into destinations of your choice.

K- Kibana: Analytics and visualization platform designed to work with Elasticsearch.which can be used to  search, view, and interact with data stored in Elasticsearch indices.

 Let's setup Nodes to send and receive Log Files

Installing Elasticsearch:

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get -y install oracle-java8-installer
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.2.deb
sudo dpkg -i elasticsearch-1.5.2.deb
sudo update-rc.d elasticsearch defaults 95 10

Installing Logstash:

wget http://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.0-1_all.deb
sudo dpkg -i logstash_1.5.0-1_all.deb
sudo update-rc.d logstash defaults 98 2


Configuring log stash:

sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private
sudo vi /etc/ssl/openssl.cnf

Find the [ v3_ca ] section in the file, and add this line under it:
subjectAltName = IP: logstash_server_ip

cd /etc/pki/tls
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

This creates cert and key for log stash and log stash forwarder 

Configuration File:
Create a log stash Configuration file in 
sudo nano /etc/logstash/conf.d/<configfile name>

input {
  lumberjack {
    host => "<logstash_server_ip>"
    port => 5090
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}
filter{
 if [type] == "syslog" {
    grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
}
}
output {
  elasticsearch {   host => "127.0.0.1" }
}

Save and Restart the lost stash. The Receiving Node should be able to parse Syslog messages and store them in ElasticSearch. In Next Tutorial, we will configure sending node to push Syslog  files to our EL Server and also setup Kibana to View Data graphically. For More information on Logstash configurations visit this post.

    By Posted On
SENIOR DEVELOPER at MICROPYRAMID

Need any Help in your Project?Let's Talk

Latest Comments
Related Articles
Understanding Logstash Parsing Configurations and options Jagadeesh V

In This Tutorial we will learn to customize Logstash to parse any type of Log Files. Logstash helps us to process logs and other event ...

Continue Reading...
ELK Stack for Parsing your Logs- Part 2 Jagadeesh V

In Previous Tutorial we looked into Setting up EL Server which can ingest your Syslog files. In this Post, you will learn about pushing your ...

Continue Reading...
Continuous Integration And Continuous Delivery With Gitlab And Docker Jagadeesh V

In this blog post, we will learn how to setup and use gitlab container registry to implement, test build and deploy your code automatically with ...

Continue Reading...

Subscribe To our news letter

Subscribe to our news letter to receive latest blog posts into your inbox. Please fill your email address in the below form.
*We don't provide your email contact details to any third parties