Kibana Dashboards & Shipping Logs: ELK Stack Part 2

Once Elasticsearch is ingesting parsed logs, the next stage is shipping logs from every host and exploring them in Kibana. In practice that means: forward logs with Filebeat or the modern Elastic Agent + Fleet, create a data view (what older docs call an index pattern), search in Discover with KQL, then build visualizations and assemble a dashboard. This is Part 2 of our ELK series and assumes you already have an Elasticsearch + Kibana cluster running on Elastic Stack 8.x or 9.x.

Key takeaways

• Elastic Agent + Fleet is the recommended shipper for new deployments in 2026 — one agent for logs, metrics, and traces, managed centrally from Kibana — while individual Beats (Filebeat, Metricbeat) still work for single-purpose, edge setups.
• Kibana renamed "index patterns" to "data views" in 8.0; a data view (e.g. logs-*) tells Kibana which indices or data streams to query.
• Discover is where you search raw logs; the default query language is KQL (Kibana Query Language), e.g. host.name : "web01" and log.level : "error".
• Build charts in Lens (drag-and-drop) and pin them onto a dashboard for an at-a-glance view of your log estate.
• Security (TLS + authentication) is on by default since Elasticsearch 8.0, so Kibana talks to Elasticsearch over HTTPS — expose Kibana only behind a reverse proxy with TLS, never on 0.0.0.0.
• Kibana is the Elastic UI; OpenSearch Dashboards is the Apache-2.0 fork. After Elastic's 2021 relicensing (SSPL / Elastic License) it added an AGPL option in 2024.

What did Part 1 cover, and where does Part 2 pick up?

Part 1 — ELK Stack log parsing: a getting-started guide set up Elasticsearch and Kibana with Docker, walked through the Filebeat → (Logstash optional) → Elasticsearch → Kibana pipeline, and showed where to parse logs — in Logstash filters or a lighter Elasticsearch ingest pipeline. For the grok and dissect details behind that parsing, see Understanding Logstash parsing configurations and options.

This part assumes parsed events are already landing in Elasticsearch. It focuses on the two jobs that come next: getting logs off your servers reliably, and turning the indexed data into searches, charts, and dashboards that people actually look at.

Should you ship logs with Beats or Elastic Agent + Fleet?

The original ELK workflow used a separate Beat for each data type — Filebeat for logs, Metricbeat for metrics, and so on — each configured by editing YAML on every host. That still works, but the modern, recommended approach is Elastic Agent: a single binary that collects logs, metrics, and traces, configured centrally through Fleet in Kibana. Instead of editing files on dozens of machines, you attach hosts to an agent policy and add Elastic Integrations (the catalog that replaces per-Beat modules) from the UI.

	Beats (e.g. Filebeat)	Elastic Agent + Fleet
Scope	One Beat per data type	Single agent for logs, metrics, traces, security
Configuration	Edit YAML on each host	Central agent policies in Kibana → Fleet
Add-ons	Per-Beat modules	Elastic Integrations (one catalog)
Management at scale	Manual / config-management tooling	Built-in fleet management, central upgrades
Best for	Minimal edge nodes, a single data type	New deployments and large fleets of hosts
Status in 2026	Supported, still widely used	Recommended default

Option A — install Filebeat from the Elastic repository

If you only need to tail a few log files, Filebeat is still the lightest option. Install it from the current Elastic 9.x APT repository (the old logstashforwarder / logstash-forwarder packages are long retired):

# Add Elastic's signing key and the 9.x APT repository (Debian/Ubuntu)
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch \
  | sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] \
  https://artifacts.elastic.co/packages/9.x/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/elastic-9.x.list

sudo apt-get update && sudo apt-get install filebeat

# Enable a pre-built module (parses common formats) and start the service
sudo filebeat modules enable system
sudo systemctl enable --now filebeat

Point Filebeat at Elasticsearch (or at Logstash on 5044 for heavy parsing) in /etc/filebeat/filebeat.yml, using the filestream input and the cluster's HTTP CA certificate over HTTPS. Part 1 shows that filebeat.yml in full, so we won't repeat it here.

Option B — enroll a host with Elastic Agent + Fleet

For anything beyond a couple of servers, prefer Elastic Agent. In Kibana, open Fleet (under Management), set up a Fleet Server if you don't have one, create or pick an agent policy, and copy the enrollment command it generates. On each host you run a single install command — the agent then pulls its configuration and integrations from Fleet and upgrades centrally:

# Download and install Elastic Agent, then enroll it into Fleet.
# Kibana → Fleet → Agents → Add agent generates the exact URL + token.
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.0.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-9.0.0-linux-x86_64.tar.gz
cd elastic-agent-9.0.0-linux-x86_64

sudo ./elastic-agent install \
  --url=https://FLEET_SERVER_HOST:8220 \
  --enrollment-token=<ENROLLMENT_TOKEN>

# Check status; the agent now ships data per its Fleet policy
sudo elastic-agent status

How do you view logs in Kibana? Data views, Discover, and KQL

Kibana doesn't query indices directly — you first define a data view, the saved definition of which indices or data streams to read. This is the single biggest naming change for anyone returning to the stack: "index patterns" were renamed "data views" in Kibana 8.0. The concept is the same — a wildcard like logs-* or filebeat-* plus a time field — but every menu, tutorial, and API written for 7.x and earlier uses the old name.

Term in older (≤7.x) docs	Current term (8.x / 9.x)	What it is
Index pattern	Data view	Definition of which indices/data streams Kibana queries (e.g. logs-*)
Lucene query string	KQL (default)	The query language in the Discover search bar
Timelion / TSVB-first charts	Lens	The default drag-and-drop visualization builder
Settings → Index Patterns	Stack Management → Data Views	Where you create and manage them

Create a data view under Stack Management → Data Views → Create data view: enter an index pattern such as logs-*, choose @timestamp as the time field, and save. Then open Discover from the main menu, pick your data view and a time range, and search with KQL in the query bar. KQL is field-aware and far friendlier than raw Lucene:

# Kibana Discover — KQL examples (type these in the search bar)

# Errors from one host
host.name : "web01" and log.level : "error"

# Failed SSH logins in the system module
event.dataset : "system.auth" and message : *Failed password*

# All HTTP 5xx responses
http.response.status_code >= 500

# Anything mentioning "out of memory", excluding a noisy host
message : "out of memory" and not host.name : "staging-box"

How do you build visualizations and dashboards in Kibana?

Discover answers "what happened just now?"; dashboards answer "what's the trend?". The workflow is to build individual panels, then arrange them on one board:

1. Open Visualize Library → Create visualization and choose Lens (the default drag-and-drop builder).
2. Pick your data view, drag a field onto the canvas, and Lens suggests a chart — for logs, common panels are a date histogram of log volume, a bar chart of log.level, and a top-values table of host.name or error messages.
3. Save each visualization, then open Dashboard → Create dashboard and add your saved panels.
4. Set a shared time range and auto-refresh, drop a KQL filter in the dashboard's query bar to scope every panel at once, and save.

The result is a single screen showing log volume over time, error breakdowns, and the noisiest hosts — exactly the original goal of "display it in Kibana with interactive graphs", now built with Lens instead of the legacy aggregation editor.

How do you expose Kibana safely?

Since Elasticsearch 8.0, security is on by default: the cluster runs with TLS and authentication, and Kibana connects to it over HTTPS using a service account. Don't undo that. The deb/rpm packages install Kibana as a systemd service (systemctl enable --now kibana) — no need for the old Supervisor recipe. Keep Kibana bound to localhost in kibana.yml (server.host: "localhost") and put a TLS-terminating reverse proxy in front of it:

# /etc/nginx/sites-available/kibana  — TLS reverse proxy in front of Kibana
server {
    listen 443 ssl;
    server_name logs.example.com;

    ssl_certificate     /etc/letsencrypt/live/logs.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/logs.example.com/privkey.pem;

    location / {
        proxy_pass http://localhost:5601;   # Kibana, bound to localhost only
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Redirect plain HTTP to HTTPS
server { listen 80; server_name logs.example.com; return 301 https://$host$request_uri; }

Add native-realm users and roles so analysts get read-only access, and never expose port 9200 or 5601 to the public internet. For the cluster side of this, see our checklist of Elasticsearch security measures and the walkthrough on securing an Elasticsearch instance, which covers the move from the old Shield plugin to today's built-in security.

Kibana, OpenSearch Dashboards, and licensing

A quick honesty note, because it affects which UI you'll actually run. In 2021 Elastic relicensed Elasticsearch and Kibana from Apache 2.0 to a source-available SSPL / Elastic License 2.0 model; AWS forked the last Apache-2.0 release into OpenSearch, whose UI is OpenSearch Dashboards. In 2024 Elastic added AGPL 3.0 as an option, restoring an OSI-approved license. Data views, Discover, and dashboards exist in both Kibana and OpenSearch Dashboards with near-identical workflows; Part 1 has the full ELK-vs-OpenSearch comparison. Elastic's paid tiers (Basic, Platinum, Enterprise) gate extras like alerting connectors and machine learning, but the core Discover-and-dashboard workflow above is available in the free Basic tier.

Frequently Asked Questions

What replaced index patterns in Kibana?

Data views replaced index patterns in Kibana 8.0. The idea is unchanged — a data view names the indices or data streams Kibana should query (such as logs-* or filebeat-*) plus a time field — but the menus moved to Stack Management → Data Views and the API endpoint became data_view. Tutorials written for 7.x and earlier still say "index pattern" for the same thing.

Should I use Filebeat or Elastic Agent?

For new deployments, prefer Elastic Agent + Fleet: one agent ships logs, metrics, and traces, and you manage it centrally with agent policies and Elastic Integrations in Kibana. Stick with individual Beats like Filebeat for minimal edge nodes, a single data type, or when you don't want to run a Fleet Server. Both are supported in 2026.

What is KQL in Kibana?

KQL (Kibana Query Language) is the default, field-aware query language in the Discover search bar. You write expressions like host.name : "web01" and log.level : "error" or http.response.status_code >= 500. It's simpler than the older Lucene query syntax, supports and/or/not and wildcards, and you can toggle back to Lucene if you need a feature KQL lacks.

How do I build a log dashboard in Kibana?

Create a data view, build individual charts in Lens (a date histogram of log volume, a breakdown of log.level, a top-values table of host.name), save each one, then open Dashboard → Create dashboard and add the saved panels. Set a shared time range and a KQL filter at the dashboard level to scope every panel at once.

Is Kibana free, and how does it relate to OpenSearch Dashboards?

Kibana's core search-and-dashboard features are available in the free Basic tier under the Elastic License / SSPL, with an AGPL option added in 2024; paid tiers (Platinum, Enterprise) add extras like advanced alerting and machine learning. OpenSearch Dashboards is the Apache-2.0 fork of Kibana that AWS maintains after the 2021 relicensing, with a near-identical Discover and dashboard experience.

Why does Kibana connect to Elasticsearch over HTTPS now?

Because security is on by default since Elasticsearch 8.0 — the cluster ships with TLS and authentication enabled, so Kibana must connect over HTTPS with a service account and trust the cluster's HTTP CA certificate. This is intentional; keep it on, bind Kibana to localhost, and place a TLS reverse proxy in front rather than exposing port 5601 directly.