Integrating Filebeat with Kibana as a Data Source

In the modern pipeline for log collection and analysis, Kibana serves as the visualization frontend for Elasticsearch. Its data efficiency is highly dependent on underlying ingestion components. Filebeat, designed specifically to replace traditional Logstash with a lightweight, high-performance architecture, aims to resolve bottlenecks in high-throughput log processing. This article delves into the technical details of deep integration between Kibana and Filebeat, covering the entire process from configuration deployment to data flow.
When building an ELK (Elasticsearch, Logstash, Kibana) or EFK (Elasticsearch, Filebeat, Kibana) stack, Filebeat plays a critical role as the "mover." It reads files, parses log content, converts it into JSON format, and sends it to Elasticsearch, thereby completing the data ingestion phase.

Compared to traditional Logstash, Filebeat offers significant advantages:

  • Lightweight Architecture: Filebeat consists of only one binary file with no complex dependency environment required, making deployment extremely simple.
  • Low Resource Consumption: Designed to minimize memory and CPU usage, it is suitable for running in resource-constrained servers or container environments.
  • Multi-Source Ingestion Capabilities: Beyond traditional log files, Filebeat supports the collection of various data sources including system events (System Events), network traffic, database auditing, etc.

Core Configuration and Pipeline Setup

To achieve effective integration between Kibana and Filebeat, it is crucial to correctly configure the filebeat.yml configuration file in Filebeat to ensure accurate routing of data to Elasticsearch for indexing and display by Kibana.

First, define an Input module within the Filebeat configuration file. Taking common file monitoring as an example:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/application/*.log
  json.keys_under_root: false
  json.add_error_key: true

This configuration specifies the path Filebeat needs to monitor and enables JSON parsing functionality.

Second, an Output module must be defined to connect to Elasticsearch:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "app-logs-%{+yyyy.MM.dd}"

Here, the node address of Elasticsearch and dynamically generated index names are specified, ensuring logs are archived daily.

Data Flow and Index Optimization

Once Filebeat is running successfully, it sends parsed log data to Elasticsearch via HTTP protocol. Kibana connects directly to the Elasticsearch cluster and queries, filters, and visualizes data through interfaces like Kibana Discover or Visualizer.

To improve overall performance, consider enabling the following optimization strategies in your configuration:

  • Batched Sending: Filebeat defaults to accumulating a certain number of documents before sending them once to Elasticsearch, effectively reducing network latency.
  • Field Mapping (Mapping): On the Elasticsearch side, reasonable mapping definitions can accelerate query speeds. For instance, timestamp fields must be defined as date type so that Kibana can perform precise time-axis sorting and aggregation analysis.

Practical Example: Monitoring Container Logs

Assume we have deployed a Docker container application requiring real-time monitoring of its standard output (stdout) logs. We can create a Filebeat configuration file named docker-container-monitor.yml:

filebeat.inputs:
- type: container
  enabled: true
  container_id: "my-app-container"
  json.keys_under_root: false
  json.add_error_key: true
  log_level: debug

output.elasticsearch:
  hosts: ["http://192.168.1.100:9200"]
  index: "docker-logs-%{+yyyy.MM.dd}"

After deployment, Filebeat automatically retrieves the container ID via Docker API and pulls log streams in real time. In Kibana, users simply need to select the docker-logs-* index to view real-time application runtime status, error stacks, and performance metrics without any additional development work.

Summary and Best Practices

Integration between Kibana and Filebeat forms the core infrastructure for observability (Observability). By reasonably configuring Filebeat's input sources and output routing in conjunction with Elasticsearch storage capabilities and Kibana analysis functions, a highly efficient and scalable log monitoring system can be constructed. In practical applications, it is recommended to regularly clean up expired indexes to free up storage space and adjust the ingestion frequency (Batch Size) of Filebeat according to business requirements to achieve an optimal balance between performance and resource consumption.