Collecting Logs with Filebeat

In building an observability platform, log collection is the starting point of data flow. Filebeat, as a lightweight log collector within the Elastic Stack (formerly known as the ELK Stack) ecosystem, was designed not to replace traditional Syslog servers or complex message queues, but to focus on efficiently and low-latently collecting local log files and sending them to Elasticsearch. Compared to Logstash, Filebeat has a more streamlined architecture; it lacks a built-in data processing pipeline (Pipeline), which gives it a significant advantage in resource consumption, making it particularly suitable for running in edge computing environments or nodes requiring rapid deployment.

The workflow of Filebeat primarily consists of three key phases: first, it monitors specified log file directories through specific Input plugins (such as fileinput); second, it reads the log lines, parses, filters, and tags them before packaging them into events; finally, these events are sent to Elasticsearch via Output plugins. This "collect-and-transmit" model requires users to perform preliminary structured cleaning before data enters the storage layer, thereby placing high demands on configuration accuracy.

Building Basic Filebeat Collection Configuration

To launch a stable Filebeat instance, one must first create a configuration file named filebeat.yml. Located in the Filebeat installation directory, this file serves as the central hub controlling its behavior. At the introductory stage, we primarily focus on two core modules: Input and Output.

Regarding the Input section, the most commonly used plugin is fileinput. It allows users to specify one or more file paths and define collection strategies. For example, we can configure Filebeat to scan .log files under the /var/log/ directory every 10 seconds and split log lines into chunks of 512 bytes each. Additionally, the multiline parameter can be set to handle multi-line log events, which is crucial for parsing application logs that contain multiple fields or span across several lines.

For the Output section, the goal is to push the collected data to Elasticsearch. The elasticsearch plugin is typically used, with corresponding cluster addresses, index names, and authentication information configured. If Elasticsearch has not yet been deployed, Filebeat also supports storing data temporarily as JSON format files for subsequent manual verification or migration to other systems.

Below is a typical example of a basic filebeat.yml configuration:

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /var/log/*.log
  json.keys_under_root: false
  json.add_error_key: true
  fields:
    host: "${host}"
    app: "my-application"

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "filebeat-logs-%{+yyyy.MM.dd}"
  username: "elastic"
  password: "your_password_here"

In the configuration above, paths defines the file pattern to monitor, json.add_error_key ensures error information is tagged separately for analysis, and the fields section injects metadata such as hostnames and application names; these tags will play a vital role in subsequent search and aggregation queries.

Practical Techniques for Parsing and Structuring Log Data

Collection is just the first step; converting unstructured raw text into machine-understandable structured data is where Filebeat truly adds value. Although Filebeat itself does not execute complex pipeline logic, it provides powerful parse functionality that allows users to clean data using regular expressions or JSON formats.

Assuming an application log has the original format: [INFO] [main] {timestamp} - User login success ip=192.168.1.1. To facilitate analysis, we need to extract fields such as timestamp, level, message content, and IP address. In filebeat.yml, this can be achieved through the processors section. For instance, use the json processor to parse log lines into JSON objects, combined with the date processor to convert strings into standard time formats, and finally utilize add_field to add custom tags.

processors:
- add_host_metadata: ~
- json:
    target: "log"
    fields_from_keys: ["timestamp", "level", "message"]
- date:
    field: "timestamp"
    formats: ["%Y-%m-%d %H:%M:%S"]
    target: "@timestamp"

Through the above configuration, the originally chaotic text logs are transformed into JSON objects with explicit fields such as @timestamp, log.level, and log.message. This structured processing not only improves log readability but also significantly enhances efficiency in building visualization dashboards and performing complex aggregation queries in Kibana.

Deployment Monitoring and Troubleshooting Strategies

Once configured, the stability of Filebeat directly relates to the reliability of the entire logging system. In production environments, it is recommended to adopt containerized deployment methods using Docker Compose or Kubernetes for orchestration. Simultaneously, a comprehensive monitoring mechanism must be established to ensure Filebeat can promptly detect and report anomalies.

Key monitoring metrics include: collection file latency (filestream.file_count), data transmission rates, and Elasticsearch write status. These metrics can be visualized using Prometheus and Grafana; if the collection rate of a node drops sharply or a large number of errors occur, one can quickly identify whether the cause is incorrect file path configuration, insufficient permissions, or network fluctuations.

Additionally, regularly reviewing log content is an indispensable step. One can view the most recently collected logs through Kibana's Discover interface to confirm if parsed fields are complete and whether there are any garbled characters or format errors in records. If specific types of logs cannot be parsed correctly, return to filebeat.yml to adjust regular expressions or parsing rules promptly, forming a closed loop of "configure-run-verify-optimize".

By following the steps above, you can quickly build an efficient and scalable Filebeat log collection system, laying a solid data foundation for subsequent log analysis, security auditing, and troubleshooting.