Filebeat Input Plugin Configuration Guide

In a log collection architecture, Logstash serves as the central hub, responsible for collecting, processing, and transporting logs from various data sources. Among the numerous Input plugins, fileinput is one of the most fundamental and widely used components, specifically designed to monitor and read file contents on the local filesystem. This article provides a detailed overview of the plugin's core concepts, key configuration parameters, and practical application scenarios to assist developers in building efficient ETL pipelines.
The Logstash fileinput plugin operates using a "monitor - parse" working mode. Upon startup, Logstash reads the path settings from the configuration file and opens the specified files for monitoring. Once new content is detected (i.e., a change occurs in the file), the plugin immediately captures these changes and processes them. This mechanism ensures high availability for log collection, effectively capturing logs even when source generation speeds are extremely high or there is a large volume of concurrent writes.

Its core logic consists of the following steps:

  1. Initialization: Load target files or directory lists based on the configuration.
  2. Monitoring Trigger: Detect file create, delete, and modify operations through file change notification mechanisms (such as Linux's inotify), or check for new content by monitoring file size changes during each read.
  3. Data Extraction: Parse newly added lines or blocks within the file to generate event objects.
  4. Pipeline Transfer: Send the processed events to subsequent Filter and Output plugins for further processing.

Key Configuration Parameters Explained

To flexibly adapt to different log scenarios, fileinput offers a rich set of configuration options. Below are several of the most common and critical parameters:

  • path (Path):
    Specifies the path to the file or directory to be monitored. Supports wildcards (e.g., *.log), allowing Logstash to automatically expand all matching files. If set to a directory mode, the plugin will recursively scan all sub-folders and files meeting the criteria within that directory.

  • start_position:
    Defines the starting position for reading the file. The default value is 0, indicating reading from the beginning; if set to -1 (or end_of_file), it reads only new content appended to the end of the file, which is highly effective for handling circular log writes and avoids consuming historical data repeatedly.

  • follow_symlinks (Follow Symlinks):
    A boolean parameter, defaulting to false. When set to true, Logstash resolves and monitors symbolic links pointing to other files; if set to false (default), it ignores these links and reads the content of the target path itself directly.

  • recode:
    Used to unify file encoding formats. If the source log file's encoding does not match what Logstash expects, this parameter can automatically perform character set conversion to prevent parsing failures or data loss caused by garbled text.

Practical Application Scenarios Example

In actual deployments, fileinput is often combined with other plugins to build complete processing workflows. The following is a typical configuration snippet demonstrating how to extract specific fields from an Nginx log directory and forward them to Elasticsearch:

input {
  file {
    path => "/var/log/nginx/*.log"
    start_position => "beginning" # Or "end_of_file"
    type => "nginx_access_log"   # Define event type for subsequent filtering
    
    codec => json { }             # Parse log lines into JSON format for easier field extraction
  }
}

filter {
  if [type] == "nginx_access_log" {
    grok {
      match => { 
        access: "%{COMBINEDAPLOGICNGINXLOG}" 
      }
    }
    
    mutate {
      rename => { 
        'access' => 'request_line' # Rename field to meet business requirements
      }
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logs-%{+YYYY.MM.dd}" # Index by date for easier archival queries
  }
  
  stdout { codec => rubydebug }   # Used to debug output partial log content
}

In the configuration above, the path parameter utilizes wildcards to achieve batch monitoring of multiple Nginx log files; setting start_position to beginning ensures full data loading (suitable for cold starts or initial synchronization), whereas in production environments it is generally recommended to set it to end_of_file for incremental capture. Combined with the grok filter, complex regular expressions automatically extract key fields such as IP addresses, URLs, and status codes, finally standardizing the output to Elasticsearch in JSON format, which significantly enhances log retrieval and analysis efficiency.

Performance Optimization and Considerations

Although fileinput is powerful, potential performance bottlenecks must be considered when handling massive volumes of logs. Firstly, monitoring an excessive number of files or dealing with extremely large single files (e.g., exceeding several GB) may lead to high memory consumption or OOM errors. It is recommended to split large files into multiple smaller ones or impose a maximum file size threshold. Secondly, frequent creation and deletion of small files generate a large volume of system notification events, consuming Logstash thread resources; in such cases, the sync parameter should be adjusted appropriately or asynchronous mode enabled to balance real-time performance with throughput.

Furthermore, ensure that path permissions in the configuration file are correct; the Logstash service account must have read permissions for the target files, otherwise input interruptions will occur and troubleshooting may become difficult. By configuring these details reasonably, the value of filesystem input plugins can be fully realized, laying a solid data foundation for log analysis systems.