Generating Data Dynamically with the Template Filter in Logstash

In the data processing pipeline of Logstash, the template filter plays a pivotal role. It allows users to dynamically determine the format or path of output data based on specific field contents within input data. This functionality is crucial for addressing data standardization issues, implementing flexible log archival strategies, and constructing complex data mapping logic. Unlike traditional fixed-rule transformations, the template filter can execute different template renderings based on conditions, thereby significantly enhancing configurability and adaptability in data processing while maintaining pipeline performance. Its core working principle involves parsing field values from input events, matching them against predefined template patterns, and generating new JSON structures or file paths accordingly.
To use the template filter, one must first understand its basic configuration syntax. This filter supports various output formats, with the most common being the generation of a new JSON object or writing data to specific file paths. In configuration files, the template type and name are typically specified via the output parameter, while concrete template definitions are associated in the input section through the template parameter.

Templates can be defined using one of the following methods:

• JSON Format: Directly define a JSON object as the template content.
• File Path: Point to a text file or JSON file containing the template content.
• Pseudo-code Pattern: Use pseudo-code logic similar to regular expressions to match fields and generate output.

For example, if you wish to dynamically adjust the structure of the output JSON based on the level field in logs, you can define the following configuration snippet:

filter {
  if [level] == "ERROR" {
    template("error_template") {
      output => {
        json => {
          level => [field("level")],
          message => [field("message")],
          timestamp => [field("@timestamp")]
        }
      }
    }
  }
}

Practical Examples of Dynamically Generating JSON Data

In practical applications, the template filter is frequently used to refactor complex log data. Suppose we receive raw logs in a disorganized format containing multiple nested fields and need to uniformly convert them into a standard-compliant JSON structure. By defining different templates, we can achieve an "one fish, many dishes" effect, meaning a single input log event can match against multiple output templates.

Below is a specific configuration example demonstrating how to generate JSON data with different structures based on the value of the source field (e.g., "web" or "db"):

filter {
  template("web_template") {
    output => {
      json => {
        type: "web_access",
        status_code: [field("status")],
        url: [field("url")],
        user_agent: [field("user_agent")]
      }
    }
  }

  template("db_template") {
    output => {
      json => {
        type: "db_query",
        query_type: [field("operation")],
        table_name: [field("table")],
        execution_time_ms: [field("duration")]
      }
    }
  }
}

In this configuration, if the source field of the input event is "web", then apply web_template; otherwise (implied context for other values not explicitly listed but handled by logic), it applies to specific conditions. If the value is "db", then apply db_template. This mechanism enables Logstash to flexibly handle diverse data sources without requiring complex code logic to achieve dynamic formatting of data.

Dynamic Write Strategies Based on File Paths

In addition to generating JSON data, the template filter is widely used to control the storage location of log files. This is particularly useful for implementing content-based log archival strategies, such as saving logs containing sensitive information separately or centrally managing logs from specific business lines by setting up path parameters allows Logstash to dynamically calculate file paths based on field contents within input data. This is often combined with glob patterns to achieve finer-grained path control. The following example demonstrates how to write logs into different directories based on the value of the environment field:

output {
  template("env_based_path") {
    path => "/var/log/%{[environment]}_app.log"
    glob => "*.log"
  }
}

In this scenario, if the environment field in the input data is "production", logs will be written to /var/log/production_app.log; if it is "staging", they will be written to /var/log/staging_app.log. This dynamic path generation capability greatly simplifies log management logic under multi-environment deployments and avoids maintenance difficulties caused by hard-coded paths.

Best Practices and Performance Considerations

When using the template filter, reasonable architectural design is crucial for ensuring system stability and performance. First, avoid defining too many template outputs within a single event to reduce processing overhead. Second, keep template files as concise as possible; avoid overly deep nested structures to minimize CPU consumption during JSON parsing and generation. Additionally, regularly review template logic to remove unused rules helps maintain the clarity and maintainability of configuration files.

For high-throughput log streams, it is recommended to decompose complex template logic into multiple steps, utilizing if statements for conditional judgment before invoking corresponding template outputs. This divide-and-conquer strategy not only improves code readability but can also optimize overall processing performance by adjusting thread counts when necessary. In summary, mastering the use of the template filter is a key skill in building efficient, flexible, and scalable Logstash data processing pipelines.