Audit Log Analysis

In Linux systems, security and observability are paramount. When a system encounters an attack, configuration errors, or performance bottlenecks, the ability to quickly pinpoint the root cause often determines the speed of recovery. The Linux kernel's built-in audit subsystem provides powerful functionality for recording critical events within the system. Mastering the methods for analyzing audit logs is a core skill essential for every Linux operations engineer and system administrator. This article will guide you from scratch to deeply understand how to configure, collect, and analyze audit logs.

Core Mechanisms of the Audit Subsystem

The Linux audit framework operates based on a "hook" mechanism. When a user-space process executes a specific system call, the kernel triggers the corresponding audit hook, thereby writing event information into log files. The monitored system calls typically involve sensitive operations such as file creation, permission modifications, and network connection establishment.

Audit data is stored by default in the /var/log/audit/ directory, primarily containing two types of files:

  • audit.log: This is the raw binary format log, containing complete context information but requiring specialized tools for parsing.
  • audit2event: Text-format logs generated through conversion, facilitating human readability and conventional grep searches.

To enable audit functionality, corresponding rule files (rules.d/*.rules) typically need to be configured in the /etc/audit/ directory. Administrators can define which system calls need to be recorded based on business requirements. For example, one can configure logging only for root user login attempts or file access to specific directories.

Common Analysis Tools and Commands

Although raw binary logs contain a massive amount of information, reading them directly is not intuitive. Linux provides a series of powerful command-line tools to process this data, with ausearch, aureport, and audit2stat being the most core.

ausearch is the most commonly used tool in audit analysis. It allows users to retrieve specific audit events by combining various conditions. Its syntax supports logical operators (such as -a, -f, -m), enabling precise filtering of records matching specific characteristics.

Here is a typical search example used to find all records where the root user performed write operations in the /etc/ directory:

ausearch -fc 1002 -fa /etc -fw write

In this command, -fc 1002 specifies the file creation system call ID, -fa indicates filtering by filename, and -fw write limits the operation type to writes.

aureport: Generating Detailed Reports

If you need a consolidated analytical report, aureport is the better choice. It can generate detailed text reports containing statistical information, timelines, and event lists, which are very suitable for security audits or incident post-mortems.

Running the following command will generate a report on all file access events within the last 24 hours:

aureport -f text -t 24h -e file_access

This command outputs a structured text file, clearly listing the time of event occurrence, user ID, process name, and specific operation paths.

audit2stat: Visualized Statistical Overview

For scenarios requiring a quick understanding of the system's overall audit status, audit2stat provides a graphical statistical perspective. It can generate bar charts or pie charts to show the distribution of different event categories, helping administrators identify high-frequency anomalous events.

Practical Exercise: Investigating Suspicious Privilege Escalation Behavior

Assume you suspect an unauthorized privilege escalation attack has occurred in the system; you need to verify this hypothesis through audit logs. We can combine the tools mentioned above to build a complete analysis workflow.

First, we need to confirm if the audit daemon (auditd) is currently running. In most distributions, this service runs by default upon system startup. If the service has stopped, please start and enable it using systemctl enable --now auditd.

Next, use the ausearch tool to search for potential privilege escalation behaviors. Common signs of privilege escalation include users attempting to execute sudo, su, or accessing sensitive files like /etc/sudoers. We can construct the following query:

ausearch -f sudo -f su -f /etc/sudoers -t all

This command will retrieve all audit records related to sudo and su command executions as well as the sudoers file.

Assuming the search results show a regular user alice attempting to access /etc/shadow at 10:45:23, this is highly likely a sign of failed privilege escalation. At this point, we can further use the -p parameter of ausearch to obtain the complete stack trace for that event, checking which process triggered the system call:

ausearch -fc 1002 -f /etc/shadow -p

By analyzing the output PID and command name, you can confirm the specific tool and method used by the attacker.

Finally, export the key logs to a text file for archiving or sharing with the security team:

ausearch -fc 1002 -f /etc/shadow > suspicious_access.log

Best Practices and Recommendations

To ensure efficient system operation and maintain the integrity of the evidence chain, it is recommended to follow the best practices below in production environments:

  1. Principle of Least Privilege: Do not enable unnecessary audit rules. Only monitor system calls that truly pose a security risk to reduce disk I/O overhead and log storage pressure.
  2. Regular Cleanup Policy: Although audit logs are critical evidence for forensics, long-term accumulation consumes significant space. It is recommended to establish retention policies (e.g., using auditctl to set -o keepfiles) to archive or clean up old logs regularly while meeting compliance requirements.
  3. Real-time Alert Configuration: Integrate complex audit queries into monitoring systems (such as Prometheus + Alertmanager or ELK Stack). Once anomalous patterns are detected (such as a large number of failed logins in a short time), trigger alerts immediately to achieve a shift from "post-event analysis" to "in-process blocking".

By mastering the techniques for analyzing Linux audit logs, you will be able to build a solid security defense line. This not only helps in quickly responding to security incidents but also enhances the overall transparency and controllability of the system.