Message Storage Mechanism in RocketMQ Core Concepts and Architecture

In the distributed architecture of RocketMQ, the message storage mechanism serves as the cornerstone ensuring system high availability, low latency, and data consistency. Unlike traditional database storage, RocketMQ adopts a disk-based log-structured storage design, making message persistence one of its core objectives. This mechanism not only guarantees that messages are not lost following node failures but also supports massive message processing capabilities through an efficient read-write separation strategy. Understanding the principles behind RocketMQ's message storage is crucial for building stable and reliable enterprise-grade messaging systems.
RocketMQ's message storage adheres to the principle of "sequential write, append-only read," a design that directly dictates its extremely high write throughput and read performance. At the physical disk level, Broker nodes serialize and store messages as files; each file corresponds to a specific Queue under a particular Topic.

Specifically, when a producer sends messages, the Broker appends data blocks (Message Blocks) into corresponding files in chronological order. This sequential writing approach significantly reduces random I/O operations, bringing disk read/write efficiency close to its theoretical limit. Meanwhile, consumers only need to sequentially read from the file header upon pulling messages, eliminating the need for complex index lookups or random access, thereby achieving efficient batch consumption.

To manage these storage files, RocketMQ introduces two key data structures: CommitLog (Commit Log) and ConsumeQueue.

• CommitLog: Responsible for storing the raw message data, including payloads, attributes, position indices, etc. It serves as the primary carrier of message persistence and employs a sequential write method.
• ConsumeQueue: Stores only the physical offset within CommitLog corresponding to each message along with its Queue ID. It does not store message content but acts as an index table for quickly locating messages in CommitLog; it also utilizes sequential write and append-only read methods.

File Lifecycle Management and Cleanup Strategies

As the volume of messages grows continuously, files within CommitLog and ConsumeQueue accumulate indefinitely, potentially reaching disk capacity limits. Consequently, RocketMQ implements a refined file lifecycle management strategy to ensure effective utilization of storage resources.

The system controls file generation and merging via configuration parameters fileSize (single file size) and maxFileSize (maximum single file size). When a file exceeds the threshold, the Broker automatically triggers cleanup mechanisms. This process consists of two phases:

1. Physical Deletion: Marks oversized files for deletion status and removes references from metadata.
2. Logical Reclamation: Executes actual disk space release operations within background threads.

Notably, RocketMQ adopts a "delete-before-write" strategy to prevent file fragmentation issues. When new messages require writing, the system checks the current maximum offset of free space (Max Offset). If this position is smaller than the starting position of the new message, it indicates that previous files have been logically deleted but not yet physically cleared; in such cases, a brand-new empty file is created directly for writing, ensuring data remains continuously contiguous.

High Availability Assurance via Storage Architecture

RocketMQ's message storage mechanism does not exist in isolation but is tightly integrated with the Master-Slave master-slave architecture to jointly build a high-availability storage system. In distributed deployments, each Broker node maintains its own CommitLog and ConsumeQueue on local disks. When a specific Broker node fails, the message data residing on it is not lost; instead, it remains preserved on the disks of other healthy nodes.

Through the Sync Replication mechanism, before synchronizing messages to local disk storage, Brokers first replicate data to slave (follower) nodes. Once the master node fails and an election selects a new leader, this new primary can immediately take over services by pulling the latest CommitLog and ConsumeQueue from followers to reconstruct the storage state. This combination of "multi-replica + sequential write" enables RocketMQ to achieve millisecond-level recovery time objectives (RTO) while guaranteeing zero message loss.

Summary

RocketMQ's message storage mechanism successfully resolves storage bottleneck issues in high-concurrency scenarios through sequential writes, dual-index structures (CommitLog and ConsumeQueue), and intelligent file lifecycle management. Its design philosophy revolves around trading space for time: leveraging the fast sequential read/write characteristics of disk to achieve extreme system throughput, while relying on multi-replica synchronous mechanisms to ensure data persistence and availability. Mastering this core concept is a prerequisite for deeply understanding RocketMQ's architecture and conducting effective operations and maintenance.