Cluster Deployment

In the field of distributed message middleware, high availability and scalability are two cornerstones of system design. As a high-performance, reliable message queue product open-sourced by Alibaba Group, one of RocketMQ's core advantages lies in its flexible cluster architecture design. Unlike many other MQ products that tend toward single-point or master-slave replication models, RocketMQ adopts a unique "multi-replica + dual-master" model to build highly available clusters. A deep understanding of RocketMQ's cluster deployment mechanism is crucial for planning production environments, ensuring business continuity, and optimizing message throughput capabilities.

Core Architecture: Multi-Replica and Dual-Master Design

RocketMQ's cluster is not merely a simple stacking of nodes but rather an orchestration based on strict topology structures; the core lies in its data redundancy strategy under "multi-master" or "semi-master" modes. Under default configurations, a standard RocketMQ Cluster consists of multiple Broker instances, where each Broker stores partial message replicas for specific Topics.

To achieve high availability, RocketMQ introduces a dual-master mechanism. This means that within a cluster there are at least two Controllers (Control Plane nodes), which jointly manage the metadata, routing information, and resource scheduling of all Brokers. These two Controllers communicate with each other in a master-slave relationship: one acts as the Master to handle requests and voting, while the other serves as the Slave to take over services or participate in elections when necessary. This design ensures that even if one control node fails, the cluster remains online and continues to provide service, avoiding downtime risks caused by single points of failure.

Simultaneously, multi-replica strategies at the data level are key to ensuring message non-loss. RocketMQ supports various replication modes, including synchronous, asynchronous, and semi-synchronous ones. In the default semi-synchronous mode, a Broker must receive an ACK (acknowledgment) from at least one Master before returning a successful response when submitting write operations to the Controller; in full synchronous mode, it requires waiting for all controllers to return their respective ACKs. This mechanism maximizes performance while minimizing the probability of data loss to its greatest extent.

Broker Node Roles and Distribution Strategies

In a cluster, Brokers are the core components that actually bear message storage and consumption. To maximize hardware resource utilization and avoid hot-spot failures, RocketMQ provides flexible deployment recommendations. Typically, multiple Broker instances can run within a single physical machine or container (e.g., 192.168.0.1:1091, 192.168.0.1:1092), and these instances collectively form logical cluster nodes.

Key points to note during deployment include:

  • Port Planning: Each Broker instance requires an independent listening port; by default, the BrokerName (e.g., broker-a, broker-b) corresponds to different port numbers.
  • Naming Conventions: Explicitly specify the Broker's Name, Port, and Controller connection addresses in configuration files. For example, a standard cluster may include two Brokers (master1, slave1) and corresponding controller nodes.
  • Resource Isolation: While multiple Brokers can be deployed on a single machine, it is recommended to distribute different Topics or partitions of the same Topic across different physical servers in production environments to prevent large amounts of messages from piling up if one server fails.

Disaster Recovery Mechanisms and Election Processings

When a critical component within a cluster fails, RocketMQ possesses a set of complete automatic recovery and master-slave switching mechanisms. The most core scenario is the crash or unavailability of the Controller Master. At this time, the Slave (standby controller) will automatically detect state anomalies and promote itself to become the new Master upon receiving requests from other surviving Brokers, completing the election process.

This process typically involves the following steps:

  1. Fault Detection: Surviving Brokers discover that the current Master Controller cannot respond to requests or times out via heartbeat mechanisms.
  2. Voting Initiation: Remaining surviving nodes (including Slaves) begin participating in voting to find a new Leader.
  3. Election Completion: The node receiving the most votes is selected as the new Master and notifies other nodes to update their routing tables.
  4. Service Restoration: The entire cluster automatically switches to the new master state within seconds, with no perception or only extremely brief latency for external business operations.

Additionally, if a Broker itself fails (e.g., due to OutOfMemory errors, full disk), its message replicas will be automatically migrated to other healthy Brokers, ensuring data persistence safety. This decentralized disaster recovery design enables RocketMQ to withstand the failure of single nodes or even multiple nodes without affecting overall availability.