Message Filtering Examples in RocketMQ

In the distributed architecture of RocketMQ, message filtering is a critical mechanism for enhancing system throughput, reducing storage costs, and optimizing consumption performance. Its core logic operates as follows: when a Broker receives messages sent by producers, it matches the message body against the configured filterExpression. Only messages that satisfy the filter conditions are persisted to their corresponding Topic; all other messages are either discarded directly or retained only in memory (depending on specific configuration strategies). This mechanism not only reduces disk I/O and index maintenance overhead but also enables consumers to precisely retrieve required data, avoiding unnecessary fetching and parsing of invalid payloads.

Typical application scenarios include:

  • Access Control: Implementing permission management within multi-tenant systems.
  • Log Management: Retaining records containing specific keywords in logging systems while discarding the rest.
  • Hot Data Handling: Targeted storage for high-concurrency scenarios involving hot data.

By designing reasonable filtering strategies, enterprise applications can significantly reduce cloud storage costs and improve overall system response speed while ensuring data integrity.

Filter Configuration Based on Regular Expressions

RocketMQ supports using regular expressions as filter conditions; this is the most fundamental and universally applicable implementation method. Users must specify the filterExpression parameter when creating or updating a Topic, which determines whether Broker will store specific messages.

Below is a concrete configuration example demonstrating how to set up filtering rules for a specific business scenario:

{
  "topic": "order_event",
  "readQueueNums": 4,
  "writeQueueNums": 8,
  "filterExpression": "order_id=1001"
}

In the configuration above, filterExpression is set to order_id=1001. This means that a message will only be persisted to disk if its body contains the field order_id=1001; all other messages carrying different order IDs are filtered out directly and do not occupy any storage space.

Regular expressions support various operators and logical combinations, such as using | for "OR" operations, & for "AND" operations, and ! for negation matching. This flexibility allows developers to construct fine-grained filtering logic based on complex business requirements without relying on additional middleware for data routing.

Multi-Tag Filtering Strategy Based on Tags

In addition to regular expressions, RocketMQ supports message filtering through the Tag mechanism. A Tag is a lightweight classification identifier commonly used for logical grouping of messages or marking specific attributes. When a Topic is configured with multiple Tags, Broker evaluates whether a message meets storage conditions based on its Tag values.

Assume we have an e-commerce order Topic where orders need to be stored differently according to their status:

{
  "topic": "order_status_monitor",
  "filterExpression": "status=success|status=paid"
}

Under this configuration, any message with a Tag of success or paid will be retained, while messages in cancelled or pending states will be filtered out. This approach is particularly suitable for scenarios requiring simultaneous monitoring of multiple business statuses; for instance, a financial audit system might only need to focus on order records that have been completed payment and successfully delivered.

Performance Optimization Recommendations for Production Deployment

Correctly configuring message filtering in production environments can yield significant performance improvements. Since unmatched messages are discarded at the generation phase, Broker avoids executing additional index construction, log flushing, or network transmission operations thereupon, thereby substantially reducing CPU and memory consumption.

When deploying in practice, adhere to the following principles:

  1. Define Filter Rules Early: Clearly specify filter conditions from the outset when creating a Topic to prevent large-scale data redundancy caused by business changes later on.
  2. Keep Expressions Concise and Efficient: Overly complex regular expressions can lead to degraded matching performance; prioritize simple field matching or common Tag combinations instead.
  3. Verify Effectiveness with Monitoring Tools: Utilize RocketMQ Dashboard to monitor message accumulation, storage size, and consumption latency for each Topic to confirm that the filtering strategy is functioning as expected.

Through the rational application of these mechanisms, developers can achieve precise control over message streams without altering existing business logic, laying a solid foundation for building high-availability, low-cost distributed systems.