Implementing Message Queues with Redis Lists

In modern high-concurrency distributed systems, message queues are critical components for decoupling services, handling traffic spikes, and ensuring data reliability. While mature middleware like RabbitMQ and Kafka exist in the industry, Redis is frequently chosen to build internal messaging systems due to its extreme performance characteristics and simple data structures, particularly in lightweight or high-frequency interaction scenarios. Among these, the Redis List structure has become an ideal choice for implementing producer-consumer models thanks to its native support for bidirectional operations. This article delves into how to efficiently construct a highly available message queue mechanism using Redis Lists.
Understanding the implementation principles of Redis Lists is a prerequisite for building reliable queues. A Redis List is an ordered doubly-linked list structure, with core operations including appending elements at either head or tail positions, removing elements at specific indices, and retrieving the list length. For most message queues based on FIFO (First-In-First-Out) principles, we primarily focus on two commands: LPUSH and RPOP:

LPUSH key value [value ...]: Pushes values to the head of the list. This typically corresponds to a producer delivering messages to the queue.
RPOP key: Removes and returns an element from the tail of the list. In standard FIFO implementations, we often need to combine LRANGE with Lua scripts or atomic operations to simulate "popping from the front" behavior, or directly utilize the doubly-linked list characteristics by treating the head as the dequeue point (requiring logic conversion such as LINDEX -1).

In practical applications, to ensure message processing orderliness, a "tail-in, head-out" strategy is typically adopted. This means producers use LPUSH to add new tasks, while consumers need to read elements from the list's head for processing. To handle empty queue scenarios or batch consumption, we must combine other commands such as LLEN (get length) and BLPOP/BRPOP (blocking pop).

Key Mechanisms for Building Highly Available Message Queues

Relying solely on basic append-and-delete operations is insufficient to meet complex requirements in production environments; mechanisms like idempotency, reliable delivery, and backpressure control must be introduced. Here are the three core elements for implementing a robust Redis List message queue:

1. Atomicity and Order Guarantee

In multi-threaded or distributed environments, non-atomic Read-Modify-Write (RMW) operations easily lead to data reordering and loss. The single-thread execution model of Redis and the atomic nature of Lua scripts guarantee operation integrity. For example, when a consumer needs to "check if the queue is empty" and decide whether to block waiting, these two actions must be encapsulated within an indivisible logical unit:

-- Pseudo-code example: Safely retrieve a message or sleep for retry
local key = "my_queue"
if redis.call('LLEN', key) > 0 then
    return 'OK' -- Queue is not empty, proceed with dequeue logic later
else
    return 'WAITING' -- Queue is empty, trigger blocking wait
end

2. Message Idempotency and Deduplication

During network jitter or consumer failure recovery, the same message might be consumed multiple times. To address this issue, pre-checks must be performed before writing to the queue, or a Redis Set (collection) can be used as a record table for processed messages. When a producer sends new tasks, it first checks if that TaskID already exists in the processed_ids set; if present, it returns success directly and skips the enqueue operation, thereby avoiding side effects caused by duplicate consumption:

-- Lua script example: Idempotency check and enqueuing
local task_id = KEYS[1]
local queue_key = 'task_queue'
local processed_set = 'processed_tasks_' .. tonumber(ARGV[1])

if redis.call('SISMEMBER', processed_set, task_id) == 1 then
    return 0 -- Already processed, skip enqueuing
else
    redis.call('LPUSH', queue_key, task_id)
    redis.call('SADD', processed_set, task_id)
    return 1 -- Successfully queued
end

3. Backpressure Control and Traffic Smoothing

When consumer processing capacity is limited while producers deliver messages too quickly, the queue can expand infinitely leading to memory overflow. In such cases, a "backpressure" mechanism must be introduced to limit the inflow rate. A simple yet effective method is to utilize the BLPOP (blocking pop) command instead of non-blocking LPOP. By setting reasonable timeout values or listening for specific events to trigger consumption logic, data can only be retrieved from the queue when consumers are ready to process it:

-- Simulate consumer pulling message logic
local key = 'my_queue'
local timeout_ms = 100 -- Max wait time of 100ms

if redis.call('BLPOP', key, timeout_ms) then
    local message = table.unpack(redis.call('LINDEX', key, -1)) -- Retrieve latest element (adjust index based on specific FIFO/LIFO requirements)
    return 'MESSAGE_RECEIVED:' .. tostring(message[1])
else
    return 'TIMEOUT' -- Timeout without receiving a message may indicate queue is full or no new tasks exist
end

Best Practices and Performance Optimization Recommendations

In actual deployment, in addition to the logical designs mentioned above, attention must also be paid to storage scale, persistence strategies, and monitoring metrics. First, Redis Lists can impact single-message read performance when data volume becomes large (e.g., exceeding tens of thousands of elements). Therefore, for task queues with long lifecycles, consider periodically moving old data into Stream structures or archiving it in a database. Second, enabling AOF persistence and setting reasonable appendfsync strategies is critical to prevent message loss due to Redis crashes; if extremely high reliability requirements exist (such as financial transactions), combining Sentinel for master-slave switching and automatic failover mechanisms is recommended.

Finally, establish a comprehensive monitoring system. In addition to conventional QPS and latency metrics, it is crucial to observe the change curve of queue length over time and the average duration of "blocking waits." If queue length continues to spike while consumer processing rates remain constant, this indicates a risk of backpressure failure; timely adjustments to producer frequency or scaling up consumer node resources are necessary.

In summary, thanks to its lightweight nature and high performance, Redis Lists can construct message queue solutions meeting the needs of most scenarios at relatively low cost. By combining the atomicity of Lua scripts, the deduplication capabilities of set structures, and traffic control mechanisms provided by blocking commands, we can build an asynchronous task processing system that is both efficient and reliable.