What to Do When You Have Too Much Redis Data

When the amount of data stored in Redis keeps growing and memory usage approaches the configured limit, you’ll start see

When the amount of data stored in Redis keeps growing and memory usage approaches the configured limit, you’ll start seeing symptoms such as slower writes, frequent key evictions, persistence stalls, and even out‑of‑memory (OOM) errors. The root cause is rarely “Redis is too small”; it’s usually a combination of:

  • Full‑dataset workloads – the application really needs every record in memory (e.g., real‑time leaderboards).
  • Stale data – old entries that are never accessed but lack an expiration time.
  • Giant keys – a small number of keys whose values occupy megabytes each (large serialized objects, big JSON blobs).
  • Uncontrolled growth – data keeps being added linearly with no cleanup mechanism.

Identifying which of these patterns applies to your environment is the first step toward a sustainable solution. Simply adding more RAM only postpones the inevitable.

Core Strategies for Tackling Excess Data

All practical approaches can be grouped into four high‑level buckets:

  1. Lifecycle Management – let data die when it’s no longer needed.
  2. Hot‑Cold Separation – keep only the most frequently accessed items in Redis, push the rest to a slower store.
  3. Horizontal Partitioning – split the dataset across multiple nodes when a single instance can’t hold the hot set.
  4. Per‑Item Slimming – redesign data structures and encoding to reduce the memory footprint of each entry.

Below we dive into each bucket, show concrete commands, and discuss when each technique shines.

1. Lifecycle Management: Expiration & Eviction

Set TTLs Wherever Possible

For any key that has a natural lifetime (session tokens, temporary caches, rate‑limit counters), assign a time‑to‑live (TTL) at creation time:

# Store a user session that expires after one hour
SET session:1001 "uid=1001" EX 3600

# Cache a rendered homepage for ten minutes
EXPIRE cache:page:home 600

TTL‑based cleanup is the cheapest way to keep memory usage in check because Redis automatically removes expired keys without any extra code.

Configure Memory Limits and Eviction Policies

Even with TTLs, you should define a hard memory ceiling and tell Redis how to behave when that ceiling is reached:

CONFIG SET maxmemory 4gb
CONFIG SET maxmemory-policy allkeys-lru

Common eviction policies and their typical use‑cases:

Policy When to Use
volatile‑ttl / volatile‑lr You have a mix of expiring cache entries and permanent data. Only keys with a TTL are candidates for eviction.
allkeys‑lru / allkeys‑lfu Pure cache scenario where any key can be evicted based on recent or frequent use.
noeviction Critical data must never be dropped; writes are rejected once memory is full.

Choosing the right policy depends on how much data you can afford to lose and whether you can tolerate write failures.

2. Hot‑Cold Separation: Tiered Storage

Redis excels at sub‑millisecond reads/writes, not at holding petabytes of historical data. A typical tiered architecture looks like this:

Layer Purpose Example Technologies
Hot (Redis) Data accessed within the last few days or hours (e.g., active sessions, current day counters). Redis (standalone or cluster)
Warm (SQL / NoSQL) Recent but less frequently accessed records (last weeks or months). MySQL, PostgreSQL, MongoDB
Cold (Object Store) Immutable, large blobs (images, logs, archived JSON). Amazon S3, Azure Blob Storage, HDFS

Implementation tips:

  • Asynchronous write‑behind – push writes to the warm store in the background, ensuring Redis stays fast.
  • Cache‑aside pattern – on a miss, fetch from the warm store, populate Redis, and set a short TTL.
  • Reference‑only storage – store only an identifier or URL in Redis, keep the full payload elsewhere. This reduces per‑key memory dramatically.

3. Horizontal Partitioning: Sharding & Clustering

When the hot dataset itself exceeds the memory capacity of a single node, you need to distribute it.

Redis Cluster (Official Solution)

  • Splits the keyspace into 16,384 hash slots.
  • Each slot is assigned to a specific node; adding nodes automatically rebalances slots.
  • Provides linear scalability: double the nodes → roughly double the capacity.
# Example of creating a 3‑node cluster (simplified)
redis-cli --cluster create 10.0.0.1:6379 10.0.0.2:6379 10.0.0.3:6379 --cluster-replicas 1

Client‑Side Sharding / Proxy Layers

If you cannot migrate existing code to the Cluster API, a proxy (e.g., Twemproxy, Codis) or a custom client‑side hash can split keys across multiple standalone instances. This approach is more invasive to maintain but avoids a full code rewrite.

Before you shard, ask yourself:

  • Do all keys really need to be in Redis, or can some be moved to a lower tier?
  • Will the added network hop affect latency requirements?
  • How will you handle cross‑shard transactions (if any)?

4. Per‑Item Slimming: Reducing the Size of Each Entry

Even without changing the overall architecture, you can often reclaim a large chunk of memory by optimizing how data is stored.

Technique How It Helps
Use hashes instead of many small keys A single hash can store dozens of related fields, reducing overhead per key.
Bitmaps / HyperLogLog Represent boolean flags or approximate cardinalities with a few bytes instead of strings.
Compress large values Store gzipped JSON or protobuf blobs; decompress only when needed.
Store IDs, not full objects Keep the primary key in Redis and fetch the full record from a relational store on demand.
Split oversized values Break a 5 MB JSON document into smaller chunks (e.g., per‑field hashes) to avoid a single massive key.

Profiling tools such as MEMORY USAGE <key> and Redis‑Insight can pinpoint the biggest memory consumers, guiding where to apply these optimizations.

Strategy Comparison

Strategy Ideal Scenario Pros Cons
Expiration & Eviction Cache‑like data with a natural TTL Low implementation cost, immediate impact Not suitable for data that must be retained forever
Hot‑Cold Separation Clear access‑pattern skew, large historical archive Balances performance and storage cost Requires extra pipelines and consistency handling
Sharding / Cluster Hot set exceeds a single node’s RAM Near‑unlimited capacity, high availability Increased operational complexity, cross‑shard coordination
Per‑Item Slimming Presence of large keys or inefficient structures No architectural changes, immediate memory savings Benefit depends on data shape; may need code changes

  1. Audit & Harden Lifecycle Controls

    • Verify that every temporary key has a TTL.
    • Choose an eviction policy that matches your data mix and set a realistic maxmemory.

  2. Eliminate Giant Keys

    • Run MEMORY USAGE across the keyspace.
    • Refactor large values: compress, split, or replace with references.

  3. Introduce Hot‑Cold Tiering

    • Identify a clear “cold” threshold (e.g., data older than 7 days).
    • Build an asynchronous pipeline that moves those records to a durable store.

  4. Scale Out Only When Needed

    • If after steps 1‑3 the hot dataset still outgrows a single node, migrate to Redis Cluster or a proxy‑based sharding solution.

In practice, the first three steps resolve the majority of memory‑pressure incidents. Sharding should be treated as a last resort, because it brings additional operational overhead.

Quick Checklist

  • All short‑lived keys have an appropriate TTL.
  • maxmemory and maxmemory-policy are set and monitored.
  • No key exceeds a predefined “large‑key” size (e.g., 1 MB).
  • Hot data retention window is documented and enforced.
  • A background job periodically archives cold data to a persistent store.
  • Capacity planning includes projected growth and the point at which a cluster would be required.

By following this systematic approach—diagnose first, then tier, then slim, and finally scale—you can keep Redis performant and reliable even as your data continues to grow.