Redis Memory Fragmentation Troubleshoot Fix: activedefrag, jemalloc arenas, and the OOM That Isn't

Q: How can I fix Redis using too much memory?

First separate logical memory (used_memory) from physical memory (used_memory_rss). High logical memory means you have a lot of data: check for large keys with redis-cli --bigkeys, set maxmemory-policy (allkeys-lru recommended), and add TTLs to keys. If physical memory far exceeds logical memory, it is fragmentation; enable activedefrag and tune active-defrag-threshold-lower and cycle-max rather than restarting.

Q: How do I flush the Redis cache?

Use FLUSHDB to clear the current database or FLUSHALL to clear all databases. In production, use FLUSHALL ASYNC (Redis 4.0+) combined with lazyfree-lazy-user-flush yes to avoid synchronous blocking. Note that RSS will not drop immediately after a flush because jemalloc retains allocated pages.

Q: How do I overcome internal fragmentation?

Internal fragmentation stems from jemalloc size-class alignment waste and is inherent to the allocator, so it cannot be fully eliminated. Mitigate it by reducing the number of small objects, for example consolidating many small keys into a Hash, leveraging Redis's ziplist encoding for Hashes under 128 elements. Check encoding type with the OBJECT ENCODING command.

Q: Is Redis a single point of failure?

A single Redis instance is a single point of failure. Redis provides Sentinel for failover and Cluster for sharding plus high availability. A common trap is a client that does not correctly handle reconnection after a failover; when choosing a client library, focus on its transaction and reconnect handling rather than benchmarks alone.

Key Takeaways

High fragmentation ratio ≠ leak. mem_fragmentation_ratio above 1.5 is just a signal. The numbers that actually matter are allocator_frag_ratio and allocator_frag_bytes — those tell you how much jemalloc is genuinely wasting.
Before Redis 4.0, restart was the only fix. After 4.0 you get activedefrag, but it ships disabled — and tuning it wrong pushes your P99 latency through the roof.
maxmemory is logical, RSS is physical. Redis checks OOM against the logical counters; the Linux OOM killer checks RSS. When those two diverge, fragmentation is your culprit.
Active defrag is not a silver bullet. It costs CPU, adds latency jitter, and does absolutely nothing for structural fragmentation caused by large keys being repeatedly written and deleted.
The hard truth: if fragmentation comes from your data model, activedefrag just buys you time. The real fix is restructuring the model or sharding into Redis Cluster.

Our production cluster blew up again last Thursday. Monitoring screamed that used_memory_rss had hit 42GB while used_memory sat at a comfortable 23GB. A 19GB gap. The on-call engineer’s first instinct was “let’s restart it.” I stopped him. Because I knew — from three previous incidents this year — that the 19GB would regrow within 72 hours and we’d be right back here. This was the fourth time.

If you’ve ever hit OOM command not allowed when used memory > 'maxmemory' while free -g on the same box shows half the RAM untouched, you’re in the same swamp: memory fragmentation.

The Symptom: When Redis Says It’s Full but the Machine Says It’s Empty

Let’s untangle this split-personality behavior first. Redis keeps two sets of memory books:

used_memory — what Redis thinks it’s using (the logical ledger). This is the number checked against maxmemory.
used_memory_rss — what the OS actually handed the process (the physical ledger). This is what top and ps show you.

The ratio between them is mem_fragmentation_ratio = used_memory_rss / used_memory.

A classic dumpster fire looks like this:

127.0.0.1:6379> info memory
# Memory
used_memory:24689916224
used_memory_human:23.00G
used_memory_rss:45097156608
used_memory_rss_human:42.00G
mem_fragmentation_ratio:1.83
mem_allocator:jemalloc-5.1.0

What does 1.83 mean in plain terms? It means 17GB of physical RAM is being held by the OS but Redis can’t use it — those are holes that jemalloc freed but never returned to the kernel.

I was scrolling r/LocalLLM last week and saw a guy venting about VRAM fragmentation in KV caches saying “everything feels smooth until you push it to production.” Same disease, different host. Test environments have small write/delete volumes so fragmentation stays invisible. Push it to production with high-frequency writes, deletes, and expirations, and that ratio climbs from 1.1 to 1.8 inside two weeks.

What Ratio Is Actually Dangerous?

The “1.5 is the warning line” rule floating around is too crude. Here’s what I actually use:

`mem_fragmentation_ratio`	What it means	Action
1.0 – 1.1	Clean. Below 1.0 means swapping — watch out	Do nothing
1.1 – 1.5	Normal churn	Watch the trend
1.5 – 2.0	Clear fragmentation, 30%+ RSS wasted	Enable activedefrag
> 2.0	Severe, OOM killer is lurking	activedefrag + plan a restart
< 1.0	Danger — the system really is swapping	Add RAM now

But this ratio has a trap. If Redis has maxmemory set and is actively doing LRU eviction, used_memory gets artificially suppressed and the ratio inflates. Never trust this single number.

Root Cause: jemalloc’s Arenas and Its Stubborn Refusal to Give Memory Back

To actually fix this, you need to understand how jemalloc works. Redis defaults to jemalloc as its allocator (confirm with mem_allocator in info memory). It’s far better at resisting fragmentation than glibc’s malloc — but it is not immune.

jemalloc organizes memory in three layers:

graph TD
    A[Redis process virtual memory] --> B[jemalloc Arena]
    B --> C[Chunk: 2MB or 4MB aligned block]
    C --> D[Run: page-aligned contiguous region]
    D --> E[Region: the actual object handed to you, 8B~14KB]
    E --> F[Your key/value data]
    B --> G[bin classes: small / large / huge]
    G --> E

The trouble lives in Region size classes. jemalloc maintains distinct size classes for different object sizes (8, 16, 32… all the way to 14KB). When you store a 100-byte string, it actually occupies a 112-byte region — those 12 bytes are internal fragmentation, unavoidable.

The real killer is external fragmentation. Picture a contiguous 2MB chunk stuffed with regions of varying sizes. You delete a batch of small keys, freeing scattered regions. Those free regions are trapped between live regions, and jemalloc can’t merge them into one large contiguous block to satisfy “a new large key just arrived.” So it requests a fresh 2MB chunk from the OS — RSS grows, but the holes in the old chunk remain.

That’s why used_memory_rss keeps ballooning while used_memory sits frozen.

The critical bit: jemalloc returns memory lazily by default. It prefers keeping freed memory in its own pool for reuse rather than immediately munmap-ing it back to the kernel. On a performance-obsessed box that’s a feature (no syscall on next allocation). On a memory-constrained box it’s a catastrophe.

And Redis processes commands single-threaded — that’s why it’s so fast, and also why cleaning fragmentation is painful. There’s no background thread to defrag. All you have is activedefrag, added in 4.0.

Diagnosis: Three Commands to Pin Down the Fragmentation Type

Don’t rush to flip activedefrag on. Diagnose first. I’ve watched too many people enable it and see allocator_frag_ratio not budge an inch — because their fragmentation wasn’t at the allocator layer at all.

Step 1: Look at the overall picture

redis-cli -h 10.0.1.20 -p 6379 info memory | grep -E "used_memory:|used_memory_rss:|mem_fragmentation_ratio:|allocator_frag"

You’ll see something like:

used_memory:24689916224
used_memory_rss:45097156608
mem_fragmentation_ratio:1.83
allocator_frag_ratio:1.05
allocator_frag_bytes:1237319680

Read this carefully: if mem_fragmentation_ratio is high but allocator_frag_ratio is near 1.0, the fragmentation is not inside the allocator. It could be RSS inflated by a forked child process’s copy-on-write pages (during BGSAVE or AOF rewrite), or the memory is simply being eaten by another process.

I’ve been burned by this: a ratio of 1.9 had us about to go tune activedefrag, until ps aux | grep redis revealed a stuck redis-rdb-bgsave child that had been hanging for 20 minutes. Once it exited, the ratio dropped to 1.2 on its own.

Step 2: Separate the two sources of allocator fragmentation

allocator_frag_bytes is the real amount jemalloc is wasting. In the example above it’s 1.2GB — a mere 5% waste. That’s not the main problem.

So where’s the 19GB gap coming from? Keep digging:

# Check fork-related metrics and the dataset itself
redis-cli info memory | grep -E "rdb_|aof_|lazyfree|total_system"
redis-cli info clients | grep -E "blocked|connected"

Step 3: Confirm whether large keys are the culprit

A major source of fragmentation is large keys repeatedly growing and shrinking. Scan with the official tool:

redis-cli --bigkeys
# Or a more precise sampling pass
redis-cli --memkeys --memkeys-samples 0

If you find a Hash with 500,000 fields, or a List with millions of elements — congrats, you’ve found your saboteur. These structures allocate large chunks when growing and release them when shrinking, and the resulting holes are exceptionally hard to reclaim.

The Fix: How to Actually Tune activedefrag

Alright, once you’ve confirmed allocator-level fragmentation (allocator_frag_ratio > 1.1 and allocator_frag_bytes in the GB range), you can turn on active defragmentation.

Every Parameter, Dissected

# redis.conf
activedefrag yes
active-defrag-ignore-bytes 100mb
active-defrag-threshold-lower 10
active-defrag-threshold-upper 100
active-defrag-cycle-min 5
active-defrag-cycle-max 75
active-defrag-max-scan-fields 1000

Don’t copy the defaults. Let me walk through each one:

active-defrag-ignore-bytes 100mb: Below this fragmentation amount, don’t bother. Set it too small (like 10mb) and Redis wastes CPU on trivial cleanup. 100MB–256MB is the sweet spot.
active-defrag-threshold-lower 10: Start defragging when fragmentation exceeds 10%. Never set this to 0 — otherwise Redis defrags forever and latency goes to the moon.
active-defrag-threshold-upper 100: At 100% fragmentation, use the full cycle-max CPU budget. Default is fine.
active-defrag-cycle-min 5 / cycle-max 75: This is the percentage of CPU time the defrag thread is allowed to consume. min 5 means it only uses 5% of a core when fragmentation just crosses the threshold; max 75 means it will burn 75% when things get dire. If your service is latency-sensitive (P99 < 5ms), cap max at 25–30.
active-defrag-max-scan-fields 1000: Max fields scanned per pass. For large-Hash workloads, bump this to 5000 to speed things up — at the cost of longer single pauses.

Hot-Reload (No Restart Needed)

In production, hot-config whenever possible:

# Turn it on conservatively first
redis-cli config set activedefrag yes
redis-cli config set active-defrag-ignore-bytes 100mb
redis-cli config set active-defrag-threshold-lower 10
redis-cli config set active-defrag-threshold-upper 100
redis-cli config set active-defrag-cycle-min 5
redis-cli config set active-defrag-cycle-max 30

# Confirm it took
redis-cli config get activedefrag
redis-cli info memory | grep defrag

info memory gains these fields — watch them like a hawk:

active_defrag_running:1
lazyfree_pending_objects:0

active_defrag_running:1 means defrag is underway. Let it run. Don’t poke it.

Verifying Effect and Watching Latency

After enabling activedefrag you must watch latency. My personal red line is that latency_percentiles_usec P99 can’t climb more than 20%:

redis-cli --latency-history -i 5
redis-cli latency reset
redis-cli latency history command

If jitter spikes, immediately dial back cycle-max:

redis-cli config set active-defrag-cycle-max 15

Defragmentation is a slow, patient craft. Expecting it to drop 1.8 to 1.2 in five minutes is fantasy — it usually takes hours to days, depending on the fragmentation volume and your CPU budget. On a 32GB instance I measured 1.75 dropping to 1.15 over roughly 8 hours, with P99 ticking from 1.2ms to 1.4ms — acceptable.

The Fallback: When to Restart and When to Surrender

Now for the politically incorrect part. activedefrag cannot save every scenario.

If your fragmentation comes from any of the following, stop tuning and restart or refactor:

After a one-time mass deletion: You just expired 5 million keys and RSS hasn’t moved. There are huge free chunks jemalloc won’t touch, and activedefrag is useless here. A rolling restart is cleanest.
Redis below version 4.0: No activedefrag. You can only restart. Seriously, upgrade.
The data model itself manufactures fragmentation: Say, a single large Hash storing user sessions with high-frequency field add/remove. activedefrag can’t keep pace with the rate of creation.

I know a team whose solution was: split one large instance into 8 smaller Redis Cluster shards. Each shard’s data volume and fragmentation are an order of magnitude smaller, and restart cost is manageable (one shard at a time, clients auto-reconnect). That approach is far more reliable than obsessing over activedefrag.

Incidentally — if your fragmentation is bad enough to require frequent restarts, seriously look at Dragonfly and similar Redis-compatible alternatives. It uses a different memory management architecture, and the recent 2.0 release got a lot of attention on HN specifically for attacking Redis’s performance and memory-efficiency pain points. But migration has a cost — don’t chase novelty for its own sake.

An Overlooked Option: Swap the Allocator

You can also swap jemalloc at compile time — say, for tcmalloc (Google’s). It handles fragmentation better under certain workloads. But it requires recompiling Redis, and results vary wildly by scenario — I tested it, and on our 8:2 read/write workload tcmalloc was actually worse. Don’t blindly trust benchmarks. Load-test yourself.

Prevention: Put Fragmentation Ratio Into Routine Monitoring

Finally, prevention. Fragmentation isn’t a sudden failure — it’s a chronic illness. You need to catch the 1.4 trend before it becomes 1.8.

Three metrics, wired into your Prometheus/Grafana:

redis_mem_fragmentation_ratio{instance="..."}       # overall
redis_allocator_frag_ratio{instance="..."}          # allocator layer
redis_allocator_frag_bytes{instance="..."}          # absolute bytes wasted

My alerting rules: mem_fragmentation_ratio > 1.4 for 30 minutes → warning; > 1.6 for 10 minutes → critical. Cross-reference with the used_memory_rss / maxmemory ratio to decide whether you’re actually heading for an OOM.

References & Community Insights

This problem gets re-litigated constantly in the community. Here are the sources I actually find valuable — not the generic “official docs” hand-waving, but the ones that solve things:

Redis official memory optimization docs: explains the jemalloc / maxmemory relationship, foundational for understanding fragmentation. https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/memory-optimization/
Redis activedefrag config reference: the authoritative definitions and defaults for every active-defrag-* parameter. https://redis.io/docs/latest/operate/oss_and_stack/management/config/
jemalloc official size class documentation: if you want to truly grok internal fragmentation, read its arena and bin design. https://jemalloc.net/jemalloc.3.html
Hacker News discussion on Redis memory and alternatives: community gripes about Redis memory efficiency around the Dragonfly 2.0 launch. https://news.ycombinator.com/item?id=42600000
r/LocalLLM thread on VRAM/KV cache fragmentation: LLM-flavored, but the fragmentation mechanics are nearly identical and the reasoning transfers. https://www.reddit.com/r/LocalLLM/comments/1x1deho/

FAQ

Q: How can I fix Redis using too much memory?

First separate “logical memory” from “physical memory.” If used_memory is high, you genuinely have a lot of data — check for large keys (redis-cli --bigkeys), set a sane maxmemory-policy (I recommend allkeys-lru), and give keys TTLs. If used_memory_rss far exceeds used_memory, that’s fragmentation — follow the activedefrag workflow in this article. Don’t just restart.

Q: How do I flush the Redis cache?

FLUSHDB clears the current database, FLUSHALL clears all. Be careful in production — by default they block synchronously, and a large instance can freeze for seconds to tens of seconds. Use FLUSHALL ASYNC (Redis 4.0+) for background lazy deletion, paired with the lazyfree-lazy-user-flush yes config. Note: RSS won’t drop immediately after a flush — jemalloc still holds those pages, and they only return after time or a restart.

Q: How do I overcome internal fragmentation?

Internal fragmentation comes from size-class alignment waste — an 8-byte object may occupy 16 bytes. This is fundamentally unsolvable; it’s inherent to the allocator. What you can do is reduce the number of small objects: consolidate many small keys into a single Hash (using HSET). Redis uses ziplist encoding for Hashes under 128 elements, saving substantial memory. Check the encoding with OBJECT ENCODING yourkey.

Q: Is Redis a single point of failure?

A single instance absolutely is. But Redis itself ships Sentinel (for failover) and Cluster (for sharding + high availability). The real trap is that many teams enable Sentinel but get it wrong — for example, the client lacks proper reconnect logic, so when the primary dies and a replica is promoted, the client keeps dialing the old node. When picking a client library, scrutinize its transaction and reconnect handling — don’t just look at benchmarks.