Request Coalescing: When a Million Requests Want the Same Data

Let me start with a situation we have all seen from the outside. Some post blows up on Reddit or a channel goes crazy on Discord, and suddenly a huge number of people are opening the exact same page at the exact same time. They are all asking your backend for the same piece of data.

The data itself is basically static for that moment. It is the same post, the same numbers, the same content for everyone. So the interesting question is, do we really need to hit the database once for every single one of these requests? Clearly not. And once you start pulling on that thread, it leads to a nice little chain of optimizations that ends in a pub/sub trick. Let me walk through the whole thinking.

---

Step 1: put Redis in front, the obvious win

The first thing anyone does is add a cache. Put Redis in front of the database and use the classic cache aside pattern.

async function getPost(id) {
  const key = `post:${id}`;

  // 1. try the cache
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  // 2. miss, go to the database
  const post = await db.query("SELECT * FROM posts WHERE id = ?", [id]);

  // 3. put it in the cache for next time
  await redis.set(key, JSON.stringify(post), "EX", 60);

  return post;
}

This already solves most of the problem. The first request fills the cache, and after that everyone reads from Redis. And Redis is comfortable with this kind of load, a single node handles well over a hundred thousand ops per second, and with Redis Cluster you are into the millions of reads per second range without breaking a sweat. So for pure reads of hot data, Redis absorbs the storm and the database barely notices.

So far so good. If this was the whole story, there would be no blog post. The problem is hiding in one specific moment.

---

Step 2: the moment it all breaks (cache stampede)

Look closely at what happens when the key is not in the cache. There are two very normal times this is true:

• The very first time anyone asks for this post (cold cache).
• The instant the key expires. Our EX 60 means every 60 seconds the key vanishes for a moment.

Now replay the viral scenario at exactly that instant. A million concurrent requests come in. They all run redis.get, they all miss (because the key is not there yet), and so they all fall through to the database at the same time. Then they all try to write the result back into Redis.

   million concurrent requests, key just expired
   │  │  │  │  │  │  │  │  │  │  │  │  │  │  │
   ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼
        all miss the cache at the same time
                     │
                     ▼
        all hammer the DATABASE at once   ← the stampede

This is called a cache stampede (also thundering herd, or dogpile). The cache was supposed to protect the database, but at the exact moment of a miss it protects nothing, and the database gets hit by thousands of identical, and often expensive, queries all at once. If that query is heavy (a big join, an aggregation), your database can fall over from a load it was never actually required to do, because remember, the answer is the same for everyone. We only needed to run it once.

And it is worse than just reads. In many designs that “populate the cache” step is not a plain set, it is a write into the database too, or an insert of a computed/derived row. Writes and inserts are much heavier than reads for a database, and now you have thousands of concurrent inserts for the same data racing each other. That is a lot of wasted, duplicated, expensive work.

So the real problem statement is, how do we make sure that when many requests want the same missing data, only one of them actually does the work, and everyone else just waits for that one result?

---

Step 3: single flight, only one request does the work

The idea has a nice name, single flight. For a given key, only allow one in flight computation at a time. Everyone else who wants the same key while it is being computed should not start their own, they should attach to the one already running.

Within a single Node.js process, you get this almost for free with a map of in flight promises:

const inFlight = new Map();

async function getPostCoalesced(id) {
  const key = `post:${id}`;

  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  // is someone already fetching this key right now?
  if (inFlight.has(key)) {
    return inFlight.get(key); // attach to the existing work, do not start your own
  }

  const promise = (async () => {
    const post = await db.query("SELECT * FROM posts WHERE id = ?", [id]);
    await redis.set(key, JSON.stringify(post), "EX", 60);
    return post;
  })();

  inFlight.set(key, promise);
  try {
    return await promise;
  } finally {
    inFlight.delete(key); // clear it once done, so the next miss can refetch
  }
}

Now if a thousand requests hit the same process during that miss window, the first one starts the database query and the other 999 simply await the same promise. One database call, one cache write, a thousand happy responses.

This is exactly what Go’s singleflight package does, and it is a genuinely underused pattern.

---

Step 4: but we have many pods, not one process

Here is the catch. In real life your app is not one process, it is 40 pods behind a load balancer. That in memory Map only coalesces requests inside one pod. Across 40 pods you still get up to 40 concurrent database calls at the miss instant, one per pod. Much better than a million, but still not the “exactly once” we want, and if the query is heavy even 40 at once can hurt.

To coordinate across all pods, we need a shared point of truth, and we already have one sitting right there, Redis.

The trick is a lock. The first request to arrive grabs a short lock in Redis, and only the holder of that lock is allowed to go to the database. SET key value NX EX ttl is perfect for this, because NX means “only set if it does not already exist”, so exactly one request across the whole fleet wins it.

async function getPostGlobal(id) {
  const key = `post:${id}`;
  const lockKey = `lock:${key}`;

  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  // try to become THE one who does the work (NX = only if not set)
  const gotLock = await redis.set(lockKey, "1", "NX", "EX", 10);

  if (gotLock) {
    // I am the chosen one. Do the heavy work exactly once.
    const post = await db.query("SELECT * FROM posts WHERE id = ?", [id]);
    await redis.set(key, JSON.stringify(post), "EX", 60);
    await redis.del(lockKey);
    return post;
  }

  // someone else is already doing it. I just have to wait for the result.
  return waitForResult(key);
}

Now exactly one request in the whole cluster touches the database. Everyone else falls into waitForResult. The only question left is, how do the waiters get the answer?

---

Step 5: polling vs pub/sub for the waiters

The simplest way to wait is to poll. Every waiting request checks Redis every so often to see if the value has appeared yet.

async function waitForResult(key) {
  for (let i = 0; i < 50; i++) {
    const cached = await redis.get(key);
    if (cached) return JSON.parse(cached);
    await sleep(50); // check again in 50ms
  }
  // fallback: still nothing, do the work ourselves so we never hang forever
}

You can even make this richer by storing a small status value, so waiters can see the stage the work is in (“pending”, “fetching”, “ready”) instead of guessing. That was actually my first instinct too, push the request state into Redis and let everyone poll it.

Polling works, but it has two annoyances. You are still making a Redis call on every poll from every waiting request, so a million waiters polling every 50ms is its own little load. And there is wasted latency, if the work finishes 1ms after your last check, you still sit idle until your next 50ms tick.

This is where pub/sub is much nicer. Instead of everyone repeatedly asking “is it ready yet?”, the waiters subscribe to a channel for that key and go quiet. When the one worker finishes, it publishes a message on that channel. Every waiting request wakes up at that exact moment and reads the now filled value.

// the worker, after it fills the cache:
await redis.set(key, JSON.stringify(post), "EX", 60);
await redis.publish(`ready:${key}`, "done"); // wake everyone up
await redis.del(lockKey);

// a waiter:
async function waitForResult(key) {
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached); // maybe it is already ready

  await subscribeOnce(`ready:${key}`); // sleep until the worker signals
  const value = await redis.get(key);
  return JSON.parse(value);
}

                    ┌─────────────────────┐
   worker (1 req) ─►│  DB query + cache    │─► publish "ready:post:42"
                    └─────────────────────┘            │
                                                        ▼
   waiter  waiter  waiter  waiter  ... all subscribed, all wake up together
      └───────┴───────┴───────┴───── then read the value from Redis, once

No busy polling, no wasted Redis calls, and the waiters get the result the instant it is ready.

---

Step 6: the tradeoff, and why it is fine

Now the honest part. Have we actually made anything faster for the user? Not really. The waiting requests still have to wait for the first request to finish its database query. If that query takes 800ms, everyone waits roughly 800ms.

But here is the thing, and this is the whole point. That 800ms was going to be paid anyway. The data genuinely takes 800ms to produce. In the naive version, every one of the million requests paid its own 800ms and piled 800ms of load onto the database a million times over. In the coalesced version, everyone pays the same single 800ms, and the database does the work once.

So we did not remove the wait, we removed the duplication. One expensive operation instead of a million, and the user experience is identical, they were going to wait that long either way. That is a fantastic trade, because the cost we cut (database meltdown) is huge and the cost we kept (a wait that was unavoidable) is something we could not have avoided anyway.

---

The sharp edges (do not skip these)

A few things that will bite you if you ship the simple version:

• The worker can die holding the lock. That is why the lock has a TTL (EX 10). If the holder crashes, the lock expires and another request can take over. Without a TTL, one crash locks that key forever.
• Waiters need a timeout and a fallback. Never wait forever for a signal that might never come (the publish could be missed if you subscribe a hair too late). If the wait times out, fall back to doing the work yourself. Correctness first, coordination second.
• There is a tiny race between subscribing and the publish. Always re-check the cache right after subscribing, in case the value landed in the gap. The snippet above does this.
• Consider stale-while-revalidate. An even smoother pattern is to serve the slightly old value while one background request refreshes it. Then nobody waits at all, they just get data that is a few seconds stale. Great when a little staleness is acceptable, which for a viral read-heavy page it usually is.
• Add jitter to your TTLs. If a lot of keys share the exact same expiry, they all stampede at the same second. A little randomness in the TTL spreads the misses out.

---

Wrapping up

The chain of reasoning is the nice part here, so let me lay it out one more time:

1. A million requests want the same data, so put Redis in front and most of them are served from cache.
2. But at a cold start or the moment the key expires, they all miss together and stampede the database, and duplicated writes and inserts are even worse than reads.
3. So use single flight, only one request does the real work. In one process a promise map is enough.
4. Across many pods, coordinate with a Redis lock (SET NX EX) so exactly one request in the whole cluster does the work.
5. Let the other requests wait, and use pub/sub instead of polling so they wake up the instant the result is ready.
6. The wait was unavoidable anyway, so the real win is turning a million expensive operations into one.

None of this is exotic. It is just Redis being used as a coordination point and not only as a cache, which is a theme worth internalizing. If you want, I have a separate deep dive on how Redis actually works on the inside, single thread, expiry, pub/sub and all, which pairs well with this one.

If you have used a different approach for cache stampedes, tell me in the comments, I would like to hear it.