The Thundering Herd Problem is not simply “too much traffic.”
It is too many actors becoming active at nearly the same time against the same constrained resource.
The key design goal is therefore:
Don’t let synchronized demand become synchronized work.
1. Start With the Mental Model
Imagine a stadium with 100,000 people.
Normal traffic: people enter through 20 gates over 30 minutes.
Thundering herd: all 100,000 people are told “Gate 7 opens at exactly 10:00:00.”
The problem is not only the number of people.
The problem is that everyone arrives together and competes for the same resource.
In distributed systems, the “gate” might be:
a database connection pool
Redis
an API endpoint
a lock
an inventory record
a downstream service
a cache-miss path
a scheduler
a message queue
a thread pool
Interview one-liner
Thundering herd = synchronized demand overwhelming a shared bottleneck.
2. What Exactly Is the Thundering Herd Problem?
A thundering herd occurs when many independent clients, processes, threads, or workers are triggered by the same event and simultaneously compete for a resource.
Typical triggers:
flowchart LR
T["Single Trigger"] --> H["Many Actors Wake Up"]
H --> R["Same Resource"]
R --> C["Contention"]
C --> O["Overload"]
O --> F["Failures / Latency"]
F --> RT["Retries"]
RT --> H
The dangerous part is the feedback loop.
A simple version:
100,000 requests ↓Database overloaded ↓Requests become slow ↓Clients timeout ↓Clients retry ↓More requests ↓Database becomes even more overloaded
The important insight
A system can collapse even when the original traffic volume was technically within its normal capacity.
The synchronization + amplification is what makes the event dangerous.
3. Normal Traffic Spike vs Thundering Herd
Characteristic
Normal Spike
Thundering Herd
Traffic shape
Gradual
Sudden / synchronized
Example
Marketing campaign
Flash sale at 7:00 PM
Work distribution
Spread out
Concentrated
Autoscaling usefulness
Usually good
May react too late
Main problem
Capacity
Synchronization + capacity
Typical mitigation
Scaling
Spread, coalesce, queue, throttle, cache
Is every traffic spike a thundering herd?
No.
A huge but smoothly increasing workload can be handled by autoscaling.
A smaller workload can cause a herd if thousands of actors become active at the same instant.
flowchart TD
U["500K Users"] --> LB["Load Balancer"]
LB --> API["Product / Order API"]
API --> INV["Inventory Service"]
API --> DB["Database"]
API --> PAY["Payment Service"]
INV --> DB
PAY --> DB
DB --> X["Connection Pool Exhausted"]
X --> Y["Timeouts"]
Y --> Z["Retries"]
Z --> API
The real bottleneck may not be the API servers.
It may be:
inventory row locking
database connection pool
payment provider
Redis
a downstream service
Scaling only the API tier may not solve this.
If 100 API instances all hammer one database, you have simply created 100 workers competing for the same bottleneck.
4.2 Viral Content
Imagine a post normally receives:
100 requests/sec
Suddenly a celebrity shares it:
100 → 100,000 requests/sec
If every request performs:
API → DB → expensive computation
then the database becomes the bottleneck.
Better
Client ↓CDN / Cache ↓API ↓Cache ↓DB only on miss
But this introduces an important question:
"If I cache the response, what should the cache key be?"
Do not use the entire HTTP request blindly as the cache key.
Define the identity of the resource whose response you are caching.
Examples:
GET /products/123key = product:123GET /posts/456key = post:456GET /users/789/profilekey = user-profile:789
If the response varies by tenant, locale, permissions, or another meaningful dimension, include that dimension:
Never cache a response under a shared key if authorization or sensitive data can make the response user-specific.
4.3 Cron Jobs
Bad:
10,000 servers ↓00:00:00 ↓All execute the same scheduled job ↓DB / API overload
This often happens because everyone uses the same cron expression:
0 * * * *
Better: jitter
Instead of:
Every worker → exactly 00:00
use:
Worker A → 00:00:07Worker B → 00:00:32Worker C → 00:01:14Worker D → 00:02:03
The total work is approximately the same.
The arrival pattern changes.
5. The Most Important Connection: Cache Stampede
Is cache stampede caused by the thundering herd problem?
Yes — a cache stampede is best understood as a specific cache-related form of the thundering herd problem.
More precisely:
Thundering herd is the broader phenomenon.
Cache stampede happens when many requests simultaneously discover that the same cached object is unavailable/stale and independently perform the expensive regeneration work.
Example
Suppose:
Cache TTL = 60 secondsTraffic = 20,000 req/sec
At 60 seconds:
Cache expires ↓20,000 requests miss ↓20,000 DB queries ↓DB overloaded
Without protection:
sequenceDiagram
participant C as Clients
participant API as API
participant Cache as Cache
participant DB as Database
C->>API: Request
API->>Cache: GET product:123
Cache-->>API: MISS
C->>API: Request
API->>Cache: GET product:123
Cache-->>API: MISS
C->>API: Request
API->>Cache: GET product:123
Cache-->>API: MISS
Note over API,DB: Thousands of requests independently recompute
API->>DB: Query product:123
API->>DB: Query product:123
API->>DB: Query product:123
The cache reduced the normal workload dramatically — but the cache miss created a synchronized burst.
6. Cache Stampede Mitigation
There are several increasingly sophisticated approaches.
6.1 Request Coalescing / Single Flight
The idea:
One request performs the expensive work; equivalent requests share its result.
flowchart TD
R["1000 Requests"] --> C["Cache"]
C --> M["MISS"]
M --> L["Single Flight / Lock"]
L --> W["One Request Computes"]
W --> DB["Database"]
DB --> V["Result"]
V --> L
L --> A["1000 Requests Receive Result"]
Instead of:
1000 requests → 1000 DB queries
you get:
1000 requests → 1 DB query
This is also called:
request coalescing
single-flight
duplicate suppression
cache locking (depending on implementation)
6.2 Distributed Lock
For multiple application instances, an in-process lock is insufficient.
Bad:
Server A → local lockServer B → local lockServer C → local lock
Each server can still enter the critical section.
A distributed coordination mechanism can make the lock shared:
Server A ─┐Server B ─┼──→ Distributed Lock → only one rebuildsServer C ─┘
Locking is not free
Consider:
lock timeout
lock ownership
lock renewal
crashed lock holder
stale lock cleanup
availability impact
what other requests do while waiting
6.3 Serve Stale While Revalidate
Instead of making everyone wait for a fresh value:
Cache has slightly stale value ↓Return stale value immediately +One worker refreshes it
This is extremely useful when slightly stale data is acceptable.
flowchart LR
R["Request"] --> C{"Cache"}
C -->|"Fresh"| F["Return"]
C -->|"Stale but usable"| S["Return stale"]
S --> BG["Background refresh"]
BG --> DB["DB"]
DB --> C
Great fit
Product catalogues, news feeds, recommendations, configuration, dashboards and other read-heavy data where a few seconds of staleness is acceptable.
6.4 TTL Jitter
Bad:
Every key:TTL = 60 seconds
If many keys were populated around the same time, they can expire together.
Better:
TTL = 60 + random(0..30) seconds
So:
Key A → 61 secKey B → 74 secKey C → 82 secKey D → 89 sec
This spreads expiration over time.
Jitter changes the shape of traffic.
It doesn’t reduce the total amount of work by itself.
It reduces synchronization.
6.5 Probabilistic Early Refresh
Don’t wait until:
TTL = 0
Instead, as the item approaches expiration, some requests probabilistically refresh it.
Conceptually:
TTL remaining ↓100s ─────────── normal 20s ─────────── refresh probability starts increasing 5s ─────────── high probability 0s ─────────── should already be refreshed
This is particularly useful for hot keys.
7. Retry Storm — A Close Relative
A retry storm is another synchronization/amplification problem.
Suppose 10,000 clients fail at:
10:00:00
and everyone retries after exactly 1 second:
10,000 retries at 10:00:01
Then:
10,000 retries at 10:00:0310,000 retries at 10:00:07...
The clients accidentally synchronize themselves.
Exponential Backoff Alone
1s2s4s8s
Better than immediately retrying.
But if every client uses exactly the same schedule, synchronization can remain.
Exponential Backoff + Jitter
Client A → random(1..2s)Client B → random(1..2s)Client C → random(1..2s)
Then subsequent attempts are also randomized.
AWS explicitly recommends exponential backoff, jitter and bounded retries, and warns that retries at multiple layers can compound into a retry storm.
Retry multiplication
Imagine:
Client retries 3 times ×API retries 3 times ×Service retries 3 times
A single logical operation can produce:
3 × 3 × 3 = 27 downstream attempts
This is why retry ownership must be deliberate.
8. Jitter Is Bigger Than “Random Delay”
Jitter is useful whenever many actors would otherwise make the same decision at the same time.
Use it for:
retries
cron jobs
scheduled workers
cache expiration
background refresh
polling
leader elections / lease renewals where appropriate
Eventually you have a queue backlog problem instead of an immediate request overload.
AWS recommends failing fast for work that cannot be completed successfully and using queues to buffer work when it can be processed later, while avoiding unbounded/stale queues
13. Batch Processing / Chunking
Your original idea of:
“Break cron jobs into chunks”
is correct.
Bad:
00:00 ↓Process 10 million records ↓One huge transaction / workload
Better:
10 million records ↓10,000-record chunks ↓Queue ↓Workers ↓Controlled concurrency
Benefits:
lower memory pressure
smaller transactions
easier retries
controlled concurrency
better progress tracking
easier horizontal scaling
14. Gradual Deployments
Gradual deployment is not a primary thundering-herd solution, but it is an important reliability technique.
Bad:
100% traffic ↓New version
Better:
1% ↓5% ↓10% ↓25% ↓50% ↓100%
This limits the blast radius if the new version has:
higher CPU usage
slower database queries
connection leaks
unexpected retry behavior
cache behavior changes
Useful patterns:
canary deployment
blue/green deployment
rolling deployment
feature flags
Think of gradual deployment as failure containment, not herd prevention.
15. Decoupling Dependencies
Your note mentioned:
“Username and password, rather than calling authentication service”
The underlying idea is dependency reduction, but be careful with security.
Suppose:
Order API ↓Authentication Service ↓Database
Every request may depend on the authentication service.
A failure there can cascade.
A common architecture instead uses:
Client ↓Identity Provider ↓Access Token ↓Order API
The API can validate a signed token locally, depending on the identity protocol and architecture.
That can remove a synchronous authentication-service call from the hot request path.
Don't copy credentials into every service.
“Avoid a network dependency” does not mean “duplicate sensitive authentication data everywhere.”
Use appropriate identity mechanisms such as signed tokens, key rotation, scoped claims and centralized identity management.
16. Is This the Same as a Sidecar?
"If I avoid calling another service by putting functionality locally, is that the same as using a sidecar?"
The sidecar is a separate process/container deployed alongside the application.
A service mesh commonly uses sidecars to provide networking capabilities such as:
retries
circuit breaking
mTLS
telemetry
traffic routing
Google notes that service meshes can provide resilience features such as retries, failover and circuit breakers while decoupling networking concerns from application logic.
Sidecar ≠ removing a dependency
A sidecar can actually introduce another network hop:
Application → Sidecar → Remote Service
It doesn’t make the remote dependency disappear.
17. The Five Main Defense Families
When answering a system-design question, group solutions into five buckets.
A. Spread the Work
Goal: remove synchronization.
Use:
jitter
randomized TTL
scheduled offsets
exponential backoff
Everyone at T ↓Everyone around T
B. Prevent Duplicate Work
Goal: don’t perform the same expensive operation thousands of times.
Use:
caching
request coalescing
single-flight
distributed locks
stale-while-revalidate
1000 requests ↓1 expensive operation
C. Control Admission
Goal: don’t allow more work into the system than it can safely handle.
Use:
rate limiting
throttling
concurrency limits
circuit breakers
load shedding
virtual waiting rooms
200K incoming ↓20K admitted
D. Buffer Work
Goal: absorb temporary bursts.
Use:
queues
Kafka
SQS
background workers
batch processing
Burst ↓Queue ↓Controlled processing
E. Add Capacity
Goal: increase the amount of work the system can handle.
Use:
pre-scaling
horizontal autoscaling
read replicas
CDN
database scaling
connection pool tuning
Capacity is only one part of the answer.
If the underlying problem is synchronization, more servers may simply produce more simultaneous requests to the same bottleneck.
18. Putting It All Together
Consider:
“A viral post suddenly receives 500K requests/sec.”
A production answer should look like:
flowchart TD
U["500K Users"] --> CDN["CDN / Edge Cache"]
CDN --> RL["Rate Limiter / Admission Control"]
RL --> API["API Fleet"]
API --> C["Distributed Cache"]
C -->|"Hit"| R["Response"]
C -->|"Miss"| SF["Single Flight"]
SF --> DB["Database"]
DB --> SF
SF --> C
API --> Q["Async Queue"]
Q --> W["Workers"]
API -.-> CB["Circuit Breaker / Bulkhead"]
These are related patterns, not interchangeable names.
21. Production Checklist
Before calling a system "herd-resistant"
Can a large number of clients become synchronized?
Are cache TTLs synchronized?
Do retries use exponential backoff?
Is retry jitter enabled?
Is there a maximum retry count/deadline?
Are retries implemented at only the appropriate layer?
Are mutating operations idempotent?
Can one cache miss cause thousands of recomputations?
Is request coalescing available for hot keys?
Can stale data be served safely?
Is there rate limiting?
Is there a concurrency limit?
Can non-critical work be queued?
Are queues bounded?
Are known events pre-scaled?
Are scheduled jobs jittered?
Are downstream dependencies isolated?
Are circuit breakers/bulkheads used where appropriate?
Is there graceful degradation?
Are P95/P99 latency and error rates monitored?
AWS specifically recommends controlling retry calls, using exponential backoff with jitter, limiting retries, and ensuring operations are idempotent. citeturn0search3
The DEV article provides useful practical examples including flash sales, cache expiry, polling, request coalescing, cache locking, jitter, early refresh and virtual waiting rooms.
“A thundering herd occurs when many clients or workers become synchronized and hit the same bottleneck at roughly the same time. It’s different from an ordinary traffic spike because synchronization can create a much sharper load peak.
A cache stampede is a common example: a hot cache entry expires and thousands of requests simultaneously miss the cache and regenerate the same data. I would use techniques such as request coalescing or single-flight so only one request performs the regeneration, serve-stale-while-revalidate when acceptable, and add TTL jitter or early refresh to avoid synchronized expiry.
For retries, I’d use exponential backoff with jitter and bounded retries. For predictable traffic I’d pre-scale; for unpredictable traffic I’d use rate limiting, concurrency limits, queues and graceful degradation. I’d also protect downstream dependencies with isolation and circuit breakers.
The key idea is not simply adding capacity — it’s preventing synchronized demand from turning into synchronized work.”