Case Studies
Design a Rate Limiter Service
Enforce "100 requests per second per API key" across many gateway nodes: local counters let a client N times through, a shared Redis counter is exact but can take everyone down, and a local fallback keeps limiting when it fails.
Intermediate 5 min read· Updated 2026-11-18
What you will learn
- See why per-node limits do not add up to a global limit.
- Implement a shared counter in Redis with atomic increments.
- Choose between failing open, failing closed and falling back.
- Return useful rate-limit responses to clients.
Before you start
Requirements
- Functional: limit each client (API key, user or IP) to a configured rate per rule; reject the excess with
429 Too Many Requestsand tell the client when to retry. - Non-functional: add at most a millisecond or two per request; the limit holds across all gateway nodes; a limiter failure must not take the API down.
- Scale: every API request passes through the limiter — it is on the hottest path there is.
Back-of-the-envelope estimates
| Quantity | Assumption | Result |
|---|---|---|
| Checks | 50,000 API requests/s | 50,000 counter increments/s |
| Redis | ~100,000 simple ops/s per node | One primary is enough; shard by key beyond that |
| Memory | 1 M active keys × a few rules × ~50 B | ~150 MB — counters expire with their window |
Evolve the design
Stage 1 of 3
A counter in each gateway
- Current architecture
- Each gateway node keeps per-key counters in memory and applies the 100/s rule on its own.
- Limitation
- The load balancer spreads one client over all nodes, and each node allows the full limit.
- Bottleneck
- The API: the scraper gets 4 × 100 = 400 req/s through and the API starts rejecting normal users.
- Improvement
- Count in one place that every node shares.
- Performance impact
- Over a thousand normal requests fail in 30 s.
- Complexity & operational cost
- Nothing extra — and the limit is wrong by a factor of the node count.
- Trade-off
- Fast and simple, but not a global limit (and it changes every time you scale).
Loading simulation…
The shared counter
-- KEYS[1] = "rl:{api_key}:{rule}:{window}" ARGV[1] = limit ARGV[2] = window seconds
local count = redis.call('INCR', KEYS[1])
if count == 1 then
redis.call('EXPIRE', KEYS[1], ARGV[2]) -- the counter disappears with its window
end
if count > tonumber(ARGV[1]) then
return {0, redis.call('TTL', KEYS[1])} -- rejected, seconds until the window resets
end
return {1, tonumber(ARGV[1]) - count} -- allowed, remaining- Atomic:
INCR+EXPIREin one script avoids a read-then-write race between gateways. - Algorithm: the fixed window is simplest; a sliding window or token bucket (stored as a hash with tokens + timestamp) smooths bursts at the window edge — see rate limiting.
- Hot keys: one huge client hits one Redis shard; a local pre-filter (drop requests far above the limit without asking Redis) protects it.
- Responses: return
429withRetry-AfterandRateLimit-*headers so well-behaved clients back off.
When the limiter fails
| Benefits | Costs | |
|---|---|---|
| Fail closed | Limits are never exceeded. | A limiter outage becomes a full API outage. |
| Fail open | Users are never blocked by the limiter itself. | Abusers and spikes reach the backend unprotected. |
| Local fallback | Stays up and still roughly limits. | Approximate while degraded; one more code path to test. |
Interview checklist
- Clarify what is limited (key, user, IP), the rules and where the limiter sits (gateway or sidecar).
- Pick an algorithm and explain why per-node counters are not enough.
- Show the atomic Redis operation and how counters expire.
- Discuss fail open vs closed vs fallback, hot keys and the 429 response.
Further reading
Related topics
Microservice Patterns
Rate Limiting
Cap how many requests a client may make per unit of time to protect services from bursts and abuse — and pick the algorithm whose behaviour at the edges you can live with.
Microservice Patterns
API Gateway
Give clients one entry point that routes requests to the right service and handles authentication, rate limiting and response aggregation in one place.
Microservice Patterns
Gatekeeper
Put a dedicated, minimal-privilege host in front of trusted services to validate and sanitise every request, so attacks and junk are rejected cheaply at the edge and the services holding secrets are never directly exposed.
Microservice Patterns
Fallback
When a dependency fails, answer with something useful instead of an error — a default, a cached value or a reduced feature — so a non-essential failure never breaks an essential page.
Microservice Patterns
Timeout
Put an upper bound on how long you wait for any remote call, so one hanging dependency cannot quietly hold every thread, connection and user request hostage.