Skip to content
System Design Lab
Case Studies

Design a Rate Limiter Service

Enforce "100 requests per second per API key" across many gateway nodes: local counters let a client N times through, a shared Redis counter is exact but can take everyone down, and a local fallback keeps limiting when it fails.

Intermediate 5 min read· Updated 2026-11-18

What you will learn

  • See why per-node limits do not add up to a global limit.
  • Implement a shared counter in Redis with atomic increments.
  • Choose between failing open, failing closed and falling back.
  • Return useful rate-limit responses to clients.

Before you start

Requirements

  • Functional: limit each client (API key, user or IP) to a configured rate per rule; reject the excess with 429 Too Many Requests and tell the client when to retry.
  • Non-functional: add at most a millisecond or two per request; the limit holds across all gateway nodes; a limiter failure must not take the API down.
  • Scale: every API request passes through the limiter — it is on the hottest path there is.

Back-of-the-envelope estimates

QuantityAssumptionResult
Checks50,000 API requests/s50,000 counter increments/s
Redis~100,000 simple ops/s per nodeOne primary is enough; shard by key beyond that
Memory1 M active keys × a few rules × ~50 B~150 MB — counters expire with their window

Evolve the design

Stage 1 of 3

A counter in each gateway

Current architecture
Each gateway node keeps per-key counters in memory and applies the 100/s rule on its own.
Limitation
The load balancer spreads one client over all nodes, and each node allows the full limit.
Bottleneck
The API: the scraper gets 4 × 100 = 400 req/s through and the API starts rejecting normal users.
Improvement
Count in one place that every node shares.
Performance impact
Over a thousand normal requests fail in 30 s.
Complexity & operational cost
Nothing extra — and the limit is wrong by a factor of the node count.
Trade-off
Fast and simple, but not a global limit (and it changes every time you scale).
Loading simulation…

The shared counter

Fixed-window counter as one atomic Redis script
-- KEYS[1] = "rl:{api_key}:{rule}:{window}"   ARGV[1] = limit   ARGV[2] = window seconds
local count = redis.call('INCR', KEYS[1])
if count == 1 then
  redis.call('EXPIRE', KEYS[1], ARGV[2])   -- the counter disappears with its window
end
if count > tonumber(ARGV[1]) then
  return {0, redis.call('TTL', KEYS[1])}   -- rejected, seconds until the window resets
end
return {1, tonumber(ARGV[1]) - count}      -- allowed, remaining
  • Atomic: INCR + EXPIRE in one script avoids a read-then-write race between gateways.
  • Algorithm: the fixed window is simplest; a sliding window or token bucket (stored as a hash with tokens + timestamp) smooths bursts at the window edge — see rate limiting.
  • Hot keys: one huge client hits one Redis shard; a local pre-filter (drop requests far above the limit without asking Redis) protects it.
  • Responses: return 429 with Retry-After and RateLimit-* headers so well-behaved clients back off.

When the limiter fails

BenefitsCosts
Fail closedLimits are never exceeded.A limiter outage becomes a full API outage.
Fail openUsers are never blocked by the limiter itself.Abusers and spikes reach the backend unprotected.
Local fallbackStays up and still roughly limits.Approximate while degraded; one more code path to test.

Interview checklist

  • Clarify what is limited (key, user, IP), the rules and where the limiter sits (gateway or sidecar).
  • Pick an algorithm and explain why per-node counters are not enough.
  • Show the atomic Redis operation and how counters expire.
  • Discuss fail open vs closed vs fallback, hot keys and the 429 response.

Further reading

Related topics