API Rate Limiting System Design: Interview Deep Dive

Every public API eventually meets a client that hammers it — a buggy retry loop, a scraper, a competitor. Rate limiting is how you stay standing. In interviews it's also a favourite standalone question ("design a rate limiter") because it's small enough to finish and deep enough to separate memorisers from thinkers.

What a rate limiter does: it enforces a policy like "100 requests per minute per user" by counting requests in a time window and rejecting or delaying the ones over the line. The interesting engineering is where the counter lives and which counting algorithm you choose.

The four algorithms worth knowing

Algorithm	How it works	Trade-off
Fixed window counter	Count per clock minute; reset on the minute	Cheap, but allows 2x burst at window edges
Sliding window log	Store a timestamp per request; count within the last N seconds	Accurate, but memory grows with request volume
Sliding window counter	Blend the current and previous fixed windows by weight	Good accuracy, low memory; the usual sweet spot
Token bucket	Tokens refill at a steady rate; each request spends one	Allows controlled bursts; smooth, widely used

A token bucket in fifteen lines

class TokenBucket {
  constructor(capacity, refillPerSec) {
    this.capacity = capacity;
    this.tokens = capacity;
    this.refillPerSec = refillPerSec;
    this.last = Date.now();
  }
  allow() {
    const now = Date.now();
    const elapsed = (now - this.last) / 1000;
    this.tokens = Math.min(
      this.capacity,
      this.tokens + elapsed * this.refillPerSec
    );
    this.last = now;
    if (this.tokens >= 1) { this.tokens -= 1; return true; }
    return false;   // caller returns HTTP 429
  }
}

The distributed problem

One bucket per server breaks the moment you have ten servers behind a load balancer — each one hands out the full allowance. The standard fix is a shared, fast store (an in-memory data store with atomic increments is the classic choice) so all servers count against the same limit. That introduces a new trade-off worth naming in the interview: a network hop per request and a hard dependency, versus slightly loose limits if you let each node approximate. Senior candidates discuss both and pick deliberately.

And when a client is rejected? Return HTTP 429 with a Retry-After header, and say so — it shows you've built APIs people actually integrate with. For more on that interface craft, see API design best practices.

Client-friendly limiters also tell callers where they stand: headers like X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset let well-behaved clients throttle themselves before they ever see a rejection. Mention them in the interview — it's a small detail that signals you designed for the developers on the other side of the wire, not just for your own protection.

Common mistakes

• Using a fixed window without mentioning the edge-burst problem. Interviewers bring it up if you don't.
• Forgetting to state what you limit by — user ID, API key, IP address? Each has failure modes (shared IPs behind NAT, key rotation).
• Storing the counters in the primary SQL database. You're adding write load to the thing you're protecting.
• No plan for the limiter's own failure. Fail-open or fail-closed is a real product decision — say which and why.

FAQ

Which algorithm should I lead with? Token bucket for general APIs (bursts are natural), sliding window counter when fairness matters more. Name the trade-off either way.

Where does the limiter live? Usually at the API gateway or as middleware, so rejection happens before your application does any real work.

Rate limiting is one of the classic deep-dive topics inside the system design spine. Practise it out loud with Aissence mock interviews.