Distributed-systems failure modes

Slow is worse than down

A dependency that is completely down fails fast: connections are refused, errors return in milliseconds, and callers can react. A dependency that is slow is far more dangerous, because every call to it holds resources while it waits: a thread, a connection from the pool, memory for the request, a slot in a queue.

Follow what happens. Service A calls service B, which has become slow. A's worker threads block waiting for B. New requests keep arriving at A, and soon every thread is waiting on B. Now A cannot serve any request, including those that never needed B. A's callers see A as slow, and the same thing happens to them. The fault travels up the call graph until the whole product is down because of one struggling component several layers deep. This is a cascading failure.

Several mechanisms feed it, and they reinforce one another.

• Resource exhaustion: threads, connection pools, file descriptors and memory are consumed by waiting requests.
• Retries multiply the load on the component that is already struggling.
• Load redistribution: when one overloaded instance dies, its traffic moves to the others, which pushes them over too.
• Cold start: instances that restart have empty caches and unopened connections, so they are slowest exactly when demand is highest.
• Health checks that kill: liveness probes fail on overloaded instances, the orchestrator restarts them, and capacity falls further.

The defensive goal is always the same. Fail fast, bound every resource, and never let one dependency's problem consume everything you have.

Timeouts and deadlines

Every call that crosses a process boundary needs a timeout. Many client libraries default to no timeout at all, which means that a hung dependency holds your thread for ever. Set two: a connection timeout, which is short because establishing a connection should take milliseconds, and a request timeout for the whole exchange.

import requests

# (connect timeout, read timeout) in seconds. Never call without one.
resp = requests.get("https://inventory.internal/stock/SKU-9917", timeout=(0.5, 2.0))

Choose the value from data, not from instinct. Look at the dependency's latency distribution and set the timeout a little above its normal high percentile, for example its p99.9. If the p99 is 200 ms, a 30-second timeout protects nothing: by the time it fires, the user has long since gone, and you held a thread the whole time. A timeout that is too tight, on the other hand, turns ordinary slowness into errors, so check how often it fires.

Timeouts should also get shorter as you go down the call chain. If the edge gives a request 3 seconds, a service two layers down that allows itself 10 seconds is doing work nobody will ever see. Deadline propagation makes this exact: the first service sets an absolute deadline, passes it along with the request, and each service checks the time remaining before starting expensive work and uses it to bound its own outgoing calls. gRPC does this natively, and with HTTP you can pass a header.

import time


class DeadlineExceeded(Exception):
    pass


def remaining(deadline: float, reserve: float = 0.05) -> float:
    """Seconds left before an absolute deadline (time.monotonic based)."""
    left = deadline - time.monotonic() - reserve
    if left <= 0:
        raise DeadlineExceeded("no time left; not starting more work")
    return left


def handle_checkout(order, deadline: float):
    stock = call_inventory(order, timeout=min(1.0, remaining(deadline)))
    payment = call_payments(order, timeout=min(2.0, remaining(deadline)))
    return confirm(order, stock, payment)

Retries that help instead of harm

Retries hide transient faults, such as a dropped packet, a pod being replaced or a brief leader election. They are also the most common way a small problem becomes a large one, because a retry is extra load, applied at the moment the system can least bear it.

The arithmetic is unforgiving. If three layers of services each make up to 3 attempts, one user request can become 3 x 3 x 3 = 27 requests at the bottom layer. A database that was 20% over capacity is suddenly facing many times its normal load, and it can never recover while the retries continue.

import random
import time

RETRYABLE = (ConnectionError, TimeoutError)


def call_with_retries(fn, max_attempts: int = 3, base: float = 0.1, cap: float = 2.0):
    """Exponential backoff with full jitter."""
    for attempt in range(max_attempts):
        try:
            return fn()
        except RETRYABLE:
            if attempt == max_attempts - 1:
                raise
            # full jitter: a random wait between 0 and the exponential ceiling
            time.sleep(random.uniform(0, min(cap, base * 2 ** attempt)))

• Exponential backoff doubles the wait ceiling after each failure, which gives the dependency room to recover.
• Jitter randomises the wait. Without it, every client that failed at the same moment retries at the same moment, and the dependency is hit by synchronised waves. This is the thundering herd. Full jitter, a uniform random wait between zero and the ceiling, spreads them out.
• Few attempts. Two or three in total. If three attempts failed, a fourth is very unlikely to succeed and certain to add load.
• Retry only what can succeed. Connection errors, timeouts, 503 and 429 are candidates. A 400, 401, 403 or 404 will fail identically every time. Honour a Retry-After header when the server sends one.
• Retry at one layer only, normally the one closest to the failure. If every layer retries, the counts multiply.

A retry budget puts a hard limit on the damage. Instead of bounding attempts per request, bound retries as a share of all traffic, for example no more than 10% of requests. In healthy times that allows every transient fault to be retried. In an outage, where everything is failing, it caps the extra load at 10% instead of 200%.

Retrying is only safe if the operation is idempotent, meaning that doing it twice has the same effect as doing it once. Reads are. "Charge this card" is not: a timeout tells you nothing about whether the charge went through. The fix is an idempotency key. The client generates a unique ID for the operation and sends it with every attempt, and the server records the keys it has processed and returns the stored result for a repeat instead of acting again.

curl -fsS -X POST https://payments.internal/v1/charges \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: 7c1f3e9a-2b44-4c1e-9d07-5a6e8b1f0c33' \
  -d '{"order_id": "o-1842", "amount": 4999, "currency": "EUR"}'

Backpressure and load shedding

Every service has a limit. What matters is what happens beyond it. A service with no protection accepts everything, queues it, slows down for everyone, runs out of memory, and dies, after which it serves nobody. A protected service recognises its limit and refuses the excess quickly and cheaply, so that the work it does accept is still served well. Serving 80% of users properly is far better than serving 100% of them a timeout.

• Bound every queue. An unbounded queue hides overload and converts it into latency and then into memory exhaustion. When a bounded queue is full, reject.
• Limit concurrency. Cap the number of in-flight requests to what the service can handle, and return 503 or 429 at once beyond that. Rejecting costs microseconds; accepting a request you cannot finish costs a thread and the client's time.
• Shed by priority. Drop the least valuable work first: background refreshes, recommendations and analytics before checkout, and unauthenticated traffic before paying customers.
• Degrade gracefully. Serve cached or simplified responses, and switch off expensive features, before refusing outright.
• Prefer newer requests under sustained overload. If the queue is long, the oldest requests have probably been abandoned by callers who timed out. Serving them is wasted work. Dropping the oldest, or serving last-in first-out, keeps the work useful.

import threading


class Overloaded(Exception):
    """Translate to HTTP 503 with a Retry-After header."""


class ConcurrencyLimiter:
    def __init__(self, max_in_flight: int):
        self._slots = threading.BoundedSemaphore(max_in_flight)

    def run(self, fn):
        if not self._slots.acquire(blocking=False):     # never wait for a slot
            raise Overloaded("at capacity; shedding load")
        try:
            return fn()
        finally:
            self._slots.release()


limiter = ConcurrencyLimiter(max_in_flight=64)

Backpressure is the same idea applied between components: a consumer that cannot keep up signals the producer to slow down, instead of letting work pile up in between. Bounded queues that block the producer, flow control in TCP and in gRPC streams, and 429 responses with Retry-After are all forms of it. Rate limiting per client, often with a token bucket, additionally stops one noisy client from consuming capacity that belongs to everyone.

Circuit breakers and bulkheads

When a dependency is clearly failing, continuing to call it wastes your resources and prevents its recovery. A circuit breaker wraps the calls and tracks failures. It has three states.

import time


class CircuitOpen(Exception):
    pass


class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, cooldown: float = 30.0):
        self.failure_threshold = failure_threshold
        self.cooldown = cooldown
        self.failures = 0
        self.opened_at = None                 # None means closed

    def call(self, fn):
        if self.opened_at is not None:
            if time.monotonic() - self.opened_at < self.cooldown:
                raise CircuitOpen("failing fast; dependency is unhealthy")
            # cool-down over: half-open, let this one trial call through
        try:
            result = fn()
        except Exception:
            self.failures += 1
            if self.opened_at is not None or self.failures >= self.failure_threshold:
                self.opened_at = time.monotonic()      # open, or re-open after a failed trial
            raise
        self.failures = 0
        self.opened_at = None                          # success closes the circuit
        return result

That sketch shows the state machine. In production, use a maintained library or a service mesh, which add rolling failure-rate windows, limits on concurrent trial calls, thread safety and metrics. Whatever you use, export the breaker's state as a metric and alert when it stays open, because an open breaker means a feature is degraded.

A breaker is only as good as its fallback. Decide in advance what each caller does when the dependency is unavailable: serve a cached value, return a sensible default, hide the feature, or queue the work for later. If the honest answer is "return an error", the dependency is a hard one, and its reliability limits yours, as the embracing-risk module calculated.

Bulkheads, named after the watertight compartments of a ship, stop one failure from flooding everything. Give each dependency its own bounded connection pool or concurrency limit, instead of one shared pool. When the recommendations service hangs, it can exhaust only its own twenty connections, and checkout carries on untouched. The same principle applies at larger scale: separate worker pools for critical and background work, separate clusters or cells for groups of customers, so that a bad deploy or a poisoned request takes out one cell and not the whole fleet.

Putting it together

These patterns are layers of one defence, and each covers a gap left by the others.

You rarely need to write these by hand. A service mesh such as Istio or Linkerd, or a proxy such as Envoy, applies timeouts, retries with budgets, circuit breaking and outlier detection uniformly, outside the application, and resilience libraries exist for every major language. What no tool can decide for you is the policy: which timeout, which calls are idempotent, which dependencies are soft, and what the fallback is. Those are design decisions, and they belong in the production readiness review.

Finally, none of this can be trusted until it has been exercised. A timeout that was never triggered and a fallback that was never used are hopes, not defences. The chaos engineering module shows how to test them on purpose.