Capacity planning & load testing

Why there is a knee

Any resource that serves requests, such as a CPU, a worker pool, a database connection or a disk, behaves like a queue. While it is mostly idle, a new request is served at once. As it gets busier, a new request increasingly arrives to find others ahead of it, and has to wait. Queueing theory gives a simple approximation for a single server with random arrivals.

response time = service time / (1 - utilisation)

service time 20 ms:
  utilisation 50%  ->  20 / 0.50 =   40 ms   (2x)
  utilisation 80%  ->  20 / 0.20 =  100 ms   (5x)
  utilisation 90%  ->  20 / 0.10 =  200 ms   (10x)
  utilisation 95%  ->  20 / 0.05 =  400 ms   (20x)
  utilisation 99%  ->  20 / 0.01 = 2000 ms   (100x)

The curve is nearly flat at first, then bends sharply upward. That bend is the knee. Going from 50% to 80% utilisation costs a little latency. Going from 90% to 95% doubles it. Real systems have many servers and less random arrivals, so the exact numbers differ, but the shape always holds. Past the knee, a small rise in traffic, a slightly slower dependency or the loss of one instance pushes you up a cliff.

Two consequences follow. First, running hot is a false economy: a fleet at 90% utilisation looks efficient, and it has no ability to absorb anything. Second, averages hide the knee. Utilisation averaged over five minutes can read 60% while one-second bursts reach 100% and queues form. Look at high percentiles of latency and at saturation signals such as queue depth and CPU throttling, not only at average utilisation.

Little's Law is the other tool worth carrying. For any stable system:

L = lambda x W

L       average number of requests in the system (concurrency)
lambda  average arrival rate (requests per second)
W       average time a request spends in the system (seconds)

Example: 400 requests/s, each taking 250 ms
  L = 400 x 0.25 = 100 requests in flight at any moment

  Each pod handles 16 concurrent requests -> 100 / 16 = 6.25 -> at least 7 pods,
  before any headroom.

  If a dependency slows and W doubles to 500 ms, L doubles to 200:
  the same traffic now needs twice the concurrency.

That last line explains many outages. Traffic did not rise, but a dependency slowed, so requests stayed in flight longer and concurrency doubled. Thread pools and connection pools sized for the old concurrency ran out, and the cascading failure from the previous module began. Size pools from Little's Law using a pessimistic latency, not the happy-path figure.

Load testing honestly

You find the knee by measurement. Different tests answer different questions.

The most important and least understood choice is the workload model. In a closed model, a fixed number of virtual users each send a request, wait for the response, then send another. When the system slows, the users slow with it, so the offered load falls just as the system gets into trouble, and the test flatters you. In an open model, requests arrive at a set rate regardless of how fast responses come back, which is how real internet traffic behaves: users do not stop arriving because your site is slow. Test with an open model, specifying an arrival rate.

The related measurement error is coordinated omission. If a tool waits for each response before sending the next request, then during a ten-second stall it records one slow request instead of the hundreds that should have been sent and would have been slow as well. The reported percentiles look far better than what users experienced. Tools that schedule requests at a constant rate, and measure from the intended send time, avoid it.

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    ramp: {
      executor: 'ramping-arrival-rate',   // open model: a request rate, not a user count
      startRate: 50,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      maxVUs: 2000,
      stages: [
        { target: 200, duration: '5m' },  // ramp to expected peak
        { target: 200, duration: '10m' }, // hold
        { target: 600, duration: '10m' }, // keep going to find the knee
      ],
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.001'],      // under 0.1% errors
    http_req_duration: ['p(99)<500'],     // p99 under 500 ms
  },
};

export default function () {
  const res = http.get('https://staging.example.com/api/products?page=1');
  check(res, { 'status is 200': (r) => r.status === 200 });
}

• Use a realistic mix of endpoints, payload sizes and data. Hitting one cached URL measures your cache, not your system.
• Test an environment like production, in size and in data volume. A query that is instant on a thousand rows may be a full scan on a hundred million.
• Watch the system, not only the tool. Record which resource saturates first: CPU, memory, a connection pool, a lock, a downstream rate limit. That is your bottleneck, and the thing to fix or scale next.
• Check recovery. After a stress test, does latency return to normal when the load drops, or does the system stay broken until restarted? The second is a serious finding.
• Do not load test a dependency you do not own without agreement. Stub third-party APIs, or you will be testing someone else's rate limiter.
• Make it routine. Run a short load test in the pipeline to catch performance regressions, and a full one before every expected peak.

Headroom and redundancy

Capacity is not sized for the average. It is sized for the peak, plus room for failure, plus room for growth and error. Work through it in order.

1. Peak demand            measured peak, e.g. 1,200 requests/s on the busiest day
2. Per-instance capacity  from a load test, at the SLO: e.g. 150 requests/s per pod
                          (the load where p99 still meets the target, NOT the breaking point)
3. Instances for peak     1,200 / 150 = 8
4. Redundancy             N+1: survive the loss of one   -> 9
                          N+2: one in maintenance AND one failing -> 10
5. Zone failure           3 zones; losing one must leave >= 8
                          -> 12 pods, 4 per zone (8 remain)
6. Growth buffer          expected growth until the next review, e.g. +20%

Step five is often the largest term, and it surprises people. With instances spread across three zones, surviving the loss of a zone means the remaining two thirds must carry the full peak, so you run at no more than about 67% of capacity in normal operation. With two zones it is 50%. That arithmetic is the real price of zone redundancy, and it is why target utilisation figures of 50 to 65% are common and sensible, not wasteful.

Remember what happens at the moment of failure. The survivors take on extra load and are moved up the latency curve towards the knee. If the fleet was at 85% before one of four instances died, the remaining three face 113% and collapse. This is the load-redistribution failure from the previous module, and headroom is the defence.

Forecasting demand

Demand comes from two sources, and they need different methods. Organic growth is the natural trend from adoption and usage. It can be estimated from history. Inorganic growth comes from events that history cannot predict: a product launch, a marketing campaign, a large customer being onboarded, a seasonal sale. It can only be learned by talking to the people planning those events, which makes capacity planning partly a communication job.

import numpy as np

# weekly peak requests/s for the last 12 weeks
peaks = np.array([810, 835, 850, 870, 905, 915, 940, 975, 990, 1020, 1050, 1075])
weeks = np.arange(len(peaks))

slope, intercept = np.polyfit(weeks, peaks, 1)          # straight-line fit
capacity = 2000                                         # tested capacity, requests/s
threshold = 0.65 * capacity                             # act before 65% of capacity

weeks_until_threshold = (threshold - intercept) / slope - weeks[-1]
print(f"growth: {slope:.0f} req/s per week")
print(f"threshold of {threshold:.0f} reached in about {weeks_until_threshold:.0f} weeks")

• Forecast the peak, not the mean. Use weekly or daily maxima, and respect seasonality: compare this December with last December, not with November.
• Compare the forecast with lead time. If adding capacity takes six weeks, because of procurement, quota increases or a database migration, you must act when the threshold is more than six weeks away. Cloud quotas and account limits are a classic hidden lead time.
• Track more than CPU. Storage, database connections, IP addresses in a subnet, licence seats, API rate limits with third parties and message queue partitions all run out, often less visibly.
• Alert on the trend. predict_linear in Prometheus turns a forecast into an early-warning ticket, such as a disk that will fill in four days.
• Revisit regularly. A forecast is wrong the day after it is made. A short monthly review of forecast against actual keeps it useful.

# disk will be full within 4 days at the current rate of growth (ticket, not page)
predict_linear(node_filesystem_avail_bytes{mountpoint="/data"}[3d], 4 * 86400) < 0

# weekly peak request rate, for the capacity dashboard
max_over_time(sum(rate(http_requests_total{job="checkout"}[5m]))[7d:5m])

Autoscaling, and its limits

Autoscaling matches capacity to demand automatically, which saves money in quiet periods and absorbs ordinary peaks. It complements capacity planning and does not replace it, because it has limits you must design around.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout
  namespace: shop
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  minReplicas: 6                 # enough for normal load with a zone lost
  maxReplicas: 30                # a ceiling that protects dependencies and the budget
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60      # may double every minute
    scaleDown:
      stabilizationWindowSeconds: 300   # scale in slowly, to avoid flapping

• It reacts, so it lags. Metrics must be scraped, a decision made, pods scheduled, perhaps a node provisioned, images pulled, the application started and its caches warmed. That takes from tens of seconds to several minutes. A spike that arrives in ten seconds is served by the capacity you already had, which is why the target is 60% and not 90%.
• Scale up fast and down slowly. Rapid scale-in followed by another spike causes flapping and needless cold starts.
• Choose the metric that reflects the bottleneck. CPU suits compute-bound services. For I/O-bound services, scale on requests per second, queue depth or in-flight requests, using custom or external metrics.
• minReplicas is a reliability setting, not a cost setting. It must cover normal load with a zone lost, without waiting for the autoscaler.
• maxReplicas protects what is downstream. Thirty pods with twenty database connections each is six hundred connections. If the database allows five hundred, the autoscaler has just caused your outage. Scale the whole chain, or put a connection pooler in between.
• Pre-scale for known events. Before a campaign or a sale, raise the minimum in advance. Do not make the autoscaler discover the launch from a wall of traffic.
• A scaled-to-zero or freshly started fleet is cold. Test how the service behaves in its first minute under load.