Horizontal Pod Autoscaling: How the Algorithm Decides, and How to Tune It

On this page

The HorizontalPodAutoscaler is usually the first autoscaler a Kubernetes team turns on and the last one it understands. A CPU target of 70 per cent is easy to write; knowing why the deployment took four minutes to scale down, or why it added eight pods at once and then sat saturated while they started, needs the algorithm. Since Kubernetes 1.23 the stable API is autoscaling/v2, and its behavior field gives direct control over how fast the HPA moves in each direction.

This article explains what the controller does on each sync, as of Kubernetes 1.29 (March 2024), which metrics to scale on, how behavior changes the outcome, and how HPA fits with the other autoscalers. The sample code replays a load trace through the same decision steps, so you can see the effect of a configuration before applying it.

What happens on each sync

The HPA sync: read metrics, compute the ratio and skip within 10 per cent tolerance, stabilise over the window, apply rate-limit policies, clamp to min and max and update the scale subresource; with the replica formula and default behavior
Five steps every 15 seconds. Most surprises come from steps 3 and 4.

The HPA controller runs inside kube-controller-manager and evaluates every HPA on a fixed interval, 15 seconds by default. For each metric it computes desiredReplicas = ceil(currentReplicas × currentMetricValue ÷ desiredMetricValue). If the ratio is within the tolerance of 1.0 (0.1 by default), it does nothing. With several metrics it takes the highest result, so a CPU metric and a queue metric can be combined safely.

Utilization targets are percentages of the pod’s resource request. A container without a CPU request gives the HPA nothing to measure against, and a request set far below real usage makes every pod look overloaded. Pods that are not yet ready, or have no metrics yet, are treated conservatively: when scaling up, not-ready pods count as using none of the target, which damps the scale-up rather than overshooting.

The raw recommendation then passes through stabilisation and rate limits. The controller keeps the recommendations from the stabilisation window; for scale-down it uses the highest of them, so a brief dip does not remove pods. Rate-limit policies cap how many pods may be added or removed per period. If you set no behavior, scale-up has no window and may add 100 per cent or 4 pods per 15 seconds, whichever is more; scale-down waits 300 seconds and may then remove up to 100 per cent per 15 seconds.

%%{init: {'theme': 'base', 'themeVariables': {'signalColor': '#0073BB', 'signalTextColor': '#0F172A', 'actorBkg': '#FFF3E0', 'actorBorder': '#FF9900', 'actorTextColor': '#232F3E', 'noteBkgColor': '#FFF8E7', 'noteBorderColor': '#FF9900', 'noteTextColor': '#232F3E', 'lineColor': '#232F3E'}}}%%
sequenceDiagram
    participant H as HPA controller
    participant M as Metrics API
    participant A as API server
    participant C as Node autoscaler
    loop Every 15 seconds
        H->>M: Read pod CPU usage
        M-->>H: Usage per pod
        Note over H: Ratio, tolerance,<br/>window, policies, min/max
        H->>A: Update replicas on /scale
    end
    A-->>C: New pods are Pending
    C->>A: Add nodes, pods scheduled
    Note over A: Pods start, pass readiness,<br/>then receive traffic

The sequence shows why scaling is slower than the 15-second sync suggests. New pods need a node, an image pull, application start-up and a passing readiness probe before they help. If the node autoscaler has to add a node, that is minutes, not seconds. The target utilisation is the buffer that covers this gap.

Choosing the metric

CPU utilisation works for services whose cost grows with request volume and where CPU is the bottleneck. Memory is rarely a good scaling signal: many runtimes do not release memory when load falls, so the HPA scales up and never back down. From Kubernetes 1.27 a ContainerResource metric (beta, on by default) can target one container’s usage, which avoids a sidecar’s CPU skewing the average.

For services limited by something other than CPU, scale on what users feel or what is queued: requests per second per pod, in-flight requests or queue depth through the custom or external metrics APIs. These need an adapter such as the Prometheus Adapter, or KEDA, which runs its own metrics server and manages an HPA for you.

⚠️
DO NOT LET VPA AND HPA FIGHT

Running the Vertical Pod Autoscaler in update mode and an HPA on CPU or memory for the same workload makes them react to each other: VPA raises the request, utilisation falls, HPA removes pods, utilisation rises. Use VPA in recommendation mode to set requests, or scale the HPA on a custom metric.

Tuning behavior

This HPA keeps one replica per Availability Zone, targets 65 per cent CPU to leave room while new pods start, scales up as fast as the default, and scales down slowly: it waits 10 minutes, then removes the smaller of 20 per cent or 2 pods per minute.

YAML40 lines
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
  namespace: orders
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 3            # one per Availability Zone
  maxReplicas: 40           # a ceiling you can afford and the database can take
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65   # leaves headroom while new pods start
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      selectPolicy: Max
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
        - type: Pods
          value: 4
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 600   # ride out dips shorter than 10 minutes
      selectPolicy: Min
      policies:
        - type: Percent
          value: 20
          periodSeconds: 60
        - type: Pods
          value: 2
          periodSeconds: 60

Whether the slow scale-down is worth it depends on traffic. The sample replays a trace through the HPA steps and reports replica-minutes (a proxy for cost), syncs spent saturated, and the number of changes:

Python157 lines
"""Replay a load trace through the HorizontalPodAutoscaler algorithm.

Implements the autoscaling/v2 decision steps the kube-controller-manager
applies on every sync (15 s by default): the replica ratio with a 0.1
tolerance, the stabilisation window, scaleUp/scaleDown rate-limit policies
with selectPolicy, and min/max replicas. Use it to see how a `behavior`
block reacts to a recorded traffic pattern before you apply it to a cluster.

Simplifications (documented so results are read correctly):
- one Resource metric with a Utilization target;
- utilisation = load / (ready replicas x capacity per pod at 100% of request);
- new pods serve traffic after `startup_seconds`; until then they are not
  counted (the real controller treats not-yet-ready pods more conservatively).
"""
from __future__ import annotations

import math
from collections import deque
from dataclasses import dataclass, field

import yaml

SYNC_SECONDS = 15
TOLERANCE = 0.1

DEFAULT_UP = {"stabilizationWindowSeconds": 0, "selectPolicy": "Max",
              "policies": [{"type": "Percent", "value": 100, "periodSeconds": 15},
                           {"type": "Pods", "value": 4, "periodSeconds": 15}]}
DEFAULT_DOWN = {"stabilizationWindowSeconds": 300, "selectPolicy": "Max",
                "policies": [{"type": "Percent", "value": 100, "periodSeconds": 15}]}


@dataclass
class Spec:
    min_replicas: int
    max_replicas: int
    target_utilization: int
    scale_up: dict = field(default_factory=lambda: dict(DEFAULT_UP))
    scale_down: dict = field(default_factory=lambda: dict(DEFAULT_DOWN))

    @classmethod
    def from_manifest(cls, text: str) -> "Spec":
        doc = yaml.safe_load(text)
        if doc.get("apiVersion") != "autoscaling/v2" or doc.get("kind") != "HorizontalPodAutoscaler":
            raise ValueError("expected an autoscaling/v2 HorizontalPodAutoscaler")
        spec = doc["spec"]
        metrics = spec.get("metrics", [])
        if len(metrics) != 1 or metrics[0].get("type") != "Resource" \
                or metrics[0]["resource"]["target"].get("type") != "Utilization":
            raise ValueError("the simulator supports exactly one Resource metric with a Utilization target")
        behavior = spec.get("behavior") or {}
        up = {**DEFAULT_UP, **(behavior.get("scaleUp") or {})}
        down = {**DEFAULT_DOWN, **(behavior.get("scaleDown") or {})}
        return cls(spec.get("minReplicas", 1), spec["maxReplicas"],
                   metrics[0]["resource"]["target"]["averageUtilization"], up, down)


def raw_recommendation(current: int, utilization: float, target: int) -> int:
    ratio = utilization / target
    if abs(ratio - 1.0) <= TOLERANCE:
        return current
    return math.ceil(current * ratio)


def _period_change(events: deque, now: int, period: int, direction: int) -> int:
    return sum(n for t, n in events if now - t < period and (n > 0) == (direction > 0))


def scale_up_limit(current: int, rules: dict, events: deque, now: int) -> int:
    limits = []
    for p in rules["policies"]:
        start = current - _period_change(events, now, p["periodSeconds"], +1)
        if p["type"] == "Pods":
            limits.append(start + p["value"])
        else:
            limits.append(math.ceil(start * (1 + p["value"] / 100)))
    if rules.get("selectPolicy") == "Disabled":
        return current
    return min(limits) if rules.get("selectPolicy") == "Min" else max(limits)


def scale_down_limit(current: int, rules: dict, events: deque, now: int) -> int:
    limits = []
    for p in rules["policies"]:
        start = current - _period_change(events, now, p["periodSeconds"], -1)  # removed pods are negative
        if p["type"] == "Pods":
            limits.append(start - p["value"])
        else:
            limits.append(math.floor(start * (1 - p["value"] / 100)))
    if rules.get("selectPolicy") == "Disabled":
        return current
    # Max = allow the biggest change = the lowest floor
    return max(limits) if rules.get("selectPolicy") == "Min" else min(limits)


def simulate(spec: Spec, load: list[float], capacity_per_pod: float,
             start_replicas: int | None = None, startup_seconds: int = 0) -> list[dict]:
    """Replay `load` (one value per sync period) and return one row per sync."""
    replicas = start_replicas or spec.min_replicas
    pending: deque = deque()            # (ready_at, count)
    recs: deque = deque()               # (time, recommendation)
    events: deque = deque()             # (time, +added / -removed)
    rows = []
    for i, demand in enumerate(load):
        now = i * SYNC_SECONDS
        while pending and pending[0][0] <= now:
            pending.popleft()
        not_ready = sum(c for _, c in pending)
        ready = max(1, replicas - not_ready)
        util = 100.0 * demand / (ready * capacity_per_pod)

        rec = raw_recommendation(ready, util, spec.target_utilization) + not_ready
        recs.append((now, rec))
        window = max(spec.scale_up["stabilizationWindowSeconds"], spec.scale_down["stabilizationWindowSeconds"])
        while recs and now - recs[0][0] > window:
            recs.popleft()
        up_window = [r for t, r in recs if now - t <= spec.scale_up["stabilizationWindowSeconds"]]
        down_window = [r for t, r in recs if now - t <= spec.scale_down["stabilizationWindowSeconds"]]
        up_rec, down_rec = min(up_window), max(down_window)
        desired = up_rec if replicas < up_rec else down_rec if replicas > down_rec else replicas

        if desired > replicas:
            desired = min(desired, scale_up_limit(replicas, spec.scale_up, events, now))
        elif desired < replicas:
            desired = max(desired, scale_down_limit(replicas, spec.scale_down, events, now))
        desired = max(spec.min_replicas, min(spec.max_replicas, desired))

        if desired != replicas:
            events.append((now, desired - replicas))
            if desired > replicas and startup_seconds:
                pending.append((now + startup_seconds, desired - replicas))
            if desired < replicas:
                # Remove not-ready pods first, as a ReplicaSet scale-down prefers them
                drop = replicas - desired
                while drop and pending:
                    t, c = pending.pop()
                    take = min(c, drop)
                    if c - take:
                        pending.append((t, c - take))
                    drop -= take
        rows.append({"t": now, "load": demand, "ready": ready, "utilization": round(util, 1),
                     "recommendation": rec, "replicas": desired})
        replicas = desired
        while events and now - events[0][0] > 1800:
            events.popleft()
    return rows


def summarise(rows: list[dict], target: int) -> dict:
    over = sum(1 for r in rows if r["utilization"] > 100)
    return {
        "peak_replicas": max(r["replicas"] for r in rows),
        "replica_minutes": round(sum(r["replicas"] for r in rows) * SYNC_SECONDS / 60, 1),
        "syncs_over_100pct": over,
        "syncs_over_target": sum(1 for r in rows if r["utilization"] > target),
        "changes": sum(1 for a, b in zip(rows, rows[1:]) if a["replicas"] != b["replicas"]),
    }

On two synthetic traces the trade-off shows clearly. With a two-minute dip between spikes and 45-second pod start-up, the default already keeps its pods through the dip (its 300-second window is longer), so the tuned version costs about 50 per cent more replica-minutes for no benefit. With a seven-minute dip and two-minute start-up, the default scales down and then spends 17 syncs saturated on the second spike; the tuned version spends 8. Run your own traces through it before choosing.

TraceBehaviorReplica-minutesSyncs saturatedSyncs over target
2-min dip, 45 s start-upDefault79248
2-min dip, 45 s start-uphpa.yaml1,20748
7-min dip, 120 s start-upDefault9631726
7-min dip, 120 s start-uphpa.yaml1,450816
💡
SCALE UP FAST, SCALE DOWN ON EVIDENCE

A short or zero scale-up window with generous policies limits saturation; the cost is a few extra pods for a few minutes. Lengthen the scale-down window only when traces show load returning soon after it falls, or when pods take long to start.

How HPA fits with the other autoscalers

HPA, VPA, Cluster Autoscaler or Karpenter, and KEDA compared on what they change, their signal, reaction time and what to watch out for
HPA changes replicas, VPA changes requests, the node autoscaler changes nodes and KEDA feeds events into an HPA.

HPA adds pods; a node autoscaler (Cluster Autoscaler, or Karpenter on AWS) adds nodes when those pods cannot be scheduled. They need each other, and both depend on accurate resource requests. KEDA builds on HPA rather than replacing it: it adds scaling to and from zero and event sources such as queues and streams. VPA sets requests, which is the input HPA utilisation targets depend on.

⚖️
RESILIENCE AND CAPACITY LIMITS

Set minReplicas to at least one pod per zone for services with availability targets, and pair the HPA with a PodDisruptionBudget so voluntary disruptions do not take replicas below what the service needs. Set maxReplicas to what downstream dependencies (databases, licences, third-party rate limits) can take, and alert when the HPA sits at its maximum.

📝
HOW THIS WAS CHECKED

hpa.yaml was validated against the Kubernetes 1.29 OpenAPI schema with kubernetes-validate==1.29.1 in strict mode. hpa_sim.py was tested with Python 3.13, pyyaml==6.0.1 and pytest==8.0.2: fourteen tests cover tolerance, default and custom rate limits, selectPolicy Min, Max and Disabled, the 300-second default window and a spike with start-up delay. The table comes from the sample’s example.py on synthetic traces; the simulator simplifies not-ready pod handling and has no metric lag, so it shows the shape of the behaviour rather than exact replica counts. Nothing was run on a cluster.

Key takeaways

  • The HPA recalculates every 15 seconds: ratio with a 10 per cent tolerance, stabilisation, rate limits, then min and max.
  • Utilisation is measured against requests; set them accurately or the HPA scales on noise.
  • Scale on CPU where CPU is the bottleneck; otherwise use requests, in-flight work or queue depth through custom or external metrics.
  • The default behavior scales up quickly and waits five minutes to scale down; change it only when your traces show why.
  • HPA needs a node autoscaler underneath it, and must not share a CPU or memory signal with VPA in update mode.

What did tuning behavior change for your services? Share your settings and traffic shape in the comments.

Glossary

HPA
HorizontalPodAutoscaler: changes a workload’s replica count based on metrics.
Sync period
How often the HPA controller evaluates each HPA; 15 seconds by default.
Tolerance
The band around the target (10 per cent by default) within which the HPA does not scale.
Stabilisation window
The period of past recommendations the HPA considers before scaling, to avoid flapping.
Scaling policy
A limit on how many pods (or what percentage) may be added or removed per period.
VPA
Vertical Pod Autoscaler: recommends or applies CPU and memory requests.
KEDA
Kubernetes Event-driven Autoscaling: scales on event sources and manages an HPA.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.