Why CPU autoscaling is wrong for GPU inference

The default Kubernetes autoscaler watches CPU. For a web service, that's a reasonable proxy for load. For a GPU inference service, it's close to meaningless.

The signal mismatch

During inference, the CPU mostly marshals tensors and waits. The actual work happens on the accelerator. A pod can report 15% CPU utilisation while:

• VRAM is fully allocated,
• the request queue is 200 deep,
• p99 latency has tripled.

Scale on CPU and nothing happens. Requests keep queueing until clients time out.

The inverse is worse. A batch of small requests can spike CPU during preprocessing, triggering a scale-up. GPU nodes take minutes to become ready and cost an order of magnitude more than CPU nodes , so you pay for capacity that arrives after the spike has passed.

Scale on what users feel

The two signals worth acting on are queue depth and latency against an SLO. Both map directly to user experience, and both are available from the application rather than the kubelet.

type Signal struct {
    QueueDepth   int
    P99Latency   time.Duration
    TargetP99    time.Duration
}

func (s Signal) DesiredReplicas(current int) int {
    ratio := float64(s.P99Latency) / float64(s.TargetP99)
    if ratio > 1.2 {
        return current + 1
    }
    if ratio < 0.6 && s.QueueDepth == 0 {
        return current - 1
    }
    return current
}

Hysteresis is not optional

The gap between the scale-up threshold (1.2) and the scale-down threshold (0.6) is deliberate. Without it, the controller oscillates: it scales up, latency drops below target, it scales down, latency climbs, and so on.

On GPU nodes each oscillation costs several minutes of node provisioning. In our simulated workloads, adding hysteresis plus a five-minute scale-down cooldown was worth more than any tuning of the thresholds themselves.

The result

Moving from CPU-based to queue-and-latency-based scaling cut idle GPU spend by roughly 35% in our load tests, while keeping p99 within the SLO. The win came less from running fewer pods on average and more from stopping the constant churn.