Model serving latency spikes: diagnosing Kubernetes bottlenecks

The traffic profile looked ordinary. But P99 went from 800ms to over 60 seconds, and the Horizontal Pod Autoscaler did exactly nothing useful about it. This is what Kubernetes-induced latency looks like when you're running stateful ML inference, and most alerting setups won't catch it until customers do.

I've spent enough nights on call to know: latency in ML serving clusters is rarely the model. It's the substrate. Cgroups throttling, KV cache evictions, kube-proxy routing decisions, control plane LIST saturation — these are the real culprits. Below is the diagnostic playbook I use when P99 jumps and the rotation lights up.

The Hidden Impact of Cgroup CPU Throttling on Inference

CPU throttling in Kubernetes is a well-known foot-gun for traditional microservices. For LLM inference, it's worse. Cgroups enforce CPU limits via a 100-millisecond quota period — that's the kernel's resolution window for CFS bandwidth control. Every 100ms, the kernel checks whether your container has exceeded its CPU quota. If it has, your threads get parked until the next period rolls over.

For a stateless HTTP service, this manifests as jitter — annoying but tolerable. For token-by-token generation in a transformer, it manifests as wall-clock latency on every single request, because the inference loop can't yield mid-decode without breaking batching semantics. A 2-CPU container that gets throttled 40% of the cycle isn't running at 1.2 CPU. It's running at 2 CPU for 60ms, then 0 CPU for 40ms. From the request's perspective, that 40ms gap is pure dead air injected into the critical path.

CPU throttling doesn't reduce throughput linearly. It shreds P99 latency by injecting synchronization stalls directly into the inference loop.

The trap is that container_cpu_usage_seconds_total looks fine. The metric averages over a window long enough to hide the 40ms gaps entirely. You need container_cpu_cfs_throttled_seconds_total and container_cpu_cfs_throttled_periods_total to see it. If the ratio of throttled periods to total periods exceeds 10%, you've already lost the latency budget — you're just waiting for the next traffic burst to surface it as a customer-visible incident.

The fix is usually one of three: raise the CPU limit, drop it entirely, or pin the pod to a dedicated node pool. Half the time, teams set CPU limits they don't need because they copied a Helm chart from 2019.

KV Cache Preemption: Why vLLM Metrics Reveal Latency Bottlenecks

Memory pressure on the KV cache is the LLM-serving equivalent of disk thrashing. vLLM manages KV cache blocks across in-flight requests; when the working set exceeds available GPU memory, blocks get preempted — evicted so other requests can claim them. The preempted request then has to recompute those tokens when it gets scheduled again.

This is tracked by vllm:num_preemptions_total. Watch this metric. When it increments, you're not getting slower inference. You're paying to do the work twice.

The latency signature is brutal. A preempted request that was 200ms into generation doesn't resume at 200ms. It jumps back to whatever token was recomputed, adding hundreds of milliseconds to P99 depending on sequence length. Under sustained memory pressure, preemptions cascade — a request gets preempted, frees its cache, competes for new blocks, gets preempted again. P99 stops being a number and becomes a distribution with a long tail you can't budget around.

The default response is to scale horizontally. But scaling vLLM pods without addressing cache locality just spreads the problem. Each new pod starts cold, fills its cache from zero, and races the other pods for KV blocks via preemption. Total preemptions across the fleet often goes up, not down. You're paying for more GPUs to generate the same tokens twice.

Load Balancing Failures in Stateful Model Serving Architectures

Default Kubernetes Services use kube-proxy in iptables or IPVS mode. For LLM serving, this is a routing disaster.

kube-proxy load balances with random or round-robin selection across pod endpoints. It knows nothing about KV cache state, request payload size, or current pod saturation. A 200-token request and a 4,000-token request have radically different resource profiles, but kube-proxy distributes them as if they were equivalent HTTP GETs.

The cost is cache fragmentation. Request A lands on Pod 1, populates 80% of its KV cache, generates 50 tokens. Request B — similar prompt, similar context — gets routed to Pod 2 because round-robin doesn't care. Pod 2 has a cold cache for this sequence. It recomputes the prefix, burns extra GPU cycles, and pushes P99 higher. Multiply by fleet size and request rate and you've got a measurable performance tax.

The pragmatic fix isn't a sidecar. It's either routing-aware proxies that track KV block occupancy across pods, or engine-level scheduling that knows the cache state before dispatching. Both are non-trivial. Both are necessary at scale. Until you have one, your horizontal scaling budget is buying you extra GPU spend, not extra reliability.

Control Plane Saturation and the Risk of Unintended Pod Restarts

The Kubernetes API server is a hidden dependency for every workload running on the cluster — including yours — even when your pods aren't talking to it directly.

When you have hundreds of pods, dozens of controllers, and aggressive scrape intervals, the API server starts serving LIST requests constantly. kubectl get pods from every CI run, from every dashboard, from every operator. Under sustained load, LIST request latencies climb. P99 for LIST operations can hit 60 seconds during control plane congestion.

This matters because exec-based liveness and readiness probes hit the API server. If your readiness probe is exec and the API server takes 55 seconds to respond, the probe times out, Kubernetes marks the pod NotReady, and traffic stops routing to it. The pod is healthy. The probe infrastructure isn't. You get cascading NotReady states across the fleet with no actual application failure, and your on-call engineer spends the next hour scaling replicas that don't need scaling.

exec-based probes are a control-plane dependency masquerading as an application health check.

The mitigation is mechanical: switch to HTTP or gRPC probes wherever possible, raise timeout thresholds well above API server P99, and decouple probe execution from application code paths. Monitor apiserver_request_duration_seconds filtered by verb=LIST and resource=pods. When that P99 exceeds 5 seconds, you're one deploy away from a fleet-wide restart event — and there's nothing in your model code that will warn you.

Advanced Observability Strategies for Production Inference

The default Prometheus stack — CPU, memory, request rate, error count — is necessary and insufficient. For inference workloads, you need inference-native metrics layered on top of infrastructure telemetry.

Required metrics for any serious LLM serving deployment:

• vllm:num_preemptions_total and per-pod KV cache utilization
• Per-request decode time vs. prefill time distribution
• TTFT (time to first token) and ITL (inter-token latency) percentiles
• GPU memory bandwidth saturation, not just VRAM occupancy
• apiserver_request_duration_seconds for control plane health
• container_cpu_cfs_throttled_periods_total for cgroup pressure

The deeper question is what to alert on. CPU usage alerts are noise. Alert on throttled period ratio crossing 15%. Alert on preemptions-per-minute rising above baseline. Alert on TTFT P99 sliding outside SLO. The SLO is the contract; everything else is diagnostic. A dashboard full of metrics you don't alert on is just decoration.

One more thing I've learned the hard way: tail-latency debugging requires correlated traces. OpenTelemetry spans should cover the entire request path — ingress, routing, pod scheduling, queue wait, prefill, decode, response. Without that, you're guessing which of the four failure modes above is biting you. With it, you can read off the timeline and point at the exact stage that broke.

This kind of strict latency budgeting isn't unique to ML inference. Latency-sensitive automation in other domains — from fraud detection pipelines to AI crypto trading bots and beyond — has treated P99 as a hard contract for years. ML serving infrastructure is still catching up.

The deploy-or-discard test is simple. If your observability stack can't tell you, within five minutes, why P99 just doubled, you don't have observability. You have dashboards. And dashboards don't page you at 3am with the answer.

The Substrate Is the Model

Most ML serving latency incidents are substrate failures, not model failures. The model is the last thing that changed and the last thing that broke. Kubernetes introduces specific failure modes — cgroup throttling, KV cache pressure, stateless load balancing, control plane saturation — that interact with stateful inference in ways the platform was never designed to handle. The default configurations assume HTTP services with bounded request lifetimes and idempotent retries. LLM serving violates every one of those assumptions.

The fix isn't to leave Kubernetes. The fix is to instrument the layers the default stack ignores. Track throttled periods, preemptions, cache locality, and API server health. Treat exec probes as technical debt. Stop trusting kube-proxy for stateful routing. Stop scaling on CPU usage alone — that metric lies.

If your team can't debug a P99 spike without paging the model authors, the gap isn't in the model. It's in the infrastructure telemetry. Fix that first. Everything else — the SLOs, the autoscaler, the cost-per-request, the on-call sanity — follows from there.