LLM inference latency: what the benchmark data shows

Memory bandwidth is the primary bottleneck for autoregressive decoding. Arithmetic intensity falls below 1 FLOP/byte for batch sizes typical of interactive workloads, placing the workload firmly in the memory-bound regime.

Decoding the bottlenecks: memory bandwidth and TTFT

LLM inference decomposes into two phases with fundamentally different compute characteristics. The prefill phase processes the entire input prompt in parallel, exploiting GPU tensor cores to deliver high arithmetic intensity. The decode phase generates tokens one at a time, with each token dependent on the previous one. This serial dependency collapses arithmetic intensity to roughly 1-2 FLOPs per byte of memory transferred, which on modern accelerators such as the NVIDIA H100 means that decode throughput is gated by HBM bandwidth (~3 TB/s on H100 SXM) rather than by raw FLOPs.

KV caching addresses part of this constraint by storing key and value tensors for previously processed tokens, eliminating redundant matrix multiplications. Without KV caching, generating 100 tokens from a 1k-token prompt would require 100x redundant computation. With KV caching, the cost of processing the prompt is paid once; subsequent decode steps read the cached KV tensors from HBM. The cached tensors themselves, however, scale linearly with sequence length and batch size, and for long-context workloads (32k+ tokens) the KV cache can exceed 50% of total VRAM consumption. This makes KV cache management — and not raw weight loading — the dominant memory planning problem for high-throughput inference servers.

The practical implication for TTFT: in single-tenant, low-concurrency deployments, prefill dominates the latency budget. A 7B-parameter model in FP16 requires loading roughly 14 GB of weights from HBM before prefill can begin. Quantizing to INT8 or FP8 reduces this footprint, shortening the model-load component of TTFT. Once prefill completes, the workload transitions to decode, and TTFT becomes irrelevant — the user-facing metric is now inter-token latency and aggregate TPS. The two metrics are governed by different hardware resources and respond to different interventions.

Scaling throughput with continuous batching architectures

Static batching — the original approach to GPU inference — accumulates requests until a fixed batch size is reached, then processes the entire batch through prefill and decode before releasing all responses. For variable-length generation, static batching leaves GPU utilization volatile: short requests finish early and idle while long requests continue generating. Measured efficiency losses in static-batched LLM inference regularly exceed 40%, and the tail latency of the slowest request dictates the response time of the entire batch.

Continuous batching, introduced in the Orca paper (USENIX OSDI 2022), restructured the scheduling loop. The scheduler processes one decode iteration at a time across all active requests, removing completed sequences from the batch and inserting new requests into vacated slots within the same iteration. Throughput gains in the Orca benchmark ranged from 2.4x to 36.7x depending on workload characteristics, with the largest improvements observed on traffic with high variance in output length — precisely the traffic pattern of chat and code-completion workloads.

The architectural successor to Orca's continuous batching is vLLM, which combined continuous scheduling with PagedAttention — a virtual-memory-inspired KV cache management scheme. PagedAttention eliminates the contiguous-memory allocation requirement for KV tensors, reducing fragmentation and allowing higher effective batch sizes. Independent benchmarks on Llama-3-8B deployments report vLLM throughput 14x-24x higher than naive static batching, with the gap widening under bursty traffic. For LLM inference latency benchmarks in 2024-2025, vLLM has become the reference implementation against which other serving frameworks are measured.

The cost-per-token calculation for vLLM vs Triton Inference Server depends on workload shape. vLLM optimizes for token-level throughput and tolerates mixed-length requests well. Triton, designed as a multi-model serving framework, offers concurrent model execution and dynamic batching across heterogeneous model types. For single-model, high-throughput scenarios, vLLM delivers higher TPS per GPU. For multi-tenant deployments serving several models concurrently with strict per-model SLOs, Triton's architecture provides more predictable resource isolation and integrates with standard MLOps pipelines (Prometheus metrics, model versioning, ensemble execution).

The quantization trade-off: balancing precision and speed

Quantization reduces the bit-width of model weights and activations, collapsing the memory footprint and increasing the rate at which parameters can be streamed from HBM. FP8 quantization on H100 GPUs yields measured throughput improvements of approximately 2x over FP16 for inference-bound workloads, driven by the doubled effective bandwidth utilization (2x more weights transferred per byte of HBM capacity). INT8, supported on a broader range of hardware including A100, delivers 1.4x-1.8x throughput improvements depending on the calibration method and the model's sensitivity to weight rounding.

AWQ (Activation-aware Weight Quantization) and GPTQ generalize the technique by protecting salient weight channels from aggressive quantization. Empirical evaluations on Llama-2-7B show AWQ maintaining perplexity within 0.1 of the FP16 baseline at INT4 precision, while naive INT4 quantization degrades perplexity by 1.5-3.0 points. The trade-off is asymmetric: the precision cost is concentrated in the calibration phase, while the throughput benefit applies to every subsequent inference. For high-volume production deployments, the amortized calibration cost becomes negligible within hours of operation.

Quantization is not a free speedup. On workloads dominated by prefill, FP8 delivers the largest gains. On memory-bandwidth-bound decode phases with small batch sizes, dequantization overhead can erase the throughput advantage on older hardware.

Three caveats apply. First, dequantization overhead on older architectures (V100, T4) can offset the memory-bandwidth gains, resulting in neutral or negative net latency. Second, KV cache quantization — quantizing the cached tensors themselves — introduces additional complexity and requires careful handling of attention softmax scaling to avoid numerical instability. Third, batched workloads amortize quantization overhead across many requests, while single-request interactive traffic exposes the full dequantization cost. Production deployments should benchmark on the actual workload distribution rather than on synthetic batch sizes; the same model can show 2x speedup at batch 32 and zero speedup at batch 1 on the same hardware.

The table illustrates the core trade-off curve. Halving the bit-width roughly doubles the throughput ceiling imposed by memory bandwidth, but each step also tightens the constraint on calibration quality. INT4 with AWQ is appropriate for high-volume, latency-tolerant batch jobs; FP8 is the default for interactive serving on H100; INT8 remains the most portable option across A100, L40S, and consumer GPUs.

Accelerating inference with speculative decoding

Speculative decoding addresses the serial nature of autoregressive generation by pairing a large target model with a small draft model. The draft model generates k candidate tokens autoregressively; the target model then verifies all k tokens in parallel using a single forward pass. Tokens that match the target's distribution are accepted; mismatches trigger rollback to the first divergent position. The technique preserves the target model's exact output distribution — a property verified in the original DeepMind and Google Research papers — making it a drop-in replacement for standard decoding.

The 2x-3x latency improvement reported in the speculative decoding literature depends on three variables: draft model size relative to target, token acceptance rate, and the arithmetic intensity of the verification pass. On Llama-2-70B paired with a Llama-2-7B draft model, measured speedups cluster around 2.1x-2.4x for workloads where the draft model achieves 60-80% token acceptance. Below 50% acceptance, the verification overhead exceeds the draft savings, and net latency increases. Acceptance rate correlates with the alignment between draft and target training distributions; out-of-distribution prompts (code, multilingual, structured output) typically yield lower acceptance and smaller speedups.

Production-grade speculative decoding implementations (Medusa, EAGLE, Speculative Streaming) generalize the technique by replacing the separate draft model with learned prediction heads attached to the target model itself. Medusa adds k parallel decoding heads to the transformer, each predicting a token at a future position. The verification pass accepts tokens where the head's prediction matches the target's distribution. Measured speedups on Llama-3-70B with Medusa reach 2.0x-2.8x without the operational overhead of maintaining a separate draft model. EAGLE extends the approach by using the target model's own hidden states as input to the prediction heads, further improving acceptance rates on out-of-distribution prompts.

The infrastructure cost of speculative decoding is non-trivial. Memory bandwidth consumed by the draft model or prediction heads competes with the target model for HBM capacity. For deployments already operating at high batch sizes, speculative decoding delivers diminishing returns, because the target model is already saturating memory bandwidth. The technique shows the largest gains in low-batch, latency-sensitive workloads — exactly the regime where TPS per request is lowest and user-perceived latency dominates. In a typical chat deployment with batch sizes of 4-16, speculative decoding delivers the largest absolute latency reduction; at batch sizes of 64+, the same techniques return marginal improvements.

Optimizing production environments: Triton and beyond

Triton Inference Server's production architecture is built around three primitives: dynamic batching, concurrent model execution, and model ensembling. Dynamic batching accumulates requests over a configurable window (typically 5-50 ms) before dispatching them as a single batch, trading a bounded latency increase for substantial throughput gains. The batching window must be tuned against TTFT SLOs: a 50 ms window adds up to 50 ms to TTFT in the worst case. For real-time conversational workloads with a 200 ms TTFT target, the window is typically capped at 20-30 ms.

Concurrent model execution allows multiple models to share a single GPU with isolated execution contexts. On an A100 80GB, deployments commonly run 2-4 models concurrently (e.g., a 7B primary, a 13B primary, and a guardrail classifier), achieving aggregate GPU utilization above 85% compared to 40-60% in single-model deployments. The throughput penalty per model is small when the models have non-overlapping memory access patterns. Triton also supports backend-specific optimizations — TensorRT for NVIDIA GPUs, ONNX Runtime for cross-platform deployments, vLLM as a Python backend — allowing each model to use the runtime best suited to its architecture.

For cost-per-token calculations, the relevant equation is: (GPU hourly cost × GPU count) / (aggregate TPS × seconds per hour). At an on-demand H100 rate of approximately $2-4/hour and aggregate throughput of 10,000 TPS, the implied cost is $0.06-$0.11 per million generated tokens for the compute component alone. Achieving this cost point requires the full stack: continuous batching, FP8 or INT8 quantization, and speculative decoding in latency-sensitive paths. Removing any single element increases cost by 1.5x-3x. The marginal cost of adding speculative decoding (draft model + verification) is recovered within hours of traffic above a few QPS, but only when batch sizes remain below the threshold where memory bandwidth is already saturated.

Kubernetes for ML deployments adds an orchestration layer on top of these primitives. Triton runs as a stateless service behind a Kubernetes Service, with HPA scaling driven by GPU utilization or queue depth metrics. Model versioning, A/B routing, and canary deployments are handled at the cluster level rather than in the inference server. For distributed training infrastructure that shares the same cluster, careful node-pool separation between training and inference workloads is necessary to prevent training jobs from starving serving pods of GPU memory.

The benchmark data indicates that production LLM inference latency is a multi-variable optimization problem with no single dominant lever. Memory bandwidth sets the ceiling for decode throughput; quantization raises that ceiling at a bounded precision cost; continuous batching fills the available bandwidth more effectively than static scheduling; speculative decoding accelerates individual request latency in low-batch regimes. The empirical work over the past 36 months — Orca's continuous batching in 2022, AWQ and speculative decoding advances in 2023, FP8 hardware support in 2024 — has expanded the achievable throughput-latency frontier by roughly an order of magnitude. The remaining engineering work centers on workload-specific tuning: selecting the batching window, the quantization format, and the speculative decoding configuration that match the request distribution, rather than applying universal defaults. A configuration optimized for chat traffic (low batch, speculative decoding enabled, FP8, short batching window) is materially different from one optimized for batch summarization (high batch, no speculative decoding, INT4 AWQ, long batching window). Production teams that benchmark on synthetic workloads consistently overestimate achievable TPS; teams that benchmark on captured production traffic approach the actual frontier.