What is LLM inference? Five factors driving model performance

Then each subsequent token arrives in tens of milliseconds, one after another, until the model stops. Let's unpack that pipeline — what is LLM inference, and what actually moves the numbers your users feel.

LLM inference is the production-time process of generating output tokens from an input prompt. For autoregressive models, generation proceeds token by token: each new token depends on every token that came before it. That dependency shapes the entire serving stack, because it forces the runtime to keep a live memory of past tokens and reuse it on every step — something training never has to worry about.

Autoregressive Token Generation and the TTFT-ITL Metric Split

Every generation is autoregressive. Token n is conditioned on tokens 1..n-1. Two latency metrics describe the user experience, and they describe different things.

Time to first token (TTFT) measures elapsed time from request submission until the first non-empty output token lands. It bundles queueing time, prompt-prefill processing, and network latency. Longer prompts push TTFT up because the full input sequence has to be processed before the KV cache is built and the first decode step can run.

Inter-token latency (ITL), also called time per output token (TPOT), is the average time between consecutive generated tokens. Output-token throughput per user is 1 / ITL in seconds.

The split matters more than it looks. A deployment can stream tokens fast once it starts and still feel broken because TTFT is two seconds. Another can return the first token in 200 ms and then crawl at 150 ms per token — slower than a person reads. Treat TTFT and ITL as separate instruments on the same dashboard, not as a single "latency" number. Reporting tokens per second alone hides which side of the experience you have actually fixed.

TTFT and ITL are independent levers. Optimizing one does not move the other.

KV-Cache Architectures and the Impact of Prefix Caching

Because each new token reuses attention calculations from earlier tokens, the runtime stores those calculations in a key-value (KV) cache instead of recomputing them. This single structure is the biggest memory consumer in long-context serving — and the biggest lever for throughput once you start hitting real context lengths.

Hugging Face documents three cache flavors with different memory and speed profiles. Dynamic Cache sits in the middle on memory. Static Cache trades higher memory usage for predictable allocation, which matters when you are pinning KV regions up front. Quantized Cache drops memory footprint the most, but there is a gotcha: cache quantization can hurt latency on short contexts when GPU memory is already plentiful, so do not assume "smaller cache = faster" without a sanity check on your actual context-length distribution.

Beyond the cache class itself, prefix caching reuses processed KV blocks when a later request has the same prompt prefix. The wins are concrete — system prompts, repeated few-shot examples, and shared chat templates all hit the same prefix across thousands of requests. vLLM caches only full blocks (no partial reuse) and supports per-request cache salting to isolate reuse in shared or multi-tenant environments. That salting is the workaround for the classic "shared prefix across users leaks cache hits between them" problem — turn it on when tenants should not share hits, leave it off when they should.

Numerical Precision Levers from FP8 to INT4 Weight Quantization

Precision is the second big lever. TensorRT supports INT8, FP8, INT4, and FP4 quantized types and documents quantization as a way to reduce model size, drop memory footprint, and accelerate computation. Each rung down the precision ladder buys memory and throughput, at the cost of numerical headroom.

The representable integer ranges tell you how tight that headroom gets. INT8 spans −128 to 127 — comfortable for most activation distributions. INT4 spans −8 to 7 — eight positive values total. Once you are at INT4, every weight has to be packed carefully.

In TensorRT's explicit quantization path, INT4 is weight-only. Supported per-block sizes are 64 and 128. That means INT4 in this path is not general activation quantization, and your deployment config should not describe it as such. FP4 exists for those who want to push further still, with the smallest memory footprint and the largest accuracy risk on the table.

The trade-off is not free. Reduced precision introduces rounding and clamping errors that affect accuracy. Validate on a held-out set that mirrors your real traffic distribution before you ship FP8 to production, and treat any drop in eval scores as a signal, not noise.

Dynamic Batching and Request Scheduling in Triton Inference Server

At serving time, requests arrive unevenly. Triton Inference Server's dynamic batching combines individual requests into a larger batch before each model execution step — and exposes the controls to tune when that combination happens.

The levers are preferred batch sizes, maximum queue delay, queue size, priorities, and time-outs. The mental model is simple: wait longer, form a bigger batch, get better GPU utilization — but pay in queueing time. Wait too short, and you ship near-empty batches that underuse the GPU and starve the very thing you bought the GPU for.

There is no universal correct setting. Workloads with predictable prompt lengths and steady traffic can hold requests briefly for a fat batch. Bursty, low-latency interactive traffic usually cannot afford the delay and benefits from smaller, more frequent batches.

A practical sanity check: instrument queue depth and average batch size at execution time. If the queue is rarely non-empty, your batcher is too aggressive on wait time; if it is constantly saturated, you are paying queueing tax on every request. Both states are fixable, but the fixes point in opposite directions.

Dynamic batching is a throughput tool with a latency tax. Calibrate the tax to your SLO, not your intuition.

Scaling LLM Fleets with Kubernetes Horizontal Pod Autoscaling

Beyond a single replica, capacity planning becomes fleet management. Triton can run multiple model instances in parallel inside one pod, and Kubernetes Horizontal Pod Autoscaler (HPA) adjusts replica counts up or down based on observed load.

HPA scales on CPU, memory, or custom metrics. When multiple metrics are configured, HPA selects the largest recommended replica count — a sanity point that prevents one conservative metric from capping scaling while another metric is screaming for capacity. The desired replica calculation is:

desiredReplicas = ceil(currentReplicas × currentMetricValue / desiredMetricValue)

Defaults worth knowing: HPA's CPU initialization period is five minutes, and the initial readiness delay is thirty seconds. Plan warm-up and cooldown windows accordingly, because cold-starting an inference replica can take longer than 30 seconds — especially if the model is large and the first request pays the full prefill cost on a freshly loaded weights file.

The Kubernetes documentation uses 60% CPU as an example autoscaling target. Do not lift that number into your LLM serving config as a default. LLMs are token-bound, not CPU-bound — GPU utilization, queue depth, and TTFT/ITL are more honest signals. Wire a custom metric exporter to feed HPA on what actually matters for your workload, and validate scaling decisions against queueing behavior rather than CPU alone.

Putting It Together

Five factors drive LLM inference performance: prompt and output length, GPU and memory footprint with numerical precision, KV-cache strategy and prefix reuse, request scheduling and batching, and capacity scaling across the fleet. None of them wins alone. A clean deployment tunes each one against a measured SLO, not against intuition or borrowed thresholds.

A practical order of operations we have seen work:

1. Establish a TTFT and ITL baseline on real traffic before touching anything.

2. Pick a KV-cache strategy that matches your context-length distribution — and turn on prefix caching if your prompts share prefixes.

3. Quantize the model with the heaviest precision your eval set still passes, then verify on held-out traffic.

4. Calibrate batch wait time against your latency budget, not against throughput alone.

5. Scale on token-level metrics — GPU utilization or queue depth — not on CPU.

That is the checklist. Now ship it, watch the dashboards, and iterate — one knob at a time.