GPU Memory Fragmentation in LLM Serving: Key Drivers

The main culprit is usually not the model weights. Weights are loaded once and remain relatively stable during serving. The unstable part is the KV cache, which grows with active sequences, changes as requests arrive and finish, and is often reserved according to a worst-case max_seq_len rather than actual usage. In traditional serving systems, this combination can waste 60% to 80% of GPU memory through over-reservation and fragmentation.

The practical question behind the phrase GPU memory fragmentation in LLM serving factors is therefore not simply “How much VRAM does this model need?” It is: how is KV cache memory allocated, how long does each allocation live, and can the serving engine reuse the remaining space without requiring one large contiguous region?

The anatomy of KV cache over-reservation

During autoregressive generation, the model does not recompute the full attention history from scratch for every new token. It stores keys and values from previous tokens in the KV cache. Each active request adds to that cache as generation proceeds.

That sounds straightforward until we size a real deployment.

A serving engine needs memory for several components at the same time:

• model weights, which occupy a mostly fixed allocation after startup;
• temporary activations used during each forward pass;
• CUDA runtime and framework allocations;
• communication buffers, especially in tensor- or pipeline-parallel deployments;
• the KV cache for every active sequence;
• workspace memory used by attention and matrix multiplication kernels.

The KV cache is the difficult component because its footprint depends on traffic. A request with a short prompt and a short completion consumes far less cache than a request that reaches the configured maximum sequence length. If the engine reserves a contiguous buffer for the maximum in advance, the unused tail remains unavailable to other requests for the lifetime of that allocation.

This is internal fragmentation. The memory belongs to the request, but part of it is empty.

A useful way to reason about the problem is to separate the advertised sequence limit from the live token count. If a request is configured for 8,192 tokens but currently needs space for 1,100, the allocation policy determines whether the remaining capacity can be reused. Under static contiguous allocation, often it cannot. Under block-based allocation, it can be assigned elsewhere.

The cost per token is not trivial for larger models. For Llama-3-70B in FP16, the total KV cache cost across all layers is approximately 160 KB per token. At 2,048 tokens, one active sequence can therefore require hundreds of megabytes of KV memory before accounting for batching and other runtime overheads. As concurrency rises, a conservative allocation strategy compounds quickly.

For LLaMA-13B in FP16, a single 2,048-token sequence can require about 1.7 GB of KV cache. The exact footprint depends on architecture and serving configuration, but the operational lesson is stable: sequence length and concurrency are first-class capacity variables. Treating them as request metadata rather than memory controls is how capacity plans become optimistic.

The model weights set the starting point. The KV cache decides whether the serving system survives real traffic.

Why maximum sequence length is an expensive default

A large max_seq_len is convenient at the API boundary. It is also a common source of stranded memory.

Suppose an endpoint accepts requests up to 16,384 tokens and the serving engine reserves cache space around that maximum. Most requests may finish at much shorter lengths, but the allocator cannot automatically turn the unused tail of one contiguous request buffer into a usable region for another request. The system has paid for the maximum even when the workload does not use it.

This is especially visible in mixed workloads:

• chat requests with short prompts and short answers;
• retrieval-augmented requests with long input contexts;
• code-generation requests with unpredictable completion lengths;
• summarization jobs that produce relatively stable output sizes;
• streaming requests that keep sequences alive while tokens are delivered slowly.

These workloads create different allocation lifetimes. A short request may finish while a long streaming request remains active. If the freed region does not match the size or position required by the next allocation, the GPU can have enough total free memory but no suitable contiguous block.

The first implementation sanity check is simple: compare the configured maximum sequence length with the actual distribution of prompt, generated, and total tokens. If the endpoint rarely approaches its limit, a static reservation policy is spending memory on an SLA that traffic does not consume.

Internal versus external fragmentation

The two forms of KV cache fragmentation are related, but they produce different symptoms.

Internal fragmentation: empty space inside an allocation

Internal fragmentation comes from reserving more space than a request currently needs. The classic pattern is a contiguous buffer sized from max_seq_len.

The allocation may be valid and physically available, but its unused portion is locked to that request. If the request grows, the reserved space is useful. If it finishes early, the reservation was unnecessary. This is a policy problem: the engine chose predictable per-request capacity over flexible reuse.

Internal fragmentation is often large in systems that allocate memory for the worst case. Across many active sequences, the unused tails can consume more memory than the tokens actually being generated.

External fragmentation: free space in the wrong shape

External fragmentation appears after allocations with different sizes and lifetimes are created and released.

Imagine three requests occupying adjacent regions. The middle request completes, leaving a gap. The next request may need more space than that gap provides, even if the sum of all free gaps is large enough. The allocator cannot place a contiguous allocation across separated regions unless the serving system supports that kind of mapping.

This is where variable output lengths matter. Two requests that arrive together may begin with similar reservations but finish at very different times. Dynamic arrivals then fill some gaps while leaving others stranded. Repeated over a long-running process, the physical layout becomes less useful even though total free memory remains visible in monitoring.

The distinction matters during incident analysis:

This classification prevents a common workaround: raising the GPU memory limit in configuration while leaving the allocation strategy unchanged. More capacity can delay the failure. It does not remove the fragmentation driver.

Why the PyTorch allocator can make OOMs confusing

PyTorch adds another layer between application logic and the CUDA driver. Its caching allocator keeps freed blocks in a pool rather than immediately returning them to the driver through cudaFree. This improves performance by avoiding repeated allocation and deallocation, but it also makes memory readings harder to interpret.

The allocator tracks memory at page granularity—typically around 2 MB. A tensor can be deleted from the application’s perspective while the underlying block remains in the caching pool for reuse. Monitoring may then show a difference between:

• memory actively allocated to live tensors;
• memory reserved by PyTorch;
• memory reported as free by the CUDA driver;
• memory that can satisfy the next allocation request.

Those numbers answer different questions.

A CUDA OOM can occur even when a dashboard appears to show free memory because the next operation needs a block with a shape or placement that the allocator cannot provide. The issue may be the KV cache policy, the framework pool, or both. We should avoid treating every OOM as proof that the model simply does not fit.

A practical debugging pass

When an inference worker starts failing, collect memory information at several points rather than only during startup:

1. After model initialization.

This gives the baseline occupied by weights, runtime state, and static workspaces. Since model weights are fixed after loading, a large increase here usually points to initialization behavior rather than request fragmentation.

2. After a low-concurrency request.

Record allocated and reserved memory after a short request and a longer request. The difference shows how the serving path grows cache usage.

3. After requests complete.

If allocated memory falls but reserved memory remains high, the caching allocator is retaining blocks. That is not automatically a leak, but it changes how much memory is immediately available for future allocations.

4. After a traffic mix with varied sequence lengths.

This is the useful stress case. Constant-size synthetic requests can hide external fragmentation because allocations have similar lifetimes and shapes.

5. At the exact failing operation.

Capture the requested allocation size, active sequence count, current KV cache usage, and allocator statistics. “GPU memory was 92% full” is not enough to identify the failure mode.

The workaround is not to call cache-clearing functions after every request. That can introduce synchronization and latency costs while leaving the underlying request-allocation pattern untouched. We want to understand whether the problem is reservation, fragmentation, concurrency, or a genuinely insufficient capacity budget.

Separate allocator behavior from serving behavior

A clean diagnosis keeps two layers distinct.

The serving engine decides how KV blocks are assigned to sequences, how batches are formed, and when requests are admitted. PyTorch and CUDA decide how those underlying allocations are requested, cached, and released. A better framework-level allocation policy can reduce application-level fragmentation without eliminating every allocator-level effect.

Conversely, changing allocator settings cannot compensate for a serving engine that reserves a large contiguous KV buffer for every request. That would be fixing the plumbing while leaving the pipe incorrectly sized.

This separation is also useful when operating across multiple teams. The platform team owns GPU pools and process lifecycle. The inference team owns sequence limits, batching, and cache policy. The application team may own prompt construction and output caps. A production OOM often crosses all three boundaries.

The same operational principle appears in other infrastructure domains: direct partnerships with global digital payment platforms reduce hidden intermediaries and make ownership clearer. In LLM serving, explicit ownership of cache allocation, admission control, and allocator telemetry provides the same benefit—fewer unexplained gaps between what the application requests and what the GPU can actually supply.

Beyond contiguous memory: how PagedAttention changes the model

PagedAttention, introduced with vLLM in 2023, addresses the central allocation problem by treating KV cache memory as fixed-size blocks rather than one contiguous buffer per sequence.

The sequence still has a logical token order. Its physical KV blocks do not need to be adjacent in GPU memory. A block table maps logical sequence positions to physical blocks, allowing the serving engine to place new blocks wherever suitable memory is available.

This is the same basic idea that makes virtual memory useful in operating systems, but applied to KV cache management. The attention kernel follows the mapping when reading the cached keys and values. The sequence sees a continuous history; the allocator sees reusable fixed-size pages.

That design removes most of the waste caused by both major fragmentation patterns:

• the engine does not need to reserve one maximum-length contiguous slab;
• completed blocks can be returned to a pool and reused by other sequences;
• sequences with different lengths can share the available physical space more efficiently;
• a new request does not need one large free region, only enough free blocks.

Under PagedAttention, memory waste is reduced to under 4%, largely confined to the unused portion of the final block in a sequence. That is a fundamentally different operating point from traditional systems that can waste 60% to 80% through static over-reservation and fragmentation.

The important nuance is that block-based allocation does not make memory unlimited. The KV cache still consumes memory in proportion to active tokens. A workload with too many long-running sequences can exhaust the block pool cleanly and predictably. The improvement is that the available memory is used for actual live tokens rather than stranded inside oversized or badly shaped allocations.

What changes in the serving loop

With contiguous allocation, the serving loop tends to ask:

Can I reserve enough memory for this whole sequence?

With paged allocation, it asks:

How many blocks does this sequence need now, and are enough blocks available?

That change supports continuous batching. Requests can enter and leave the active batch without requiring the entire batch to be rebuilt around a single fixed shape. As one sequence completes, its blocks become available for another request. The scheduler can then trade available blocks against latency, throughput, and fairness.

There is still engineering work around block size, kernel compatibility, prefix sharing, and scheduler behavior. Smaller blocks reduce tail waste but may increase metadata and mapping overhead. Larger blocks reduce bookkeeping but can leave more unused capacity in the final block. The right setting depends on the engine and workload; the core win comes from avoiding large per-sequence contiguous reservations.

Modern block-based systems such as vLLM and SGLang should not be described as suffering from the same high internal fragmentation pattern as traditional contiguous allocators. They can still experience capacity pressure, runtime overhead, or other bottlenecks, but PagedAttention specifically addresses the KV cache fragmentation problem.

Quantifying the cost in a real deployment

A useful memory budget starts with tokens, not requests.

“Thirty concurrent requests” is not a complete capacity statement. Thirty short requests and thirty long-context streaming requests can have very different KV footprints. We need at least four dimensions:

• maximum active sequences;
• prompt tokens per sequence;
• generated tokens per sequence;
• request lifetime and arrival pattern.

The last dimension is easy to miss. A streaming response that generates tokens slowly may hold its KV blocks for much longer than a batch request that finishes quickly. Long-lived connections reduce block turnover and can dominate peak usage even when their token rate is modest.

For a rough first-pass estimate, calculate the live token count across active sequences and multiply it by the model’s KV cache cost per token. Then add static weights, runtime overhead, communication buffers, and headroom. This is not a substitute for a benchmark, but it exposes bad assumptions early.

The 160 KB-per-token figure for Llama-3-70B in FP16 illustrates the scale. At 1,000 active tokens, the KV cache alone is already substantial. At tens of thousands of active tokens, the cache becomes a primary consumer of GPU memory rather than a secondary detail.

For LLaMA-13B in FP16, the approximately 1.7 GB required by a single 2,048-token sequence is another useful warning. If a capacity plan assumes that a 24 GB GPU can host many such sequences after loading the model, it may be ignoring the cache entirely. The weights, KV cache, temporary workspaces, and framework overhead all compete for the same device.

The controls that actually move the number

The most effective controls are operational rather than cosmetic:

1. Set sequence limits from observed traffic.

Do not make max_seq_len a symbolic promise that every request can consume unlimited memory. Split endpoints by workload where necessary—short chat, long-context retrieval, and batch summarization rarely deserve identical limits.

2. Use block-based KV cache allocation.

This is the direct fix for contiguous-buffer waste. It allows physical memory to be reused across sequences with different lengths and lifetimes.

3. Limit active tokens, not only request count.

A request-count limit treats a 200-token interaction and a 16,000-token interaction as equivalent. Token-based admission control is closer to the resource being consumed.

4. Measure prompt and generation lengths separately.

Long prompts increase the initial cache footprint. Long generations keep adding tokens and extend the allocation lifetime. The two patterns can require different scheduling policies.

5. Track allocated, reserved, and usable memory.

A single GPU utilization percentage does not explain allocator state. Export KV blocks in use, free blocks, active sequences, tokens per sequence, and failed allocation sizes.

6. Test with churn.

Mix short and long requests, vary completion lengths, and include cancellations and streaming. A static benchmark with uniform sequence sizes is a poor test for fragmentation.

7. Keep headroom for temporary workspaces.

Filling the device with KV blocks can maximize nominal throughput while leaving no room for a kernel workspace or a larger-than-usual batch. The result is an OOM at the worst possible point—under live traffic.

8. Consider quantization as a capacity lever, not a fragmentation fix.

Lower-precision weights can free space for KV cache and concurrency. They do not, by themselves, solve poor KV allocation. If the engine still reserves contiguous maximum-length buffers, the same pattern remains.

Monitoring signals for production

A production dashboard should make the allocation story visible before users see a 500 error. At minimum, correlate:

• GPU allocated memory;
• GPU reserved memory;
• KV cache blocks used and available;
• active sequence count;
• total live tokens;
• prompt-token and generation-token distributions;
• request duration and time to first token;
• allocation failures and worker restarts;
• batch size or scheduler token budget.

The useful alert is not simply “GPU memory above 90%.” It is closer to: free KV blocks are falling while active tokens remain below the expected capacity envelope, or reserved memory is growing after requests complete, or allocation failures are concentrated after high-churn traffic.

That correlation points us toward a specific workaround. If live tokens are low but blocks are unavailable, inspect block release and allocator reuse. If live tokens are high, tighten admission control or reduce sequence limits. If reserved memory is high but allocated memory is low, inspect the framework pool and process lifecycle. If the model barely fits at startup, no allocator policy will create the headroom needed for production variance.

A deployment sanity check

Before shipping an LLM endpoint, we should be able to answer these questions with metrics rather than guesses:

• How many KV-cache bytes does one token consume for this model and precision?
• What is the maximum live-token budget per GPU?
• Does the engine allocate contiguous buffers or fixed-size blocks?
• What fraction of requests approach the configured sequence limit?
• How much memory remains for temporary kernels at peak concurrency?
• Do cancellations and streaming requests release cache blocks promptly?
• Can a mixed-length load test reproduce the same allocator state as production?
• Which team owns the response when the process reports free memory but allocation fails?

The last question matters because GPU OOMs are often treated as isolated application bugs. In practice, they can emerge from the interaction between model configuration, scheduler policy, framework caching, and traffic shape. A clean incident report should identify the layer that ran out of usable capacity.

The practical conclusion

The key drivers of GPU memory fragmentation in LLM serving are not mysterious. Traditional systems reserve KV cache memory for maximum sequence lengths, store it in large contiguous regions, and then struggle when request sizes and lifetimes diverge. Internal fragmentation leaves unused space inside active reservations. External fragmentation leaves free space split into regions that cannot satisfy the next request. PyTorch’s caching allocator can make the situation harder to read by retaining freed blocks and exposing several different definitions of “free” memory.

The fix is to manage KV cache as a dynamic, block-based resource. PagedAttention reduces waste to under 4% by separating logical token order from physical placement. Token-aware admission control, realistic sequence limits, mixed-length load tests, and allocator-level observability complete the solution.

If a serving system fails with a CUDA OOM while monitoring still shows free VRAM, start with the allocation shape—not the headline capacity. Check how many live tokens exist, how KV blocks are assigned, what PyTorch has reserved, and whether the next request needs one contiguous region. That path usually leads to a concrete engineering fix instead of another round of blindly adding GPUs.