GPU Memory Footprint: Formula for LLM Deployment

Long context windows and concurrent requests can make the KV cache larger than the weights allocated to a single request.

The practical GPU memory footprint estimation formula is:

Total VRAM ≈ Model Weights + KV Cache + Activation Memory + Framework Overhead

For production sizing, a further 10–20% headroom is typically added for dynamic activations, workspace buffers, memory fragmentation, and runtime overhead. The calculation is therefore not a file-size exercise. It is a capacity model for a changing workload.

Deconstructing the VRAM budget: static and dynamic allocation

The first distinction is between memory that remains mostly fixed and memory that expands with traffic.

Model weights are static for a given checkpoint and precision. A 70-billion-parameter model stored in FP16 requires approximately 140 GB for weights alone:

70B parameters × 2 bytes = 140 GB

That figure excludes the memory required to process tokens. The inference engine still needs space for intermediate activations and the attention state associated with every active sequence.

The KV cache is dynamic. Its size increases with:

• Context length per request.
• Number of concurrent sequences.
• Number of transformer layers.
• Number of key-value heads.
• Attention head dimension.
• KV-cache precision.
• The serving architecture and its memory-management strategy.

This distinction changes the deployment decision. A model may fit on a GPU for a short single-request test and fail under production concurrency. The checkpoint has not changed. The workload has.

Model weights determine the static floor. The KV cache determines how quickly that floor becomes operationally irrelevant.

A deployment estimate should therefore separate at least four components:

The estimate becomes less reliable when these components are collapsed into one number. A model card may report a checkpoint size. That is not a serving requirement.

Quantifying model weights from FP16 to INT4

The weight-memory calculation is direct:

Model weight memory = Parameter count × bytes per parameter

The precision determines the byte count. The basic comparison is:

These values describe the weights only. They do not include KV cache or runtime allocations.

FP16 and BF16 use two bytes per parameter. They remain common because modern accelerators provide optimized matrix operations for these formats, and because they generally preserve model behavior better than more aggressive quantization. The memory cost is still substantial for large checkpoints.

FP8 and INT8 reduce the static weight allocation by half relative to FP16. The practical result depends on the implementation. Quantization support must exist in the model-serving framework, GPU kernels, and execution path. A nominally smaller checkpoint does not automatically produce a proportionally smaller end-to-end deployment footprint.

INT4 and AWQ reduce weight storage to approximately half a byte per parameter. A 70B checkpoint therefore requires roughly 35 GB for weights before runtime overhead. That can make single-GPU deployment possible where FP16 cannot fit. It does not remove the KV-cache constraint. A long-context, high-concurrency workload can still exceed the remaining VRAM.

A compact transformer model memory footprint calculation should preserve the distinction between parameter efficiency and usable serving capacity. Quantization improves the static term. It does not eliminate dynamic memory growth.

A worked weight example

Consider a 13B model:

• FP16/BF16: 13B × 2 bytes = approximately 26 GB.
• FP8/INT8: 13B × 1 byte = approximately 13 GB.
• INT4/AWQ: 13B × 0.5 bytes = approximately 6.5 GB.

A GPU with 24 GB of VRAM may appear suitable for the INT4 version. That conclusion is incomplete. The available capacity after loading the weights must also cover KV cache, activations, CUDA workspaces, and framework overhead. At meaningful context lengths or concurrency, the usable margin can disappear quickly.

The KV cache scaling problem

The KV cache stores the key and value tensors generated by the attention layers for tokens already processed. It prevents the runtime from recomputing the full attention history for every new token.

The per-token memory formula is:

KV cache bytes per token = 2 × number of layers × number of KV heads × head dimension × precision bytes

The factor of two accounts for the key and value tensors.

For Llama 3.1 70B in FP16, the documented configuration is:

• 80 layers.
• 8 KV heads.
• 128 dimensions per head.
• 2 bytes per value.

The calculation is:

2 × 80 × 8 × 128 × 2 = 327,680 bytes per token

That equals approximately 0.33 MB per token. For a full 131,072-token context window, one request consumes approximately 42.95 GB of KV-cache memory.

This is the central reason long-context inference changes the hardware calculation. The model weights remain fixed. The cache grows with the sequence.

The request-level estimate is:

KV cache per request = bytes per token × active tokens

For multiple concurrent sequences:

Total KV cache = bytes per token × active tokens per request × number of concurrent requests

The concurrency multiplier is not optional. Ten requests at the same context length require roughly ten times the request-level KV allocation, before accounting for batching behavior and allocator details.

A deployment with a 131,072-token maximum context does not necessarily allocate the full 42.95 GB for every request at startup. The exact behavior depends on the serving engine and cache-management policy. However, capacity planning must account for the maximum active token volume the system is expected to support. A theoretical context limit becomes an operational memory requirement when traffic reaches it.

Why model size alone fails

Suppose the 70B model above is quantized to INT4. The weights consume approximately 35 GB. That leaves substantial nominal capacity on a high-memory accelerator. But an FP16 KV cache for a long sequence can consume more memory than the quantized weights. If the runtime stores the cache at a different precision from the weights, the memory reduction from quantization applies primarily to the static model term.

This produces several non-obvious outcomes:

1. Quantizing weights does not proportionally reduce KV-cache memory.

The cache precision and architecture determine its own allocation.

2. Grouped-query or multi-query attention changes the cache term.

The number of KV heads appears directly in the formula. Fewer KV heads reduce per-token cache consumption.

3. A larger context window increases memory linearly.

Doubling active tokens doubles the KV-cache requirement under the same architecture and precision.

4. Concurrency and context length compound each other.

Doubling both produces approximately four times the cache demand.

5. Batch size is not a sufficient proxy for cache usage.

Requests can have different prompt lengths and generation lengths. Active token count is the more informative variable.

The KV-cache memory size formula is linear in active tokens, but production traffic is not uniform. Capacity must be sized for the token distribution, not the shortest benchmark prompt.

PagedAttention-style implementations and continuous batching can improve utilization by managing the cache in blocks and admitting requests dynamically. They do not violate the underlying memory requirement. They reduce waste and improve scheduling. They do not make the stored keys and values free.

Fine-tuning requires a different memory model

Inference and fine-tuning should not use the same GPU VRAM calculator. Fine-tuning introduces gradients and optimizer states, which can dominate the memory budget.

For full fine-tuning in Float32 with AdamW, the static memory baseline is:

• Model memory: 4 bytes per parameter.
• Gradient memory: 4 bytes per parameter.
• Optimizer memory: 8 bytes per parameter.

The total is:

4 + 4 + 8 = 16 bytes per parameter

This is the baseline before activation memory.

For a 70B-parameter model:

70B × 16 bytes = approximately 1.12 TB

That figure is not a complete training requirement. Activations, temporary buffers, data-parallel replicas, communication storage, and framework overhead are additional. It does show why a full fine-tuning job cannot be planned from the inference checkpoint size.

The optimizer term is particularly costly. AdamW maintains additional state for parameter updates. The model may therefore require several times more memory during training than during inference, even before sequence length and batch size are considered.

Parameter-efficient fine-tuning methods can change the calculation by freezing most base weights and training only a smaller set of parameters. The exact savings depend on the method and implementation. LoRA-style adapters, for example, reduce trainable parameter and optimizer-state memory, but the frozen base model still must be loaded, and forward-pass activations remain part of the budget.

Activation memory also behaves differently during training. The runtime may retain intermediate tensors for backpropagation. Sequence length, microbatch size, checkpointing strategy, and model architecture all affect this term. A training configuration that fits at a short sequence length can fail when the context window is expanded without any change to the checkpoint.

Tensor parallelism and per-GPU capacity

Eight-way tensor parallelism partitions the model weights across GPUs. It also partitions the KV cache according to the assigned attention heads. This spreads the static and cache allocations across the tensor-parallel group.

The per-GPU calculation is not simply the total model memory divided by eight in every implementation. Communication buffers, replicated components, uneven head partitioning, and runtime workspace allocations create deviations. Activation memory also remains a separate term. The supplied deployment baseline places activation memory around 25% of the per-GPU partitioned weight size in an eight-way tensor-parallel configuration, but that should be treated as an estimate rather than a universal constant.

Tensor parallelism solves capacity constraints by using more devices. It introduces computational overhead through inter-GPU communication. The relevant trade-off is therefore not only whether the model fits, but whether the communication fabric can sustain the target latency and throughput.

NVLink-class interconnects and PCIe-only configurations can produce materially different serving behavior. A memory plan that is valid at the capacity level may still be operationally poor if cross-device synchronization becomes the dominant latency component.

Operational headroom and framework overhead

The formula becomes useful only when it includes memory that is not visible in the model architecture.

Serving runtimes allocate workspace buffers for matrix multiplication, attention kernels, quantized operations, tensor-parallel communication, and graph execution. CUDA allocators may reserve blocks that are not immediately occupied. Fragmentation can prevent a new allocation even when the reported free-memory number appears sufficient.

A standard planning rule adds approximately 10–20% headroom for dynamic activations, workspace buffers, and CUDA or framework overhead:

Required VRAM ≈ Model Weights + KV Cache + 10–20% operational headroom

This expression is a practical approximation. It should not be interpreted as a replacement for runtime profiling. The exact static CUDA allocation depends on the driver, framework version, kernel selection, allocator, and model implementation. Customized non-PagedAttention KV-cache systems can also exhibit implementation-specific fragmentation.

The headroom requirement changes with the serving mode:

• Single-request, short-context inference has a relatively small dynamic component.
• Continuous batching improves throughput but increases the number of active cache blocks.
• Long-context generation shifts the budget toward KV cache.
• Quantized inference reduces static weight memory but may require specialized workspaces.
• Tensor-parallel serving adds communication buffers and synchronization overhead.
• Fine-tuning adds gradients, optimizer states, and retained activations.

A deployment plan should track peak allocation, not the average allocation. Average utilization can conceal outlier prompts, burst concurrency, and generation queues that trigger an out-of-memory failure.

A practical estimation sequence

The following sequence is sufficient for an initial capacity model:

1. Identify the parameter count and weight precision.

Multiply parameters by 2 bytes for FP16/BF16, 1 byte for FP8/INT8, or 0.5 bytes for INT4/AWQ.

2. Calculate per-token KV-cache consumption.

Use layers, KV heads, head dimension, and cache precision. Do not substitute total attention heads if the architecture uses grouped-query attention.

3. Estimate active tokens.

Include prompt tokens already resident in the cache and generated tokens retained by the runtime.

4. Multiply by concurrent sequences.

Use expected peak concurrency, not the average request rate.

5. Add activation and framework overhead.

Apply the 10–20% rule as an initial headroom estimate, then validate the result through profiling.

6. Account for partitioning.

For tensor parallelism, distribute weights and cache across the configured GPUs while reserving memory for communication and non-partitioned components.

7. Test the failure boundary.

Benchmark the longest relevant context and the highest expected concurrency. A short prompt benchmark is not a capacity test.

The output should be expressed as a range rather than a single falsely precise number. Model architecture and precision provide deterministic inputs. Runtime allocation and workload shape do not.

From formula to deployment decision

A GPU VRAM calculator for inference is useful only if it exposes its assumptions. At minimum, it should accept parameter count, weight precision, layer count, KV-head count, head dimension, cache precision, context length, and concurrency. A tool that requests only model size cannot estimate the dynamic cache term.

The main deployment configurations produce different bottlenecks:

This also affects model-serving framework selection. A runtime with efficient paged cache management may handle variable-length requests more effectively than one that reserves large contiguous buffers. A framework optimized for static batch shapes may deliver predictable performance but waste memory on irregular traffic. The architectural trade-off is between utilization, latency variance, implementation complexity, and peak capacity.

The benchmark should report more than tokens per second. A credible evaluation includes:

• Peak VRAM allocation.
• Context length.
• Number of concurrent sequences.
• KV-cache precision.
• Weight precision.
• Batch or continuous-batching policy.
• Time to first token.
• Inter-token latency.
• Out-of-memory behavior under peak load.

Without these variables, throughput numbers are difficult to compare. A high tokens-per-second result at a short context and a single request says little about production capacity.

Limitations of the estimation model

The formula provides a planning baseline, not an exact allocation trace.

It does not determine the precise CUDA memory reservation for a specific driver and runtime combination. It also cannot predict the exact fragmentation overhead of a proprietary cache implementation without profiling. Kernel selection can change workspace requirements. Quantization libraries can use temporary buffers that are absent from the checkpoint representation. Tensor-parallel layouts can produce uneven allocations.

The formula also assumes that the relevant memory terms are measurable independently. In practice, frameworks may fuse operations, reuse activation buffers, preallocate cache pools, or reserve memory for future batches. The observed peak can therefore differ from the arithmetic sum.

Those limitations do not make the calculation unnecessary. They define its proper use. The estimate narrows the hardware search and identifies the dominant term. Runtime profiling then verifies the boundary under the actual serving workload.

Conclusion

LLM GPU memory planning starts with model weights but cannot end there. The correct baseline is:

Total VRAM = Model Weights + KV Cache + Activation Memory + Framework Overhead

Weight memory scales with parameter count and precision. KV-cache memory scales with layers, KV heads, head dimension, cache precision, active tokens, and concurrency. Fine-tuning adds gradients and optimizer states, reaching a baseline of 16 bytes per parameter for Float32 with AdamW before activation memory.

The operational decision is determined by the largest active term. For short-context inference, weights may dominate. For long-context or concurrent serving, the KV cache can become the limiting resource. For full fine-tuning, optimizer and gradient memory usually redefine the hardware requirement.

A GPU is therefore adequate only when it fits the model, the workload, and the runtime overhead at peak conditions. Anything less is checkpoint accounting, not deployment sizing.