Hugging Face Transformers tutorial: measuring model efficiency

A change can improve one while degrading the others. Mixed precision may reduce memory without reducing end-to-end latency. ONNX Runtime may improve a fixed-shape encoder workload while adding no value to an autoregressive generation path.

This Hugging Face Transformers tutorial treats optimization as a measurement problem. The baseline must be fixed. The device, checkpoint, precision mode, batch size, sequence length, decoding configuration, and padding policy must remain constant across variants. Without that control, a reported speedup is not an experimental result.

A model optimization is only valid relative to an identical baseline, identical inputs, and synchronized GPU timing.

Precision timing with PyTorch Benchmark Utils

The most common failure in Hugging Face inference performance metrics is timing CUDA work with a one-off time.time() call around model(**inputs). CUDA execution is asynchronous. Python can record the launch of GPU kernels before those kernels have completed. The resulting number is often a dispatch time, not model latency.

torch.utils.benchmark.Timer is the more appropriate instrument for controlled PyTorch measurements. It performs warmups, manages thread-pool settings for reproducibility, and synchronizes asynchronous accelerator work where necessary. Its blocked_autorange() method repeatedly executes the statement for at least 0.2 seconds by default, selecting a block size that keeps timer overhead below 0.1% of total computation.

A minimal benchmark design for a Transformers model has four constraints:

1. Put the model in evaluation mode. Use model.eval() before inference measurement. Dropout and other training-only behavior must not affect the result.

2. Disable gradient tracking. Run the forward pass inside torch.inference_mode() or torch.no_grad(). Inference-mode execution removes autograd overhead and prevents activation retention that belongs to training, not deployment.

3. Reuse representative device-resident inputs. Tokenization, host-to-device transfer, JSON parsing, and request handling are different components of serving latency. Measure them separately unless the target metric is explicitly end-to-end request latency.

4. Benchmark the actual output path. Encoder classification, token classification, embedding extraction, and text generation have different execution profiles. A forward pass returning logits is not equivalent to model.generate() with beam search, KV-cache updates, and stopping criteria.

For a sequence-classification implementation, the timed statement should contain the model forward call on already prepared tensors. The timer setup should construct the model, move it to CUDA, move input_ids and attention_mask to the same device, and enter inference mode. The benchmark statement then executes only the operation under comparison.

The key result is not a single wall-clock value. Timer produces a distribution. Median latency is generally more stable than the minimum. The interquartile range matters when system-level noise, dynamic GPU clocks, CPU contention, or memory allocation behavior affect the workload.

A practical reporting record should include:

For batch inference, report both batch latency and throughput. If a batch of 32 requests completes in 64 milliseconds, the batch latency is 64 milliseconds while throughput is 500 examples per second. That does not imply every request experiences 2 milliseconds of latency. A production queue, tokenizer cost, batching window, and network transport alter observed service latency.

The distinction is more severe for causal language models. Prefill and decode should be measured independently. Prefill processes the input prompt and typically benefits from parallel matrix operations. Decode produces one or a small number of tokens per iteration and is constrained by repeated kernel launches, attention over the growing context, and memory bandwidth. A benchmark that averages the two into one number hides the actual bottleneck.

Timing pipeline code without contaminating the result

The high-level pipeline() API is useful for application assembly. It is not automatically the right unit for kernel-level benchmarking. A pipeline can include tokenizer execution, preprocessing, output postprocessing, device transfers, and task-specific formatting. Those costs are relevant to end-to-end serving, but they should not be mixed with raw model timing without labeling.

A defensible Hugging Face pipeline optimization workflow uses two measurements:

• Model-only latency: tokenized tensors already on the accelerator; measures the model execution path.
• Application latency: raw text input through tokenizer, model, decoding or postprocessing, and returned Python object; measures user-visible work inside the process.

The delta between them identifies whether optimization effort belongs in CUDA kernels, tokenization, batching, serialization, or generation settings. Optimizing an ONNX graph does not repair a CPU-bound tokenizer stage.

Decoding Hugging Face Trainer memory metrics

The built-in memory report in Trainer is useful, but it is routinely overstated. It does not represent total GPU process consumption. It reports PyTorch-tracked allocations and is limited to rank 0 and GPU 0. In distributed training, a clean-looking rank-0 number says little about memory asymmetry across workers.

Trainer separates two values that should not be conflated:

• Allocated-memory delta: memory still allocated at the end of the measured stage relative to its start.
• Peak-memory delta: the additional temporary allocation required at the high-water mark during that stage.

Adding these values gives the memory needed to complete that measured stage. Reading only the allocated delta can understate a training step whose intermediate activations temporarily consume substantial memory. Reading only the peak delta can omit persistent allocations retained after initialization.

The first CUDA operation is another source of false interpretation. CUDA kernel loading can consume approximately 0.5–2 GB of GPU memory before the model’s steady-state workload is established. That cost may appear in initialization, training, evaluation, or inference depending on when the first accelerator call occurs. It should not be attributed casually to a model layer or a Trainer configuration.

For a custom Trainer loop in PyTorch, memory measurement should distinguish at least three questions:

1. How much memory does the process reserve? This is operational capacity planning. PyTorch allocator behavior matters.

2. How much memory does PyTorch allocate for tensors? This is closer to model, optimizer, gradient, and activation consumption.

3. What is the peak allocation during a representative step? This determines whether a specific batch size will fit without out-of-memory failure.

The useful pattern is to reset peak-memory statistics after warmup and before the measured stage, execute several representative steps, then read peak allocation. A single first step is not a stable training measurement because compilation effects, allocator initialization, kernel loading, and dataloader startup may dominate it.

Trainer memory metrics are stage-local PyTorch allocations, not a complete audit of every byte resident on a GPU.

Why distributed training makes the default report incomplete

Data parallelism, tensor parallel training, and optimizer sharding modify the memory topology. The model may be replicated, parameter shards may be distributed, and activations can remain local to each rank. A rank-0-only statistic cannot establish cluster-wide peak consumption.

For multi-GPU Hugging Face model training scripts, record memory per rank and identify the maximum rather than the mean. The limiting worker determines whether the job fits. This matters especially when input lengths vary, when the final batch differs in shape, or when a pipeline stage has an uneven layer allocation.

External allocations also remain outside Trainer’s narrow view. CUDA extensions, communication libraries, custom kernels, and framework internals can reserve memory that PyTorch’s allocation counters do not describe. The correct conclusion from a Trainer report is therefore bounded: it describes tracked PyTorch memory for the measured process and device, not total device occupancy.

Profiling GPU activity and FLOPs for model bottlenecks

Latency tells the reader that a model is slow. The PyTorch Profiler identifies where the time and memory are spent. It can capture CPU and CUDA activity, tensor allocation and deallocation through profile_memory=True, and estimated FLOPs for selected operations through with_flops=True.

The last qualifier is material. FLOP estimates are available for specific operation classes, including matrix multiplication and 2D convolution. A Transformer workload contains much more than GEMMs: layer normalization, softmax, indexing, embedding lookup, attention-mask handling, data movement, and Python-side logic. A profiler FLOP total is useful for comparing compatible variants. It is not a universal predictor of latency.

A focused profiling pass for a Hugging Face model should answer the following:

• Is time concentrated in matrix multiplications, attention operations, normalization, or output projection?
• Does CPU time indicate tokenizer work, dataloader stalls, Python control flow, or synchronization?
• Are unexpected tensor allocations occurring inside each forward pass?
• Does a candidate optimization reduce a dominant operator or merely move work elsewhere?
• Does generation spend time in prefill, per-token decode, sampling, or cache handling?

Shape recording and stack tracing are powerful diagnostics, but they carry profiling overhead. A trace collected with record_shapes=True and stack traces enabled is not an unprofiled production run. It should be used to locate a bottleneck, followed by a clean timing benchmark to quantify the final effect.

This separation is methodologically necessary. Profiling produces attribution. Benchmarking produces performance numbers. Treating a profiler trace as a final latency measurement confuses instrumentation cost with model cost.

FLOPs are not a latency metric

Two model variants can have similar estimated FLOPs and materially different latency. The causes include kernel fusion, tensor layout, memory bandwidth, cache behavior, sequence shape, GPU architecture, and launch overhead. Small batches and autoregressive decode are particularly resistant to simplistic FLOP-based ranking.

A practical ablation study should therefore include both operator-level and wall-clock evidence:

The architecture must remain stable across the ablation. Changing the checkpoint, tokenizer, maximum length, and precision in one experiment yields an anecdote, not a result.

Memory overheads in mixed-precision training

Mixed precision is often summarized as “half the memory.” That is inaccurate for standard training configurations. Parameters are only one component of memory consumption. Optimizer states, gradients, activations, temporary tensors, and allocator behavior can dominate the final footprint.

For mixed-precision training, the documented accounting baseline is approximately:

This explains why a 4-billion-parameter model is not an 8 GB training workload simply because its FP16 weights occupy roughly 8 GB. Under the stated mixed-precision assumptions, parameter copies, Adam states, and gradients already impose a much larger static footprint. Activations then scale with batch size, sequence length, model depth, and hidden dimension.

The often-cited illustrative configuration—a 4-billion-parameter model trained with mixed precision at batch size 16—requires roughly 85 GB of GPU memory. It is not a universal threshold. It is evidence that training memory cannot be inferred from checkpoint size alone.

Gradient checkpointing: a measurable exchange

Gradient checkpointing reduces activation memory by storing only selected activations during the forward pass and recomputing others during backpropagation. The mechanism is direct. Less retained state lowers memory pressure. More recomputation raises computational overhead.

The expected training-speed penalty is approximately 20%, although the observed number depends on the architecture, checkpoint segmentation, sequence length, hardware, and optimizer configuration. It should be benchmarked as a paired experiment, not enabled by default because an out-of-memory error occurred.

The relevant comparison is not “does checkpointing work?” It does. The relevant comparison is whether the saved memory permits a batch-size increase, a longer context length, or a larger model that compensates for the throughput loss. If it merely adds 20% step time while the original batch already fits comfortably, it is an unfavorable trade.

Other memory interventions should be evaluated with the same discipline:

• Reducing sequence length lowers attention and activation cost, but changes the training distribution.
• Reducing microbatch size lowers peak memory, but can reduce throughput and alter optimization dynamics if accumulation changes the effective batch.
• Optimizer changes can reduce state memory, but affect convergence behavior and implementation complexity.
• Sharding changes per-device capacity, but introduces communication overhead and multi-rank observability requirements.
• Activation offloading can relieve GPU pressure while moving the bottleneck to transfer bandwidth.

None of these is a free efficiency gain. Each reallocates a constraint.

Optimizing inference with ONNX Runtime levels

Exporting a Transformers model to ONNX Runtime is a backend change, not a performance result. The exported graph must first be validated for numerical and functional equivalence on representative inputs. Only then can it enter a benchmark against the original PyTorch path.

Hugging Face Optimum defines four ONNX Runtime optimization levels:

O4 requires CUDA/GPU use. It should not be described as a generic ONNX optimization for CPU inference. Similarly, no optimization level guarantees a speedup. The result depends on the model family, hardware, input shape, provider configuration, export graph, and the exact workload being timed.

Attention fusion also introduces a semantic constraint. The documented defaults assume right padding for BERT-like models and left padding for GPT-like generative models unless use_raw_attention_mask=True is used. Changing padding direction to satisfy a backend assumption without validating outputs is not an optimization. It is a correctness risk.

A robust ONNX evaluation sequence is sequential:

1. Export one fixed checkpoint and preserve the original tokenizer configuration.

2. Build a fixed corpus of representative tokenized inputs, including relevant sequence lengths.

3. Verify output shape and numerical tolerance against the PyTorch baseline.

4. Benchmark PyTorch and ONNX Runtime with the same device, precision, batch size, and inputs.

5. Record median latency, throughput, and peak memory separately.

6. Repeat for O1 through O4 only where the execution provider supports the level.

7. Validate padding and attention-mask behavior after each graph transformation.

For encoder models, static or narrowly bucketed input shapes may allow stronger graph specialization. For variable-length traffic, dynamic axes and realistic padding distributions matter more than an artificial fixed-length benchmark. For decoder-only generation, the exported path must also be evaluated against the generation semantics actually used in deployment. A fast logits-only ONNX forward pass does not establish faster token generation.

The implementation standard: measure the constraint being changed

The useful output of a Transformers library code implementation is not a claim that a model is “optimized.” It is a compact record of what changed and which measured constraint moved.

A baseline table for each experiment should contain the checkpoint, task path, input shape, precision, batch size, device, PyTorch version, export provider where applicable, median latency, throughput, and peak memory. For training, add step time, effective batch size, optimizer, checkpointing state, and per-rank memory. For generation, split prompt processing from token decode.

That record makes failures informative. If ONNX Runtime reduces encoder latency but increases memory, the trade-off is visible. If gradient checkpointing allows the required context length but lowers tokens per second, the cost is explicit. If a profiler identifies attention as dominant while a change only reduces tokenizer time, the optimization has been applied to the wrong layer.

The practical rule is narrow. Synchronize CUDA. Warm up the workload. Fix input shapes and decoding settings. Separate model execution from application overhead. Treat memory counters as scoped measurements. Validate exports before timing them. Then retain only the changes that improve the metric the deployment actually constrains.