PyTorch tutorial for beginners: tensor operation verification

Loss may even decrease. The implementation is still incorrect.

This PyTorch tutorial for beginners treats tensor verification as an engineering discipline rather than a debugging afterthought. A tensor operation should be tested against an explicit contract: output shape, metadata, numerical tolerance, broadcast behavior, and autograd semantics. Value comparison alone is insufficient.

A tensor is not validated when its numbers look plausible. It is validated when its values, structure, precision, and execution context match the intended contract.

Anatomy of a tensor: shape is only one field

A PyTorch tensor carries more state than a multidimensional array. At minimum, every verification path should inspect:

• shape or size(), which defines the dimensional contract consumed by the next operation;
• dtype, which determines numerical precision and, in some cases, whether gradients can exist;
• device, which determines where the operation executes;
• layout, which matters when code moves beyond ordinary dense strided tensors;
• requires_grad, which determines whether autograd records the operation;
• values, assessed with a comparison rule appropriate to the operation.

The common beginner pattern is to print tensor.shape and proceed. This catches rank errors but misses many failures. A model can accept a (32, 128) float64 tensor when the intended training path uses float32. It can produce a CPU target tensor while logits reside on CUDA. It can silently detach a branch that was expected to contribute to the gradient graph.

A minimal operation contract can be expressed in prose before it is expressed in tests:

1. Define the input metadata. For example: float32, CUDA, rank 2, batch dimension first.

2. Define the expected output shape.

3. Define whether the output should preserve the input dtype and device.

4. Define whether gradient tracking should survive the transformation.

5. Define a numerical reference and an acceptable error bound.

This is the practical core of PyTorch tensor basics. The tensor’s values are only one component of correctness.

Consider a normalization function. Its output may have the correct batch and feature dimensions, yet contain float64 values because a scalar constant was created with a mismatched dtype. The immediate cost is additional computational overhead. The downstream cost can be harder to diagnose: mixed-precision training behaves differently, kernel selection changes, and a strict test may fail only on a particular accelerator.

For a dense training implementation, tensor inspection should be local. Validate immediately after the operation that creates risk: a reshape, reduction, cast, device transfer, mask application, or custom CUDA-adjacent function. Inspecting only final model outputs produces poor fault localization.

Numerical integrity: allclose and assert_close test different contracts

Exact equality is a narrow tool. torch.equal(a, b) returns True only when tensors have the same size and corresponding elements. It does not distinguish dtypes. A float32 tensor and a float64 tensor with equal represented values can pass. It also treats NaN values as unequal, including NaN at the same position in both tensors.

That makes torch.equal() appropriate for discrete or structurally exact results: integer index tensors, Boolean masks, deterministic permutations, and some shape-derived outputs. It is not a general correctness test for floating-point operations.

Floating-point verification requires tolerances. torch.allclose(actual, expected) applies the following condition elementwise:

|actual - expected| <= atol + rtol × |expected|

Its defaults are rtol=1e-05 and atol=1e-08. These values are not a universal correctness threshold. They are defaults. A reduction over a long sequence, a mixed-precision matrix product, and a simple elementwise addition have different error profiles.

torch.testing.assert_close() is usually the stronger default for test code. It checks numerical closeness but can also validate device, dtype, and layout. It raises an AssertionError rather than returning a Boolean, which produces a useful failure boundary in automated tests.

For float32, the default tolerance behavior in torch.testing.assert_close() is stricter than torch.allclose() in relative terms: default rtol is 1.3e-06, with atol=1e-05. Float16 and bfloat16 require looser treatment because their representational precision is lower. The default relative tolerance for float16 is 1e-03; for bfloat16 it is 1.6e-02.

The distinction matters in mixed precision training. A test that uses float32 tolerances for a bfloat16 activation path can reject a valid result. The opposite error is worse: applying bfloat16-scale tolerance to a float32 reference can conceal a regression.

A defensible test has a reference, an expected precision regime, and explicit tolerances where defaults do not represent the operation. For example:

• Elementwise transformations should normally use relatively strict tolerances because error accumulation is limited.
• Matrix multiplication can require a larger tolerance, particularly under reduced precision or different backend kernels.
• Reductions need attention to tensor size and accumulation order. Parallel hardware does not guarantee one fixed summation order.
• Probabilities near zero need a meaningful absolute tolerance. Relative error becomes unstable when the reference value approaches zero.

The test should not state that two outputs are “close enough” without defining close enough. This is the same distinction found in the working process of an equity research analyst: an assertion without an explicit comparison basis is not an analysis result.

Tolerance is part of the model’s numerical specification, not a parameter inserted to silence a failing test.

A practical reference strategy

For simple tensor operations, calculate the expected result from a small hand-auditable input. Small tensors expose indexing and broadcasting errors better than large random arrays.

For more complex functions, use two implementations:

• a production path, potentially vectorized or fused;
• a slow reference path, written for transparency rather than throughput.

Compare their outputs with torch.testing.assert_close(). This is an ablation study at the operation level. It isolates the optimization from the mathematical intent. If the vectorized implementation differs from the transparent reference, the issue is local. There is no need to inspect an entire model training trace.

Randomized tests remain useful, but random inputs should include edge cases: zeros, negative values, repeated values, very small magnitudes, large magnitudes within the intended range, and non-contiguous slices where relevant.

Broadcasting is a shape transformation with constraints

PyTorch follows NumPy-style broadcasting rules for many operations. Dimensions are compared from right to left. Two aligned dimensions are compatible when:

• their sizes are equal;
• one of the sizes is 1;
• one tensor does not have that dimension.

Broadcasting can expand a size-one dimension without copying tensor data. This is parameter efficient. It avoids materializing repeated values. It also creates a recurring implementation error: code assumes that successful out-of-place broadcasting implies that the equivalent in-place operation will work.

It does not.

An in-place operation cannot change the shape of the destination tensor. If a target tensor would need expansion to accommodate the broadcasted result, an underscore-suffixed operation such as add_() can raise a RuntimeError, even when a + b succeeds.

This difference should appear in tensor tests. A shape validation test should not only test the successful output. It should state whether an operation is intended to mutate its input.

A useful broadcasting test set includes three cases:

1. Compatible trailing dimensions. Confirm the result shape and values for a standard batch-by-feature operation, such as (batch, features) combined with (features,).

2. Size-one expansion. Confirm that a tensor with shape (batch, 1) produces the intended feature-wise result when combined with (batch, features).

3. Invalid or in-place expansion. Confirm that the implementation fails rather than silently reshaping or mutating an incompatible target.

The third case is often omitted. It should not be. PyTorch model debugging benefits from tests that establish expected failure modes, especially around custom loss functions, attention masks, and normalization parameters.

A related issue is accidental rank collapse. A reduction such as mean(dim=1) removes dimension 1 unless keepdim=True is supplied. The resulting values can be mathematically correct but structurally incompatible with a later broadcast. Testing the output shape directly after reductions is cheaper than diagnosing a mismatch several layers later.

Autograd history and dtype transitions change the contract

Tensor creation is not neutral. torch.tensor(data) copies input data and creates a leaf tensor with no autograd history. This is often correct for raw Python or NumPy data. It is not the right mechanism for preserving the computational relationship of an existing tensor.

When the input is already a tensor, the explicit pattern is clearer: use clone() when a copy is required, detach() when graph separation is intentional, and requires_grad_() when a leaf must track gradients. Each operation changes the graph contract differently.

This matters for PyTorch gradient checking and for ordinary model code. An accidental detach() can produce a stable forward pass and a broken optimizer step. An unnecessary clone can increase memory use. Reconstructing an intermediate with torch.tensor(existing_tensor) can remove history and create a new leaf, which is almost never the intended behavior inside a differentiable model path.

Tests for gradient-sensitive operations should verify more than output values:

• The output should have requires_grad=True when it depends on a gradient-tracked floating-point input.
• The intended leaf tensor should receive a finite gradient after backward propagation.
• Detached outputs should be tested explicitly where detachment is part of the design, such as a target-network branch or a metric calculation.
• Integer casts should be treated as graph boundaries.

The last point has a hard PyTorch constraint. Only floating-point and complex tensors can require gradients. If Tensor.to() converts a gradient-tracked tensor to an integer dtype, the returned tensor has requires_grad=False.

That behavior is correct. The error is conceptual: using integer conversion in a path that was expected to remain differentiable. Token IDs and class labels should be integer tensors. Activations, logits, losses, and learned parameters should not become integers.

Device and dtype assertions should be local

A common failure mode in training scripts is constructing a new tensor inside forward() without aligning it to the input device and dtype. This can produce a CPU-versus-CUDA error immediately. More subtly, it can introduce float64 constants into float32 computation.

For constants derived from an existing activation, preserve local context. For tests, assert the output’s dtype and device rather than assuming inheritance. torch.testing.assert_close() provides this behavior by default, while a raw allclose() call does not.

This is also relevant to ONNX export and inference parity. A model may pass a CPU float32 unit test while the deployment path uses half precision, a distinct operator set, or different reduction behavior. The verification suite should first establish correctness for one declared execution contract, then add target-specific parity tests. Combining all environments into one permissive tolerance obscures the source of divergence.

NaN and infinity require explicit failure handling

NaN and infinity are not ordinary numerical discrepancies. They are state failures. A tolerance comparison is not the first diagnostic.

torch.isfinite() returns a Boolean tensor identifying finite values. For real tensors, NaN, positive infinity, and negative infinity are all non-finite. In a training loop, finite checks should be placed around the points where instability enters the computation:

1. Inputs after preprocessing and device transfer.

2. Logits or intermediate activations after numerically sensitive layers.

3. Loss values before backward propagation.

4. Gradients after backward propagation and before the optimizer step.

5. Updated parameters after the optimizer step when instability is under investigation.

The objective is not to leave permanent assertions around every operation in production. The objective is to identify the first non-finite tensor. Once that boundary is known, diagnosis becomes tractable.

NaNs can result from division by zero, invalid logarithms, overflow in exponentials, unstable normalization, corrupted input data, or an excessive update scale. The location matters. A NaN in the loss may originate in logits. A NaN in logits may originate in an earlier residual branch. Testing only the final scalar loss provides little information.

When NaNs are expected in a controlled test—for example, to verify masking or missing-value propagation—comparison functions need explicit configuration. Both torch.allclose() and torch.testing.assert_close() treat NaNs as unequal by default. Set equal_nan=True only when identical NaN placement is part of the expected output contract. It should not be used to normalize unexplained training failures.

Verification belongs beside the implementation

A reliable PyTorch training loop verification strategy is not a large end-to-end test with one final loss assertion. It is a sequence of local contracts. Shapes validate interface compatibility. Metadata validates execution compatibility. numerical comparisons validate mathematical behavior. Gradient checks validate differentiability. finite checks validate stability.

The resulting code is easier to optimize. A CUDA kernel optimization, mixed-precision conversion, or tensor-parallel refactor can be evaluated against a reference implementation without conflating speed with correctness. The computational overhead of targeted tests is small relative to the cost of training a model on an invalid tensor path.

The practical standard is straightforward: every nontrivial tensor operation should have a declared output contract, and every optimization should be measured against a reference before it enters a training run. That is not defensive programming. It is the minimum experimental control required for executable machine learning code.