Stable Diffusion Models: How to Calculate FID and CLIP Scores

Evaluation protocols partition into two regimes. Distribution-level metrics quantify whether the model's sample distribution matches a target dataset's feature statistics. Sample-level metrics quantify per-prompt semantic fidelity. A reported "Stable Diffusion FID of 7.2" carries interpretable meaning only when paired with the Inception-v3 checkpoint version, the reference dataset identifier, the prompt corpus source, and the sample size. The subsequent sections dissect each metric's construction, enumerate the minimum requirements for statistical validity, and identify the failure modes that invalidate the majority of open-source benchmark reports.

The Role of FID in Measuring Generative Distribution Similarity

FID computes the Fréchet distance, equivalent to the Wasserstein-2 distance between multivariate Gaussians, fitted to feature distributions extracted from real and generated image sets. The reference implementation extracts a 2048-dimensional feature vector from the pool3 layer of an Inception-v3 network pre-trained on ImageNet classification. The metric captures distributional divergence between the two feature sets. It does not measure text-prompt fidelity, object counting accuracy, compositional correctness, or per-sample visual artifacts.

FID measures distributional similarity between real and generated images; it does not measure text-image alignment. A model can produce visually implausible samples yet receive a low FID if the marginal feature distribution matches.

The FID pipeline for Stable Diffusion comprises three components: a fixed feature extractor, a fixed reference image set, and a generated sample set produced from a held-out prompt corpus. Each component introduces independent variance. Reference dataset selection alone shifts reported FID by several points, since the COCO validation split and LAION-5B subsets differ in object category distribution, lighting, and aspect ratios. Cross-paper FID comparisons therefore require identical reference sets, identical prompt corpora, and identical sample counts. Numerical comparisons absent these specifications yield no valid inference about relative model quality.

Technical Requirements for Computing Stable FID Scores

The FID estimator's variance scales inversely with sample size. Empirical guidelines recommend a minimum of 30,000 generated images paired with an equivalent reference set, with 50,000 preferred for sub-point precision. Below 10,000 samples, the estimator produces high-variance estimates and frequently yields counter-intuitive rankings where a visibly superior model receives a worse numerical score. The Gaussian fit assumption further degrades at low sample counts, as the empirical covariance matrix becomes singular or poorly conditioned.

Several parameters require explicit specification in any FID report:

• Feature extractor: Inception-v3 trained on ImageNet, pool3 layer (2048-dim output). FID-CLIP and other variants exist but produce non-comparable scores.
• Reference dataset: COCO-2017 validation split (5,000 images, often upsampled to 30,000 via center crops), LAION-5B subsets, or a model-specific held-out set. Cross-dataset FID deltas exceed intra-dataset variance.
• Sample count: ≥30,000 generated images, matched to reference set size where possible.
• Prompt corpus: identical prompts across compared models. MSCOCO captions, PartiPrompts, and DrawBench each yield different FID distributions.
• Image resolution: matched to model output. Mismatched resolutions introduce resampling artifacts affecting FID.
• Sampler configuration: identical samplers, step counts, and classifier-free guidance values across compared checkpoints.

Omission of any parameter renders the reported FID non-reproducible and non-comparable.

Quantifying Text-Image Alignment with CLIP Score Metrics

CLIP Score measures the cosine similarity between CLIP image and text embeddings for corresponding prompt-image pairs, typically scaled by a factor of 100. The metric captures semantic alignment at the sample level. It does not capture image quality, distributional coverage, or compositional correctness. A generated image may exhibit severe artifacts and still receive a high CLIP Score if the artifact distribution correlates with high CLIP image embedding norms.

CLIP Score quantifies text-image semantic alignment; it does not quantify image quality. A high CLIP score on a corrupt image indicates embedding correlation, not perceptual fidelity.

The standard implementation computes per-sample CLIP scores across a prompt corpus and reports the mean. Variance, percentiles, and per-category breakdowns carry additional diagnostic value. CLIP Score depends strongly on the CLIP model variant: OpenCLIP-ViT-B/32, OpenCLIP-ViT-L/14, and OpenAI CLIP-ViT-L/14 produce non-comparable numerical ranges. Cross-variant comparisons constitute a methodological error and propagate through leaderboards that aggregate scores across heterogeneous CLIP backbones.

Distinguishing Between CLIP-T and CLIP-I Evaluation Scenarios

CLIP-T (text-to-image) applies to text-conditioned generation. The score uses the text prompt embedding as the reference vector against the generated image's CLIP embedding. This metric dominates text-to-image benchmark reporting for Stable Diffusion models. CLIP-I (image-to-image) applies to image-conditioned tasks: inpainting, image variation, ControlNet outputs, and structural conditioning. The score uses a source image's CLIP embedding as the reference vector against the generated image's embedding, measuring preservation of source visual identity.

The two metrics answer different empirical questions. CLIP-T quantifies prompt adherence. CLIP-I quantifies identity preservation. A ControlNet output may score highly on CLIP-I while scoring poorly on CLIP-T if the prompt requested modifications incompatible with the source layout. Stable Diffusion evaluation reports frequently conflate the two metrics, particularly in image variation benchmarks where CLIP-T is reported in lieu of the appropriate CLIP-I. Correct metric selection depends on the conditioning modality of the generation pipeline under evaluation.

Comparative Properties of FID and CLIP Score

The table clarifies that the metrics measure orthogonal axes. Neither alone provides a sufficient characterization of Stable Diffusion model performance. Distribution-level divergence (high FID) and sample-level alignment failure (low CLIP Score) indicate different model deficiencies requiring different remediation strategies: architectural adjustments for distributional coverage versus training data or text encoder improvements for prompt adherence.

Common Pitfalls in Benchmarking Generative Image Architectures

Benchmarking errors in the Stable Diffusion ecosystem follow recurrent patterns:

1. Insufficient sample count. Reports citing FID below 5,000 generated samples should be discarded, as the estimator exhibits high variance in this regime and rankings reverse frequently at sub-10,000 counts.

2. Reference set opacity. Reports omitting the reference dataset identifier preclude reproduction. FID against COCO-2017 validation differs from FID against LAION-5B aesthetic subsets by several points on identical generated samples.

3. Feature extractor substitution. FID-Inception and FID-CLIP compute different metrics. Cross-variant numerical comparison constitutes a methodological error and remains common in compiled leaderboards.

4. Resolution mismatch. Generating at 512×512 and computing FID against 256×256 reference images introduces resampling artifacts that bias FID upward and distort cross-model comparisons.

5. Prompt corpus conflation. FID on PartiPrompts differs from FID on MSCOCO captions. Numbers across prompt sets are not comparable, even with identical sample counts and reference datasets.

6. CLIP variant substitution. CLIP scores from OpenCLIP-ViT-L/14 and OpenAI CLIP-ViT-L/14 occupy different numerical ranges. Cross-variant comparison yields invalid inference about relative model performance.

7. CLIP Score gaming. Adversarial prompts and degenerate outputs (solid-color images with high CLIP norms) inflate CLIP scores without perceptual improvement. The metric does not penalize low-quality samples.

8. Single-metric reliance. FID alone does not capture text alignment. CLIP Score alone does not capture distribution coverage. Comprehensive evaluation requires both, plus task-specific measures such as CLIP-I for conditional generation.

These pitfalls propagate through leaderboards, technical reports, and comparative analyses. Practitioners consuming reported metrics should require the full specification: feature extractor, reference set, sample count, prompt corpus, and resolution. Models ranked highly on incomplete specifications carry minimal empirical weight.

Synthesis

Stable Diffusion evaluation demands multi-metric protocols with explicit specification of all measurement parameters. FID requires Inception-v3 pool3 features, a documented reference dataset, and sample counts exceeding 30,000 for statistical validity. CLIP Score requires paired prompt-image samples, a specified CLIP variant, and reporting of the score's distribution rather than only the mean. CLIP-T and CLIP-I measure distinct properties and must be selected according to the conditioning modality of the generation task.

The methodological hygiene required for valid Stable Diffusion benchmarking remains inconsistent across published comparisons. Reported metrics lacking the full specification should be treated as uninterpretable. Until standardized evaluation protocols become normative across the ecosystem, model rankings derived from incomplete metric reports carry no reliable empirical content. The orthogonality of FID and CLIP Score means that ablation studies isolating single-metric improvements risk optimizing one axis at the expense of the other, producing models that excel at reported metrics while degrading on unmeasured properties.