Benchmark Shows Attribution Methods Depend Heavily on Model Architecture

The controlled benchmark evaluates 13 attribution methods across five dimensions and eight representative backbones. Its central result is narrow but operationally important: attribution performance is strongly architecture-dependent.

For practitioners building explanation tooling around vision models, this weakens the assumption that a method validated on one backbone will retain the same properties on another. The benchmark does not support a universal attribution ranking. It supports architecture-specific evaluation.

The benchmark tests transferability, not marketing claims

The study is framed around a concrete methodological question: whether explainability behavior remains stable when the underlying model architecture changes. CNNs and Vision Transformers organize visual information differently, so a result observed on one family cannot be treated as a direct proxy for the other.

The reported evaluation covers 13 attribution methods, five assessment dimensions, and eight CNN and Vision Transformer backbones. Those dimensions are not detailed in the available source summary. The confirmed result is therefore the cross-architecture dependency itself, rather than a complete ranking of methods or a claim that one technique dominates the field.

That distinction matters. A single attribution visualization can appear plausible while still failing a broader evaluation. If its measured behavior changes with the backbone, the visualization is not an architecture-neutral property of the input or task. It is partly a property of the interaction between the explanation method and the model.

The benchmark consequently treats explainability as an empirical component of a system, not as a detachable interface layer.

What the result changes for model evaluation

The practical implication is a change in experimental design. Attribution methods should be evaluated on the exact model family and backbone used in deployment. Reusing a configuration from a CNN for a Vision Transformer, or the reverse, is not justified by the benchmark’s result.

A minimum comparison should therefore hold the evaluation protocol constant while changing the attribution method and backbone. The five reported dimensions provide the study’s evaluation frame, but the available evidence does not specify their definitions. Implementations should not infer that one dimension, such as visual plausibility, is sufficient to establish explanation quality.

This is a parameter-efficiency issue in the broader sense: adding an explanation method does not automatically add a reliable interpretability capability. It adds another component whose behavior must be characterized. The relevant unit of analysis is the model–attribution pair.

For teams maintaining reproducible research code, the benchmark also implies that attribution settings belong in the model configuration rather than in a generic post-processing utility. A method should be versioned and reported alongside the backbone. Otherwise, comparisons across experiments may combine architecture effects with explanation-method effects.

Limitations and the next verification step

The available arXiv summary provides the study scope and its primary finding, but not the individual scores, the names of all 13 methods, the five dimensions, or the eight backbones. It is therefore not possible to identify the best-performing method, quantify the transfer gap, or determine whether the observed dependence is uniform across all evaluation criteria.

That limits the immediate claim. The study establishes that architecture matters in attribution benchmarking; it does not, from the available evidence, establish a universal failure mode for explainability or a definitive replacement method.

The next practical step is controlled replication. Run the same attribution candidates against the CNN and Vision Transformer backbones used in a target workflow, preserve the evaluation protocol, and record results per architecture. Any explanation pipeline that reports aggregate performance without this split risks hiding the exact dependency identified by the benchmark.