Multimodal large language models by the numbers

The benchmark record supports a narrower conclusion than vendor narratives imply. Vision-language model performance has improved sharply on structured visual reasoning, document interpretation, and mathematical tasks. It has not established reliable expert-level reasoning across domains. The decisive variables are task construction, answer space, training-data overlap, tool access, and whether the model receives domain adaptation.

The shift to expert-level reasoning: why MMMU-Pro matters

MMMU was designed to test multimodal reasoning across college-level disciplines. Its successor, MMMU-Pro, applies a stricter protocol. It uses vision-only inputs and expands multiple-choice questions from four options to ten. This modification is not cosmetic. It changes the probability structure of guessing and reduces the utility of shallow answer-elimination heuristics.

The measured consequence is substantial. Models experience a 16.8% to 26.9% performance drop on MMMU-Pro relative to the original MMMU benchmark. This gap is one of the clearest current indicators that multimodal model accuracy comparisons cannot be transferred across benchmark versions without examining the evaluation protocol.

A model that performs well on a four-option benchmark can benefit from partial semantic recognition. It may identify two implausible answers, match a familiar diagram type, or exploit language priors in the prompt. Ten-option evaluation makes these shortcuts less effective. Vision-only input removes another possible channel: recovering critical information through surrounding text or text-heavy prompt construction.

The benchmark therefore places more pressure on the actual pipeline:

1. Visual extraction. The model must recover values, labels, spatial relations, notation, or chart structure from the image rather than infer them from a text paraphrase.

2. Cross-modal binding. The extracted visual entities must remain correctly linked to the problem statement. A variable in a diagram is not useful if it is assigned to the wrong axis, component, or condition.

3. Latent-space reasoning. The model must transform the visual representation into a domain-specific reasoning trajectory rather than retrieve an answer pattern associated with the image class.

4. Answer discrimination. With ten candidates, coarse conceptual alignment is insufficient. The model must reject answers that are nearly correct but fail on a sign, boundary condition, unit, or inference step.

The leading MMMU-Pro scores remain high in absolute terms. Gemini 3.5 Flash records 83.6%. GPT-5.2 records 79.5% without tools and 80.4% with Python. The limited improvement from Python in this comparison is informative. The bottleneck is not always arithmetic execution. In many multimodal tasks, the failure occurs upstream: an incorrect reading of the figure, a missing constraint, or an invalid formalization of the visual input cannot be repaired by a calculator.

MMMU-Pro measures more than model knowledge. It measures whether visual evidence survives the path from perception to answer selection.

This is also why a single aggregate MMMU-Pro score should not be read as a general certificate of expert competence. Aggregate reporting compresses heterogeneous failure modes. Diagram interpretation, engineering schematics, clinical images, plots, and symbolic notation occupy different regions of the latent space. Similar accuracy can emerge from materially different error distributions.

For implementation teams, the practical implication is direct. MMMU-Pro is a more defensible screening benchmark than its predecessor for general multimodal reasoning. It is not a deployment validation set for a high-stakes vertical.

Performance gaps in specialized domains: the SurgVU case study

The surgical AI comparison provides the more restrictive result. In zero-shot evaluation on SurgVU, Claude Sonnet 4.6 achieved 23.05% exact-match accuracy. Gemini 3.1 Pro Preview reached 22.46%. These scores are not evidence of clinical readiness. They show that broad frontier-model capability does not automatically transfer to specialized visual and procedural reasoning.

The contrast with parameter-efficient adaptation is more consequential. LoRA fine-tuning of Gemma 3 27B increased SurgVU exact-match accuracy to 50.61%. The absolute gain is large. It does not mean that fine-tuning resolves the problem. A result near 50% exact match still leaves a substantial error rate. But the direction of the ablation is unambiguous: task-specific adaptation contributed more than zero-shot scale in this setting.

Three technical distinctions explain this spread.

First, surgical tasks contain vocabulary and visual conventions that are weakly represented in broad multimodal pretraining. Anatomical orientation, operative instruments, sequential workflow, and exception conditions require joint representations that are unlikely to emerge consistently from generic image-text corpora.

Second, exact-match evaluation is intentionally unforgiving. A model may produce a medically adjacent explanation while failing to identify the required structure, action, or procedure stage. In a conversational setting, such output may appear plausible. Under strict scoring, it is correctly registered as wrong.

Third, LoRA changes the adaptation economics. Instead of retraining the full parameter set, low-rank updates alter a limited subset of weight directions. The result is parameter efficiency, lower computational overhead, and a practical path for organizations that need domain calibration without full-scale foundation-model training. The trade-off is narrower. A tuned model can improve on the target distribution while losing robustness outside it if the adaptation data are limited or overly homogeneous.

This distinction is routinely blurred in multimodal LLM reporting. Zero-shot scores estimate transfer. Fine-tuned scores estimate adaptation potential. They answer different questions and should not appear in the same ranking without explicit labeling.

A reasonable evaluation protocol for a specialized vision-language model should therefore separate:

• zero-shot baseline performance;
• prompt-engineered performance under a frozen model;
• tool-augmented performance, if tools are allowed in production;
• parameter-efficient tuning results;
• held-out performance by subdomain, image modality, and difficulty;
• calibration and abstention behavior, not only exact-match accuracy.

The final item is often absent. It should not be. A model that can identify uncertainty on ambiguous cases is operationally different from a model that returns a confident but unsupported answer.

Open-weight versus proprietary: the Gemma 4 and Gemini 3.5 landscape

The open-weight segment has narrowed the gap on broad multimodal benchmarks. Gemma 4 31B, a dense model with 30.7 billion parameters, records 76.9% on MMMU-Pro and 85.6% on MATH-Vision. Its predecessor, Gemma 3 27B, records 49.7% and 46.0% on the same measures.

The difference is too large to attribute to parameter count alone. The two models are near each other in nominal dense parameter scale, while the benchmark delta exceeds 27 percentage points on MMMU-Pro and 39 points on MATH-Vision. Architecture, training mixture, visual encoder integration, post-training, and data curation are more plausible explanatory variables. Public parameter counts do not expose those factors fully.

Gemma 4 also includes a mixture-of-experts configuration: Gemma 4 26B A4B MoE has 25.2 billion total parameters but activates 3.8 billion per token. This is an important distinction for deployment analysis. Total parameter count estimates storage and aggregate capacity. Active parameters per token more directly affect inference-time computational overhead. A sparse model can therefore offer a different latency-throughput trade-off from a dense model with a comparable total footprint.

The comparison should stop there. It is not valid to infer that Gemma 4 has equivalent general capabilities to proprietary models from one or two benchmark columns. Proprietary model sizes, training-data composition, and internal routing strategies are undisclosed. Their architecture cannot be normalized against a 30.7B dense parameter count.

However, the operational value of open weights is not reducible to leaderboard position. Open-weight models permit controlled fine-tuning, inference profiling, quantization experiments, and reproducible ablation studies. A team can inspect how accuracy changes as image resolution, context budget, decoding strategy, or LoRA rank changes. That degree of experimental control is unavailable, or materially constrained, with closed APIs.

This is the relevant decision boundary:

• Choose a proprietary model when the task matches publicly demonstrated general benchmarks, the API constraints are acceptable, and maximum out-of-the-box accuracy is the primary requirement.
• Choose an open-weight model when data locality, reproducible evaluation, parameter-efficient tuning, or serving control materially affect the system design.
• Do not select either class from a single broad benchmark. The benchmark must resemble the target task’s visual modality, answer format, and tolerance for error.

Open weights change the experimental surface. They do not eliminate the need for a target-domain evaluation set.

Mathematical visual reasoning: beyond human baselines

MathVista is frequently cited because it provides an intuitive headline: GPT-4o reached 63.8%, above a reported human average of 60.3%. The newer scores are considerably higher. Gemini 3.5 Flash reached 90.4% on MathVista testmini, while Gemini 3 Pro Preview reached 89.4%.

These results establish progress in a defined task family. They do not establish a general human-equivalence threshold. Mathematical visual reasoning benchmarks typically reward a specific sequence: parse the visual artifact, formalize the problem, execute the calculation, and map the result into the expected response. Models have improved at each stage, particularly when diagrams, charts, and standard educational notation resemble patterns seen during training.

The human baseline requires equal scrutiny. A human average is conditional on the participant group, time budget, instructions, input quality, and scoring protocol. It is not a universal capability ceiling. More importantly, the model and human may arrive at identical answers through different processes. A correct model answer can result from robust reasoning, pattern retrieval, answer-format priors, or incidental benchmark overlap. Aggregate accuracy cannot distinguish these mechanisms.

MATH-Vision offers a second view. Gemma 4 31B reaches 85.6%, far ahead of Gemma 3 27B at 46.0%. The result indicates a substantial gain in visual mathematical processing. It does not reveal which subskills improved. Without a published ablation study across diagram types, OCR sensitivity, multi-step depth, and distractor structure, the score remains a compressed metric.

For model developers, the useful follow-up experiments are specific:

1. Image perturbation tests. Alter resolution, crop margins, label placement, and font rendering. A robust vision-language model should preserve its reasoning when irrelevant visual properties change.

2. Counterfactual diagrams. Modify one numerical value or geometric relation while keeping the language constant. This tests whether the model reads the image or retrieves a familiar solution pattern.

3. Intermediate representation audits. Require structured extraction of entities, values, and relations before answer generation. Errors can then be localized to perception, formalization, or inference.

4. Answer-option permutation. Shuffle multiple-choice candidates. Large score variation after permutation indicates dependence on positional or lexical artifacts.

5. Tool-isolated evaluation. Measure the model with and without code execution. This separates visual interpretation quality from symbolic computation quality.

Without these controls, high MathVista-style scores can overstate reasoning depth. The model may be highly capable within the benchmark distribution while remaining fragile under modest shifts in notation or visual layout.

Mitigating contamination with MMBench-Live and StatEval

Benchmark contamination is now a central measurement problem. Static public benchmarks are useful for longitudinal comparison, but their items can enter training corpora, synthetic data pipelines, prompt collections, or post-training evaluation loops. Once that occurs, a score partly measures exposure rather than generalization.

MMBench-Live addresses this failure mode through continuous updates. Introduced with 5.9K newly generated evaluation instances, it is designed to reduce contamination risk and preserve cross-version ranking stability. The design is more important than the raw item count. A living benchmark imposes recurring generalization tests rather than allowing models to optimize against a fixed public target.

That does not make MMBench-Live contamination-proof. No benchmark can make that claim when model training pipelines are opaque and internet-scale. It does create a stronger evaluation regime: items can be refreshed, task distributions can evolve, and suspicious performance jumps can be investigated against newly generated data.

StatEval supplies a different corrective. It evaluates statistical reasoning rather than general visual understanding and contains 13,817 foundational problems plus 2,374 research-level proof tasks. The high-difficulty subset is revealing. GPT-5-mini reaches 57.62% on research-level proof tasks, while Gemini 2.5-flash reaches 51.14%. Both results are below 58%.

The implication is not that these models lack statistical knowledge. It is that research-level proof construction is a distinct workload. It requires formal assumptions, valid derivations, and sensitivity to quantifiers and edge conditions. Language fluency can conceal failure here because an invalid proof often remains locally coherent.

StatEval also demonstrates why broad multimodal LLM benchmark datasets should be paired with domain benchmarks. A model can score above 80% on MMMU-Pro and still fail frequently on a task requiring formal statistical argument. These are not contradictory outcomes. They occupy different reasoning regimes.

The strongest evaluation stacks now use three layers:

• Static public benchmarks for historical comparability and broad peer comparison.
• Fresh or continuously updated benchmarks for contamination resistance.
• Private task-specific test sets for deployment decisions, with samples that reflect actual data sources, error costs, and output constraints.

A fourth layer is often necessary for regulated or high-consequence settings: prospective evaluation after deployment conditions are simulated. Offline benchmark accuracy does not measure interface failures, retrieval errors, image acquisition noise, latency limits, or operator overreliance.

The current benchmark record

Multimodal large language models are no longer limited to superficial image captioning and chart description. Their performance on MMMU-Pro, MathVista, and MATH-Vision shows substantial improvement in visual reasoning. Gemma 4 31B demonstrates that open-weight systems can now approach proprietary frontier results on selected broad evaluations. Gemini 3.5 Flash leads the reported MMMU-Pro comparison at 83.6%.

The limitations are equally measurable. MMMU-Pro removes 16.8% to 26.9% of performance relative to the easier MMMU setup. StatEval research-level proof accuracy remains below 58% for the reported advanced models. Surgical zero-shot exact-match results remain below 25%, while LoRA adaptation raises an open model to 50.61% rather than to a reliability threshold.

The relevant metric is therefore not a universal leaderboard rank. It is the gap between benchmark conditions and the intended workload. Model selection should begin with the target image modality, domain vocabulary, answer format, tool policy, and cost of a wrong output. Only then do multimodal model accuracy comparisons become operationally meaningful.