What are large language models? Architecture and scale

The operational definition is architectural and procedural: a Transformer-based model pretrained on broad data at substantial compute scale, then adapted for multiple downstream tasks.

Parameter count remains visible because it is easy to report. It is not sufficient evidence of capability. Training dataset volume, compute allocation, alignment procedure, and evaluation design materially alter the result. The Chinchilla result made this constraint explicit: a 70-billion-parameter model trained on 1.3 trillion tokens, using the same reported training compute as the 280-billion-parameter Gopher model, outperformed the larger model on nearly every measured task.

The relevant unit of analysis is therefore not “how many parameters?” It is the full training and deployment system.

The Transformer paradigm: beyond recurrence and convolution

Modern LLMs are built primarily on the Transformer architecture introduced in 2017. Its central departure from prior sequence models was structural. It discarded recurrence and convolutions in favor of attention mechanisms.

Recurrent neural networks process a sequence step by step. This creates an inherently sequential computation path. Long-distance dependencies are difficult to preserve, and training cannot fully exploit parallel hardware. Convolutional approaches improve some local pattern extraction but impose their own receptive-field constraints.

Self-attention changes the unit of computation. Each token can assign weights to other tokens in the input context. A representation of a word is therefore conditional on the surrounding sequence rather than fixed in isolation. Multi-head attention allows the model to learn several relationship patterns in parallel: syntax, local agreement, positional associations, entity references, or more diffuse semantic dependencies.

The original Transformer paper established that this architecture could produce competitive translation results without recurrence. It reported 28.4 BLEU on WMT 2014 English-to-German and 41.8 BLEU on English-to-French. Those scores are historically relevant, but the larger consequence was computational. Attention-based models could be trained at scales that recurrent architectures handled less efficiently.

A typical decoder-only LLM applies this structure autoregressively. Given a token sequence, it estimates the probability distribution for the next token. Repeating that process produces text, code, structured outputs, or other tokenized sequences. GPT-3 is the canonical large-scale example: an autoregressive model with 175 billion parameters.

The architecture does not encode factual verification, causal reasoning, or a stable world model as separate components. It optimizes next-token prediction over its training distribution. Apparent reasoning behavior can emerge from this objective and from scale, but the mechanism remains probabilistic sequence modeling.

An LLM is not a parameter count with a chat interface. It is a pretrained Transformer, a data regime, a compute budget, an adaptation stack, and an inference system.

Scaling laws: parameter counts require token context

Early scaling discussions often reduced model progress to larger parameter counts. This was incomplete. Empirical scaling research found that language-model cross-entropy loss followed power-law relationships with model size, dataset size, and training compute across experiments spanning more than seven orders of magnitude.

The conclusion is narrow but consequential. Performance depends on several coupled resources. Increasing one resource while holding the others poorly allocated can produce a suboptimal model.

The Chinchilla experiment provided a clear comparison. DeepMind trained Chinchilla with 70 billion parameters and 1.3 trillion tokens. It used the same reported training compute as Gopher, a 280-billion-parameter model. Chinchilla outperformed Gopher on nearly all measured tasks. The result does not establish a permanent token-to-parameter ratio for every architecture or corpus. It does reject the simplistic claim that a larger parameter count is intrinsically better.

This distinction affects how LLM parameter counts should be interpreted in model releases. A 70B model and a 175B model cannot be compared meaningfully from size alone. Relevant missing variables include:

• The number and composition of training tokens.
• The data filtering and deduplication pipeline.
• The context length and tokenizer design.
• The optimization schedule and total training compute.
• The post-training procedure.
• The benchmark protocol, including prompt formatting and few-shot examples.

A raw parameter figure is useful for estimating memory pressure and computational overhead. It is weak evidence for task quality.

The same principle applies to headline metrics outside machine learning. A single visible number can conceal the denominator, selection rules, and measurement window. That failure mode is also evident in the numbers behind play-to-earn crypto gaming: activity metrics need context before they can support a market conclusion. LLM benchmarks have the same methodological dependency.

Foundation models and the multitask benchmark landscape

A foundation model is trained on broad data at scale and can be adapted to multiple downstream applications. The category is wider than language. BERT, GPT-3, and DALL-E all fit the general concept despite operating across different modalities and objectives.

Large language models are therefore a subset of foundation models, not a synonym for them.

The practical appeal of a foundation model is transfer. Rather than training separate models from scratch for summarization, extraction, translation, classification, code completion, or question answering, a pretrained model can be prompted, fine-tuned, or otherwise adapted. This changes the economics of ML development. It does not remove the need for task-specific evaluation.

GPT-3 illustrated one adaptation path through in-context learning. Its few-shot experiments supplied task descriptions and demonstrations as text in the prompt. The model performed the task without gradient updates or task-specific fine-tuning. This was operationally significant because behavior could be modified at inference time. It was not equivalent to proof of stable competence across domains.

MMLU remains a useful reference point for the limitations of broad benchmark claims. The benchmark covers 57 subjects across STEM, humanities, social sciences, and other fields. In the original study, few-shot GPT-3 with 175 billion parameters scored 43.9%. Smaller GPT-3 variants were close to the 25% random-choice baseline for four-option questions.

The benchmark showed variation across subjects rather than a single general capability level. The original study also found that evaluated models remained below expert-level accuracy on every task and often failed to recognize when their answers were incorrect.

This is a calibration problem as much as an accuracy problem. A model that produces an answer in fluent prose does not expose the probability that its internal prediction is wrong in a deployment-reliable way. Benchmark performance cannot independently establish factuality, safety, domain competence, or reliability under distribution shift.

What a benchmark score can and cannot support

Benchmark results support comparative claims under a specified protocol. They can indicate that one model performed better than another on a fixed task set, prompt template, decoding configuration, and evaluation split.

They do not support broader conclusions without additional evidence.

1. MMLU is not a measure of intelligence. It samples 57 academic and professional subjects. It does not exhaust real-world tasks, adversarial inputs, tool use, long-context retrieval, or production workflows.

2. Few-shot results are prompt-dependent. Demonstration choice, ordering, answer formatting, and token budget can alter outcomes. A score is partly a property of the evaluation harness.

3. Aggregate accuracy hides failure concentration. A model can achieve a respectable mean score while failing systematically in a high-risk subject subset.

4. Multiple-choice success does not prove calibration. Selecting the correct option differs from recognizing uncertainty, citing evidence, or declining an unsupported answer.

5. Contamination remains a methodological concern. Broad web-scale pretraining complicates the distinction between generalization and memorized benchmark exposure unless the evaluation design addresses it directly.

Foundation model benchmarks measure behavior under a protocol. They do not certify competence outside that protocol.

Alignment and adaptation: post-training changes the observed model

Pretraining produces a broad next-token predictor. It does not necessarily produce an assistant that follows user instructions, ranks responses according to preference, or declines undesirable outputs. Those behaviors are largely shaped during post-training.

InstructGPT is a standard example of this distinction. Its procedure combined supervised fine-tuning on human demonstrations with reinforcement learning from rankings of model outputs. The result was not merely a larger base model. It was a model optimized against a different objective stack.

In the reported human evaluations, a 1.3-billion-parameter InstructGPT model was preferred over outputs from 175-billion-parameter GPT-3 on the studied prompt distribution. This does not imply that 1.3B parameters generally outperform 175B. It demonstrates that preference alignment can dominate raw scale for a defined interaction task.

The result has two practical implications. First, base-model benchmarks and instruction-following benchmarks should not be treated as interchangeable. Second, reinforcement learning from human feedback does not eliminate hallucinations, simple mistakes, toxicity, or instruction-following failures. It changes the observed behavior distribution. It does not establish correctness.

Fine-tuning introduces a separate resource question. Full fine-tuning updates the pretrained model’s entire parameter set. For large models, this creates substantial optimizer-state and GPU-memory costs. Parameter-efficient tuning methods attempt to reduce that burden.

LoRA, or Low-Rank Adaptation, freezes pretrained weights and inserts trainable low-rank matrices into Transformer layers. The update is represented in a lower-dimensional latent space rather than by independently changing every original weight. In the GPT-3 175B comparison reported by the LoRA paper, this approach used up to 10,000 times fewer trainable parameters and required three times less GPU memory than Adam-based full fine-tuning.

The numbers are meaningful, but their scope is specific. Parameter efficiency depends on target modules, rank selection, task complexity, optimizer settings, and serving architecture. LoRA reduces adaptation cost; it does not make the base model free to host, evaluate, or govern.

Adaptation methods solve different constraints

The correct method depends on the deployment objective. Prompting is suitable when task changes are frequent and outputs can be monitored. LoRA is attractive when multiple domain variants must share one base model. Full fine-tuning remains relevant where adaptation requirements exceed the capacity of lightweight adapters. Alignment methods are necessary when the product is evaluated by interaction quality rather than raw continuation likelihood.

Deployment realities: quantization and inference efficiency

Training scale is only one side of the LLM cost structure. Inference determines whether a model can be deployed within hardware, latency, and throughput constraints.

A 175B-parameter model in FP16 requires a large memory footprint before accounting for runtime activations and key-value caches. Autoregressive generation compounds the issue: each output token requires another forward pass, and longer contexts increase attention-related cost. A model can be benchmark-competitive while remaining economically impractical for a target serving environment.

Quantization addresses weight-storage and inference constraints by representing model weights with fewer bits. GPTQ is a post-training weight-quantization method designed for generative pretrained Transformers. Its reported experiments quantized a 175B-parameter GPT model to 3-bit or 4-bit weights in approximately four GPU hours with negligible accuracy degradation under those test conditions. The paper also reported FP16 end-to-end speedups of about 3.25× on NVIDIA A100 hardware and 4.5× on NVIDIA A6000 hardware.

These figures should not be generalized beyond the tested setup. Quantization quality varies with model architecture, calibration data, task sensitivity, kernel implementation, hardware, context length, and decoding workload. A negligible degradation on one benchmark may be material for structured extraction, multilingual generation, code completion, or safety classification.

The deployment trade-off is not simply precision versus accuracy. It is a compound interaction among:

• Weight precision and model-memory residency.
• Batch size and latency targets.
• Prompt length and key-value cache growth.
• GPU architecture and available inference kernels.
• Output length distribution.
• Accuracy tolerance for the application.
• The cost of fallback, retrieval, validation, or human review.

A quantized model is not automatically the efficient model. If dequantization overhead, memory bandwidth, or unsupported kernels dominate the serving path, theoretical compression may not translate into lower end-to-end latency. The only defensible comparison is a workload-specific measurement.

Scale is an allocation problem, not a label

The question “what are large language models?” has a narrow answer. They are usually Transformer-based language models pretrained at scale on broad corpora and adapted for many tasks. The broader answer is more useful: they are systems whose behavior emerges from coupled choices in architecture, data volume, compute allocation, post-training, evaluation, and deployment.

The Transformer made large-scale sequence modeling computationally viable. Scaling laws showed that parameter count cannot be separated from training tokens and compute. Chinchilla demonstrated that a smaller model can outperform a larger one under a better compute allocation. MMLU showed that broad multitask scores remain incomplete and uneven. InstructGPT showed that alignment can change user preference more than raw parameter scale. LoRA and GPTQ showed that adaptation and inference constraints can be reduced without retraining or serving every weight in full precision.

None of these results establishes a universal best model size, training ratio, benchmark, adaptation method, or quantization format. That is the practical conclusion. LLM evaluation should begin with the workload and end with measured behavior under that workload. Parameter count belongs in the report. It should not be the report.