Decoding LLM Architecture: A Functional Guide to Model Parameters

A Coursera explainer segments LLM parameters into five functional categories—weights, biases, hyperparameters, temperature, and activation functions—positioning each as a configurable axis for model capacity and overfitting control. For practitioners evaluating recent open-weight releases against closed-system benchmarks, the taxonomy functions as a verification scaffold rather than a conceptual primer.

Parameter classes as empirical axes

The framework decomposes weights and biases as the learned representational layer, hyperparameters (learning rate, batch size) as the optimization trajectory, temperature as the sampling distribution control, and activation functions as the non-linear signal transformation scope. The implication for benchmarking is direct: reported gains on one axis—say, parameter count—do not generalize across activation regimes or temperature settings. Replication requires ablation across categories, not aggregate reporting.

Concurrent releases testing the boundaries

Ornith-1.5, released on August 19, 2026 under the MIT License, ships three variants spanning parameter scales: Ornith-1.5-397B (MoE, 397B total parameters), Ornith-1.5-35B-A3B (MoE, 35B total / 3B active), and Ornith-1.5-9B (Dense). Reported benchmark comparisons place the 397B variant on par with Claude Opus 4.8 across multiple evaluations and ahead in several. The 9B Dense variant is reported to outperform Gemma-4-31B despite a substantially smaller parameter footprint, while the 35B-A3B MoE configuration is reported to exceed both Muse-Glimmer-30B and Gemma-4-31B at comparable total scales. A quantized Ornith-1.5-9B-Mobile variant targets smartphone inference, providing a concrete test bed for activation behavior under reduced precision.

Prime Intellect's NanoGPT Speedrun Frontier, launched separately, evaluates 18 autonomous coding models on their capacity to optimize a small language model trainer without human intervention. By constraining the optimization surface to hyperparameter selection, the benchmark operates as a controlled ablation environment for autonomous research agents.

Harvey's Tenet, an open-weight legal AI model, adds a domain-specific vertical test case to the parameter taxonomy.

Verification priorities

Three empirical questions warrant attention. First, the active-versus-total parameter ratio in MoE architectures: the 35B-A3B configuration implies an inference compute profile closer to a 3B dense model, complicating direct comparisons against dense baselines at similar total parameter counts. Second, quantization-induced behavioral shifts on the Ornith-1.5-9B-Mobile variant—activation function outputs under reduced precision require independent measurement. Third, reproducibility of vendor-reported benchmark scores against closed models; the Ornith-1.5-397B comparisons against Claude Opus 4.8 await independent replication.

Readers seeking deeper architectural context on these parameter trade-offs may consult long-form analyses of model design choices for background material.