Pipette: A New Open-Source Framework for Benchmarking AI Models on Edge Devices

Liquid AI and Artificial Analysis Launch Pipette On-Device AI Benchmark Suite

The suite ships with native iOS and Android clients and a public dataset of more than 1,000 lab-verified configurations. For anyone shipping quantized models to phones, laptops, or embedded hardware, this is the kind of empirical tooling we have been waiting for.

Why on-device benchmarking is a systems problem

Pipette's core premise is straightforward: on-device behavior is a property of the deployed system, not the model in isolation. Capability scores from server-class FP16 runs don't necessarily predict how a model will behave when it is quantized, dropped onto a phone SoC, and asked to decode under thermal constraints. Pipette addresses that gap with versioned benchmark definitions and published measurement protocols designed for reproducibility.

Quality is measured against three well-known benchmarks — IFBench, GPQA Diamond, and MATH-500 — with FP16 or BF16 reference values where available. The currently published quality scores come from llama.cpp evaluation runs on NVIDIA H100 80GB systems, paired with compatible on-device performance measurements in the dashboard. That gives us a clean way to compare how much of a reference artifact's capability survives quantization.

Numbers worth a sanity check

A few figures from the release are worth pulling apart before you pick a model for your next edge deployment:

• Context scaling for similar-size models diverges sharply. At Q4_K_M on a Galaxy S26 Ultra, Granite-4.0-H-350M retains 78.4% of its decode throughput from 256 to 4,096 input tokens. Granite-4.0-350M, by contrast, retains only 33.8%. Two 350M models, very different scaling behavior — classic gotcha.
• Sparse activation is not free memory. LFM2.5-8B-A1B decodes 2.4x faster than Qwen3.5-4B and 2.6x faster than Ministral-3B-Instruct-2512 at 2,048 input tokens on the same device, despite activating only 1.5B of its 8.5B parameters per token. It still peaks at 5.29 GiB because all expert weights contribute to memory. Speed without the memory savings.
• Direct speed–quality trade-off inside one phone class. At Q4_K_M, MiniCPM5-1B completes a 2,048-input / 256-output workload in 3.47 seconds versus 4.12 seconds for LFM2.5-1.2B-Instruct — a 15.8% reduction in elapsed time on iPhone-class hardware.

What to verify on your stack

Before you adopt any number from Pipette as your own deployment decision, run this short checklist:

1. Confirm the exact quantization format (Q4_K_M is not the only option, and behavior shifts across formats).

2. Match the deployment configuration — runtime, processor, memory budget, thinking mode — to your target device.

3. Re-measure thermal and power state, since published numbers assume a fresh device state.

4. Re-run quality on IFBench, GPQA Diamond, and MATH-500 with your chosen artifact to confirm the reference delta still holds.

Let's be honest — most of us have been guessing at on-device performance from a spec sheet. Pipette gives us a reproducible baseline, which is the only way to stop arguing about which 1B model is fastest and start measuring it.