Our Model Testing

Every model recommendation on this site traces back to a number we measured ourselves. This page is the front door to that testing: what we run, how we run it, and what we've learned that the leaderboards won't tell you.

The short version of why: we run a local AI fleet as a delegation sidekick - a set of models on our own hardware that a frontier orchestrator (Claude) hands bounded work to, so paid API tokens get spent only where frontier reasoning earns them. That architecture only works if you know, precisely, what each local model can do and how fast. Guessing isn't knowing. So we test.

What we test on

The current rig is a pair of modded 48GB RTX 4090s - 96GB of VRAM between them - with 128GB RAM, serving with vLLM in Docker and orchestrated by houtini-lm. Not datacentre hardware, deliberately: the point is what's achievable on a machine an enthusiast or a small studio can realistically build. The numbers below were measured on a single card; tensor parallel and dual-card figures will follow.

The settings matter more than the card, and almost nobody publishes them. Both cards are capped at 330W by measurement, not left on the default. The core clocks are locked to 2200 MHz before anything loads - on modded 48GB cards that is a stability requirement, not a tuning choice, and it holds on both dies at once under a full dual-GPU sweep (2190 MHz, 97-98% utilisation, around 193W a card). Most presets run an fp8 KV cache, which is the trick that lets a 131k to 262k token context actually fit. The stack underneath is driver 610.88 on CUDA 13.3, WSL2, vLLM in Docker, and presets are single-GPU unless a row says TP=2. Change any of those and the numbers move, which is exactly why they are on the page.

How we test

We test in three layers, and it is worth knowing which number you are looking at.

The first is our own rig harness, and it is the one that matters most here. Five scenarios, chosen to look like real delegation traffic rather than a generic serving benchmark: single-stream decode, cold time-to-first-token on a 16k code-shaped prompt, the same prompt again to show the prefix cache working, decode speed with 16k already resident, and four requests running at once. Temperature 0 throughout, token counts read straight from the server. The script is the spec - run it and you get our runs.

The second layer is vLLM's own vllm bench serve, pinned to fixed parameters (ShareGPT V3, 64 prompts, seed 42, eight-way concurrency) so you can point it at your own hardware and compare like for like. Our healthy qwen reference at those settings is 261.7 tok/s output, 838ms mean time-to-first-token, and 25.6ms per output token.

The third layer is quality, and we deliberately leave it out of the routine sweep. A speed table is a speed table; it never implies one model is smarter than another. When a throughput crown is genuinely contested we run a bounded task set as a one-off and label it as exactly that.

The method is frozen as v1. Rows are only comparable within a method version, so when the method changes, the version does too.

A few results that illustrate why we bother: a model with nearly a million downloads and a strong benchmark pedigree produced code that doesn't compile on a 20-line real-world task. An unbenchmarked community model won that same face-off. And the single biggest speedup we found came not from a clever flag but from choosing a different quantisation file of the same model - a 3x difference the model cards don't mention.

The current numbers

The fleet as it stands, measured 9 August 2026 on the single 48GB card at the standing 330W power policy. A scheduled sweep re-runs these weekly and this table tracks it.

The number to look at is our first dual-GPU resident: Nemotron 3 Super 120B (TP=2 across both cards, so it sits outside the single-card table above). It holds 62.7 tok/s decode with 16k tokens resident, against 19.4 for the 27B - a 3.2x gap, and it happens because essentially nothing degrades as the context fills up. That is the Mamba2 hybrid architecture doing exactly what it says on the tin, and for long-document work it is the most interesting figure on the page. At the other end, Muse Glimmer 30B is in the table above as an honest split: the best prefill in the fleet by a distance (+75% on qwen) and the worst decode. A prefill specialist, not a daily driver.

Decode is single-stream at temperature 0; prefill and TTFT are a 16k-token prompt; repeat TTFT shows prefix caching working. The last row is the same 27B before tuning - the gap between 18.8 and 66 is what the deep-dive below explains.

See what's trending on Hugging Face

The fleet above is what we run; this is what everyone else is pulling right now. It reads the Hugging Face trending list live, and the dropdown filters it to the models that will actually fit your card. Handy for spotting the next thing worth benchmarking.

The frontier models, as reported

The local fleet exists to take work off paid APIs, so the other half of the measurement question is what those APIs cost and deliver. Unlike everything above, these are reported numbers - vendor list pricing and public leaderboards, not our harness - and we keep that distinction loud on purpose. List prices move, so the frontier comparison on our research page now tracks them live from OpenRouter's public feed; the table below keeps the figures as they stood in July 2026, when we wrote this. The gauntlet runs against the frontier APIs (same TypeScript task, same tool-calling gate - the harness doesn't care that the endpoint is remote) are queued; when they land, this table gets our numbers.

The chart is the whole economics argument in one picture: two orders of magnitude separate the cheapest and most expensive output tokens on the frontier, and a local token costs none of it. The hardware isn't free, of course - our GPU guide covers what the cards cost and which make sense - but once a card is on the desk, every token it decodes takes work off the metered tiers above. That's the comparison that matters: not local versus cloud as a religion, but which tier each job deserves.

The routing conclusion is ours, and it's a three-tier ladder: the local fleet for bounded work (free after hardware, measured), DeepSeek V4 Pro as the candidate middle tier for high-volume agent loops (unverified by us, so no real work routes there yet), and Claude for orchestration and anything where being wrong costs more than the tokens. Tool-calling reliability - the thing our whole delegation architecture depends on - is where the frontier lead is least contested.

Pro tips from working with these models

The genuinely useful part of running a fleet is everything the documentation leaves out. Here are the traps that cost us real time, each with the fix and the number that proves it.

• Restart the engine between benchmark runs. A vLLM engine driven into a bad scheduling state keeps returning 200s and keeps answering benchmarks - just 60x slower - while the container health stays green. Swap presets and it restarts for you; re-run the same preset and it does not, so do it by hand. Skip it and you can spend hours measuring a wedged engine and blaming the model.
• Never benchmark speculative decoding on random tokens. --dataset-name random collapsed an MTP preset to about 1 tok/s that serves real text at 262. Speculative decoding predicts likely continuations, and random tokens reject every draft. Use ShareGPT or your own real prompts.
• Read three things on the model card before you load it: the recommended sampling, how to switch reasoning off, and whether it fences code. On one 120B, generic defaults gave us 2 of 6 build phases and 58,747 output tokens; the same model with its documented settings gave 6 of 6 and 18,481. Three settings, 68% fewer tokens, five times the usable output.
• Load time is not disk speed. A 77GB model took 41 minutes off a Windows bind mount and looked like a slow filesystem. Then dd measured that same mount at 530MB/s - about two and a half minutes for 77GB, so 95% of the load was compute (dequantisation, sharding, graph capture), not I/O. If load time needs fixing, profile it first.
• Set `HF_TOKEN`. Anonymous HuggingFace pulls hit a rate limit that drops the loader into a silent backoff - no error, no message, just a process that looks hung. The token removes it and roughly doubles download speed as a bonus.
• Large download dying with a CAS decode error? Retry with HF_HUB_DISABLE_XET=1; it resumes from what is already on disk.
• Check architecture support before you pull tens of gigabytes. A one-line docker run against the image's ModelRegistry tells you whether the model will even load before you spend an hour downloading it.
• Day-one models often need a model-specific image tag, not :latest, and the CUDA version baked into that image can outrun your driver - in which case the container dies at the nvidia prestart hook before vLLM even starts.
• A model that crashes in multimodal profiling can still be fine for text. --limit-mm-per-prompt.video 0 --limit-mm-per-prompt.image 0 gets you a working text endpoint while the vision path is broken upstream. Worth trying before you write a model off.
• Speculative decoding plus a big batched-token budget can deadlock. 16384 with MTP wedged on a 16k cold prefill - engine at 95% CPU, GPU pegged but drawing only ~72W, health still green. 8192 is both safe and faster on that preset, so there is nothing to trade.
• Coarse progress bars look exactly like a hang. Shard loading reports per shard, and a 120B shard can take 150+ seconds. Split the log on carriage returns and check whether VRAM or read_bytes is still moving before you call it frozen.
• If your harness records config next to results, filter the secrets. Ours captured the whole .env into a git-tracked results file. Fine until a token goes in it.
• Comparing models? Send an identical request body. A single model-specific flag - a thinking toggle, say - silently rigs a token comparison, because one model pays for its chain of thought and the other gets a free pass.

The thread running under all of these is the same, and it is the one discipline worth taking away: a number that was inferred rather than measured gets written down as a fact, and every decision after it inherits the mistake. A slow load blamed on a filesystem, a wedged engine blamed on the model - measure the thing instead. dd takes fifteen seconds.

Use this with Claude

The practical way to use these findings is houtini-lm , our open-source MCP server that connects Claude to your local models. Claude orchestrates; your hardware executes; your API bill shrinks. It works with LM Studio out of the box and with a vLLM stack like ours - the model routing in it is built directly from the measurements on this page.

If you'd like help building this kind of setup for your team - local inference, agentic workflows, or the measurement discipline itself - that's exactly what our consulting and agentic AI services cover. Our tools page has everything we've open-sourced.

The full write-up

Two articles carry the detail. The vLLM tuning deep-dive covers the kernel trap, speculative decoding, the failures, and every number with its methodology. The journey update tells the story around it - why we left LM Studio, the model stable, and the face-off that deleted two models in an evening. The harness itself lives in the repo, because a benchmark you can't interrogate is just an opinion with digits.

Questions about the testing, or a model you think we should run? Get in touch .