Our Model Testing

Every model recommendation on this site traces back to a number we measured ourselves. This page is the front door to that testing: what we run, how we run it, and what we've learned that the leaderboards won't tell you.

The short version of why: we run a local AI fleet as a delegation sidekick - a set of models on our own hardware that a frontier orchestrator (Claude) hands bounded work to, so paid API tokens get spent only where frontier reasoning earns them. That architecture only works if you know, precisely, what each local model can do and how fast. Guessing isn't knowing. So we test.

What we test on

The current rig is a single modded 48GB RTX 4090 (a second identical card lands imminently - tensor parallel and dual-server numbers will follow), 128GB RAM, serving with vLLM in Docker. Not datacentre hardware, deliberately: the point is what's achievable on a machine an enthusiast or a small studio can actually build.

How we test

Every model faces the same harness: single-stream decode speed, cold and cached time-to-first-token on a 16k prompt, prefill throughput, and concurrent aggregate. Speed alone doesn't earn a place - models must also pass a structured tool-calling gate (a real tools request that has to come back parsed, not as text soup), vision models must read text out of images we generate, and coding models face a real TypeScript task judged on whether the output compiles. One variable changes per run. Failures get recorded alongside wins - some of our most useful findings are the things that didn't work.

A few results that illustrate why we bother: a model with nearly a million downloads and a strong benchmark pedigree produced code that doesn't compile on a 20-line real-world task. An unbenchmarked community model won that same face-off. And the single biggest speedup we found came not from a clever flag but from choosing a different quantisation file of the same model - a 3x difference the model cards don't mention.

The current numbers

The fleet as it stands, measured 20 July 2026 on the single 48GB card at stock power. A scheduled sweep re-runs these weekly at the standing 330W power policy and this table tracks it.

Decode is single-stream at temperature 0; prefill and TTFT are a 16k-token prompt; repeat TTFT shows prefix caching working. The last row is the same 27B before tuning - the gap between 18.8 and 66 is what the deep-dive below explains.

The frontier models, as reported

The local fleet exists to take work off paid APIs, so the other half of the measurement question is what those APIs cost and deliver. Unlike everything above, these are reported numbers - vendor pricing and public leaderboards as of July 2026, not our harness - and we keep that distinction loud on purpose. The gauntlet runs against the frontier APIs (same TypeScript task, same tool-calling gate - the harness doesn't care that the endpoint is remote) are queued; when they land, this table gets our numbers.

The chart is the whole economics argument in one picture: two orders of magnitude separate the cheapest and most expensive output tokens on the frontier, and a local token costs none of it. The hardware isn't free, of course - our GPU guide covers what the cards cost and which make sense - but once a card is on the desk, every token it decodes takes work off the metered tiers above. That's the comparison that matters: not local versus cloud as a religion, but which tier each job deserves.

The routing conclusion is ours, and it's a three-tier ladder: the local fleet for bounded work (free after hardware, measured), DeepSeek V4 Pro as the candidate middle tier for high-volume agent loops (unverified by us, so no real work routes there yet), and Claude for orchestration and anything where being wrong costs more than the tokens. Tool-calling reliability - the thing our whole delegation architecture depends on - is where the frontier lead is least contested.

Use this with Claude

The practical way to use these findings is houtini-lm , our open-source MCP server that connects Claude to your local models. Claude orchestrates; your hardware executes; your API bill shrinks. It works with LM Studio out of the box and with a vLLM stack like ours - the model routing in it is built directly from the measurements on this page.

If you'd like help building this kind of setup for your team - local inference, agentic workflows, or the measurement discipline itself - that's exactly what our consulting and agentic AI services cover. Our tools page has everything we've open-sourced.

The full write-up

Two articles carry the detail. The vLLM tuning deep-dive covers the kernel trap, speculative decoding, the failures, and every number with its methodology. The journey update tells the story around it - why we left LM Studio, the model stable, and the face-off that deleted two models in an evening. The harness itself lives in the repo, because a benchmark you can't interrogate is just an opinion with digits.

Questions about the testing, or a model you think we should run? Get in touch .