The broken rules of local LLM inference

Before I ran language models on this machine, I mined Ethereum. Water-cooled rigs, 3090s, the lot - I wound it down early in the end, partly because a hot UK summer and a room full of GPUs are a miserable combination. But the craft stuck with me. Once a rig was built, most of the actual skill was in locking the core clocks and capping the power: do it properly and the card ran cooler and hashed higher. Counter-intuitive the first time you watch it happen, then obvious forever after.

So when I found myself with two RTX 4090s serving language models, that instinct turned into a proper list of questions. The cards will pull anywhere from 300 to 600W if you let them - do they actually need that power to hit their best tokens per second (tok/s)? Can a pair of consumer cards run the big models at all, the 70Bs and the 120Bs, or is that strictly datacentre work? When you've got two cards, how should you split a model across them? And underneath all of it, the question that makes the rest answerable: what does a good number even look like, so you know when you've got it right?

These are engineering questions, not prompting ones, and that distinction is the whole reason I bothered. There's a difference between knowing how to use ChatGPT and knowing how to build a machine that runs a model at its best - the second is a real skill, largely untapped, and this rig is where I went to learn it. So the method was the plain one: ask each question, then measure the answer instead of assuming it. Five days later I had them - and several of the answers killed rules I'd taken as settled. Five, in the end, and three were mine, written into my own documentation as fact, which is the part I'd rather skip past. What follows is those questions, how I answered them, and what died along the way. Every number comes from the campaign's frozen dataset (study-dataset.json, downloadable from the full benchmark hub ).

New to running models locally? This one assumes you're ready to get your hands into vLLM and Docker. If you've never tried it, LM Studio is the gentlest way to run your first local model in an afternoon, and our vLLM setup guide is the on-ramp to a rig like this - come back when you fancy something more involved. If that's already familiar ground, read on.

The rig, and why it exists

The hardware is two modded RTX 4090s with 48GB of VRAM each - 96GB across the pair. I paid about £3,100 a card, so roughly £6,200 for the rig's compute. The mod is load-bearing context, not a footnote: these are rebuilt cards with doubled memory, and mine will blue-screen from microsecond power spikes if the clocks run unlocked. That's why the production setting is a 330W cap with the core clock locked to 0-2200MHz. Hold that thought; it becomes an experiment shortly.

The reason the rig exists at all is economics. The single-card route to 96GB is NVIDIA's RTX PRO 6000 Blackwell - 96GB of GDDR7, an exact match for this rig's ceiling - and the UK retail price when I checked was £12,959.99, with some PNY OEM listings sitting nearer £7,700. Same VRAM ceiling, about half the retail money. The toll you pay instead is two-card complexity: an x16 slot and an x4 slot, tensor-parallel decisions, a power policy, load-order rules. This study is, in effect, the manual for paying that toll.

For the rest of the picture: GPU 0 sits on the x16 link and drives the displays, GPU 1 on the x4; 128GB of system RAM; Windows 11 with Docker Desktop on WSL2. About as far, I think, as a home machine goes in 2026 without the budget getting properly frightening.

Five days, twelve models, one pinned engine

The campaign ran from the 15th to the 19th of August: I verified twelve open-weight models end to end, excluded three with their reasons recorded, all on vLLM 0.26.0 pinned by image digest. The pinning is itself a scar - an earlier compose file said latest, the stack upgraded itself from 0.25.1 mid-week with nobody choosing it, and a week of benchmarks became comparable to nothing before or after. I learned that one the expensive way.

My method was onboard-before-bench: I loaded each model alone, captured the real errors, retried with corrected settings, and recorded the working flags before any timing run touched it. I needed the discipline, too - in the first onboarding pass, four of eight models failed to load at the first attempt, each for a different reason, and none of them the model's fault.

The fleet order at short context runs Nemotron-3.5-Lightning at 158.4 tok/s, gpt-oss-120b at 153.4, GLM-4.7-Flash at 134.5, and on down to Qwen3.8-27B-FP8 at 16.6. The spread between fastest and slowest is nearly 10x, and parameter count explains almost none of it.

What explains rather more is the quantisation format. The same Qwen3.8-27B decodes at 35.8 tok/s as AWQ-INT4 and 16.6 as FP8-dynamic - the same model, double the speed, purely down to which file you happened to download. Llama-3.3-70B shows the same direction, 23.2 against 19.1. Decode speed is bytes-touched-per-token, not parameter count, and the format that touches fewer bytes wins.

And one "accelerator" turned out to do nothing at all. MTP speculative decoding had bought a real 1.9x on the previous vLLM; on the pinned 0.26.0 it measured dead neutral - around 47 tok/s whether I switched it on or off, across every context I tried. The draft head was healthy, guessing right about 69% of the time; the engine had simply regressed to the point where running the drafter cost as much as it saved. It stays off in production, helped along by a deadlock it can still trigger on a long cold prefill. Getting to that flat number honestly took three instruments and two wrong answers, which is a story for the postmortem below.

Do the cards need all that power?

This is the mining question, and I'd been waiting months to run it properly. I'd caught the headline of it once before, capping these same cards to 330W in an earlier tuning pass and watching the big models barely notice, but that run moved power and clock together. This time I isolated them. I gave it three models, three conditions and six repeats each: A is the production setting (330W cap, clocks locked), B raises the cap to 450W to isolate the power limit, C keeps the 330W cap but unlocks the clocks.

The decode number never moved. Qwen3.8-27B ran 35.1, 35.1 and 35.5 tok/s across A, B and C. Llama-3.3-70B, split across both cards, ran 34.9, 35.2 and 35.1. What did move was the electricity: unlocking the clocks pushed the Qwen's mean draw from 279.8W to 327.3W - 17% more power for a 1% decode change - and on the 70B the unlock added roughly 90W of draw for well under 1%. The lock saves 90W for nothing.

The raw run, straight off the bench: three power and clock conditions on the 70B, decode barely flinching while the wattage moves. The clocks reading 2640 whether locked or not is exactly the enforcement question I have left open.

So rule one - you must let the cards run flat-out for their best tok/s - is dead, at least for single-stream decode on this rig. The mining discipline transfers straight across: lock the clocks, cap the power, lose nothing. Cooler cards, quieter room, smaller bill.

One thing I cannot yet prove: whether the clock lock on GPU 0 holds under load. nvidia-smi accepts the lock command and reports success, but clocks.max.graphics reports the hardware ceiling whether you've locked it or not, and during the Llama runs the "locked" clocks read 2640MHz and above. The power-draw deltas stand on their own; the lock-enforcement question has a rerun scheduled, and until it lands I'm treating it as open. The first Llama pass, which scoped its conditions to one GPU of a two-GPU model, sits quarantined in the dataset rather than deleted - you can go and look at it.

Should you split a model that fits on one card?

The standing doctrine says no: tensor parallelism across an x4 link means every decode step pays an all-reduce over a slow bus, so a model that fits on one card should stay on one card. I believed it, and it's written down in more than one place as though it were physics.

So I measured it, on Qwen3.8-27B (which fits comfortably on a single 48GB card): TP1 decoded at 34.4 tok/s at long context, TP2 at 48.9. That's 42.2% faster from splitting a model that never needed splitting - the doubled memory bandwidth of two cards beats the x4 all-reduce tax with room to spare. Pipeline parallelism, the arrangement the doctrine would prefer if you must use two cards, came in at 32.9 - slower than not splitting at all, because the pipeline bubble costs more than the all-reduce saves. As a side benefit, the TP2 load finished in 9.2 minutes against TP1's 14.6.

The topology sweep on the bench, both models: TP2 taking a clean +42.2% at long context, and the full run log beneath it - power verified, engine warmed, production restored.

The scope hedge, before anyone re-architects production on my say-so: this is one model, one context, single stream. The concurrency test - what TP2 does under four simultaneous streams - hasn't run yet.

The formats Ada supposedly cannot run

For weeks my own repository stated, as fact: Ada has no kernels for block-FP8, NVFP4 or MXFP4. Three models were rejected on that basis, and I'd written an automated gate to enforce the rule on future runs - a wrong sentence about to be inherited by every later decision, mechanically.

Then I asked the engine. vLLM carries a minimum-capability table for every quantisation method it supports, and my cards report compute capability 8.9 (Ada). MXFP4: minimum 80, supported. NVFP4: minimum 75, supported. Block-FP8: minimum 75, supported. Every format the rule forbade is permitted by the engine's own capability gate, on this exact GPU. Nothing in my repo had ever measured any of it - the rule was an inference from a plausible proxy, written down as fact, and it survived because written-down things do.

A permitted code path is not a fast one, though. The capability probe refutes "no kernels"; it doesn't establish "good". That still needs a number, and the number arrived the next day.

The 120B that should not have fit

The companion rule said MXFP4 upconverts to bf16 on Ada - that a model shipped in the format would balloon back to full size in VRAM, making it pointless on consumer cards. If that rule were true, what happened on the 18th could not have: OpenAI's gpt-oss-120b, a 117B-parameter MoE shipped natively in MXFP4, loaded across both cards and served at its full native context of 131,072 tokens in 92.7GB. In bf16 the weights alone would be roughly 234GB. They cannot exist inside 96GB of VRAM, so the weights demonstrably stayed 4-bit. A claim that had ruled models out for six weeks died in a single 1,334-second load.

Then the sweep put a number on it: 153.4 tok/s at short context, still 140.7 with 16k of prompt in the cache. For scale, that's a 120B-class model pacing the fastest thing on the rig - the 30B Nemotron-3.5-Lightning at 158.4 - on hardware that cost me £6,200. The trick is the same bytes-touched arithmetic as the quant pairs above: the MoE activates only 4 experts per token, so the big model reads like a small one.

Here are the flags behind that 153 tok/s, lifted straight from the verified config:

# gpt-oss-120b on the dual-4090 rig, vLLM 0.26.0
--tensor-parallel-size 2            # both cards
--max-model-len 131072              # full native context
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92
# standing defaults for every model on this rig:
#   VLLM_USE_V2_MODEL_RUNNER=0        (MoE and multimodal models crash on the V2 runner under WSL2)
#   engine pinned by digest:          vllm/vllm-openai@sha256:ffb2d59b...

The full docker invocation, and the per-model flags for all twelve, live in the benchmark hub .

And the part I'd rather not admit: the first version of my verdict tool got this backwards. It read vLLM's memory allocation - 92% of each card, filled by design with weights plus preallocated KV cache - as evidence of upconversion. The instrument measured its own configuration, inside the tool I'd built specifically to settle an instrument dispute. Read the number, not the verdict.

The mount I had already ruled out

The last rule was mine too, and it had a benchmark behind it, which is what made it dangerous. Model loading on this rig had felt slow forever, and the obvious suspect was the WSL2 bind mount - Docker on Windows reaches files on the Windows side through 9P, a network filesystem protocol borrowed from Plan 9, and 9P pays a latency cost on every file operation. I'd tested it, back on the 12th, with dd: 530MB/s sequential reads. Fine, I concluded, migration buys nothing, and the migration script got marked SUPERSEDED on the strength of it.

The controlled A/B says otherwise. Same model, same flags, same engine, the mount as the only variable, alternating order, two repeats each: bind mount 763 and 762 seconds; Docker volume on ext4, 203 and 204. A 3.75x difference, reproduced to the second. The dd number was perfectly accurate - about a workload vLLM never runs. The loader's access pattern is thousands of small operations, each paying 9P's per-operation latency, and a sequential-throughput test never sees any of it. An instrument can be exactly right about the wrong question.

I moved the fleet's weights to a proper volume on the 19th. The one-off copy cost 175 seconds per model, which the first load repays three times over.

What five dead rules have in common

Line the five up and the anatomy is identical. Each began as an inference from a plausible proxy - a spec-sheet fact, a sequential benchmark, a forum consensus, an older architecture's behaviour. Each got written down without its provenance attached. And once written, each was inherited by every later decision as though it had been measured, to the point that I twice built automation in good faith to enforce a rule that measurement was about to kill. The gate is the frightening part: a wrong sentence in a doc costs you when you read it; a wrong rule in a gate costs you forever, silently.

What made the campaign slow wasn't the models, either. Twelve distinct instrument failures cost me more time than any genuine incompatibility - my error capture kept the wrong end of the log, my monitor reported success having checked nothing, my GPU sampler indexed characters instead of fields and never threw. The MTP number above is the cleanest of the lot, and the reason I can state it plainly now. My first sweep booked that switch as a 44% slowdown, because the counter was tallying server-sent-event chunks as though each carried one token. That is exactly right for ordinary decoding and about 2.4x wrong under speculative decoding, which bundles several tokens into every chunk - so the accelerator got recorded as a brake. A second, hastier check then over-corrected to a 26% gain. Only counting from the model's own reported token usage, across three contexts, gave the flat truth. Three instruments, one switch, and just one of them honest. That story runs long enough to be its own piece, and it will be, later this week. The rules that fall out of it are short enough to carry now: suspect the instrument before the subject, read the number rather than the verdict, and remember the thing reporting status is not always the thing doing the work.

Where this goes next

Every number above, plus the full twelve-model table with per-model flags, load times, measured contexts and the raw dataset download, lives in the benchmark hub - that page is the living asset, and it updates as the rig produces data. The deeper dives publish across this week as siblings: the VRAM traps that stopped a 16.6GB model loading on a 48GB card, the full instrument postmortem, and the production runbook with every flag we ship.

None of this stays niche for long. The workstation on a serious desk in a few years will be AI-ready by default - a big GPU, a big local model, quick enough that running one stops feeling like an occasion. When the hardware is a given, the only thing that separates people is what they choose to do with it.

It's one thing to be able to use chat - well done, you can type. Configuring a model to run at its best in a serious harness like vLLM, on hardware you own, with settings you can defend because you measured them, is a different discipline, and it looks a lot less like prompting and a lot more like the mining bench: lock the clocks, cap the power, trust nothing you haven't watched under load. The cards never needed to run flat-out. I suspect a fair few of the other rules we're all carrying will go the same way, and I'd rather be holding the stopwatch when they do.

If you're running local models and carrying a rule you've never measured, I'd like to hear about it - drop me a line, and if it's testable on 96GB of Ada, it goes on the list.