The Dual-4090 96GB vLLM Benchmark & Runbook

Two modded RTX 4090s, 96GB of VRAM between them, and a question that shaped a five-day measurement campaign: can a £6,200 modified Ada rig match enterprise MoE throughput? This page is where that question gets answered with a stopwatch rather than a forum thread. It's the permanent measurement record for the rig - the twelve-model benchmark table, the power, topology and quantisation experiments, the method that produced them, and the raw dataset - and it stays live, updating as the rig produces new data. The story of how the campaign went, and the five standing rules it killed (three of them mine, which still stings a little), is a separate piece: the story behind the numbers . This page is the reference that story stands on.

Every figure below comes from one frozen dataset, and you can take it away with you - the full campaign record, per-run rows included:

Download the raw dataset (JSON)

The rig

The hardware is two modded RTX 4090s with 48GB of VRAM each - 96GB across the pair, roughly £3,100 a card, £6,200 for the rig's compute. GPU 0 sits on a PCIe x16 link and drives the displays; GPU 1 is on an x4. Behind them: 128GB of system RAM, Windows 11 with Docker Desktop on WSL2, and vLLM 0.26.0 pinned by image digest (the pinning matters - more in the method section). The production power policy is a 330W cap with the core clock locked to 0-2200MHz, which started life as a stability requirement on these rebuilt cards and turned out, under measurement, to cost nothing.

The comparison that frames everything here: the single-card route to 96GB is NVIDIA's RTX PRO 6000 Blackwell, an exact VRAM match, at £12,959.99 UK retail when I checked. Same ceiling, about half the money - and the toll you pay instead is two-card complexity. Tensor-parallel decisions, a power policy, an x4 link with opinions of its own. This page is, in effect, the manual for paying that toll.

The fleet: twelve models, three contexts

Twelve open-weight models verified end to end between the 15th and 19th of August - US, Chinese and European vendors, 1.2B to 120B, dense and MoE, five quantisation formats. Each row carries the model's weight footprint, decode speed in tokens per second (tok/s) at three measured context depths, load time, tensor-parallel setting, native context and MoE expert count. The table is sortable; click a column header and have a play.

Three cells in the 16k column are empty, and they stay empty rather than getting a guessed value. LFM2.5-1.2B and Llama-3.3-70B-AWQ overran their serving context at that round - the 16k label measured nearly double its nominal size on some tokenizers (see method), and both requests came back HTTP 400. GLM-4.7-Flash is stranger: it returns "no tokens generated" at around 16k of context, and I haven't diagnosed why yet. That one's on the open list.

One reading note on the footprint column, because it caught my own tooling out: footprint is the weight files on disk, validated against the shard indexes, not VRAM while serving. At --gpu-memory-utilization 0.92 vLLM fills 92% of each card by design - weights plus preallocated KV cache, up to the target - so serving VRAM reads about 45GB per card for every model regardless of size. It measures the setting, not the model. The one serving-VRAM figure this page keeps is gpt-oss-120b's 92.7GB total, retained as the MXFP4 existence proof (a 117B model whose bf16 weights would need roughly 234GB, serving inside 96GB - the weights demonstrably stayed 4-bit).

Power and clock: what does decode cost?

The production setting locks the clocks and caps the power at 330W, an instinct I carried over from Ethereum mining. The experiment isolates the two: condition A is the production setting (330W cap, clocks locked), B raises the cap to 450W, C keeps the 330W cap but unlocks the clocks. Three models, six repeats per condition.

Decode never moved. Qwen3.8-27B ran 35.1, 35.1 and 35.5 tok/s across A, B and C; Llama-3.3-70B, split across both cards, ran 34.9, 35.2 and 35.1. The electricity moved instead: unlocking the clocks pushed the Qwen's mean draw from 279.8W to 327.3W - 17% more power for a 1% decode change - and on the 70B the unlock added roughly 90W of draw, from 561.9W to 652.4W, for well under 1%. The lock saves 90W for nothing. Cooler cards, quieter room, smaller bill, same tok/s.

A third model, the little LFM2.5-1.2B, ran the same conditions but one of its passes collided with a background download; that data stays in the dataset flagged as a disk-contention exhibit, not as evidence. And a caveat I want on the record: nvidia-smi accepts the lock command on GPU 0 and reports success, but the "locked" clocks read 2640MHz and above under the Llama runs, so whether the lock is enforced under load on that card is unproven. The power-draw deltas stand on their own; the enforcement question has a rerun scheduled.

Topology: one card, two cards, or a pipeline?

The standing doctrine says never tensor-parallelise a model that fits on one card, because every decode step pays an all-reduce over the x4 link. Measured, on Qwen3.8-27B (which fits comfortably on a single 48GB card): TP1 decoded at 34.4 tok/s at long context, TP2 at 48.9 - a 42.2% gain from splitting a model that never needed splitting. The doubled memory bandwidth beats the x4 tax with room to spare. Pipeline parallelism, the arrangement the doctrine would prefer if you must use two cards, came in at 32.9 - slower than not splitting at all. Llama-3.3-70B tells the same story at short context: TP2 35.3, PP2 21.8. As a side benefit, the TP2 load finished in 9.2 minutes against TP1's 14.6.

Scope, before you re-architect anything on my say-so: one model, one context, single stream. What TP2 does under four simultaneous streams is a pending test, and I'd want that number before moving production.

Quantisation: the format is the speed

The cleanest controlled pair in the dataset: the same Qwen3.8-27B decodes at 35.8 tok/s as AWQ-INT4 and 16.6 as FP8-dynamic. Same model, same card, roughly double the speed, purely down to which file you downloaded. Llama-3.3-70B shows the same direction, 23.2 against 19.1. Decode is memory-bandwidth-bound - bytes-touched-per-token, not parameter count - and the format that touches fewer bytes wins. It's also why the fleet table's speed order looks nothing like its size order.

The doctrine ledger

Five standing rules of local LLM inference met the measurements above and died. The ledger, each with its refuting number:

1. "Never tensor-parallelise a model that fits one card." REFUTED - TP2 +42.2% over TP1 on Qwen3.8; PP2 32.7% slower than TP2.
2. "Ada has no kernels for block-FP8, NVFP4 or MXFP4." REFUTED - vLLM's own capability table permits every one of those formats at compute capability 8.9, probed on these exact cards. Nothing had ever been measured; the rule was an inference written down as fact.
3. "MXFP4 upconverts to bf16 on Ada." REFUTED - gpt-oss-120b, a 117B MoE shipped natively in MXFP4, serves at its full 131,072-token context in 92.7GB total. The bf16 weights would be ~234GB and cannot exist in 96GB. Then the sweep measured it at 153.4 tok/s.
4. "The 9P bind mount is irrelevant to load time." REFUTED - a controlled A/B with the mount as the only variable: 763 and 762 seconds on the WSL2 bind mount, 203 and 204 on an ext4 Docker volume. 3.75x, reproduced to the second.
5. "The cards must run flat-out for best performance." REFUTED - the power and clock section above. Decode flat across all three conditions; the lock saves ~90W for nothing.

And one deliberate non-entry: MTP speculative decoding is not a doctrine kill, in either direction - it measured neutral on this engine. Around 47 tok/s whether switched on or off, across ~0, ~4k and ~15k of context (47.4/46.9/45.9 off against 51.1/44.6/47.8 on - within the noise floor). The draft head itself is healthy, accepting 69% of its guesses; the same preset bought a measured ~1.9x on the 0.25.x engine, so this is a version regression to neutral, not a reversal of the idea. It stays off in production anyway, because on 0.26.0 it can deadlock on a long cold prefill while the health endpoint carries on returning 200. A neutral result held to the same evidential standard as the kills - that's the deal on this page.

Method and provenance

This is the "why you can trust these numbers" section, and it's mostly a list of scars.

Onboard before you bench. Every model was loaded alone first, its real errors captured, its settings corrected and retried, and its working flags recorded - before any timing run touched it. Four of the first eight models failed their first load attempt, each for a different reason, so the discipline paid for itself immediately.

The engine is pinned by digest. An earlier compose file said latest, and the stack upgraded itself from 0.25.1 to 0.26.0 mid-week with nobody choosing it - a week of benchmarks became comparable to nothing before or after. Every number on this page is vLLM 0.26.0, pinned by image digest, and the MTP regression above is exactly why the version matters.

Context is measured, not requested. Every context figure comes from the API's own usage.prompt_tokens, not from the label on the request. The distinction earns its keep: rounds before the 18th of August requested roughly 1.9x their nominal label (a chars-per-token guess that different tokenizers treated very differently), and the measured truth was stored per row throughout - which is also why the "16k" round overflowed two models' serving contexts.

One model in VRAM at a time, with a preflight check first, because production vLLM releases VRAM when idle and reloads on request - a free-looking card can fill mid-load.

Sampling comes from each model's own generation_config.json, never from the client, and the thinking toggle is set server-side using that model's own key. Vendor-recommended settings, verified per model.

The exclusions are recorded, not buried. Three models didn't make the fleet: Kimi-Linear-48B (its tokenizer requires executing Python shipped with the weights - a policy exclusion, not a technical one), Muse-Glimmer-30B (architecture absent from the pinned 0.26.0 registry), and a community AWQ conversion of gpt-oss-120b whose MoE experts were never quantised at all (replaced by the official MXFP4 release).

And the instruments failed more than the models did. Twelve distinct instrument failures are logged in the dataset, verbatim - an error capture that kept the wrong end of the log, a GPU sampler that indexed characters instead of fields, a token counter that tallied SSE chunks and booked MTP's neutral result as a 44% slowdown before a hastier check over-corrected it to a gain. The rules that fell out: suspect the instrument before the subject, and read the number, not the verdict. The full postmortem is its own article, later this week.

Open questions

Stated as open, because pretending otherwise is how the doctrines above got written in the first place:

• GLM-4.7-Flash returns no tokens at ~16k context. A soft quirk, reproducible, undiagnosed.
• GPU 0's clock lock is unproven under load. The command succeeds; the clocks read 2640MHz anyway. A rerun with elevation and load is scheduled.
• Nemotron-3.5-Lightning at 158.4 tok/s is 3.4x the production daily driver - which raises the obvious question of whether it should be the daily driver. The delegation gauntlet that decides it hasn't run yet.
• TP2 under concurrency. The +42.2% is single-stream; the production question needs four simultaneous streams measured before anything moves.

Reproducing these numbers

The runbook half of this page's job. If you're building toward a rig like this, the vLLM setup guide is the on-ramp, and the first tuning pass on these same 48GB cards is the step-by-step that predates this campaign - the campaign is the measurement layer on top of it. The frozen dataset at the top of this page carries every per-run row, every verified launch config, the exclusions and the instrument catalogue, so you can check my arithmetic or rerun any experiment against your own hardware. The full production runbook - the exact flags we ship per model, with the reasoning attached - publishes as this cluster's final piece later this week.

If you run something on comparable hardware and get a number that disagrees with this table, I want to hear about it - that's what a living benchmark page is for. Drop me a line.