The dual-4090 runbook: my working vLLM settings for twelve local models

No two open source LLMs are the same. During my research I've discovered just how different, in setup and api request / response, they can be. So if you're swapping models with expectations of performance, only to be disappointed, read this rather than spending a whole night trying to get your head around what's going on.

Below are my working vLLM settings for twelve open-weight models on a dual-4090 rig, and for each one the part the settings don't explain: what tripped me up, and how I got it running. The broken rules and the VRAM traps are the background if you want it; this is the hands-on part.

First, though, a quick word on where all this configuration lives - because "OpenAI-compatible" hides most of it.

The doorway and the dials

"OpenAI-compatible" describes the doorway, not the room. It tells you the URL and the JSON shape, and that part is useful - one client talks to all twelve models. But whether a model runs, and runs right, is decided somewhere else: in the settings you hand the server when you launch it, per model.

Those settings come from three places, and only the first one feels like configuration. Your launch flags are the obvious ones - how many cards to split across, how much context to allow, the KV-cache precision, how hard to fill the VRAM. The model's own files are the sneaky ones: config.json holds the architecture, generation_config.json holds the vendor's recommended sampling, and vLLM reads that file by default - so if your client sets temperature, you're quietly overriding the people who trained the model. And the chat template, a Jinja file, is where the switch for turning reasoning on and off lives, which is why that switch is named something different on nearly every model. Then there's the rig discipline you set once and forget: pin the engine by digest so it can't upgrade itself under you, turn the V2 runner off because it crashes MoE models on WSL2, and never pass --trust-remote-code.

One rig decision isn't a launch flag at all, and for the big models it matters more than most of the flags that are: where the weights live. On Windows, Docker reaches host files through a network filesystem (9P) that charges a latency toll on every one of the thousands of small reads a model load makes - enough that a 120B can take the better part of an hour just to come up. Keep the weights in a proper Docker volume instead of a bind mount and that toll disappears: the biggest models on this rig load five to nine times faster for it, and the full before-and-after is in the broken rules of local LLM inference . If you're stuck on a bind mount and can't move them, --safetensors-load-strategy=prefetch claws back roughly a quarter of the time on 9P - it never switches on by itself there, so you have to ask for it - but moving the weights still wins by a distance.

The per-model differences belong in the server launch, never in your client. Get that right and a single identical request talks to everything. Get it wrong and you're comparing harnesses while you think you're comparing models - and you won't spot it, because the request still looks completely standard.

Right. The models. This one first, because it trips two assumptions at once.

Nemotron-3-Super-120B: the model that ships code it never runs

Here is what serves it, on both cards:

# NVIDIA-Nemotron-3-Super-120B-A12B-AWQ-4bit, vLLM 0.26.0, dual 4090
--tensor-parallel-size 2            # 80.7GB of weights - it needs both cards
--max-model-len 32768
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92
--reasoning-parser nemotron_v3      # v3, not v2 - the parser is versioned
# NO --trust-remote-code            # it ships .py files it never executes (below)
# standing rig defaults:
#   VLLM_USE_V2_MODEL_RUNNER=0       # the V2 runner crashes this MoE under WSL2
#   engine pinned by image digest

This 120B looks like it needs --trust-remote-code, and it doesn't. That flag lets a downloaded model run its own Python, so it's one to keep off by default - which is why a model that appears to want it is worth a second look before you throw it away. It looks that way because the model ships four .py files with an auto_map in config.json pointing at them, which is the usual sign a model wants to run its own code. I was wary enough to write a fleet audit that rejects any model with an auto_map, and it flagged this 120B as unsafe - while it was loading perfectly, four shards from done, with no --trust-remote-code set.

What's going on is simpler than it looks. vLLM has NemotronHForCausalLM in its own registry and runs the model with its own code; the .py files just sit there, never used. Shipping code and running code aren't the same thing, and my audit was checking the wrong one - whether Python was present, not whether any of it runs. So the fix is to do nothing: leave --trust-remote-code off and let vLLM use its native path. (I changed the audit to a warning that asks the right question - does it serve without the flag? If it does, nothing ran.) One other thing this family catches you on is turning thinking off. The documented way, a "detailed thinking off" system prompt, is from Nemotron 2; v3 ignores it silently, so the model keeps thinking and quietly inflates your token counts. On v3 you pass the toggle as a chat_template_kwargs instead, and you set the parser to nemotron_v3, not v2, or it won't start.

Did that fix it? It did. It comes up across both cards in about three minutes from the volume, and it barely slows down as the context fills, which is unusual for a model this size. The rule worth keeping: don't treat "the repo has Python" as "the model runs Python". Check whether it serves without --trust-remote-code. If it does, the code never ran, and you've kept a model you'd otherwise have thrown away.

gpt-oss-120b: the 120B that fit, and the one that lied about it

# openai/gpt-oss-120b (official, native MXFP4), vLLM 0.26.0, dual 4090
--tensor-parallel-size 2            # 65.2GB of weights across both cards
--max-model-len 131072              # full native context - and it fits
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92
# thinking toggle for this family is the kwarg "thinking" (not enable_thinking)

The config above is boring, and that's the headline. A 117-billion-parameter model, shipped in MXFP4, loads across both cards at its full 131k context in about 92.7GB and holds 153 tokens a second. Boring is the win here, because for weeks my own notes said MXFP4 gets upconverted to full precision on Ada, which would mean around 234GB of weights and no chance of fitting 96GB. It fit. (The weights stayed 4-bit - 234GB can't live inside 96GB - and a rule I'd carried for six weeks died in one load. That story is in the broken rules ; here you just need the flags.)

The part that will bite you is the version I tried first. Before the official release landed I grabbed a community "4-bit" AWQ conversion of the same model, and it ran out of memory on load - which makes no sense for something that's meant to be quantised. So I opened the file, and there it was: the metadata showed the MoE experts, the bulk of the model, were never quantised at all. F16, every one. The name said 4-bit; the weights said full precision, so vLLM took the unquantised path and filled both cards. The fix was to stop trusting the name, read the metadata, and use the official MXFP4 build. Did that work? Yes - the official build is the one up at the top, the 117B that fit inside 96GB. A quant's name is a claim and the file is the truth - and it costs you nothing to check which, before you spend an hour downloading the wrong one.

The two Llama-3.3-70Bs: same model, one flag, two different contracts

I ran this one twice, once in AWQ and once in FP8, and the two builds behave so differently that it's the clearest lesson on the rig in how much rides on which file you download, not which model.

# Llama-3.3-70B-AWQ - fits ONE card, but not at full context
--tensor-parallel-size 1
--max-model-len 21110               # 39.8GB weights leaves ~4GB for KV - see below
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92
# no reasoning parser - Llama 3.3 isn't a reasoning model

# Llama-3.3-70B-FP8 - needs BOTH cards, and quietly loses a feature
--tensor-parallel-size 2
--max-model-len 32768
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92

Take the AWQ build first, because the number in that comment is the whole VRAM-traps story in one line. It's 39.8GB of weights, which slides onto a single 48GB card with room to spare - and then won't serve its own advertised 131,072 context, because the weights are only half of what has to fit and the KV cache is the other half. On one card it tops out around 21,110 tokens. That's not a bug, it's arithmetic nobody prints on the box, and I've written it up properly in the VRAM traps - for the runbook, just know that the odd-looking 21110 is measured, not a typo.

Then the FP8 build caught me out. It's 72.7GB, so it needs both cards and loads about two-thirds slower (1,536 seconds against 940) - fair enough for the bigger file. What you would not expect is that the FP8 build's chat template doesn't handle tools and the AWQ build's does. Same model, same maker, same weights underneath, and whether you can call functions comes down to which quantisation someone uploaded. The quantiser changed the contract, silently, with nothing to warn you. So I keep both and keep them straight: reach for AWQ when you want tools and a single card's context is enough, FP8 across both cards when you want the longer window and don't. Check tool support per build, not per model - the model card won't tell you, because as far as it's concerned they're the same thing.

GLM-4.7-Flash: the kilobyte in the wrong memory

# GLM-4.7-Flash-GPTQ, vLLM 0.26.0, one card
--tensor-parallel-size 1
--max-model-len 32768
--kv-cache-dtype auto               # NOT fp8 - and that single word is the fix
--gpu-memory-utilization 0.92

Every other model here runs an fp8 KV cache and this one, alone, does not - and if you've read the VRAM traps you already know why, so I won't tell the whole tale twice. The short version for the runbook: a 16.6GB model refused to load on a 48GB card, short by exactly one kilobyte of shared memory - a scratchpad inside each SM, nothing to do with VRAM - because the fp8 KV-cache kernel was tuned for a bigger card. Serving with --kv-cache-dtype auto picks a different kernel entirely. Did that fix it? It loaded in 441 seconds and behaved - one word in one flag. I've left that comment loud in the config because it's the single least guessable line in this whole runbook, and the day it caught me I lost the best part of an evening to it.

The two Qwen3.8-27Bs: the temperature that binned an evening

# Qwen3.8-27B-AWQ - the faster of the pair, oddly
--tensor-parallel-size 1
--max-model-len 32768
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92
--reasoning-parser qwen3
# tool parser qwen3_xml; thinking toggle is the kwarg enable_thinking
# and DO NOT pin temperature - the vendor default is 1.0 (yes, really)

# Qwen3.8-27B-FP8 - same model, roughly half the speed
--tensor-parallel-size 1
--max-model-len 32768
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92
--reasoning-parser qwen3

Two things here, and the first cost me an evening - so let me save you the same one. Qwen ships a generation_config.json next to the weights with the sampling the people who trained the model recommend, and vLLM reads that file by default. For this family the recommended temperature is 1.0. That feels wrong, because you've spent years being told to run low temperatures when you want a deterministic answer - but it isn't wrong: Qwen explicitly warn against low-temperature decoding for the family, because of how they post-trained it. My harness had a profiles table pinning Qwen to 0.2, under a tidy comment claiming that was "vendor-documented". It wasn't. And a client that sets temperature silently overrides the file on every request, so I ran a whole evening of measurements off-distribution before I noticed and binned the lot. So: don't set temperature at all unless you honestly know better than the people who trained the model. The file already told you, and overriding it makes no noise.

The second thing is the AWQ-versus-FP8 comparison, which comes out the counter-intuitive way round. The FP8 build is the slightly bigger file (30.9GB against 27.7) and decodes at roughly half the speed - about 16.6 tokens a second where the AWQ manages 35.8. Decode speed is really bytes-touched-per-token, not parameter count, and AWQ's four-bit weights touch fewer bytes per token than FP8's eight, so on a memory-bandwidth-bound job the "smaller" precision wins going away. If you only take one Qwen, take the AWQ. I keep both mostly so I can point at the gap.

Nemotron-3.5-Lightning-30B: the one that doesn't slow down

# Nemotron-3.5-Lightning-30B-FP8, one card
--tensor-parallel-size 1
--max-model-len 32768
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92
--reasoning-parser nemotron_v3      # same family, same versioned parser as the 120B

This is the fastest thing in the fleet - 158 tokens a second at short context - and the reason I keep coming back to it is what happens as the context fills up, which on most models is where the speed quietly drains away. It barely moves. That's the Mamba2 hybrid architecture doing what it's built to do, and for anything long-document it's the most interesting model on the rig. Configuration-wise it's a well-behaved member of the Nemotron family, so it inherits the same lesson as its big sibling: nemotron_v3 for the parser, the thinking toggle through the chat-template kwarg rather than the deprecated "detailed thinking off" system prompt, and no --trust-remote-code. If you've set up the 120B you already know this one; it's the same shape at a third of the weight.

Ternary-Bonsai-27B: the featherweight

# Ternary-Bonsai-27B-AWQ, one card
--tensor-parallel-size 1
--max-model-len 32768
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92

Not every model on this rig put up a fight, and this is one that didn't. It's a 27B that lands at 18.7GB - lighter than several models half its size - and loads in 218 seconds from the volume, which is quick for its size. It's built on the same Qwen3.5 architecture as the pair above, so it configures the same way and needs nothing special. I've kept it in because a runbook that only showed you the models that fought back would leave you braced for trouble that mostly isn't there: once you know a family's habits, most models in it just come up. This one just came up, first time.

Devstral-Small-24B: the coding model that behaved

# Devstral-Small-2-24B-AWQ, one card
--tensor-parallel-size 1
--max-model-len 32768
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92

Devstral is a coding-focused model on a Mistral multimodal architecture, 32.1GB in AWQ, and it's the other one I can be brief about because it didn't misbehave - it loaded in about 700 seconds and got on with it. The only thing worth flagging, and it applies to any of the multimodal-capable models here, is that if a model ever crashes during vision profiling but you only want it for text, you don't have to give up on it: --limit-mm-per-prompt.image 0 gets you a working text endpoint while the vision path sorts itself out upstream. I didn't need it for Devstral in the end, but it's the trick I'd reach for first, and it's saved a couple of the others.

LFM2.5-1.2B: a small model with surprisingly big tokens

# LFM2.5-1.2B-Instruct, one card - and check your context maths
--tensor-parallel-size 1
--max-model-len 32768
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92
# note: this model declares NO vendor sampling - engine defaults apply

Don't be fooled by the size. This is the little one - 2.3GB, loads in under two minutes, and it's quick - but it taught me a lesson that has nothing to do with speed. The first time I benchmarked it, a prompt that had comfortably fit every other model came back HTTP 400 Bad Request, and I spent a while assuming I'd fat-fingered a flag. I hadn't. LFM2.5 has a much smaller vocabulary than the Qwen family - about 65,000 tokens against 248,000 - and a smaller vocabulary chops the same words into more pieces. The identical text that Qwen encoded in 29,715 tokens came to 33,139 here, an 11.5% jump, and that was enough to overflow a 32,768 window that had held everything else. So "128k context" is not a fixed amount of text; it's a fixed number of tokens, and how much text that buys you depends on the model. The other quiet trap: LFM2.5 declares no vendor sampling at all, so the comfortable habit of "just trust generation_config.json" isn't universal either - you have to check, model by model, whether there's anything in the file to trust.

Llama-4-Scout: ten million tokens you almost certainly don't want

# Llama-4-Scout-INT4, dual card - CAP the context or pay for it
--tensor-parallel-size 2
--max-model-len 32768               # NOT the native 10,485,760
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92

Scout is the one that will happily hurt you with a default. It's a 64.9GB mixture-of-experts model across both cards, and its model card advertises a native context of 10,485,760 tokens - ten million. Accept that and let the engine size a KV cache for it, and it will try, and you will have a bad time. The whole config for this model is really one act of restraint: set --max-model-len to something a human needs, here 32,768, so the engine reserves cache for that and not for the ten-million-token conversation you are never going to have. Beyond that it's a well-behaved Llama-4, and it's worth knowing that parallel tool calls work on Llama 4 where they don't on Llama 3 - one of those quiet capability jumps between versions that the shared API shape hides completely.

The three I couldn't serve

A runbook that only listed wins would be dishonest, because three models beat me, and each failure is its own small lesson. Kimi-Linear-48B is the one that still nags. Its architecture is natively supported - KimiLinearForCausalLM is right there in the engine's registry - so "vLLM doesn't support it" is the obvious reading and the wrong one. The wall is the tokenizer: it ships a raw tiktoken.model and some custom Python and no tokenizer.json, which means there's no fast-tokenizer fallback, and the only way to serve it is to run the shipped code or hand-convert the tokenizer and accept the risk of quietly skewing every token count I publish. I checked whether the official upstream repo was any cleaner - it wasn't, it ships the identical files - so there was no better source to fall back to. I dropped it. Native support and a servable model turn out to be two different things.

Muse-Glimmer-30B was simpler and more frustrating: a Meta release six days old when I tried it, and MuseGlimmerForConditionalGeneration is simply absent from the pinned engine's registry. No card tells you that; the engine tells you, by refusing. It only ever loaded on a custom development image, which is a different rig than the one this runbook describes, so it doesn't get a config here.

And the community gpt-oss-120b AWQ from earlier - the one whose experts were never quantised. I've counted that as a failure too, because I lost real time to it before the metadata gave it away, and "the label was wrong" is exactly the kind of thing that should cost the person who shipped the label, not the person who trusted it.

Where to go from here

If there's one thread running through all twelve, it's the one from the top: the request shape is the doorway, and everything that decides whether a model runs - the quant that fits, the parser it needs, the switch that turns its thinking off, the sampling it was trained for - is a per-model fact you set at the server, once, at launch. Keep that discipline and one byte-identical client talks to your whole fleet. Break it and you'll spend your evenings comparing harnesses while you think you're comparing models.

Every one of these configs, with the load times and measured contexts behind them, sits in the benchmark hub , and the fit calculator there will tell you whether any of them lands on the card you've got. And if you're carrying a model that's fighting you the way a few of these fought me, send it over - if it runs on 96GB of Ada, it goes on the list.