How to set up vLLM in Docker: serve an open-weight model on your own GPU

By the end of this guide you'll have vLLM running in Docker on your own NVIDIA GPU, serving an open-weight model through an OpenAI-compatible endpoint (the same API shape ChatGPT clients speak, so your existing tools connect without changes), verified with a curl call. Mine has been up serving the delegation fleet for most of a day as I write this. The software is free; the cost is disk space for model weights, an hour of your time, and the electricity, of course. I moved to vLLM after months on LM Studio because my tool calls kept leaking into chat as raw XML - the assistant would print <function=read_file> at me and carry on as if nothing had happened. vLLM ships a parser for each model family's tool-call dialect, and that one feature is why this stack got rebuilt. The rest of this is the build.

What you need before you start

• An NVIDIA GPU. I run a pair of modded 48GB RTX 4090s (the dual-slot blower cards built for the Chinese AI market), but one card with 12-16GB runs a usable 8-14B model. If you're weighing up hardware, the GPU guide covers which cards make sense and why VRAM is the number that matters.
• Docker with GPU support. On Windows that means Docker Desktop with the WSL2 backend and a current NVIDIA driver; on Linux, the NVIDIA Container Toolkit. The whole setup runs under WSL2 because this machine does more than serve models, and it works - with two ceilings covered in the gotchas.
• Disk for weights. A quantised 27B model is 15-20GB, and you will end up with more than one. Budget 100GB and thank yourself later.
• A terminal you're comfortable in. Everything below is compose files and curl.

You don't need a Python environment, a CUDA toolkit install, or an account with anyone. The vLLM Docker image carries its own runtime, and model weights come down from Hugging Face on first load.

The compose file

One file runs the whole thing. This is mine, verbatim:

services:
  vllm:
    image: vllm/vllm-openai:latest
    ipc: host
    shm_size: 16g
    ports:
      - "8000:8000"
    volumes:
      - hf-cache:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      --model ${MODEL}
      --served-model-name ${SERVED_NAME}
      --max-model-len ${MAX_MODEL_LEN}
      --kv-cache-dtype ${KV_CACHE_DTYPE}
      --gpu-memory-utilization ${GPU_UTIL}
      --tensor-parallel-size ${TP_SIZE}
      ${EXTRA_ARGS}

Every model-specific value is a ${VARIABLE} read from a .env file, so the compose file never changes; the .env does. That's what makes model swapping a one-command job later, and it's the difference between a serving setup and a pile of shell history you're afraid to touch.

shm_size: 16g matters because vLLM moves tensors through shared memory and the Docker default (64MB) is nowhere near enough. And the hf-cache volume keeps downloaded weights outside the container, so pulling a new vLLM image doesn't cost you a 20GB re-download.

Start it with docker compose up -d, then watch the logs while the model loads:

docker logs -f vllm

First load of a new model downloads the weights, so give it time. When you see the Uvicorn startup line, the endpoint is live on port 8000.

Presets - how do you swap models?

vLLM loads one model at a time. That's the constraint of the design, and you work with it rather than around it: each model gets its own .env preset, and a small script rewrites .env and restarts the container.

.\swap-model.ps1              # no arguments lists the presets
.\swap-model.ps1 qwen3.6-27b-awq

A preset is nothing clever - it's the environment block for one model. Here's the daily driver, Qwen3.6-27B in cyankiwi's AWQ build (community quantisers are part of the story on this beat, not a footnote):

MODEL=cyankiwi/Qwen3.6-27B-AWQ-INT4
SERVED_NAME=qwen3.6-27b
MAX_MODEL_LEN=131072
KV_CACHE_DTYPE=fp8
GPU_UTIL=0.92
TP_SIZE=1
EXTRA_ARGS=--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching --max-num-batched-tokens 8192 --max-num-seqs 16 --speculative-config.method mtp --speculative-config.num_speculative_tokens 2

A swap takes a minute or two once weights are cached. The one-model-at-a-time rule shapes how you work: vision and coding in the same minute means either two machines or a compromise model, and pretending otherwise wastes an afternoon.

That --tool-call-parser qwen3_xml flag is the fix for the XML-in-chat failure that started this rebuild. Every model family speaks its own tool-call dialect - Qwen wraps calls in XML, Gemma uses a bracket format, others emit Hermes-style JSON - and the parser flag tells vLLM which translator to run so your client receives clean OpenAI-schema tool calls. Set it per preset to match the model family (qwen3_xml, gemma4, hermes, lfm2). Get it wrong, or leave it off, and you're back to raw dialect leaking into prose.

The flags that earned their place

None of the settings above came from a forum post. I benchmark every configuration change with a script that measures five things per run - decode speed, prefill, time to first token on a cold 16k prompt, the same prompt repeated, and throughput with four concurrent requests - and logs a JSON line with the full config alongside the result, so any number here traces back to the exact flags that produced it. Change one thing, re-run, compare. Slower than reading Reddit. More useful.

The universal baseline that showed no regressions on any preset:

--enable-prefix-caching --max-num-batched-tokens 8192 --max-num-seqs 16

Two findings from the benching shaped everything else.

Prefix caching is the orchestration flag

Orchestrated calls repeat the same system prompt and tool definitions on every request, and on a repeated 16k-token prompt the time to first token drops from 3.92 seconds to 0.08. If your workload is agentic - same preamble, different question - this flag alone changes what the machine feels like.

kv-cache-dtype fp8 halves what the context cache eats

The KV cache (the model's working memory for your prompt) competes with the weights for VRAM, and storing it at 8-bit is how a 27B model runs 131k context on a single 48GB card. On the quant format itself: AWQ beat FP8 on every one of the four models I tested both ways, by 1.53x to 2.76x on decode. The 27B runs 36 tokens per second (tok/s) as the official FP8 build and 56-60 as the AWQ build with speculative decoding (a small draft head guesses the next couple of tokens; the full model verifies them in one pass). Same model, same card - the format is the difference.

The trade-off in the AWQ choice: cold prefill on long prompts is slower on those kernels. Prefix caching absorbs that on repeated contexts; a one-off 16k paste will feel it.

One more measured result, because it saves you money: capping the card at 330W against its 370W default cost nothing - 133.9 tok/s against 133.1 on the 35B. Power capping for a 24/7 serving box is free performance-per-watt.

Verify it works

Start with the cheapest check there is - asking the server what it thinks it's serving:

curl http://127.0.0.1:8000/v1/models

Here's what mine returns with the daily driver loaded:

{"object":"list","data":[{"id":"qwen3.6-27b","object":"model","owned_by":"vllm","root":"cyankiwi/Qwen3.6-27B-AWQ-INT4","max_model_len":131072}]}

Three fields matter. id is the served-name, the string that goes in the model field of every request you make (it's whatever you set SERVEDNAME to in the preset, so if the name reads oddly you've only yourself to blame). root shows the real Hugging Face repo behind the alias - useful once you're a few presets deep and can't remember whether "qwen3.6-27b" points at the AWQ build or the FP8 one. And `maxmodel_len` is the context window the server is running, worth a glance because it's set per preset and easy to lose track of.

One small thing before you copy that command: it says 127.0.0.1 rather than localhost, and that's deliberate. Under WSL2's mirrored networking, localhost can resolve to IPv6 (::1) while the container binds IPv4 only, so localhost:8000 can silently connect to nothing at all - no error worth reading, just a request going nowhere while the container sits there perfectly healthy. Use 127.0.0.1 and save yourself the confusion; I speak from experience.

With the model confirmed, the proof call, run after every swap:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [{"role": "user", "content": "Extract the product names from this text as JSON: ..."}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'

That enable_thinking: false took me longer to find than I'd prefer to admit. Reasoning models think before they answer, and for plumbing calls - extraction, formatting, tool dispatch - the thinking phase is pure latency: 0.8 seconds with it off against 3.7 with it on, identical output. Thinking is for problems, not plumbing. Default it off for orchestrated work and switch it on when the task earns it.

What comes back follows the OpenAI chat-completion schema: a choices array carrying the message, and a usage block with prompt_tokens and completion_tokens - the same usage field my benchmarking reads its token counts from. When the model makes a tool call, finish_reason comes back as tool_calls and the arguments arrive as clean JSON in function.arguments, which is the whole point of the per-model tool-call parser from the presets section. If you ever see tool-call dialect sitting in the message text instead, the parser flag is wrong for the model family.

A response with a choices array means you're serving. Anything that speaks the OpenAI API - and nearly everything does - can now point at http://127.0.0.1:8000/v1 instead of a cloud endpoint.

How the models differ

Once the endpoint answers, the next thing you'll do is go shopping on Hugging Face for a second model, and the naming soup starts immediately: 27B, 35B-A3B, AWQ, FP8, QAT, GGUF. The short version is that two things decide how a model behaves on your card - the architecture (dense or mixture-of-experts) and the quantisation format - and both show up plainly in the measurements. So rather than walk you through the theory, let me teach it from the fleet's own numbers.

Dense and MoE - why a bigger model can be faster

A dense model touches every parameter for every token it generates. A mixture-of-experts (MoE) model stores a large total parameter count but routes each token through a small slice of it, the "active" parameters, and you can read the split straight off the name: 35B-A3B means 35 billion parameters, about 3 billion active per token.

The consequence is the bit that catches people out. VRAM is sized by the total - all 35 billion have to sit on the card - but decode speed is governed by the active slice, because decode is memory-bandwidth-bound: the card's bandwidth divided by the bytes it touches per token. Decode speed is bytes-touched-per-token, not parameter count. Same rig, same benchmark harness:

The 35B MoE is physically bigger than my 27B daily driver and decodes roughly twice as fast, 56-60 against 120-134 tok/s. The Gemma pair is starker still: 40-42 tok/s for the dense 31B against ~155 for the 26B MoE - about 3.7x, from a model that takes up less room on the card. And the bottom row is the one I had to sit with for a minute: a 120B out-decoding the daily driver, which no model that size had managed on this rig before. (More on that one in the next section - it needs both cards, so it's not where you start.)

So why not run MoE for everything?

Because the speed has a quality cost on open-ended work. I A/B'd the 27B dense against the 35B MoE on the same coding tasks, and on bounded codegen they tied - both passed 12 of 12 assertions, the MoE about 1.6x faster, which on that evidence makes the MoE the obvious pick. Then I gave both an open-ended code review with no max_tokens cap, and the MoE wouldn't stop. It ground out roughly 19k tokens of ever-more-marginal findings, left the literal word "SEVERITY" sitting in the output where a severity rating was supposed to go, and cited the same line reference more than fifteen times. The dense 27B wrote its review and finished.

My rule since: cap MoE models, and any model you haven't measured, with max_tokens - and leave the cap off only for models you've watched self-terminate cleanly. (In fairness to the 35B: on a later, real review of a 7.9k-token diff, with an explicit "stop when you've covered what matters" instruction, the runaway didn't reproduce. Make of that what you will; my cap stays on.)

Quant formats - the ones I run, and the trap

The quant format is how the weights get compressed onto your card, and kernel support decides your speed as much as the bit-count does. The formats I allow on this rig:

• AWQ (INT4). My default for weights. Its Marlin kernels are tuned for Ada (the 40-series), and on the models I've tested both ways AWQ decoded 1.53x to 2.76x faster than FP8. The trade is cold prefill on a long prompt, which runs roughly 2x slower on those kernels - prefix caching absorbs that on repeated contexts; a one-off 16k paste will feel it.
• FP8 - but only FP8-dynamic, never FP8-block. This is the kernel trap. The official Qwen/Qwen3.6-27B-FP8 checkpoint ships block-format FP8 (128x128 blocks), which has no tuned kernel on a 4090 and falls back to generic Triton - so the AWQ build of the very same model decodes well over twice as fast (the flags section above has the exact spread). Block-FP8 is a Hopper/Blackwell format; on Ada, FP8-dynamic (per-tensor) is the variant to download.
• QAT (W4A16). Quantisation-aware training - the model learned its weights knowing they'd be quantised. My accuracy-first default for code work; the Gemma build in the table above is one of these.
• BF16. Unquantised, and only on the menu if the model fits inside 24GB.

Two formats you'll see everywhere that I skip: GGUF is the llama.cpp and LM Studio format and isn't on the vLLM path at all, so the community coder repos that only ship GGUF or MLX quants are a pass for this stack. And GPTQ I don't use - AWQ, QAT, FP8-dynamic and BF16 cover everything I serve.

Will it fit your card?

It's only a rule of thumb, but: a 4-bit quant costs roughly 0.6GB of VRAM per billion parameters, plus a little headroom. My real footprints bear it out - the 120B AWQ lands at 77GB (0.64GB per billion) and the 26B at 16GB (0.62). Thirty seconds of arithmetic before you commit to a 40GB download.

One last dial while we're on memory. The fp8 KV cache from the compose file halves what the context cache eats, which is how the 27B runs 131k context on one card - but it buys that memory at an accuracy cost on code work, and my accuracy-first Gemma preset deliberately keeps its KV cache at bf16 for exactly that reason.

The best model in the stable right now

Before I name names, one line on how "best" gets decided here, so the word carries some weight. Every number below comes from a frozen, versioned benchmark harness (v1) that measures five delegation-shaped scenarios - single-stream decode, cold and warm time-to-first-token on a ~16k prompt, decode at 16k context, and four concurrent requests - all at temperature 0, with token counts read from the server's own usage field. Quality gets scored in a separate layer, deliberately, because a speed table never implies a quality ranking. With that said, you get two answers from me, because the fleet optimises for different jobs.

The incumbent: Qwen3.6-27B

Qwen3.6-27B in cyankiwi's AWQ-INT4 build - the daily driver this whole guide has been configured around. Dense, 56-60 tok/s decode, and the best model in the stable at code review, evaluation and bug-checking. It's also the only model I've measured that self-terminates cleanly on open-ended judgement work, which, after the MoE runaway story above, is the property I now price highest. If you're setting up exactly one preset, make it this one.

The challenger: Nemotron 3 Super 120B

NVIDIA Nemotron 3 Super 120B - cyankiwi again, an AWQ-4bit quant of NVIDIA's BF16 release. 120 billion parameters with about 12 billion active per token, and it lands at 77GB spread across both cards with tensor parallelism (so this one is strictly a two-card pick - the section below covers the wiring). I measured it on 2026-08-11, and the decode line is the one I keep going back to: 61.9 tok/s, faster than the daily driver. At 16k context it holds 62.7 tok/s against the 27B's 19.4 - a 3.2x gap, because its Mamba2-hybrid attention barely degrades as the context grows. Four concurrent requests come through at 169.7 tok/s combined. Best in the stable at raw decode, at long-context decode and at concurrency.

It isn't crowned yet. The quality gauntlet against the Qwen is still pending, and speed is not judgement - the runaway MoE earlier in this piece is exactly why those two get measured separately.

The oddball: Muse Glimmer 30B

For contrast, because it shows how lopsided a model can be: Muse Glimmer 30B is the fleet's prefill specialist. Prefill runs 3550 tok/s, up 75% on the 27B, and the cold 16k time-to-first-token is a quick 4.71 seconds. Decode, though, sits at the bottom of the fleet at 16.3 tok/s - the repo name explains it if you've read the quant section, because this was a day-one FP8-block build with no draft head, and day-one dev-stack numbers are floors, not ceilings.

Every measured run behind these numbers is published on the model testing page , per model and per scenario, if you want to check my working.

Running a model across two GPUs

You don't need this to start, and it's not the place to start. But the second card is why my box exists in its current form.

--tensor-parallel-size 2 splits each layer of the model across both cards, and it's how an 80B model becomes a daily tool rather than a demo: Qwen3-Coder-Next 80B at AWQ 8-bit runs 118.4 tok/s across the pair, 314.8 combined with four concurrent requests. Numbers like that from consumer hardware still catch me off guard.

The consumer-board caveat comes wired into the dual-GPU presets before anything else:

NCCL_P2P_DISABLE=1

Peer-to-peer GPU transfers hang on consumer motherboards - the cards try to talk directly, the board can't do it properly, and the container sits there forever. With P2P disabled, traffic routes through the PCIe bus and system memory, which works everywhere and costs you scaling: expect 0.6-0.75x per additional card over PCIe, not the doubling the flag name implies.

Gotchas

Each one of these wasted a bit of my time - I'm sharing so you don't have to go through the same thing.

Tool calls appear in chat as XML or bracketed text

The parser flag is missing or doesn't match the model family. --tool-call-parser qwen3_xml for Qwen models, gemma4 for Gemma, hermes for the JSON-dialect families. This is the failure that made me leave LM Studio, and vLLM only fixes it when you tell it which dialect to expect.

The engine wedges on a long prompt, and /health still returns 200

This is the nasty one, because the container looks alive while serving nothing - engine core spinning at 95% CPU, GPU pegged at 100% while drawing idle-level wattage. Turns out, raising --max-num-batched-tokens from 8192 to 16384 with speculative decoding enabled deadlocks the engine on a 16k cold prefill. I reproduced it twice before I believed it. Keep 8192 on any preset with MTP - which is no sacrifice, because 8192 also measured faster on prefill (2002 tok/s against 1715). Why exactly the combination wedges, I can't tell you yet; the fix is solid, the mechanism is still an open question on my board.

Some FP8 model builds refuse to load on RTX 40-series cards

On this rig, block-quantised FP8 builds hit a missing-kernel error on the 40-series card; FP8-dynamic and AWQ builds loaded fine. Current vLLM docs say Ada block-FP8 support exists, so the wall is probably version- or WSL2-specific rather than universal - but if you hit it, don't fight it: the AWQ route is what the benching favoured anyway.

Newer engine features complain about unified memory under WSL2

Some of vLLM's newer runner paths want a memory mode Docker Desktop can't provide. If a flag combination that works for Linux people fails for you with memory errors, this is probably the wall you've hit; drop the feature rather than fighting the platform.

Leave the port exposed and it's somebody else's free compute

vLLM has no authentication worth the name. The compose file above binds port 8000 to every interface, which is right for a home network behind a router and wrong for anything with a public address. If the box is reachable from the internet, put the endpoint behind a reverse proxy with an API key, or bind it to 127.0.0.1:8000:8000 and tunnel in. GPU time is exactly the kind of thing the internet's background scanners are looking for.

A model that flew in benchmarks crawls on your prompts

Check whether your prompts repeat a common prefix. Without --enable-prefix-caching, every request pays full prefill; with it, the repeat TTFT collapses. The flag is in the baseline for a reason.

Where to go from here

You've just opened up a world of choices - here's what you could do next.

If the endpoint is the goal, you're done - point your tools at http://localhost:8000/v1 and get on with your work. The settings above are a starting configuration that measured well on my hardware; the tuning write-up covers the longer story of squeezing a 48GB card, including the quantisation-format trap that cost an earlier build 40% of its speed.

If you got here for the same reason I did - taking bulk work off a metered frontier-model bill - the next piece is the orchestration layer. houtini-lm is the MCP server I built to let Claude delegate bounded tasks to this endpoint: Claude does the reasoning and judgement, the local model does the volume, and the lifetime token counter tells you what the arrangement is worth. The full backend write-up covers wiring the two together, and the measured runs show what each model in the stable earns its place doing.

Either way: the endpoint on port 8000 doesn't care which model is behind it, and that's the point of the whole design. Bench your task, pick your preset, swap in a minute. The GPU you already own does the rest.