How to set up vLLM: Docker, compose, and the flags that earn their keep

vLLM is what you move to when one person chatting stops being the job. It serves many requests at once properly, and it is a worse experience for everything else: Linux, Docker, CUDA, and flags that refuse to load a model rather than quietly coping.

This gets you from nothing to a server answering requests, then to the configuration I run daily. Budget an evening for the first run, mostly spent downloading. Every command here runs on my own bench, and each number comes from a results file that records the configuration alongside the result - the same data published on whichaipc.com , where the hardware testing lives.

Is it worth it for you?

Settle this before you spend the evening. The question is whether several requests will ever be in flight at once.

If it is you, one chat window, one question at a time, LM Studio gives you nearly the same single-stream speed and none of this work. My own numbers make the case for the other side: a single request against the 80B returns 118.4 tokens per second, while four concurrent requests aggregate to 314.8. That second number is the whole argument for vLLM. If you will never see it, you are doing a lot of work to reach the first.

It earns its place with more than one person or agent hitting the model, with agent frameworks that fan out requests, and with bulk jobs over thousands of documents.

What you need before you start

• An NVIDIA GPU with enough VRAM for your model. 24GB runs a good 27B at 4-bit. Our bench is two modded 48GB RTX 4090s.
• Linux, or Windows with WSL2. Everything after WSL2 is identical to running on Linux, which is the point of doing it that way.
• Docker plus the NVIDIA Container Toolkit. The toolkit is what lets a container see the GPU.
• Disk. Model weights are tens of gigabytes each and you will collect several.
• A Hugging Face token if you want gated models such as Llama.

You do not need the CUDA toolkit on the host. The driver plus the container toolkit is enough - the container brings its own CUDA.

Getting the GPU visible to Docker

1. Install WSL2, if you're on Windows. In an admin PowerShell, then reboot.

wsl --install -d Ubuntu

1. Install Docker inside Ubuntu (or on your Linux box). Log out and back in after the group change or the next command fails on permissions.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

1. Install the NVIDIA Container Toolkit. Skipping this is the most common reason a container starts and then reports no devices.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

1. Prove the GPU is visible from inside a container. If this does not print your card, nothing below will work.

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

First run: one command

This pulls the official image and serves an 8B model. The first run is a large download - the image and the weights.

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90

Three of those matter more than they look:

• --ipc=host - without it you hit shared-memory errors as soon as the model is anything but tiny. If sharing the host namespace bothers you, use --shm-size 16g instead.
• -v ~/.cache/huggingface - keeps the model cache outside the container. Skip it and every restart re-downloads tens of gigabytes.
• --max-model-len - caps context. vLLM reserves memory for the full window at startup, so an uncapped model can refuse to load on a card that would otherwise hold it comfortably.

Gated models need a token: add -e HF_TOKEN=hf_....

Check it works. The API is OpenAI-compatible, which is the other reason to run it.

curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"Explain VRAM in two sentences."}],"max_tokens":120}'

If the first returns your model and the second returns text, you have a working server. Anything pointed at http://localhost:8000/v1, with any API key string, will now talk to it.

The compose file I run

The single command is fine for trying it. For anything you come back to, put it in compose with the model-specific parts in a .env, so switching models is an edit and a restart rather than a remembered incantation.

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm
    restart: unless-stopped
    ipc: host
    shm_size: 16g
    ulimits:
      memlock: -1
      stack: 67108864
    ports:
      - "8000:8000"
    volumes:
      - hf-cache:/root/.cache/huggingface
    environment:
      - HF_HUB_ENABLE_HF_TRANSFER=1
      - NCCL_P2P_DISABLE=${NCCL_P2P_DISABLE:-0}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      --model ${MODEL}
      --served-model-name ${SERVED_NAME}
      --max-model-len ${MAX_MODEL_LEN}
      --kv-cache-dtype ${KV_CACHE_DTYPE}
      --gpu-memory-utilization ${GPU_UTIL}
      --tensor-parallel-size ${TP_SIZE}
      ${EXTRA_ARGS}
    healthcheck:
      test: ["CMD", "curl", "-sf", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 600s

volumes:
  hf-cache:

Two additions worth keeping. HF_HUB_ENABLE_HF_TRANSFER=1 speeds up the weight downloads, which matters when they are 40GB. And the healthcheck has a ten-minute start_period on purpose - a large model genuinely takes that long to load, and a shorter grace period marks a perfectly healthy container as failed while it is still working.

A preset per model

Keep a .env per model rather than editing flags by hand, because the flags that differ between models are exactly the ones that stop it loading. Ours is a small library of them:

MODEL=Qwen/Qwen3.6-27B-AWQ
SERVED_NAME=qwen3.6-27b
MAX_MODEL_LEN=131072
KV_CACHE_DTYPE=fp8
GPU_UTIL=0.92
TP_SIZE=1
EXTRA_ARGS=--enable-prefix-caching --max-num-batched-tokens 8192 --max-num-seqs 16

Swapping is then a copy and a restart, a minute or two once weights are cached:

cp presets/qwen3.6-27b-awq.env .env
docker compose up -d --force-recreate vllm

Run one model at a time. vLLM claims most of the card at startup, so two servers on one GPU will fight over memory.

The flags that earn their keep

The three I put on everything:

--enable-prefix-caching --max-num-batched-tokens 8192 --max-num-seqs 16

Those showed no regression across any preset I tested, so they are simply always on now.

Two findings worth taking on their own. Architecture beats size: an 8B mixture-of-experts model decoded at 176.3 tok/s on my bench while a dense 27B managed 60.9. You are shopping for the largest MoE you can hold, which is a cheaper list than the largest model you can hold. And power capping is close to free: the same 35B returned 133.9 tok/s at 330W and 133.1 at 370W, which is inside run-to-run noise.

The deeper version of all this - the kernel trap on Ada cards, speculative decoding, what Windows costs you - is in the tuning write-up .

Gotchas that cost me time

Peer-to-peer hangs on consumer boards. Running tensor parallel across two 4090s, I watched the process sit there doing nothing at all. Consumer motherboards do not do GPU-to-GPU transfers the way the library expects, and the failure is a silent hang rather than an error:

environment:
  - NCCL_P2P_DISABLE=1

Tensor parallel crash-looping under WSL2. My 35B refused TP=2 entirely, crash-looping on startup with nothing useful in the logs. The cause turned out to be CUDA graph capture under WSL2 rather than anything about the model or the flags. If you are on WSL2 and a multi-GPU start dies immediately, that is where to look first.

Models that fit elsewhere refuse to load here. Nearly always the context reservation. Cap --max-model-len or set --kv-cache-dtype fp8 before concluding the card is too small.

A debugging flag contaminated a benchmark. I had a run showing a 35B dropping from 133 tok/s on one card to 14.9 across two, which would have been a startling finding about tensor parallelism. It was wrong. That run also carried --enforce-eager, which disables CUDA graphs and costs a great deal on its own, and the two effects could not be separated afterwards. It is published as contaminated rather than as a result, and the clean re-run is still outstanding. Check your flags before you believe your own numbers.

Numbers move between reboots. The same 35B preset that benched at 133.9 tok/s returned 118-123 after a reboot, an 8-12% drop I have not fully explained. Bench a config more than once before you trust a small difference.

Leave it exposed and it is somebody else's free compute

vLLM has no authentication worth the name. Bind it to localhost, or put something in front of it before it goes anywhere near a network you share.

Where to go from here

You have a server. Three directions from here.

For a chat interface, Open WebUI is the usual answer and it is one more container pointed at http://host.docker.internal:8000/v1. For wiring it into Claude so the local model takes the bulky work off your token bill, houtini-lm does exactly that, and the journey write-up covers how I got there. For which models are worth loading, the measured fleet has the current numbers.

And if this reads like more machine than you need, it probably is. LM Studio covers single-user work properly and takes an afternoon rather than an evening.