Tagged: inference

How to set up vLLM in Docker: serve an open-weight model on your own GPU

vLLM in Docker, from empty machine to an OpenAI-compatible endpoint you can curl: the compose file I run, the flags that survived benchmarking, and the deadlock that hides behind a healthy /health check.