- · local-llm · vllm
How to set up vLLM in Docker: serve an open-weight model on your own GPU
vLLM in Docker, from empty machine to an OpenAI-compatible endpoint you can curl: the compose file I run, the flags that survived benchmarking, and the deadlock that hides behind a healthy /health check.
Read article - · local-llm · gpu
Tuning vLLM on a Modded 48GB RTX 4090: From 18 to 133 Tokens per Second
Three days of benchmarking vLLM on a modded 48GB RTX 4090: the quantisation trap that costs Ada owners 40% of their speed, the free 1.9x from speculative decoding, the KV compression that never served once, and what cutting 120 watts costs you. Every number measured, every failure kept.
Read article - · claude-code · mcp
Giving Claude a Local Sidekick: a vLLM Journey Update
I want Claude to have a powerful local sidekick through houtini-lm. I'm not there yet, but this month I got noticeably closer - a vLLM rebuild, a benching spree on the 48GB 4090, and a face-off that deleted two models in an evening.
Read article