Operating · accepting briefs Work with me

Tagged: vllm

9 articles tagged vllm.

How to Run Qwen3-Coder-Next Locally: My vLLM Settings for Two 48GB RTX 4090s

I loaded Qwen3-Coder-Next across my two modded 48GB RTX 4090s with vLLM, and today's post covers the vLLM settings for 192k of context, and how it did on the jobs I use AI for. It answered questions on a whole 86k-token repo in 26 seconds, where Qwen3.8-27B on its 32k one-card preset couldn't take the prompt at all, and it scoped an upgrade in under 30 seconds against about 3 minutes for the 27B. But it got facts wrong, and given git tools it committed on a failing test in 2 of 3 runs.

Local LLMs · vllm
How to Benchmark vLLM: Find the Best Model, Quant and Settings for Your GPU

In today's post I'm sharing how I benchmark local models on vLLM by changing one thing at a time, along with the traps that quietly skew the numbers and my results from 21 setups on two 48GB RTX 4090Ds. The harness, vllm-bench, is free, so you can run the same tests on your own card.

local-llm · vllm
vLLM settings for a pair of RTX 4090s: the flags I run for twelve local models, and why

The flag-by-flag vLLM settings page for a dual-4090 rig: the baseline every preset shares, the six flags that change per model, the rig discipline you set once, and one verified launch block for each of twelve open-weight models with the reason attached.

local-llm · vllm
KV Cache Explained: Why a Local LLM Won't Load on Your GPU

In today's post I'm taking a closer look at the KV cache, the memory a model keeps of the conversation so far, and why 'does it fit in VRAM?' is the wrong question when a local LLM won't load. It's what refused a 70B model its own 131k context on my 48GB card, and I'll show you how to size it, the vLLM settings that control it, and the kilobyte of GPU shared memory that stopped a 16GB model outright.

local-llm · vllm
LLM Inference Optimization: What Made Two RTX 4090s Faster

In today's post I'm going through what made local LLM inference faster on my two modded 48GB RTX 4090s, in the order it mattered: the file format, how you split a model across cards, locking the clocks at boot, speculative decoding and load times, with the numbers from my own testing.

local-llm · vllm
Muse Glimmer 30B vs qwen: my day-one local benchmark on a dual RTX 4090 rig

Meta dropped Muse Glimmer 30B and I spent the day trying to unseat my qwen daily driver on the dual-4090 rig. Four apparent hangs, one day-one bug, and a same-harness bench later, I had my answer - and it wasn't the coronation I'd half expected. So, is it better than my daily driver?

gpu · hardware
Best Local LLM for Coding, Plus the API and Image Models I Use

In today's post I'm taking a closer look at the best LLM for coding, both local and over an API. Here are the local and API models in my setup, the job each one does and the tests behind it, plus the image models I use.

claude-code · local-llm
How to set up vLLM in Docker on Windows (WSL2): serve an open-weight model on your own GPU

vLLM doesn't run on Windows natively, so this is the Docker Desktop and WSL2 route my rig has served on since July: the compose file, the image pin that stops it upgrading itself, the walls WSL2 puts in your way, and how the settings were found, with Claude writing the test harness.

local-llm · vllm
How to run houtini-lm on vLLM

I moved my local models off my old server and onto vLLM in Docker. In today's post we're taking a closer look at pointing houtini-lm at it, and why thinking goes off for a model served under an alias.

claude-code · mcp