Tag All articles

Tagged: vllm

7 articles tagged vllm.

vLLM settings for a pair of RTX 4090s: the flags I run for twelve local models, and why

The flag-by-flag vLLM settings page for a dual-4090 rig: the baseline every preset shares, the six flags that change per model, the rig discipline you set once, and one verified launch block for each of twelve open-weight models with the reason attached.

· local-llm · vllm
The VRAM traps: why a 16GB model wouldn't load on a 48GB card

A 16GB model would not load on my 48GB card, and the reason was a shortfall of one kilobyte in a memory no spec sheet mentions. These are the fit traps a VRAM figure will never warn you about.

· local-llm · vllm
The broken rules of local LLM inference

I used to lock the clocks on my mining GPUs. The same instinct just helped kill five rules of local LLM inference on a £6,200, 96GB rig.

· local-llm · vllm
Muse Glimmer 30B vs qwen: my day-one local benchmark on a dual RTX 4090 rig

Meta dropped Muse Glimmer 30B and I spent the day trying to unseat my qwen daily driver on the dual-4090 rig. Four apparent hangs, one day-one bug, and a same-harness bench later, I had my answer - and it wasn't the coronation I'd half expected. So, is it better than my daily driver?

· gpu · hardware
The best local coding setup isn't one model: how I route across Claude, Kimi and my own rig

The question I get asked is which local model is best for coding. Wrong question. The setup that works routes three tiers - Claude reasons, Kimi builds, and a Qwen coder on my own rig does the volume for nothing. Here's the whole thing, wired up.

· claude-code · local-llm
How to set up vLLM in Docker on Windows (WSL2): serve an open-weight model on your own GPU

vLLM doesn't run on Windows natively, so this is the Docker Desktop and WSL2 route my rig has served on since July: the compose file, the image pin that stops it upgrading itself, the walls WSL2 puts in your way, and how the settings were found, with Claude writing the test harness.

· local-llm · vllm
Moving houtini-lm to vLLM: What I learned

I decommissioned Hopper (my local LLM bootstrapped server) and moved my local models to a two-card 4090 rig with vLLM on Docker. It's so much faster - but houtini-lm spat its dummy. Two bugs, one hiding behind the other, and how v3.2.1 fixes it.

· claude-code · mcp

Tagged: vllm

vLLM settings for a pair of RTX 4090s: the flags I run for twelve local models, and why

The VRAM traps: why a 16GB model wouldn't load on a 48GB card

The broken rules of local LLM inference

Muse Glimmer 30B vs qwen: my day-one local benchmark on a dual RTX 4090 rig

The best local coding setup isn't one model: how I route across Claude, Kimi and my own rig

How to set up vLLM in Docker on Windows (WSL2): serve an open-weight model on your own GPU

Moving houtini-lm to vLLM: What I learned