Operating · accepting briefs Work with me

Local AI

7 articles in Local AI.

vLLM settings for a pair of RTX 4090s: the flags I run for twelve local models, and why

The flag-by-flag vLLM settings page for a dual-4090 rig: the baseline every preset shares, the six flags that change per model, the rig discipline you set once, and one verified launch block for each of twelve open-weight models with the reason attached.

local-llm · vllm
KV Cache Explained: Why a Local LLM Won't Load on Your GPU

In today's post I'm taking a closer look at the KV cache, the memory a model keeps of the conversation so far, and why 'does it fit in VRAM?' is the wrong question when a local LLM won't load. It's what refused a 70B model its own 131k context on my 48GB card, and I'll show you how to size it, the vLLM settings that control it, and the kilobyte of GPU shared memory that stopped a 16GB model outright.

local-llm · vllm
LLM Inference Optimization: What Made Two RTX 4090s Faster

In today's post I'm going through what made local LLM inference faster on my two modded 48GB RTX 4090s, in the order it mattered: the file format, how you split a model across cards, locking the clocks at boot, speculative decoding and load times, with the numbers from my own testing.

local-llm · vllm
Muse Glimmer 30B vs qwen: my day-one local benchmark on a dual RTX 4090 rig

Meta dropped Muse Glimmer 30B and I spent the day trying to unseat my qwen daily driver on the dual-4090 rig. Four apparent hangs, one day-one bug, and a same-harness bench later, I had my answer - and it wasn't the coronation I'd half expected. So, is it better than my daily driver?

gpu · hardware
Best Local LLM for Coding, Plus the API and Image Models I Use

In today's post I'm taking a closer look at the best LLM for coding, both local and over an API. Here are the local and API models in my setup, the job each one does and the tests behind it, plus the image models I use.

claude-code · local-llm
How to set up vLLM in Docker on Windows (WSL2): serve an open-weight model on your own GPU

vLLM doesn't run on Windows natively, so this is the Docker Desktop and WSL2 route my rig has served on since July: the compose file, the image pin that stops it upgrading itself, the walls WSL2 puts in your way, and how the settings were found, with Claude writing the test harness.

local-llm · vllm
How to run houtini-lm on vLLM

I moved my local models off my old server and onto vLLM in Docker. In today's post we're taking a closer look at pointing houtini-lm at it, and why thinking goes off for a model served under an alias.

claude-code · mcp