Operating · accepting briefs Work with me

Tagged: local-llm

9 articles tagged local-llm.

How to Benchmark vLLM: Find the Best Model, Quant and Settings for Your GPU

In today's post I'm sharing how I benchmark local models on vLLM by changing one thing at a time, along with the traps that quietly skew the numbers and my results from 21 setups on two 48GB RTX 4090Ds. The harness, vllm-bench, is free, so you can run the same tests on your own card.

local-llm · vllm
vLLM settings for a pair of RTX 4090s: the flags I run for twelve local models, and why

The flag-by-flag vLLM settings page for a dual-4090 rig: the baseline every preset shares, the six flags that change per model, the rig discipline you set once, and one verified launch block for each of twelve open-weight models with the reason attached.

local-llm · vllm
KV Cache Explained: Why a Local LLM Won't Load on Your GPU

In today's post I'm taking a closer look at the KV cache, the memory a model keeps of the conversation so far, and why 'does it fit in VRAM?' is the wrong question when a local LLM won't load. It's what refused a 70B model its own 131k context on my 48GB card, and I'll show you how to size it, the vLLM settings that control it, and the kilobyte of GPU shared memory that stopped a 16GB model outright.

local-llm · vllm
LLM Inference Optimization: What Made Two RTX 4090s Faster

In today's post I'm going through what made local LLM inference faster on my two modded 48GB RTX 4090s, in the order it mattered: the file format, how you split a model across cards, locking the clocks at boot, speculative decoding and load times, with the numbers from my own testing.

local-llm · vllm
Best Local LLM for Coding, Plus the API and Image Models I Use

In today's post I'm taking a closer look at the best LLM for coding, both local and over an API. Here are the local and API models in my setup, the job each one does and the tests behind it, plus the image models I use.

claude-code · local-llm
How to set up vLLM in Docker on Windows (WSL2): serve an open-weight model on your own GPU

vLLM doesn't run on Windows natively, so this is the Docker Desktop and WSL2 route my rig has served on since July: the compose file, the image pin that stops it upgrading itself, the walls WSL2 puts in your way, and how the settings were found, with Claude writing the test harness.

local-llm · vllm
How to run houtini-lm on vLLM

I moved my local models off my old server and onto vLLM in Docker. In today's post we're taking a closer look at pointing houtini-lm at it, and why thinking goes off for a model served under an alias.

claude-code · mcp
Best GPU for Local LLMs in 2026: What I'd Buy and Why

In today's post I'm going through the best GPU for running LLMs locally, and why memory bandwidth and VRAM decide it. I start with the cheapest VRAM you can buy and the cheapest 32GB cards, then the modded 48GB RTX 4090s I run, and finish on when an RTX PRO 6000 or a DGX Spark makes more sense.

local-llm · gpu
How to set up LM Studio: GPU offload, developer mode and your first local model

Turn a gaming GPU into a working local AI model with LM Studio - free, no billing, no API key. In today's guide, we install it, point it at the GPU, download a first model, and then go through the settings: GPU offload, context length, Flash Attention, temperature and top-k, and the Developer mode that unlocks them.

claude-code · local-llm