Giving Claude a Local Sidekick: a vLLM Journey Update

For a few months now I've been chasing the same idea: Claude in charge, and a properly capable local model doing the heavy lifting underneath it, connected through houtini-lm . Partly this is about performance and cost - I've written about what Claude Code does to a token bill - but if I'm honest with myself it's mostly curiosity. I want to know how good a sidekick I can build. I'm not there yet. This month I got noticeably closer, and this is the update.

The LM Studio wall

The journey so far, briefly. I started where most people start, with LM Studio , and for everyday chat it was fine. The problems began when I wanted the local model to do real work - tool calls, structured output, the things an actual sidekick needs. I kept hitting the same intermittent failure: the model would decide to call a tool, wrap the call in its own XML dialect, and the server would pass the whole thing through as plain text. Sometimes it worked. Sometimes my assistant printed <function=read_file> at me and carried on as if nothing had happened. Crashes I can live with. Silent, intermittent failures I really dislike.

It took me longer than I'd like to admit to work out that the models weren't the problem. Every model family speaks its own tool-call dialect - Qwen wraps calls in XML, Gemma has a bracket format, others emit Hermes-style JSON - and something on the server has to translate each one into the standard OpenAI schema. LM Studio wasn't doing that reliably. Once I understood that, the decision more or less made itself.

Why vLLM

vLLM ships a parser for each dialect, and that alone would have been enough for me. But it also does continuous batching (several requests properly in flight at once, not queued politely), and its KV cache can run at fp8, which halves the memory the context eats. That last one matters more than it sounds: it's the difference between a 27B model with a cramped context and the same model holding 131k tokens on one card.

The trade is that vLLM loads one model at a time, and swapping means a container restart. A minute or two once the weights are cached. I went back and forth on whether that constraint would annoy me. For now, it mostly doesn't - more on that below.

The hardware, for now

Everything here runs on one RTX 4090 with 48GB of VRAM - one of the modded dual-slot blower cards built for the Chinese AI market. Its twin is on the way, which will take the box to 96GB and open up tensor parallelism (one bigger model split across both cards). I've already wired in the consumer-hardware gotcha for that day: peer-to-peer transfers hang on consumer boards, so the dual-GPU preset carries NCCL_P2P_DISABLE=1 before the second card has even arrived. The realistic expectation over PCIe is 0.6-0.75x scaling per card, not the 2x you'd hope for. We'll see. I've covered which GPUs make sense for local LLMs separately if you're weighing up cards.

The serving stack itself is vLLM in Docker under WSL2, one compose file, with every model-specific flag coming from a .env that a swap script writes:

services:
  vllm:
    image: vllm/vllm-openai:latest
    ipc: host
    shm_size: 16g
    ports:
      - "8000:8000"
    volumes:
      - hf-cache:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      --model ${MODEL}
      --served-model-name ${SERVED_NAME}
      --max-model-len ${MAX_MODEL_LEN}
      --kv-cache-dtype ${KV_CACHE_DTYPE}
      --gpu-memory-utilization ${GPU_UTIL}
      --tensor-parallel-size ${TP_SIZE}
      ${EXTRA_ARGS}

volumes:
  hf-cache:

Benching: change one thing, run it again

This is where most of the month went. I wrote a bench script that measures five things per configuration - decode tok/s, prefill tok/s, time to first token on a cold 16k prompt, the same prompt repeated, and throughput with four concurrent requests - and logs every run as a JSON line with the full config attached. Then I did the boring thing: change one flag, run it again, compare. KV cache dtype. Batched-token limits. Speculative-decode depth. Thinking on versus off. Small changes, tested and retested, until a baseline emerged that didn't regress on any preset:

--enable-prefix-caching --max-num-batched-tokens 8192 --max-num-seqs 16

One number from that process stopped me. With prefix caching on, a repeated 16k-token prompt drops from 3.92 seconds to first token down to 0.08. I went back through the logs to check I hadn't mislabelled a run. It's real - and it makes sense once you think about what orchestrated calls look like, because the system prompt and tool definitions are identical every time. The engine already has them cached. For the sidekick use case, that one flag might be worth more than any model choice.

The stable as it stands

Since only one model can be loaded at a time, what I've ended up with is a stable of presets, each benched, each with a job:

The 61 tok/s on the coding driver is two tricks stacked, and I measured them separately to be sure. AWQ quantisation on Marlin kernels trades slower cold prefill for faster decode - prefix caching absorbs the prefill cost on repeated contexts. MTP speculative decoding (the model drafting two tokens ahead) measured 1.9x on its own. Stock FP8 gets 36 tok/s. Both together, 61. Same model, same card.

The face-off that surprised me

Decode speed doesn't tell you whether the code compiles, so I ran the small coding models head-to-head on the same task: a TypeScript MCP tool handler, judged on whether the output compiled and followed the spec.

I expected the well-reviewed 9B to take it. Instead it produced TypeScript that didn't compile - an undeclared variable, error handling that broke the spec, validation outside the try block. Deleted the weights that evening. A second model benched at a spectacular 174 tok/s, then spent its entire token budget deliberating and never finished the code. Also deleted. The winner was a Gemma-based 12B fine-tune I hadn't rated: a correct, compiling handler in 12.3 seconds against the 27B's 55, at what I'd judge 95% of the quality. It now gets the quick TypeScript jobs, and I'm still slightly suspicious of how well it did.

The lesson I keep re-learning: bench your task, not the leaderboard's.

Hooking it up to Claude

The bridge is houtini-lm , the MCP server I built for exactly this - it points at any OpenAI-compatible endpoint and gives Claude a delegation tool, which is why everything above had to speak the OpenAI schema in the first place. At startup it profiles every model the server offers (a HuggingFace lookup, cached in SQLite) and tracks measured speed per model across calls, so the orchestrator is choosing a sidekick from evidence rather than vibes. Here's its discover call, run while writing this:

Status: ONLINE
Endpoint: http://localhost:8000 (OpenAI-compatible)
Active model: qwen3.6-35b-a3b
Context window: 131,072 tokens

Claude quota saved - lifetime: 150,562 tokens / 198 calls

That last line is the scoreboard. Not a fortune yet - but every one of those calls was work Claude didn't burn frontier tokens on, and the number only moves one way. I've written before about pushing this idea to its extreme ; the stable above is the more moderate, daily-driver version of the same ambition.

Two client-side things I learned the slow way. Thinking is per-request (chat_template_kwargs: {"enable_thinking": false}), and for tool calls it's the difference between 0.8 and 3.7 seconds with identical output - default it off for delegated work. And the --tool-call-parser flag has to match the model family (qwen3_xml, gemma4, hermes), or you're straight back to XML leaking into prose, which is the exact failure this whole rebuild was meant to bury.

Where I've got to (and what hasn't worked yet)

So: is Claude's sidekick powerful yet? Not really, no. It's useful - bounded jobs like extraction, drafts, code stubs and reviews come back good, and the fast lane returns tool calls in under a second. But local inference still runs 3-30x slower than the frontier model, the one-model-at-a-time constraint means vision and coding can't happen in the same minute, and WSL2 has ceilings I keep bumping into (some newer vLLM runner paths want unified memory Docker Desktop can't provide, and FP8-block quants fall over on Ada cards - FP8-dynamic or AWQ instead). The sidekick I want - one that can hold up its end of real agentic coding - is probably the 80B Qwen3-Coder-Next, and that needs the second card.

Which lands soon. Then I get to find out whether PCIe tensor parallelism holds the promised 0.6-0.75x or falls on its face, whether the 80B is worth the VRAM it costs, and whether the swap-model workflow survives contact with two GPUs. I'll write it up either way - if you're building something similar, or you've already hit the dual-GPU wall I'm walking towards, I'd love to compare notes.