Moving houtini-lm to vLLM: What I learned

I decommissioned Hopper (my local LLM bootstrapped server) and moved my local models to a two-card 4090 rig with vLLM on Docker. It's so much faster - but with all code changes bugs are a risk and unfortunately houtini-lm spat its dummy; no worries - we're fixed and now support vLLM hosted local models too.

Clearly there was something different about the JSON output from vLLM despite the "OpenAI API compatibility". The bug was mine, there were two of them, and the second was hiding behind the first.

houtini-lm in a nutshell

houtini-lm is an MCP (Model Context Protocol - the standard that lets Claude talk to external tools) server. I built it so Claude can hand the bounded, bulky pieces of a job - boilerplate, a code review, an extraction, a first draft - to a model running on your own hardware. Your Claude token bill shrinks by whatever those pieces would have cost. On my own machine I've watched it save over 150,000 tokens across roughly 200 calls, and it prints the running total in the footer of every response, so the saving is never a matter of faith. My plan is that Claude does the reasoning and orchestration; the local model executes. Keep that split in mind - it turns out to be the whole story.

But everything broke!

For most of its life I ran houtini-lm against LM Studio and Ollama. They are the two friendliest local backends, and they do a lot of quiet work to behave like the OpenAI API, so a tool like mine can pretend there's only one API in the world to talk to. Then I moved the rig to vLLM in Docker, across two 4090s modded to 48GB each, and the daily driver at the time became Qwen3-Coder-Next 80B, serving at around 101 tokens per second (tok/s) across both cards. The vLLM tuning work covers how that rig came together, and the setup walkthrough the steps to stand one up.

vLLM is a stricter animal. It's built for throughput and production serving, not for keeping one desktop user comfortable, and it doesn't paper over the rough edges of the OpenAI-compatible surface the way the desktop backends do. As I learned, no two versions of OpenAI API support may be the same.

The output was a bit garbled - here's what I was getting:

// the response shape (illustrative - content empty, answer in the wrong field):
{ "content": "", "reasoning_content": "Okay, so the task is to..." }

That last part is the tell. A reasoning model produces two streams, its hidden thinking and its final answer, and a well-behaved API keeps them apart: the answer goes in a content field, the thinking (if it's exposed at all) in a separate reasoning_content field. I'd built houtini-lm with a defensive fallback - if content is empty, surface reasoning_content rather than hand Claude a blank. Most days that fallback is a safety net. Here it was a smoke alarm. The model was doing the work and then filing it in the wrong drawer.

Cause one: the flag in the wrong place

When you want a thinking model to execute a task rather than deliberate about it, you tell it to skip the thinking. For the Qwen family the switch is a parameter, enable_thinking, set to false. houtini-lm had been setting it for a long time, and on LM Studio and Ollama it worked.

vLLM ignores this! No error, no warning - just a thinking model that carried on thinking as though I'd never asked it to stop.

Turns out, the flag was in the wrong place. LM Studio and Ollama read enable_thinking as a top-level parameter on the request. vLLM doesn't look there at all; it only honours the toggle nested inside a chat_template_kwargs object, because to vLLM that flag is an argument to the model's chat template, not a property of the request. Same word, same intent, two different drawers:

// What LM Studio and Ollama read:
{ "enable_thinking": false }

// What vLLM reads (and only this):
{ "chat_template_kwargs": { "enable_thinking": false } }

It's one of the most-reported Qwen3-on-vLLM issues of 2025, and in the same vLLM issue thread where the mechanism gets pulled apart, someone points out that the identical flag vLLM insists on nesting is accepted top-level by Alibaba's own DashScope endpoint. Same parameter, different drawer, depending on who's doing the serving. Hold that thought - it's the whole lesson, and I'll come back to it.

The fix for this half was small. I had houtini-lm send enable_thinking: false in both shapes now - top-level for the desktop backends, nested for vLLM - so the instruction lands wherever the backend happens to look. I checked it directly: the same request with the nested shape came back clean, the answer in content where it belonged.

Fixed. Except it wasn't.

The fix that didn't work

I made the change, restarted, ran the delegation again, and the answer came back empty. Again.

This is the bit I want to keep in, because it nearly didn't happen. The shape fix was correct - I had a clean response in hand proving it. Every reasonable instinct said write the changelog and move on. If I had, I'd have shipped a package documenting a fix for a bug that was, in ordinary use, still completely broken, because the fix was right but it was never running.

For a while I couldn't see daylight between the isolated test that passed and the real delegation that didn't. The test sent the nested flag by hand. Real delegation didn't. The question was why not.

Cause two: a model hiding under an alias

houtini-lm doesn't send the no-think toggle to every model. It shouldn't - plenty of models have no thinking mode to switch off, and the flag is just noise to them. So the tool decides first: it works out whether the model is a thinking model, and only then takes the branch that sends the toggle. That branch was where my shape fix lived.

It works out the answer by reading the model's metadata from Hugging Face - the public model registry - keyed on the model's name. There's the trap I fell into. LM Studio and Ollama serve models under their real, registry-resolvable names. vLLM serves them under whatever short alias you gave the server at launch. Mine is coder-next. When I traced it, there's no model called coder-next on Hugging Face, so the lookup came back empty, so a genuine thinking model was filed as non-thinking, so the toggle branch never ran, so the shape fix inside it never fired. The model kept thinking, the answer kept landing in reasoning_content, and the blank replies kept coming. Two bugs, stacked, the first hiding the second - and the only thing between me and shipping a still-broken tool was checking the real path instead of the tidy one.

The proper fix is an explicit override. v3.2.1 adds an environment variable, HOUTINI_LM_THINKING, with three settings: auto (the default - detect from metadata, as before), off, and on. Set it to off and the no-think path runs regardless of what the detection thinks it found:

HOUTINI_LM_THINKING=off

That's the right setting in two situations at the same time, and they happen to be the same situation from two angles: when an orchestrator like Claude is doing the reasoning and the local model only needs to execute, and when you're serving through vLLM under an alias that detection can never resolve. One flag covers both. I verified it end to end against the live coder-next server - the log shows the toggle firing on every call now, and the answer comes back in content.

I also wrote a regression test, test-vllm-thinking.mjs, that pins the vLLM contract - the nested flag, the alias, the expectation that content isn't empty - so the next refactor that quietly breaks this fails loudly instead. It skips cleanly on a machine with no GPU, so CI stays green without pretending it has hardware it doesn't.

What this taught me

Strip out the specifics and a few things are worth keeping, whether or not you ever run houtini-lm.

"OpenAI-compatible" is a family, not a standard.

The phrase gets used as though it were a guarantee, and it isn't. It means a backend broadly speaks the OpenAI request and response shape - enough that a naive client works for the easy case. It doesn't mean two backends agree on where a vendor-specific flag lives. My whole first bug was one flag in one place too high in the JSON, and the sourced proof that this is the rule rather than my bad luck is that same DashScope detail: the parameter vLLM demands nested, Alibaba accepts top-level. Porting a tool between backends is exactly the exercise that flushes these out. The only reliable way through is to read each backend's own docs rather than trust the family name.

The final check is the point, not the formality.

The fix that should have worked returned empty, and the only reason I caught it was checking the real end-to-end path instead of the isolated test that had just gone green. A passing unit test is evidence that one path works. It isn't evidence that the path your users actually take works. Shipping the first fix would have been worse than shipping nothing, because it would have carried a changelog line swearing the problem was solved.

Serving aliases defeat metadata detection.

Any tool that identifies a model by looking its name up in a registry breaks the moment a server hands that model out under a custom name, and vLLM is far from the only server that does. Clever detection is worth having as a default. It also needs a manual override sitting behind it, because there's always a setup the cleverness can't see. The override isn't a crutch for the lazy - it's the correct tool for a whole class of legitimate configs.

No-think is the right default for a local model that's only executing.

When Claude has already done the reasoning and the local model just has to carry out a bounded instruction, letting it think is waste - it spends the token budget deliberating over nothing, and it's what walked me into the blank-content trap in the first place. Switched off, the same tool call came back in 0.8 seconds instead of 3.7. Faster, cheaper, and it steps around the failure entirely. If your local model is executing rather than deciding, tell it not to think.

Where to get it

A few weeks on from all this, vLLM stuck. It's the backend the rig runs on now, with LM Studio and Ollama still there for the desktop. The fleet's moved on from that first 80B, too: a Qwen3.6-27B does the everyday delegation, and a 120B Nemotron takes the long-context jobs across both cards (a 120-billion-parameter model out-decoding the 27B still feels like it shouldn't be allowed, but that's a story for the model testing page ). Every one of them still gets enable_thinking nested inside chat_template_kwargs, because the lesson didn't age a day.

v3.2.1 is live on npm as @houtini/lm and on GitHub , with both fixes, the regression test, and a docs/VLLM-BACKEND.md that writes the caveats down for anyone else pointing the tool at vLLM. To be clear about what did and didn't change: vLLM is now a first-class, tested backend, and LM Studio, Ollama and OpenRouter are all still supported exactly as before. This wasn't a migration off anything. It was making the strict backend work as well as the forgiving ones - and finding out, on the way, that the forgiving ones had been hiding my bugs for months.

If you want the wider picture of why the delegation is worth the bother - the fleet, the models, the economics of a local machine doing the bulk work while Claude conducts - the model-testing page has the measured version. And if you're running Claude Code and watching the token bill climb, the case for delegating in the first place is where to start.