Private AI on our own hardware

Extraction, first drafts, code review, classification. The expensive reasoning engine was spending half its day typing.

There was a second problem, and for some of our client work it's the bigger one: some data shouldn't be sent to a third-party API at all.

Build a private AI tier and wire it into the same workflow. Two open-weight models run concurrently on our own GPUs - a 27B and a 31B, one per card, 96 GB of VRAM between them - served by vLLM through an OpenAI-compatible interface, so the rest of the stack can't tell it isn't a cloud API. A router sits in front, with the local tier and two cheap cloud tiers behind it, and the delegation layer picks the right one per job.

Getting vLLM right on consumer-class hardware is its own discipline, and we've published the whole journey: the same model went from 18.8 to 66 tokens a second on one card once the quantisation-format trap was understood - a 3.2x difference between "technically running" and "worth using". The write-ups are on this site if you want the detail: setting up vLLM, tuning it on a 48GB RTX 4090, and the migration story.

The division of labour is the point: the frontier model scopes the work and judges the output; the local models do the typing-heavy volume at zero marginal cost per call, at about 58 tokens a second with a 128,000-token context - long documents in a single local pass, at conversational speed. Nothing on the sensitive path leaves the building.

The delegation layer itself is houtini-lm, our open-source MCP server - @houtini/lm on npm - whose own ledger shows a couple of hundred thousand tokens of bounded work already offloaded on one workstation alone. The rig under my desk has been up 33 hours straight as I write this, both GPUs serving. (The wife thinks I'm mad, and she's probably right.)

The competitor price monitor needed 12,576 candidate product pairs judged: is this competitor's castor the same product as the client's, or just similar? That's exactly the shape of work this tier exists for - high-volume, bounded, and involving a client's commercial data. The two local models graded every pair, free, and agreed on 88.3% of them; only the 1,467 disagreements went to a paid cloud model as tiebreak. Every verdict carries a reason of twelve words or fewer, so a human can audit why the machine paired two products. Paid tokens spent on 12% of the work, client data kept on our own hardware for the rest, and the whole grading pass ran while we got on with something else.

Live, in daily use on our own work. This isn't a packaged product - it's the capability we build for others, running on our own bench first. The savings figure to date is modest, and I'll own that - the value isn't last month's bill. It's that the volume work now has a home that costs nothing per call and shares nothing with anyone.