Private AI on my own hardware

Extraction, first drafts, code review, classification. The expensive reasoning engine was spending half its day typing.

There was a second problem, and for some of my client work it's the bigger one: some data shouldn't be sent to a third-party API at all.

Build a private AI tier and wire it into the same workflow. Two open-weight models run concurrently on my own GPUs - a 27B and a 31B, one per card, 96 GB of VRAM between them - served by vLLM through an OpenAI-compatible interface, so the rest of the stack can't tell it isn't a cloud API. A router sits in front, with the local tier and two cheap cloud tiers behind it, and the delegation layer picks the right one per job.

Getting vLLM right on consumer-class hardware is its own discipline, and I've published the whole journey: the same model went from 18.8 to 66 tokens a second on one card once the quantisation-format trap was understood - a 3.2x difference between "technically running" and "worth using". The write-ups are on this site if you want the detail: setting up vLLM, tuning it on a 48GB RTX 4090, and the migration story.

The division of labour is the point: the frontier model scopes the work and judges the output; the local models do the typing-heavy volume at zero marginal cost per call, at about 58 tokens a second with a 128,000-token context - long documents in a single local pass, at conversational speed. Nothing on the sensitive path leaves the building.

The delegation layer itself is houtini-lm, my open-source MCP server - @houtini/lm on npm - whose own ledger shows a couple of hundred thousand tokens of bounded work already offloaded on one workstation alone. The rig under my desk has been up 33 hours straight as I write this, both GPUs serving. (The wife thinks I'm mad, and she's probably right.)

A competitor price monitor I built needed 12,576 candidate product pairs judged: is this competitor's castor the same product as the client's, or just similar? That's exactly the shape of work this tier exists for - high-volume, bounded, and involving a client's commercial data. The two local models graded every pair, free, and agreed on 88.3% of them; only the 1,467 disagreements went to a paid cloud model as tiebreak. Every verdict carries a reason of twelve words or fewer, so a human can audit why the machine paired two products. Paid tokens spent on 12% of the work, client data kept on my own hardware for the rest, and the whole grading pass ran while I got on with something else.

Live, in daily use on my own work. This isn't a packaged product - it's the capability I build for others, running on my own bench first. The savings figure to date is modest, and I'll own that - the value isn't last month's bill. It's that the volume work now has a home that costs nothing per call and shares nothing with anyone.