Operating · accepting briefs Work with me

Tagged: quantisation

4 articles tagged quantisation.

How to Benchmark vLLM: Find the Best Model, Quant and Settings for Your GPU

In today's post I'm sharing how I benchmark local models on vLLM by changing one thing at a time, along with the traps that quietly skew the numbers and my results from 21 setups on two 48GB RTX 4090Ds. The harness, vllm-bench, is free, so you can run the same tests on your own card.

local-llm · vllm
vLLM settings for a pair of RTX 4090s: the flags I run for twelve local models, and why

The flag-by-flag vLLM settings page for a dual-4090 rig: the baseline every preset shares, the six flags that change per model, the rig discipline you set once, and one verified launch block for each of twelve open-weight models with the reason attached.

local-llm · vllm
KV Cache Explained: Why a Local LLM Won't Load on Your GPU

In today's post I'm taking a closer look at the KV cache, the memory a model keeps of the conversation so far, and why 'does it fit in VRAM?' is the wrong question when a local LLM won't load. It's what refused a 70B model its own 131k context on my 48GB card, and I'll show you how to size it, the vLLM settings that control it, and the kilobyte of GPU shared memory that stopped a 16GB model outright.

local-llm · vllm
LLM Inference Optimization: What Made Two RTX 4090s Faster

In today's post I'm going through what made local LLM inference faster on my two modded 48GB RTX 4090s, in the order it mattered: the file format, how you split a model across cards, locking the clocks at boot, speculative decoding and load times, with the numbers from my own testing.

local-llm · vllm