3 articles tagged quantisation.
The flag-by-flag vLLM settings page for a dual-4090 rig: the baseline every preset shares, the six flags that change per model, the rig discipline you set once, and one verified launch block for each of twelve open-weight models with the reason attached.
· local-llm · vllm
A 16GB model would not load on my 48GB card, and the reason was a shortfall of one kilobyte in a memory no spec sheet mentions. These are the fit traps a VRAM figure will never warn you about.
I used to lock the clocks on my mining GPUs. The same instinct just helped kill five rules of local LLM inference on a £6,200, 96GB rig.