1 article tagged memory-bandwidth.
In today's post I'm going through the best GPU for running LLMs locally, and why memory bandwidth and VRAM decide it. I start with the cheapest VRAM you can buy and the cheapest 32GB cards, then the modded 48GB RTX 4090s I run, and finish on when an RTX PRO 6000 or a DGX Spark makes more sense.
local-llm · gpu