Best GPUs for Running Local LLMs (2026): Memory Bandwidth, VRAM and the Cards Worth Buying

I have run local models on more GPUs than I can sensibly list: every flavour of the RTX 30-series through the mining years, multi-GPU rigs on risers, a Threadripper workstation stacked with RTX 4500 Ada cards, and - since July - a pair of modded 48GB RTX 4090s, one of which I have benchmarked half to death . This guide is what all of that taught me about spending money on inference hardware.

Two specs decide a local-LLM GPU purchase: memory bandwidth (GB/s) and VRAM capacity (GB). Bandwidth governs how fast tokens come out. VRAM governs how big a model fits. Three years ago the answer was simpler - buy the most VRAM you could afford and live with whatever else came with it. The mid-2026 market is stranger: prices have detached from MSRPs, the most interesting 48GB card comes from an unofficial supply chain, and a third spec has quietly joined the first two - whether your card's architecture has fast kernels for the quantisation format you plan to run.

Why memory bandwidth matters more than compute (mostly)

LLM token generation is memory-bandwidth bound, not compute bound. During the decode phase - every token after the first - the GPU spends most of its time waiting on memory reads, not doing arithmetic. A card with 60% more TFLOPS and the same bandwidth generates tokens at very nearly the same speed as the slower card. A card with 78% more bandwidth and similar compute generates them close to 78% faster. The TFLOPS column is mostly a vanity stat for this workload.

The caveat: prefill is compute-bound. Prefill is what happens before any tokens come back - the model reads and embeds your prompt to build the KV cache. On a short chat prompt it is invisible. On a 16k-token context with a big RAG document it absolutely is not, and it scales with TFLOPS rather than GB/s. Interactive chat rarely cares; agentic workflows that re-process long contexts every turn very much do. My own benchmarks show the same split from the other direction: an AWQ-quantised model on my 4090 decodes far faster than the FP8 build of the same model, but pays for it with roughly half the cold prefill speed - the kernels trade one phase against the other.

I lived the bandwidth ceiling for a year on the workstation tier: the RTX 4500 Ada cards in my old Threadripper rig topped out at 432 GB/s each, well below a 3090's 936 GB/s, and you felt it on every long generation. Workstation cards trade raw bandwidth for ECC, density and blower cooling. That machine is sold now, but the lesson transferred straight into the current build: buy bandwidth first.

The spec sheet that matters:

Hardware Corner's RTX 5090 LLM benchmarks measure 102.7 tokens per second on Qwen3 14B at Q4_K and 16k context, with the 4090 at 77% of that across their test set - tracking the bandwidth ratio almost exactly. My own numbers tell the same story inside one card: decode speed moves with bytes-touched-per-token, which is why a 35B mixture-of-experts on my 4090 decodes at 133 tok/s while the dense 27B manages 61 - the MoE touches a ninth of the bytes. Pure bandwidth maths is the upper bound, not the number you will see on a meter; expect 70-90% of theoretical scaling once kernel and framework overhead eat their share.

Prioritise GB/s for decode speed, TFLOPS for prefill speed, GB for model size. If a card's spec sheet leads with RT cores or gaming benchmarks, that's a marketing document, not a buying guide for this workload.

Check the quant format before you buy

This is the buying criterion the tier lists don't mention, and it cost me three days of confusion, so it gets its own section.

The same GPU can be fast or slow on the same model depending on which quantisation format the checkpoint uses, because kernel optimisation is per-architecture. I loaded the official Qwen3.6-27B FP8 release on my 48GB 4090 and got 18.8 tok/s - baffling for a card with 1 TB/s of bandwidth. The reason: official Qwen FP8 checkpoints are block-format FP8, which has beautifully tuned kernels for datacentre Hopper and Blackwell cards and none for Ada, the architecture every 4090 owner is sitting on. The engine falls back to a generic kernel and quietly loses you 40% or more. Swapping to an AWQ INT4 build of the same model - Marlin kernels, superbly tuned for Ada - took the same card to 60.9 tok/s with speculative decoding stacked on top. Same silicon, 3.2x difference, decided entirely by which file I downloaded.

The buying consequence: a consumer Ada card (any 4090, including the modded 48GB ones) wants AWQ, GPTQ or QAT-style quants; it will underperform on block-FP8 releases. Blackwell cards (5090, Pro 6000) carry the newer kernel support and don't hit this particular wall. If your plan is "run the official FP8 releases," that's a genuine point for the 5090 tier that no spec table shows. The full investigation - with every measurement - is in the vLLM tuning write-up .

How much VRAM you need

VRAM decides which models you can run and at what quality. Practical tiers in 2026 at 4-bit quantisation (the practitioner default), with some context budget kept aside:

Q4KM (the 4-bit quantisation most practitioners reach for) is the practical floor. It compresses model weights to roughly 4 bits each with minimal quality loss. Q6 and Q8 keep more precision and produce noticeably better output for coding and reasoning - the difference is worth chasing if your work depends on the model. 96GB matters not because you suddenly run bigger models, but because you run the same 70B at Q6 instead of Q4, and that quality bump is what justifies the price.

One pleasant surprise from my own testing: context is cheaper than the headline maths suggests on some modern architectures. Qwen3.6's hybrid attention keeps full KV cache on only a quarter of its layers, which is how the full 262k context window fits in about 8.6GB on my 48GB card. Worth checking the architecture before you assume long context needs a bigger GPU.

The question to ask when sizing up is not "what model do I want to run today" but "what model do I want to keep running comfortably in 18 months." Models keep growing. 13B was the daily driver in 2024. 30-34B is the daily driver in 2026. Plan for the next jump, because the one after that is already in training.

The cards worth buying in 2026

Five GPUs cover the practical range. Anything else is either worth waiting a generation on, or has been superseded enough that the secondhand market is the better route in.

Used RTX 3090 - the floor (~$700-1,000)

24GB GDDR6X. 936 GB/s. From 2020 and still earning its place in 2026.

I ran a 3090 in my development machine for years - DeBERTa scoring passes over article drafts, Granite embeddings, everything in the 7B-13B range that wanted fast responses. Before that I mined on three of them and rebuilt more 3090-class cards than I care to remember, so I know exactly how they behave at sustained load: hot, loud, and relentlessly dependable if you respect the VRAM temperatures. That machine has since been consolidated into the 4090 build, but nothing in the years I ran it changed the verdict: for a $700-1,000 used card, it does an unreasonable amount of work.

Used 3090s are the cleanest sub-$1000 path to 24GB. The 4080 (16GB) and 4070 Ti (12GB) get marketed as "AI-friendly" cards, but the VRAM ceiling hits fast on anything 13B and bigger. Once you have hit that ceiling, no amount of marketing copy puts you back under it.

Why a 3090 still beats most newer mid-range cards for LLM inference:

• 24GB is the practical minimum for meaningful work with 13-30B models
• 936 GB/s - only ~7% behind the 4090 for token generation
• Wide aftermarket: easy to source, easy to replace, easy to pair
• 350W TDP runs hot and wants a 1000W PSU, which is the tradeoff

Under $1000, this is the answer. There is no second place worth mentioning.

Dual RTX 3090 - the value play (~$1,500-1,900)

Two used 3090s give you 48GB combined and let you run 70B models at Q4. Cheapest path to the 70B tier, full stop, and the highest VRAM per pound on the market.

My multi-GPU miles come from adjacent rigs rather than a 3090 pair specifically: mining rigs with more cards than sense, and later the Threadripper workstation running multiple RTX 4500 Adas - so PCIe topology, PSU sizing and cooling-at-sustained-load are things I have lived with rather than read about. The specific 1.6-1.8x NVLink tensor-parallelism number below is a 3090-pair result from the practitioner reports, not something I have measured myself. The framing holds though: NVLink is the reason the dual-3090 configuration still works.

Why did NVIDIA strip NVLink out of the 4090 and 5090? To protect datacentre sales. The 3090 supports it (a 112 GB/s bidirectional bridge between two cards), and that interconnect is what lets tensor parallelism on a two-card setup scale properly. Without it you fall back to splitting the model across the PCIe bus, which is roughly one-thirtieth the bandwidth of VRAM.

The scaling story depends on the parallelism strategy your inference engine uses:

• Tensor parallelism over NVLink (the right config for two 3090s) - close to linear on token generation, often 1.6-1.8x on a 2-card setup
• Tensor parallelism over bare PCIe (the no-NVLink fallback: 4090 or 5090 pairs) - the practitioner consensus is 0.6-0.75x scaling per card, and on consumer boards you must disable peer-to-peer transfers (NCCL_P2P_DISABLE=1) or the whole thing silently hangs. I have this wired into my own dual-card config before the second 4090 has even benched.

The standing rule from my own fleet: never split a model across two cards if it fits on one. Two independent cards, each serving its own model, always warm, beats one model paying the interconnect tax - tensor parallelism is for the model that will not fit any other way. Bandwidth does not combine either way; what combines is capacity, and 48GB at this price is the real value.

Other caveats:

• A 3-slot NVLink bridge runs ~$80-150 - factor it in; you cannot run NVLink without one
• Two physical x16 slots and a 1300W+ PSU is the realistic minimum (see the power-cap note under Common Mistakes - it helps more than you'd think)
• Runs hot, runs loud, draws 700W under sustained load. Not a quiet build.
• Three-slot clearance per card; stacking two axial-cooler 3090s without space throttles the top card inside minutes

RTX 4090 - the awkward middle (~$2,200-2,500 used)

24GB GDDR6X. 1,008 GB/s. I skipped the stock 4090 entirely: by the time I outgrew 24GB, the modded 48GB versions of the same silicon existed, and doubling the VRAM for similar money was the obvious move. That is the 4090's problem in mid-2026 in one sentence.

Used prices have not come down the way everyone expected - they have drifted up to $2,200-2,500 in the general GPU squeeze, which puts a used 24GB 4090 uncomfortably close to a modded 48GB one. Hardware Corner's ranking puts it at 77% of the 5090's throughput, which tracks the bandwidth ratio.

The remaining case: mature drivers, mature cooling, comfortable in a 750W PSU, and if one lands at a properly good price it is still a fine card. But at current prices, most buyers should either step down to a used 3090, or across to the modded 48GB tier.

Modded RTX 4090 48GB - the one I bought. Twice. (~$2,500-3,400)

48GB GDDR6X on a single card at standard 4090 bandwidth. These are the Chinese-market rebuilds - the memory doubled by populating both sides of a longer PCB, dual-slot blower coolers, standard NVIDIA drivers via patched BIOS. When I last updated this article, I wrote that I'd love a pair but the workstation budget wasn't there. The budget got there: I now run two, and one of them has been through the most thorough benchmark programme I have ever inflicted on a piece of hardware .

What I can tell you from living with them rather than reading the teardowns:

• The performance is real. ~1 TB/s of bandwidth behaves exactly as the spec suggests: my daily-driver 27B decodes at 61-66 tok/s (AWQ plus speculative decoding), a 35B MoE at 133, and the full 262k context window fits with room to spare. Every number, with methodology, lives on our model testing page .
• Thermals deserve respect but are manageable. Clamshell VRAM means double the memory chips in the same envelope. Mine run capped at 330W - and here is the finding that surprised me: the cap costs the big models nothing measurable on decode, because memory-bound workloads barely notice clock speed - while the card runs meaningfully cooler. Hardware Corner's teardown measured 86°C memory and a 65 dB blower at full tilt; capped, mine are far more civilised.
• The supply-chain risk is the honest price of entry. Refurbished dies, no NVIDIA warranty, the modder's reputation instead. I did my homework, accepted the risk profile, and would do it again - but I would not recommend it to anyone who needs warranty support or predictable replacement.

The market has moved in this card's favour: at $2,500-3,400 it now costs less than a street-priced 5090 while carrying 50% more VRAM. The nearest warrantied-lineage alternative is a used RTX A6000 48GB at $2,600-3,800 - ECC and official silicon, but 768 GB/s against the modded card's ~1 TB/s, so you trade roughly a quarter of your decode speed for the peace of mind. Both are defensible; I took the bandwidth.

For someone who needs 48GB on a single card and accepts the unofficial-supply risk profile, this is - by my own money, twice - the buy of the current market.

RTX 5090 - the fastest card you can't buy at MSRP (~$3,600-3,850 street)

32GB GDDR7. 1,792 GB/s. The biggest bandwidth jump NVIDIA has shipped in years - and a $1,999 MSRP that has become fiction, with street prices at $3,600-3,850 and partner cards well beyond in the ongoing memory squeeze.

I looked hard at the 5090 and went the modded-4090 route instead: for the same money I got 48GB rather than 32GB, and my workloads are VRAM-hungry more than they are latency-critical. That is a workload call, not a verdict - if your daily model fits in 32GB and you want maximum single-user speed, the 5090 is a different tier. Hardware Corner's tested benchmarks measure 102.7 tok/s on Qwen3 14B at Q4_K with 16k context, and their summary - "significantly higher performance and more usable context than its predecessors" - matches everything the bandwidth spec implies. It also carries Blackwell's kernel support, so the block-FP8 trap that bites Ada owners does not apply.

Why the 5090 lands for speed-first buyers:

• 78% more bandwidth than the 4090, and it shows up in tokens per second
• 32GB fits 30B-class models at Q6 (quality territory) instead of Q4
• GDDR7 and current-generation kernel support are where the ecosystem is heading

Caveats:

• Street pricing: budget $3,600+, not the MSRP you read at launch
• 575W TDP wants a 1000W+ PSU and serious cooling (though see the power-cap note - inference rarely needs the full envelope)
• 32GB does not cover 70B at Q4, which pushes you back to multi-GPU or the 48GB tier

If the money is fixed and the choice is 5090 versus modded 48GB 4090, decide by workload: speed on models that fit 32GB → 5090; capacity for 70B-class → the modded card. I took capacity.

RTX Pro 6000 Blackwell - workstation tier (~$13,000-20,000)

96GB GDDR7 ECC. 1,792 GB/s. Same bandwidth as the 5090, three times the VRAM - and a price that has climbed out of enthusiast reach entirely: $13,250-20,000 in the current market, up from the $8,500-10,000 this article quoted in June.

I ran the workstation tier for a year - the RTX 4500 Ada generation, one rung down the ladder - so the case for ECC, blower density and sustained-load reliability is one I have lived rather than imagined. As Linus Tech Tips put it in their February 2026 hands-on : "the RTX Pro 6000 can fit much larger models in VRAM than the 5090 could ever dream of running efficiently." For LLM work, that sentence is the card's entire thesis. Hardware Corner puts it at 94% of the 5090's throughput - identical bandwidth, the difference is all VRAM ceiling:

• 70B at Q6 or Q8 instead of Q4 - meaningfully better output for coding and reasoning
• 120B-class models at Q4, and multi-user serving from a single card
• ECC memory for long-running work where bit-flips compound

Two workflow caveats from the LTT review that spec sheets will not tell you: the card ships DisplayPort-only (budget a quality active adapter if your monitor is HDMI), and Pro-line drivers update on a slower cadence than GeForce - fine for a dedicated inference box, an irritation for mixed use.

At the new price, though, the arithmetic has changed: two modded 48GB 4090s deliver the same 96GB total for well under half the money, if you can live with two cards, no warranty and no ECC. That is in fact the build I am completing right now. The Pro 6000 remains the answer for small-team serving where warranty and single-card simplicity are worth paying for - but for a solo practitioner the gap is now wide enough that I'd want a business reason, not an enthusiasm, to sign that invoice.

What about workstation cards, AMD, Apple Silicon, or a unified-memory box?

NVIDIA RTX 4500/5000/6000 Ada Generation - the workstation line below the Pro 6000 - is the tier I ran until this summer: multiple 4500 Adas (24GB ECC, 432 GB/s, 200W each) in a Threadripper box. The trade-offs are exactly as advertised: lower bandwidth per card, ECC, blower coolers made for stacking, low TDP, certified drivers. It served a 120k-context coding model reliably for a year, and I sold the machine only because two modded 4090s do the same job faster with less complexity. Buy this tier when ECC, density and sustained reliability matter more than raw single-user throughput; buy consumer cards when tokens per second per pound is the metric.

AMD RX 7900 XTX (24GB, ~$1,100) deserves an honest mention it did not get in the June version of this article: with ROCm maturing and NVIDIA prices inflated, it is the cheapest new 24GB card by a distance. The catch is the software time-tax - quant-format support and day-one model compatibility still trail CUDA, and if the kernel-trap section above taught you anything, it is that format support decides your real speed. I have not run one; practitioners I trust describe the experience as "fine once you've paid the setup weekend." AMD MI300X/MI325X remain datacentre answers to a question home builders are not asking.

Apple Silicon is the interesting outsider I still have not tested personally. An M4 Ultra Mac Studio offers up to 192GB of unified memory at ~1 TB/s - 4090-class bandwidth with a memory ceiling no consumer NVIDIA card touches. MLX keeps improving; it remains behind CUDA for depth. Already own one? Get productive on it before buying a rig. Buying specifically for LLM work? NVIDIA still prices better per unit of fast memory and supports everything first.

Unified-memory boxes are the new fourth option. NVIDIA's DGX Spark (128GB unified, ~$4,699) and the AMD Strix Halo mini-PCs (Ryzen AI Max+ 395, up to 96GB allocatable as VRAM, ~$3,000) both run 70B-class models on the desk with no discrete GPU at all - slower per token than a big card, far bigger memory ceilings per pound. I have covered that whole category, including where the trade-offs bite, in the AI mini-PCs guide .

What I'd buy

Three I'd steer away from:

• Any current-generation card under 24GB. Excellent gaming cards; the 16GB ceiling hits fast on anything 13B and bigger.
• A used 4090 at near-modded-48GB prices. Same silicon, half the VRAM, similar money. The market has made this card hard to justify.
• A workstation card you do not need. If your largest workload is 30B at Q4, spend the difference on RAM, NVMe, and a faster consumer card.

Common mistakes

1. Spending on TFLOPS instead of bandwidth. The TFLOPS column gets the marketing real estate. The GB/s column governs your experience.

2. Ignoring quantisation-format kernel support. The same card can be 3x faster on the same model depending on the checkpoint format - see the quant-format section above. Ada wants AWQ/Marlin; block-FP8 belongs to Hopper and Blackwell. Check what you plan to run before you pick the silicon.

3. Buying for the model you run today. Models keep growing, faster than most people expect. 16GB because "Mistral 7B fits" is a one-year decision. 24GB minimum is a three-year decision. I bought my 3090 in 2022 when 13B was the sensible target; it carried me to 30B-class workloads four years later.

4. Forgetting the rest of the rig - and overestimating the power you need. A 5090 in a 750W PSU browns out under load. But the flip side, from my own sweep: decode is memory-bound, so capping my 48GB 4090s at 330W from their 450W ceiling cost the big models nothing measurable while cutting heat and giving the PSU transient headroom for the second card. Cap your card, size the PSU for transients, plan three-slot clearance.

5. Assuming dual-GPU scales linearly. Two 3090s with NVLink get 1.6-1.8x on generation. Consumer cards without NVLink tensor-parallelise over PCIe at 0.6-0.75x per card - and on consumer boards you need NCCL_P2P_DISABLE=1 or it hangs. The capacity is the win either way. And if the model fits on one card, run it on one card.

Where to go from here

Pick the tier that matches your budget and your model-size goal. Stuck between two adjacent tiers? Almost always, buy the bigger one - the VRAM headroom pays back across the card's life.

A 24GB rig runs everything from 13B through 30-34B at usable Q4, which is where most practitioner work happens. A 48GB rig opens the 70B door - and in mid-2026 it is cheaper to walk through than it has ever been. A 96GB rig opens everything currently shipping as open weights.

For software: LM Studio remains the cleanest on-ramp - get a model running there first. When you outgrow it (for me that moment was tool-calling reliability), the vLLM tuning write-up documents exactly what the graduation looks like, measurement by measurement, on the same 48GB card this guide recommends. For everything around the GPU, the best PCs for local AI guide covers the build.

Get the bandwidth right, check the quant format, and the rest is configuration.