The VRAM traps: why a 16GB model wouldn't load on a 48GB card

Say you've got a 48GB card and a 16GB model. It fits - obviously it fits, there's thirty-odd gigabytes of headroom - so you point the server at it, hit go, and it dies on load. That one happened to me on this rig, and the cause turned out to be a shortfall of one kilobyte, in a kind of memory most people don't know a GPU has.

I spent five days in August benchmarking twelve open-weight models on a pair of modded RTX 4090s, and roughly the first day of that wasn't benchmarking at all - it was getting the things to load in the first place. Four of the first eight failed on the opening attempt, each for a different reason, none of them the model's fault. The question underneath all of it - the one that decides whether you buy the right card, and whether you lose an evening per model - isn't "does it fit in VRAM", which is what everyone asks. It's will it run? Three things kept catching me out: a memory that isn't the memory you think, a context length the model advertises but can't serve, and a label on the box that means less than it says. Here's each one, with the number that proved it, so you can spot them before they cost you the evening they cost me.

If you've not run a local model at all yet, this one assumes you're already comfortable with vLLM and a container or two. If you're not there, LM Studio will get your first model running in an afternoon and you can come back for the deep end. Everyone else, read on.

The kilobyte that stopped it loading

The model was GLM-4.7-Flash-GPTQ. I had 16.6GB of weights going onto a card with 48GB, so I wasn't braced for trouble - not a download problem, not a VRAM problem by any sane reading of the numbers. It got most of the way up and then fell over in KV-cache initialisation - the KV cache being the running memory of the conversation, the bit every token you generate has to write itself into - with this:

triton.runtime.errors.OutOfResources: out of resource: shared memory,
Required: 102400, Hardware limit: 101376

I read the two numbers a couple of times before they landed. It wanted 102,400 bytes and the hardware had 101,376. It was short by 1,024 bytes. One kilobyte. And the resource it ran out of wasn't VRAM at all - it was shared memory, which is a different thing entirely, and the reason this failure is so baffling the first time you meet it.

A GPU has a memory hierarchy, and what I'd never had to reckon with is that "out of memory" can mean any level of it:

I'd run local models on this rig for months and never once had to know shared memory existed. It's a tiny, extremely fast scratchpad physically inside each Streaming Multiprocessor - the SM, the GPU's basic compute unit. A kernel loads a tile of data into it, all the threads in the block hammer on that tile at speed, then it writes back; it's the whole trick behind FlashAttention-style kernels (keep the working tile in shared memory instead of trudging out to VRAM for every operation). Fast, and finite. And a kernel declares its shared-memory needs up front: if the hardware can't provide that much, the kernel does not run. No spilling, no slower-but-works fallback - a hard allocation, checked the instant the thing launches, which is why my model died on arrival rather than limping.

It's also per-SM, which is why my first instinct - throw the other card at it - does nothing. A second card can't help you, because the limit isn't a total you can add to - it's a ceiling on each unit. Ten cards each short by a kilobyte are still, every one of them, short by a kilobyte. This is the rare case where throwing another GPU at the problem is no help at all, and I'd have wasted a good while learning that if the error had been any less specific.

So why did the kernel want 100 KB in the first place? (I went looking for someone to blame and there isn't one - just a sensible decision made for a different card.) Because bigger tiles are faster - more work done per trip out to VRAM - so whoever wrote the kernel picks the largest tile the hardware they're aiming at can hold. Aim at a datacentre part and the budget is roughly double. Run that same kernel on a consumer card and it asks for more than exists, and falls over on arrival.

The kernel wanted 102,400 bytes - exactly 100 KB, a round number a human typed - and it was almost certainly tuned against a 163 KB or 227 KB budget, where 100 KB is a comfortable fit rather than a hair's-breadth overrun. On my Ada cards (compute capability 8.9), that leaves me a kilobyte short.

Which raises the obvious worry: are Ada cards about to become e-waste? No. And the reason no is the whole lesson. This isn't a generational limit that Blackwell quietly fixes - it's a market-segmentation limit. A 3090 has the same 99 KB per SM as a 4090; NVIDIA has held consumer parts at roughly 100 KB across several generations while the datacentre parts climbed to 163 and then 227. The line runs between GeForce and datacentre, not between old and new. Buying a newer consumer card does not obviously fix this, and I'd want to measure a consumer Blackwell part before spending a penny on the assumption - which is exactly the kind of number this whole project refuses to state without checking.

And the constraint is software, not silicon. The error message itself tells you as much: "Reducing block sizes or `numstages` may help." Most Triton kernels ship several tuned configurations and an autotuner picks one that fits; this failure is what happens when no_ configuration in the set fits under 99 KB - an oversight in a tuning table, a bug report, not a law of physics.

# GLM-4.7-Flash-GPTQ: the FP8 KV-cache kernel wanted 100KB of shared memory.
# Take the KV cache off FP8 and vLLM drops onto a different kernel entirely.
--kv-cache-dtype auto        # was fp8 - loaded in 441s and served clean

The fix took thirty seconds and cost nothing (though I'll admit I stared at that one-kilobyte gap for a good deal longer than thirty seconds before I worked out where it came from). The request had come from the FP8 KV-cache kernel, so I served the model with --kv-cache-dtype auto instead, it dropped onto a different code path, loaded in 441 seconds, and behaved itself. One flag.

The forecast, then, isn't obsolescence. It's a maintenance tax that grows slowly: as more kernel work targets datacentre hardware, consumer configurations get less attention, and the odds of hitting an untuned path drift up. You'll meet it as the occasional model-plus-flag combination that fails on arrival, each one individually fixable, each one costing you an evening if you don't know this failure mode exists. Which is the entire reason I'm writing it down. The card isn't obsolete. Some kernels just haven't been tuned for it.

"Does it fit in 48GB?" is the wrong question

The next one caught me despite the sums looking watertight, which is exactly the problem: it looks like arithmetic, and the arithmetic lies. You weigh the weights, you see they're smaller than the card, you conclude it fits. But the weights are only half of what has to live on the card. The other half is the KV cache, and it grows with the context length you run.

Llama-3.3-70B in AWQ is 39.8GB of weights. Its model card advertises a native context of 131,072 tokens. I asked for that context on a single 48GB card, and the engine refused to start it outright:

20.0 GiB KV cache is needed, which is larger than the available KV cache
memory (3.58 GiB) ... estimated maximum model length is 23456.

Thirty-nine gigabytes of weights leaves about four for the cache, and a full 131k context wants twenty. It fits, and it will not serve what its own card promises. Even a routine 32k request gets capped; the config I verified on one card runs it at 21,110 tokens, not 131,072. To get the long context I had to spread it across both cards. So the number you want, before you buy anything or point a server at anything, is not "weights versus VRAM" - it's weights plus KV cache at the context length you're going to run, and that second term is the one nobody prints on the box.

This is also where parameter count finally stops meaning anything, and I had to unlearn it. A 120B model, in the right format, is smaller on the card than a 70B. OpenAI's gpt-oss-120b in native MXFP4 is 65.2GB; Llama-3.3-70B in FP8 is 72.7GB. The 120B is the lighter one. Mixture-of-experts models - where only a few of the model's many expert sub-networks fire for any given token - break the old rule of thumb that parameter count predicts footprint, and they broke it a while ago. I still catch myself reaching for it out of habit; if you're doing the same, you're sizing hardware by a proxy that retired.

The label on the box lies twice

So parameter count is out. I moved on to trusting the quant label instead, which lasted about as long. Surely "4-bit" is honest, at least - a 27B model at four bits a weight is about thirteen and a half gigabytes, and you can plan around that?

You can't, and here's the first lie. Qwen3.8-27B in AWQ lands at 27.7GB, not the ~13.5GB the arithmetic promises (I sized a card off that arithmetic once and came up short - a cheap mistake to make, an annoying one to catch), because "4-bit" only ever describes some of the model. The attention layers, the lm_head, the whole vision tower - they stay in bf16, full fat. The headline number is the compression on the bulk of the weights; the real footprint is that plus everything the quantiser left alone, and the gap between them is the difference between fitting and not.

The second lie is worse, because the label can be outright false. I pulled a community "4-bit" build of gpt-oss-120b - an AWQ conversion - and it OOM'd on load, which made no sense for a model that was supposed to be quantised. When I checked the actual metadata on HuggingFace, the MoE experts - the bulk of the model - were never quantised at all. F16, 100% of them. The conversion had 4-bit in its name and full-precision weights in its files, so the engine dutifully sent it down the unquantised fused-MoE path and ran out of memory doing it. I dropped it and used OpenAI's official MXFP4 release instead, which is the one that later served a 117-billion-parameter model inside 96GB and did 153 tokens per second (tok/s) doing it. The lesson isn't "community quants are bad" - plenty are excellent. It's that the label is a claim, and the file is the truth.

The things only the engine can tell you

By this point in the campaign a pattern had set in, and I'd stopped trusting anything I hadn't asked the engine directly: the fact you need is one the spec sheet cannot give you, and the engine can, if you ask it instead of assuming.

Take kernel support, which is the sibling to the whole shared-memory story above. For weeks my own notes said, flatly, that Ada had no kernels for formats like MXFP4 and NVFP4 - so I'd rejected models on that basis. Then I stopped guessing and asked the engine, which carries a minimum-capability table for every quantisation method it supports:

$ python quant-support.py          # ask the engine; my cards report capability 89
mxfp4          min capability 80  ->  supported
nvfp4          min capability 75  ->  supported
fp8_per_block  min capability 75  ->  supported
awq            min capability 75  ->  supported

Every format I'd written off is permitted, on this exact GPU. (The full story of that particular wrong rule, and four others, is in the broken rules of local LLM inference - this is the part about how you find out rather than that it's true.) The caveat that is the whole discipline in one line: a permitted code path is not a fast one. The probe refutes "no kernels"; it does not promise "good". That still needs a number.

Then there's whether the engine has ever heard of your model. MuseGlimmerForConditionalGeneration, a Meta release six days old when I tried it, is simply absent from the pinned engine's registry - it only ever loaded on a custom development image. No model card tells you that; the engine tells you, by refusing. And "natively supported" is not the same as "servable" either. Kimi-Linear-48B is a vLLM-native architecture and I still couldn't serve it, because its tokenizer ships only a raw tiktoken.model and some custom Python, with no tokenizer.json for the fast-tokenizer fallback. Native as an architecture, dead as a thing you can run, and the only way I found out was trying. (A related trick, if you ever need vLLM's list of valid parsers: pass a deliberately wrong one and read the KeyError. The registry is lazy-loaded and reads empty if you ask it politely.)

None of these are on any spec sheet, model card, or VRAM calculator, and every one of them cost me a failed load before I learned to ask first. They're engine facts, and the engine is the only thing that knows them. Probe, don't assume - the same rule that runs through the whole campaign.

And what fits today may not fit next week

One more, and it's the one that unsettles me most, because it moves the ground under a working setup. A model that loaded last week may not load today, through nothing you did.

gemma4-31b-qat had been loading fine on my rig for a fortnight. Then it started failing - by 0.72GiB, a maddeningly small miss - after the serving engine upgraded itself from 0.25.1 to 0.26.0 with nobody choosing it. (My compose file said image: ...:latest and every model swap ran --force-recreate; between those two, the stack re-pulled a newer build overnight, and I found out by reading the version string, not by deciding anything.) You can force the model back in by raising the memory-utilisation ceiling, but you pay for the room: prefill throughput dropped from 1,384 to 938 tok/s once I did. So "it fits" is not even a stable fact across a Tuesday. Pin the engine by digest, or accept that your fit calculation has a moving part you didn't put there. I pin it now.

So how do you know it'll fit?

Line up everything that caught me and the shape is the same each time: the number that would have saved me was one the box could never print. So the working answer to "will it fit" is a short list of habits, not a glance at a VRAM figure.

• "Out of memory" is not one error. VRAM, L2 and shared memory are three different resources with three different limits, and only one of them is on the spec sheet. Read the required-versus-limit numbers in the traceback: 102400 against 101376 tells you immediately it's a tuning overrun, not a capacity problem, and a model wanting twice the budget would be a completely different conversation.
• Size it as weights plus KV cache at your context, never as weights against VRAM. The cache is the term that grows, and it's the term that refuses a 70B its own advertised context on a single card.
• Trust the file, not the label. Parameter count doesn't predict footprint, "4-bit" isn't four bits across the whole model, and a quant's name is a claim its metadata may not keep.
• Ask the engine the things only it knows - kernel support, registry membership, tokenizer viability - rather than inheriting a rule you never measured. And when a model-plus-flag combination fails, try the adjacent flag before you conclude the model is incompatible. The kilobyte failure was one server option away from a clean load.
• Consumer versus datacentre is the real dividing line, not old versus new. A 3090 and a 4090 share the ceiling I hit; an H100 wouldn't have blinked. That's about market segmentation, not technology, and it's worth knowing before you spend money to "upgrade" past a problem a newer consumer card carries too.

None of this is exotic once you've been bitten - and I have been, by every one of these. It's just that none of it is written on the thing you're buying.

Where this goes next

Back to where we started: the 48GB card and the 16GB model. It fits - that was never in question. Whether it runs is a different question, and now you know how to ask it: what kind of memory does the kernel want, how much cache does your context need, and is the label on the file telling the truth. Those are the three I check now, and asking them turns most of the evening I lost into one you get to keep.

Every model in this piece, with its verified flags, load times and measured contexts, lives in the benchmark hub - the raw dataset is downloadable there if you want to check my working. If you want the step-by-step of setting these flags on the same cards, the vLLM tuning walk-through is the how-to underneath this campaign, and the GPU buyer's guide covers why 48GB consumer cards earn their place despite the traps.

I still can't fully close the shared-memory question. Whether consumer Blackwell moves the 99 KB ceiling is a number I haven't measured, and until I have, I'd treat "a newer card fixes it" as a hope rather than a fact. If you've hit a fit failure that a VRAM figure never warned you about - the kilobyte kind, the context kind, the mislabelled-file kind - I'd like to hear it. If it's reproducible on 96GB of Ada, it goes on the list.