A 13B parameter model is running on a GPU with 48GB of VRAM. The model weights take roughly 26GB when running in FP16.
So the calculation looks simple:
48GB GPU − 26GB model = ~22GB available
Then you put around 10 users on it. Everything should be fine.
Until the GPU crashes with:
CUDA out of memory
This is where many teams make the wrong diagnosis.
They start thinking:
“Maybe the GPU isn't big enough.”
Often, that isn't the real problem.
This is usually a memory management + inference architecture problem.
The model weights are only one part of GPU memory consumption. During inference, the server also needs memory for KV cache, temporary computation, request handling, CUDA allocations and other runtime requirements.
And some of those requirements change depending on what your users are doing.
That is where things can go wrong.
1. The Mistake: Looking Only at Model Size
Suppose you have:
13B Model
↓
FP16
↓
~26GB weights
Your GPU:
48GB VRAM
It is tempting to think:
48GB ├──────────────────────────────┤ │ Model ~26GB │ ├──────────────────────────────┤ │ Remaining ~22GB │ └──────────────────────────────┘
But inference isn't simply:
Load model → receive request → generate response
A more realistic picture is:
GPU MEMORY
│
┌────────────┼────────────┐
↓ ↓ ↓
Weights KV Cache Runtime /
~26GB Variable Temporary
Memory
│ │ │
└────────────┼─────────────┘
↓
Total VRAM Used
The weights are relatively predictable.
The other allocations can change with workload.
And with multiple users, those changes can happen simultaneously.
The first important lesson is:
Model size is fixed. Request memory isn't.
2. The Hidden Memory Consumer: KV Cache
During generation, the model needs to retain information about the tokens it has already processed.
That information is stored in the KV cache.
The important thing is:
KV cache grows with context length and the number of active sequences.
Consider two users.
User A:
Prompt: 100 tokens Response: 200 tokens Total: ~300 tokens
Small request.
User B:
Prompt: 15,000 tokens Response: 2,000 tokens Total: ~17,000 tokens
Very different memory requirement.
Now imagine ten users doing different things at the same time:
User 1 → 500 tokens User 2 → 1,200 tokens User 3 → 8,000 tokens User 4 → 700 tokens User 5 → 12,000 tokens User 6 → 2,000 tokens User 7 → 400 tokens User 8 → 9,000 tokens User 9 → 1,000 tokens User 10 → 6,000 tokens
The model is still the same 13B model.
But the memory requirements are completely different.
The GPU may effectively look like:
48GB GPU ┌──────────────────────────────────────────────┐ │ Model weights │ │ ██████████████████████████ │ │ │ │ KV Cache │ │ ███████ █████████████ ███ ████████ █████ │ │ │ │ Runtime / temporary allocations │ │ ████ █████ ███ │ │ │ │ Free memory │ │ ██ │ └──────────────────────────────────────────────┘
The model weights are not the whole story.
3. Ten Users Don't Necessarily Mean Ten Small Requests
This is another common assumption.
Someone says:
“We're only expecting ten concurrent users.”
That number alone tells you very little.
Ten users asking:
"Hello"
is completely different from ten users sending:
10,000–30,000 token documents + large conversation history + long responses
The second workload can put enormous pressure on GPU memory.
So instead of asking:
“How many users can this GPU handle?”
Ask:
“How many tokens can this GPU safely serve concurrently?”
That is a much more useful question.
Concurrency needs to be understood together with:
- Input length
- Output length
- Context length
- KV-cache usage
- Batch size
- Model precision
- GPU memory
- Runtime overhead
Ten lightweight users may be perfectly fine.
Ten users with very large contexts may not be.
4. The Second Problem: Uncontrolled Concurrency
Now look at a very simple application:
@app.post("/generate")
async def generate(req):
return model.generate(req.prompt)
It looks perfectly reasonable.
But what happens when ten requests arrive together?
Potentially:
Request 1 ───────────────→ GPU Request 2 ───────────────→ GPU Request 3 ───────────────→ GPU Request 4 ───────────────→ GPU Request 5 ───────────────→ GPU Request 6 ───────────────→ GPU Request 7 ───────────────→ GPU Request 8 ───────────────→ GPU Request 9 ───────────────→ GPU Request 10 ──────────────→ GPU
There is no intelligent traffic management here.
There may be:
- No request queue
- No memory-aware scheduling
- No backpressure
- No context limit
- No concurrency limit
- No per-request token budget
The application effectively says:
“Everyone come in at once.”
And the GPU says:
“I don't have enough memory.”
Then:
CUDA out of memory
5. What You Actually Want
Instead of:
10 requests
↓
10 independent generation jobs
↓
GPU
You want something closer to:
Incoming Requests
│
↓
┌─────────────────┐
│ Request Queue │
└────────┬────────┘
│
↓
Memory-aware
Scheduler
│
┌─────────────┼─────────────┐
↓ ↓ ↓
Request A Request B Request C
│ │ │
└─────────────┼─────────────┘
↓
GPU
The server decides what can safely run together.
This is where dedicated inference engines become extremely useful.
6. Continuous Batching Changes the Game
One of the biggest improvements is continuous batching.
Traditional processing may look like:
Batch 1 User A ────────────────→ Finish Batch 2 User B ────────────────→ Finish Batch 3 User C ────────────────→ Finish
That can leave resources underused.
Continuous batching allows requests to enter and leave the active batch dynamically:
Time →
────────────────────────────────────────────
User A ████████████████████
User B ███████████████
User C █████████████████
User D ███████
User E █████████████
↓
Dynamic scheduling
↓
GPU
Common inference stacks include:
- vLLM
- TensorRT-LLM
- Hugging Face TGI
The exact choice depends on your model, hardware, quantization, workload and deployment requirements.
But batching is not magic.
Simply enabling batching doesn't mean:
“Now we can put unlimited users on the GPU.”
You still need memory limits.
7. Control the KV Cache
This is one of the most practical fixes.
You need limits around:
Maximum input tokens
Don't allow an individual request to consume an unreasonable amount of context.
max_input_tokens = controlled
Maximum output tokens
Don't let every request generate enormous responses.
max_output_tokens = controlled
Maximum context length
Define how much total context your service actually needs.
context window
↓
must fit within
↓
your GPU memory budget
Maximum concurrent sequences
Don't blindly accept unlimited active requests.
GPU capacity
↓
memory budget
↓
maximum active sequences
This is much safer than:
Users → unlimited → GPU
8. Don't Run the GPU at 100%
Another mistake is trying to squeeze every last GB out of the GPU.
Suppose your monitoring says:
48GB total 47.8GB used
It may look efficient.
It isn't necessarily healthy.
Your runtime still needs room for allocations and workload changes.
A safer approach is to maintain a memory reserve.
For example:
48GB GPU │ ├── Model ├── KV cache ├── Runtime allocations ├── Temporary memory └── Safety reserve
Depending on your workload and runtime, keeping roughly 10–15% as a practical starting reserve can be reasonable, but it should be validated through actual load testing rather than treated as a universal rule.
The objective is simple:
Don't operate permanently on the edge of failure.
9. GPU Memory Fragmentation Can Make Things Worse
Now comes another subtle problem.
Imagine the allocator has memory arranged something like this:
GPU Memory
[USED][FREE][USED][FREE][USED][FREE][USED]
↑
scattered
Suppose you have:
15GB total free
but the next operation needs a sufficiently large allocation and the allocator can't satisfy it efficiently.
You can end up with an allocation failure even though your monitoring appears to show plenty of memory remaining.
Modern CUDA and runtime allocators have mechanisms to reduce fragmentation, so don't automatically blame fragmentation whenever you see an OOM.
But it is a real consideration, particularly with workloads involving changing allocation sizes and long-running processes.
The practical lesson:
Watch allocation behavior, not just the headline “free VRAM” number.
10. Observability Is Not Optional
If your production LLM server occasionally crashes, don't rely on:
“It crashed around 2 PM.”
You need to know what happened before the crash.
At minimum, monitor:
GPU memory used GPU memory available GPU utilization Active requests Queued requests Input tokens Output tokens Request duration Maximum context length Errors / OOM events
For example:
nvidia-smi --query-gpu=memory.used,memory.total \ --format=csv -l 1
This gives you a basic view of memory consumption.
But application-level metrics are equally important.
You want to be able to answer:
OOM occurred
↓
How many users were active?
↓
How many tokens were they processing?
↓
What was the KV-cache usage?
↓
What was the GPU memory usage?
↓
Which request caused the spike?
Without that information, you're debugging blind.
11. Add a Queue Instead of Letting Everyone Hit the GPU
If the GPU is already busy, the application shouldn't necessarily start another generation immediately.
A queue gives you control:
20 incoming requests
│
↓
┌─────────────┐
│ QUEUE │
└──────┬──────┘
│
┌──────────┴──────────┐
↓ ↓
Can run safely? Too much load?
│ │
YES WAIT
│ │
↓ ↓
GPU Queue
This is called backpressure.
The idea is simple:
When the GPU is full, don't keep pushing work into it.
Make users wait a little rather than making the entire service crash.
A controlled 5-second wait is usually much better than a 30-second outage.
12. Reject Oversized Requests Early
This is another simple but powerful protection.
Suppose your application normally handles:
1,000–4,000 tokens
Then someone submits:
50,000 tokens
Don't discover the problem after you've already started allocating GPU memory.
Check the request first.
Incoming request
│
↓
Token count?
│
├── Within limit → Queue
│
└── Too large → Reject / ask user to shorten
This protects everyone else using the system.
13. The Architecture we Would Use
For a 13B model on a 48GB GPU, I would think about the system like this:
USERS
│
↓
┌───────────────┐
│ API SERVER │
└───────┬───────┘
│
↓
Request validation
│
┌───────────┴───────────┐
│ │
Too large Valid
│ │
Reject ↓
┌───────────┐
│ QUEUE │
└─────┬─────┘
│
↓
Memory-aware scheduler
│
↓
Continuous batching
│
↓
Inference engine
│
↓
48GB GPU
│
┌─────────────┴─────────────┐
│ │
Model weights KV cache
│ │
└─────────────┬─────────────┘
↓
Monitoring
This is a much healthier architecture than:
HTTP Request
↓
model.generate()
↓
GPU
14. What we Would Measure Before Buying a Bigger GPU
This is perhaps the most important takeaway.
If your 13B model is crashing on a 48GB GPU, don't immediately order a larger GPU.
First measure:
1. Model weight memory 2. KV-cache consumption 3. Maximum context length 4. Average input tokens 5. Average output tokens 6. Peak concurrent requests 7. GPU memory usage under load 8. Request queue depth 9. Generation throughput 10. OOM frequency
Then run progressively larger workloads:
1 user ↓ 2 users ↓ 4 users ↓ 6 users ↓ 8 users ↓ 10 users ↓ 15 users ↓ 20 users
At every step record:
VRAM Latency Tokens/sec Queue time Throughput Errors
You will quickly discover where the actual limit is.
A Practical Example
Imagine your test produces this:
| Concurrent users | VRAM | Result |
| 1 | 30GB | Stable |
| 2 | 32GB | Stable |
| 4 | 35GB | Stable |
| 6 | 39GB | Stable |
| 8 | 43GB | Slow |
| 10 | 47GB+ | OOM |
The answer isn't necessarily:
“Buy a 64GB GPU.”
The first question should be:
“Why did memory grow from 30GB to 47GB?”
Maybe:
Long contexts
+
Large KV cache
+
Too many active sequences
+
No request limits
+
No memory reserve
Fix those first.
You may find that the same 48GB GPU can serve the workload reliably with better scheduling and sensible limits.
KV Cache Doesn't Grow Exponentially
You'll sometimes see explanations saying:
“KV cache grows exponentially.”
That's not the right way to describe it.
For a given model configuration, KV-cache memory generally grows roughly linearly with the number of cached tokens and active sequences.
The problem is that the number of tokens being served concurrently can become surprisingly large.
So the better statement is:
KV cache grows with context length and concurrent sequences, and that growth can quickly consume the remaining VRAM.
That's a much more useful way to think about it.
The Real Lesson
When an LLM crashes with CUDA OOM, don't immediately ask:
“Is my model too big?”
Ask:
“What else is consuming my GPU memory?”
Then investigate:
Model weights
+
KV cache
+
Runtime memory
+
Temporary allocations
+
Concurrency
+
Context length
+
Allocator behavior
The model might fit perfectly.
Your serving architecture might not.
And that distinction can save you from unnecessarily buying expensive GPU hardware.
The Architecture in One Picture
┌─────────────────────┐
│ USERS │
└──────────┬──────────┘
│
↓
┌─────────────────────┐
│ API / Gateway │
└──────────┬──────────┘
│
Validate request
│
┌─────────────┴─────────────┐
│ │
Too many tokens Valid request
│ │
Reject ↓
┌──────────────┐
│ Request Queue│
└──────┬───────┘
│
↓
┌────────────────┐
│ Memory-aware │
│ Scheduling │
└───────┬────────┘
│
↓
Continuous Batching
│
↓
┌────────────────┐
│ Inference │
│ Engine │
└───────┬────────┘
│
↓
┌──────────────┐
│ 48GB GPU │
│ │
│ Model │
│ KV Cache │
│ Runtime │
│ Reserve │
└──────────────┘
│
↓
Monitoring