Running an LLM through an API is relatively straightforward. You send a prompt, receive tokens, and pay for usage.
Running your own LLM infrastructure is different.
Once you deploy models such as Llama, Qwen, Mistral, DeepSeek, Gemma, or your own fine-tuned model, you become responsible for much more than the prompt:
- model selection
- inference parameters
- context-window management
- GPU memory
- KV cache
- batching
- concurrency
- quantisation
- model loading
- request scheduling
- latency
- throughput
- observability
- infrastructure utilisation
- cloud cost
A model producing a good answer does not necessarily mean the system is efficient.
You could have an LLM server using an expensive GPU while processing only a fraction of its potential workload. You could send thousands of unnecessary tokens with every request. You could configure a 32K context window when most requests use fewer than 2K tokens.
Or you could simply be generating far more tokens than your application actually needs.
This article explains the major parameters involved in LLM inference, how they affect output and infrastructure, and how developers can design self-hosted AI systems that use compute more efficiently.
First, Understand What Happens During LLM Inference
Consider a simple request:
System: You are a technical support assistant. User: Explain why my Nginx server returns HTTP 502.
Before the model generates anything, several things happen.
The text is converted into tokens.
The model processes those input tokens during a stage commonly called prefill.
It then begins generating output tokens during the decode stage.
Conceptually:
Prompt ↓ Tokenizer ↓ Input Tokens ↓ Prefill ↓ KV Cache ↓ Decode ↓ Output Tokens
These stages have different performance characteristics.
Prefill involves processing the input context and can make heavy use of GPU compute.
Decode generates tokens sequentially and is often constrained by memory bandwidth and the model's ability to repeatedly access its weights and cached attention state.
This distinction matters because:
2,000 input tokens + 100 output tokens
and
100 input tokens + 2,000 output tokens
are both 2,100-token requests, but they do not necessarily create the same workload.
Understanding this is the foundation of efficient LLM hosting.
Temperature
temperature controls how strongly the model favours high-probability tokens during sampling.
Example:
{
"temperature": 0.2
}
Lower temperature generally produces more predictable output.
Higher temperature increases variation.
For example, consider:
Complete this sentence: The database failed because...
At a low temperature, the model may consistently generate something like:
The database failed because the connection pool was exhausted.
At a higher temperature, responses may vary more:
The database failed because replication lag caused the failover process to time out.
or:
The database failed because an unexpected schema migration locked several tables.
Typical usage
For code generation, extraction, classification, technical support, and structured output, lower temperatures are usually appropriate.
For brainstorming and creative generation, higher temperatures can be useful.
Example starting points:
Code generation → 0.0–0.3 Technical Q&A → 0.1–0.4 General assistant → 0.4–0.8 Brainstorming → 0.7–1.0+
These are guidelines, not universal rules. Different models respond differently to sampling parameters.
Does temperature reduce GPU cost?
Not significantly by itself.
A lower temperature does not magically make the transformer cheaper to execute.
The larger cost reduction usually comes from generating fewer tokens, using a smaller model, improving batching, or reducing unnecessary context.
Top-P
top_p, also known as nucleus sampling, limits sampling to a set of tokens whose cumulative probability reaches a specified threshold.
Example:
{
"top_p": 0.9
}
Imagine the next-token probabilities are:
server 35% application 25% database 15% process 10% container 5% ...
Instead of sampling across the entire vocabulary, top_p restricts sampling to the most probable candidates until their cumulative probability reaches the threshold.
A lower value makes the model more conservative.
A higher value gives the model access to more alternatives.
In production systems, avoid changing every sampling parameter simultaneously without evaluation.
A configuration such as:
{
"temperature": 0.2,
"top_p": 0.9
}
can work well for technical tasks, but benchmark against your model and workload.
Top-K
top_k limits the candidate set to the K most probable tokens before sampling.
Example:
{
"top_k": 40
}
Only the 40 highest-probability candidate tokens are considered.
Compare:
top_k = 1
with:
top_k = 50
With top_k = 1, generation effectively selects the highest-probability candidate at each sampling step.
With top_k = 50, the model can sample from a larger candidate set.
Not every inference API exposes top_k, and implementations can differ, so verify how your serving engine handles it.
Maximum Output Tokens
This is one of the most important parameters for infrastructure efficiency.
Depending on the API, you may see:
{
"max_tokens": 500
}
or:
{
"max_completion_tokens": 500
}
or:
{
"max_new_tokens": 500
}
The exact name depends on the inference engine.
The concept is the same: limit how many tokens the model may generate.
Suppose your application only needs:
{
"status": "critical",
"risk": 87,
"reason": "SSH exposed publicly"
}
Giving the model a 4,000-token generation budget makes little sense.
If the task normally requires fewer than 200 tokens, start with something closer to:
{
"max_tokens": 250
}
Why?
Because generation is expensive.
Each additional generated token requires another decode step.
If your application processes millions of requests, unnecessary generation becomes unnecessary infrastructure cost.
Stop Sequences
Stop sequences allow generation to terminate when the model reaches a known delimiter.
Example:
{
"stop": ["</result>"]
}
Suppose your expected response is:
<result> risk=high score=92 </result>
Once the model generates:
</result>
the server can stop generation.
Without a stopping condition, the model may continue:
Additional explanation: The reason this happened...
That additional text might be completely useless to your application.
Efficient inference is often less about making tokens cheaper and more about avoiding unnecessary tokens entirely.
Repetition and Frequency Controls
Models can occasionally enter repetitive patterns.
For example:
The service failed because the database connection failed. The database connection failed because the connection failed. The connection failed because...
Depending on the inference engine, controls may include:
frequency_penalty presence_penalty repetition_penalty
These parameters discourage repeated tokens, phrases, or previously used concepts.
They are useful when repetition is an observed failure mode, but aggressive penalties can reduce answer quality.
Do not add penalties simply because the API exposes them.
Evaluate them against your actual workload.
Seed and Determinism
Some inference engines expose a seed.
Example:
{
"seed": 42
}
A fixed seed can improve reproducibility when debugging or evaluating model behaviour.
However, a seed does not necessarily guarantee identical output across:
- different hardware
- different model versions
- different inference engines
- different kernels
- different quantisation methods
- different batching conditions
Use it as an evaluation and debugging control rather than assuming complete determinism.
Context Window
This is where LLM efficiency becomes much more interesting.
Suppose a model supports:
128,000 tokens
That does not mean every request should use 128,000 tokens.
Consider an application that sends:
System prompt 1,500 tokens Conversation 8,000 tokens Retrieved docs 7,000 tokens User query 200 tokens -------------------------------- Total 16,700 tokens
Now imagine most of the relevant information was contained in only 3,000 tokens.
You are repeatedly asking the model to process thousands of unnecessary tokens.
A large context window is a capacity, not a target.
The Hidden Cost of Large System Prompts
Developers often focus on user prompts while ignoring the system prompt.
Suppose every request includes a system instruction containing:
4,000 tokens
And your application receives:
100,000 requests/day
That means:
4,000 × 100,000 = 400,000,000 system-prompt tokens/day
before considering user input, retrieved documents, tool definitions, conversation history, or generated output.
This is why prompt architecture is also infrastructure architecture.
Ask whether the model really needs:
4,000 tokens of instructions
or whether the same behaviour can be achieved with:
800 tokens
Well-designed prompts can improve both model reliability and infrastructure efficiency.
Conversation History Is Not Free Memory
Consider a support assistant with a long conversation.
Message 1 Message 2 Message 3 ... Message 50
A naïve implementation may send all 50 messages back to the model for every new turn.
By turn 50, the request could contain tens of thousands of tokens.
Instead, manage conversation state intentionally.
For example:
Recent conversation
+
Conversation summary
+
Relevant historical facts
+
Current user request
Instead of:
Entire conversation since the beginning
A practical context pipeline might look like:
Incoming request
↓
Retrieve recent messages
↓
Retrieve relevant memories
↓
Retrieve relevant documents
↓
Apply token budget
↓
Construct prompt
↓
LLM
Context management should be treated as a system component, not just prompt concatenation.
RAG Should Retrieve Relevant Context, Not Maximum Context
Retrieval-Augmented Generation can reduce the need to send entire knowledge bases to the model.
But badly configured RAG can create another problem.
Suppose your vector database returns:
20 documents × 1,000 tokens
That adds approximately:
20,000 tokens
to the request.
If only three documents matter, most of that context is waste.
A better retrieval pipeline may use:
Query ↓ Vector Search ↓ Candidate Documents ↓ Metadata Filtering ↓ Reranking ↓ Top Relevant Chunks ↓ LLM
Retrieval quality often matters more than retrieval quantity.
Model Size Matters
Suppose you have access to:
7B model 14B model 32B model 70B model
A common mistake is:
The 70B model is better, so every request should use 70B.
That may be unnecessary.
Consider these tasks:
Extract IP address from log Classify ticket category Summarise 500-word incident Generate Bash script Analyse complex architecture problem
They do not necessarily require the same model.
A smaller model may handle:
classification extraction simple summarisation format conversion routing
while a larger model handles:
complex reasoning architecture analysis difficult debugging long-form generation
This creates a model-routing architecture.
Incoming Request
↓
Router
↙ ↓ ↘
Small Medium Large
Model Model Model
The objective is not:
Use the cheapest model.
It is:
Use the smallest model that reliably satisfies the task.
Quantisation
Model weights are commonly stored or served using formats such as:
FP32 FP16 BF16 FP8 INT8 INT4
Quantisation reduces numerical precision to reduce memory and potentially improve inference efficiency.
For illustration, a 70-billion-parameter model using FP16 weights requires roughly:
70B × 2 bytes ≈ 140 GB
for the raw model weights alone.
At 8-bit precision, conceptually:
70B × 1 byte ≈ 70 GB
At 4-bit precision:
70B × 0.5 byte ≈ 35 GB
Actual runtime memory requirements differ because inference also needs memory for:
- KV cache
- activations
- runtime buffers
- CUDA graphs
- framework overhead
- batching
- temporary operations
Quantisation can dramatically change the hardware required to serve a model.
But benchmark quality and performance.
A smaller representation is not automatically faster in every hardware/software combination.
KV Cache: One of the Most Important Parts of LLM Serving
During autoregressive generation, the transformer repeatedly attends to previous tokens.
Recomputing everything from scratch for every generated token would be extremely inefficient.
Inference engines therefore maintain a Key-Value cache, commonly called the KV cache.
Conceptually:
Prompt tokens
↓
Transformer
↓
KV Cache
↓
Generate token 1
↓
Update cache
↓
Generate token 2
↓
Update cache
KV-cache consumption generally grows with factors such as:
number of concurrent sequences × sequence length × model architecture × cache precision
This creates an important trade-off.
Large context windows and high concurrency can consume substantial GPU memory even when the model weights fit comfortably.
That is why:
Model fits in GPU memory
does not necessarily mean:
Production workload fits in GPU memory
Batching
Imagine a GPU receives these requests independently:
Request A Request B Request C Request D
If they are processed inefficiently, GPU capacity may remain underutilised.
Batching allows multiple requests to share GPU execution.
A ┐ B ├── Batch → GPU C ┤ D ┘
Modern LLM servers often use techniques such as continuous batching, where requests can enter and leave the active batch dynamically.
This is important because LLM requests have different lengths.
One request may generate:
50 tokens
while another generates:
1,500 tokens
Static batching can waste capacity waiting for long-running sequences.
Continuous batching improves utilisation by scheduling active sequences more dynamically.
Throughput vs Latency
These two metrics should not be confused.
Latency measures how long an individual request takes.
Throughput measures how much work the system can process over time.
You may measure:
requests/second
or:
tokens/second
Important LLM metrics include:
TTFT = Time To First Token TPOT = Time Per Output Token ITL = Inter-Token Latency
An interactive chatbot may prioritise low TTFT.
A batch-processing system may care much more about aggregate throughput.
The optimal infrastructure configuration therefore depends on workload.
Streaming
For interactive applications, streaming can significantly improve perceived performance.
Instead of waiting for:
Complete 1,000-token response
the application starts displaying tokens as they are generated.
Model ↓ Token ↓ Token ↓ Token ↓ Client
Streaming does not necessarily reduce total computation.
But it can significantly improve user experience because users see output earlier.
Concurrency
Suppose your server handles:
1 request
very quickly.
That does not mean it can handle:
100 simultaneous requests
efficiently.
Concurrency affects:
- GPU memory
- KV cache
- batching
- queue depth
- TTFT
- throughput
- tail latency
Benchmark production systems at multiple concurrency levels.
For example:
Concurrency 1 Concurrency 4 Concurrency 8 Concurrency 16 Concurrency 32 Concurrency 64
Then measure:
TTFT tokens/sec requests/sec GPU utilisation GPU memory queue time p95 latency p99 latency
Optimising only single-request latency can lead to poor production economics.
GPU Utilisation
Imagine paying for a GPU instance while seeing:
GPU utilisation: 22%
for most of the day.
You may have an infrastructure utilisation problem.
Potential causes include:
- insufficient batching
- low traffic
- CPU preprocessing bottlenecks
- slow tokenisation
- network latency
- inefficient serving engine
- excessive synchronisation
- model too large for the workload
- poor request scheduling
- storage bottlenecks
The objective is not to force the GPU to 100% utilisation constantly.
Instead, determine whether the infrastructure is delivering acceptable latency and throughput for the cost of the hardware.
CPU and RAM Still Matter
LLM hosting discussions often focus exclusively on GPUs.
But the complete pipeline may include:
API Gateway
↓
Authentication
↓
Request validation
↓
Tokenizer
↓
Retrieval
↓
Reranking
↓
Prompt construction
↓
GPU inference
↓
Post-processing
↓
Response
The GPU is only one component.
A slow CPU, insufficient RAM, inefficient retrieval service, or overloaded vector database can leave expensive GPU hardware waiting.
Optimise the whole inference path.
Use an Efficient Serving Engine
Loading a model using a basic Python script is useful for experimentation.
Production serving is different.
Popular inference engines include:
- vLLM
- Hugging Face Text Generation Inference
- TensorRT-LLM
- SGLang
- llama.cpp for suitable workloads
Production engines can provide capabilities such as:
continuous batching KV-cache management prefix caching quantisation tensor parallelism streaming request scheduling OpenAI-compatible APIs metrics
The serving engine can materially affect how much useful work you obtain from the same GPU.
Prefix Caching
Consider an application where every request starts with the same large system prompt:
System prompt: 3,000 tokens User prompt: 200 tokens
If supported by the inference stack, prefix caching can allow repeated prompt prefixes to reuse previously computed state rather than repeatedly processing the same prefix.
This can be especially valuable for workloads with:
- long shared system prompts
- common instruction prefixes
- repeated document prefixes
- multi-turn conversations
Caching effectiveness depends heavily on prompt structure.
For example, keeping stable content first:
Stable system instructions Stable tools Variable user content
is generally more cache-friendly than introducing changing information near the beginning of the prompt.
Prompt Design Is Infrastructure Optimisation
Consider this instruction:
You are an AI assistant responsible for carefully analysing the following log information. You should examine each log line in detail and determine whether there are potential security issues...
versus:
Analyse the logs for security risks. Return JSON: severity, finding, evidence.
If both produce equivalent quality for your model, the second version is cheaper to process and easier to maintain.
At large scale, reducing prompt size by even a few hundred tokens can matter.
Prompt engineering is therefore not just about answer quality.
It can affect:
latency memory throughput GPU utilisation cloud cost
Avoid Sending Images When You Do Not Need Them
Multimodal models introduce another source of compute usage.
Suppose an application sends:
7 images
with every request.
Ask:
Does the model actually need all seven?
A better architecture might first perform:
Image filtering
↓
Deduplication
↓
Resize/compression
↓
Select relevant images
↓
Vision model
If seven images contain nearly identical information, sending all seven wastes inference capacity.
For document-processing pipelines, consider whether OCR or a specialised vision model can extract relevant information before invoking the larger multimodal model.
Do Not Use an LLM for Everything
Suppose you need to detect whether a string is an IPv4 address.
You could ask a 14B model:
Is 192.168.1.25 an IP address?
But this may be solved with ordinary code.
import ipaddress
ipaddress.ip_address("192.168.1.25")
LLMs should be used where probabilistic language or reasoning provides value.
Use conventional software for:
validation exact arithmetic regex matching database lookup sorting simple transformations deterministic rules
One of the easiest ways to reduce LLM infrastructure cost is simply not invoking the model when software can solve the task more efficiently.
Cache Responses
Suppose users repeatedly ask:
How do I reset my password?
If the context and answer are effectively identical, repeatedly running GPU inference may be unnecessary.
A cache layer can look like:
Request ↓ Cache lookup ↓ HIT ─────→ Response MISS ↓ LLM ↓ Cache ↓ Response
Caching strategies may include:
exact prompt caching semantic caching prefix caching application-level caching retrieval caching
Each solves a different problem.
Autoscaling: Avoid Paying for Idle GPUs
Suppose your traffic pattern looks like:
09:00 High 12:00 High 17:00 Medium 23:00 Low 03:00 Almost zero
Running the same GPU capacity 24/7 may be wasteful.
Depending on latency requirements, infrastructure can scale according to:
request queue active sequences token throughput GPU utilisation KV-cache pressure request rate
But GPU autoscaling has complications.
Model loading may take significant time.
Cold starts can be expensive.
Therefore, a production architecture might maintain:
Minimum warm capacity
+
Autoscaled workers
rather than scaling everything to zero.
Model Routing Can Reduce Cloud Cost Dramatically
Consider this architecture:
┌→ Small Model
Request → Router ├→ Medium Model
└→ Large Model
Suppose 10,000 daily requests consist of:
6,000 simple 3,000 medium 1,000 complex
Sending all 10,000 requests to the largest model means paying large-model infrastructure costs for simple extraction and classification tasks.
Instead:
Simple → 7B/8B model Medium → 14B/32B model Complex → larger reasoning model
Routing can be based on:
- request type
- prompt length
- confidence
- user tier
- latency requirements
- task complexity
- previous model failure
- expected output length
The routing layer itself can use rules, classifiers, or a smaller model.
Design a Token Budget
Every application should have a token-budget strategy.
For example:
Maximum context: 16,000 System prompt: 1,000 Conversation: 3,000 Retrieved context: 6,000 User input: 1,000 Output reserve: 2,000 Safety buffer: 3,000
Before calling the model:
if estimated_tokens > context_budget:
trim_or_summarise()
Do not wait for the inference server to reject requests because the context exceeded the model limit.
Context management belongs in the application layer.
Measure Cost Per Useful Output
GPU utilisation alone is not enough.
Imagine two configurations.
Configuration A
GPU utilisation: 90% Throughput: 2,000 tokens/sec Error rate: 8%
Configuration B
GPU utilisation: 75% Throughput: 1,700 tokens/sec Error rate: 0.5%
Configuration A appears more efficient until failed or low-quality outputs are considered.
Useful metrics may include:
cost/request cost/1M tokens cost/successful request tokens/request TTFT tokens/sec queue time GPU memory utilisation cache-hit ratio model-routing distribution quality score
The goal is not maximum hardware utilisation.
The goal is maximum useful output per unit of infrastructure cost while meeting quality and latency targets.
A More Efficient Production Architecture
Instead of:
Application
↓
70B Model
↓
GPU
consider:
┌───────────────┐
│ API Gateway │
└───────┬───────┘
↓
┌───────────────┐
│ Request │
│ Classification│
└───────┬───────┘
↓
┌───────────────┐
│ Cache Check │
└───────┬───────┘
↓
┌───────────────┐
│ Context │
│ Builder │
└───────┬───────┘
↓
┌───────────────┐
│ Model Router │
└───┬────┬─────┘
│ │
┌─────┘ └─────┐
↓ ↓
Small Model Large Model
│ │
└───────┬────────┘
↓
Inference Engine
↓
GPU Pool
↓
Observability
Now the platform can make decisions based on workload instead of blindly forwarding every request to one expensive model.
Example: Optimising an AI Log Analysis Platform
Imagine a cybersecurity application analysing server logs.
A naïve architecture sends every log line directly to a large LLM.
Nginx Logs
Apache Logs
SSH Logs
System Logs
↓
Large LLM
This is expensive and unnecessary.
A better pipeline could be:
Logs ↓ Parser ↓ Deduplication ↓ Rule Engine ↓ Anomaly Detection ↓ Relevant Events ↓ Small LLM ↓ Complex Cases ↓ Large LLM
Suppose a server produces:
1,000,000 log lines/day
but filtering identifies only:
8,000 relevant events
and grouping/deduplication reduces those to:
600 incidents
The LLM may only need to analyse those 600 incident groups.
That is a fundamentally different infrastructure requirement from sending one million log records into an LLM.
Benchmark Before Choosing Infrastructure
Do not select GPU infrastructure because:
This GPU has more VRAM.
Build a representative benchmark.
For example:
Model: 14B Quantisation: FP8 Prompt lengths: 512 2K 8K 16K Output lengths: 100 500 1K Concurrency: 1 8 32 64
Measure:
TTFT TPOT tokens/sec requests/sec GPU memory GPU utilisation power consumption where relevant error rate quality
Then calculate infrastructure economics.
Cost per hour ÷ successful requests per hour = infrastructure cost per successful request
This gives you a meaningful comparison between hardware configurations.
Optimise the Model and Infrastructure Together
LLM efficiency has several layers.
Application
↓
Prompt
↓
Context
↓
Model
↓
Inference Engine
↓
GPU
↓
Cloud Infrastructure
Optimising only one layer is rarely enough.
A shorter prompt cannot fix an inference server with poor scheduling.
A faster GPU cannot fix an application sending 20,000 irrelevant tokens per request.
Quantisation cannot fix unnecessary model calls.
Autoscaling cannot fix a 70B model being used for simple classification.
Treat AI infrastructure as a complete system.
A Practical Optimisation Checklist
Before scaling your self-hosted LLM environment, examine the following.
Model
Is this model larger than necessary? Can simpler requests use a smaller model? Can the model be quantised safely?
Prompt
Is the system prompt unnecessarily large? Are instructions repeated? Can structured output reduce generation?
Context
Are old conversations always included? Is RAG retrieving too many chunks? Can irrelevant context be removed?
Generation
Is max_tokens too high? Can stop sequences terminate generation earlier? Are responses longer than the application needs?
Serving
Is continuous batching enabled? Is KV-cache utilisation monitored? Can prefix caching help? Is concurrency tuned?
Infrastructure
Is the GPU appropriate for the model? Is GPU memory being used effectively? Are CPU or network bottlenecks starving the GPU? Can workers autoscale?
Architecture
Can deterministic logic replace some LLM calls? Can requests be cached? Can different models handle different task classes?
Observability
Are input/output tokens measured? Is TTFT tracked? Is throughput tracked? Is queue time tracked? Is cost/request calculated?
If you cannot answer these questions from telemetry, infrastructure optimisation becomes guesswork.
Where BitHost Can Help
Deploying an LLM is relatively easy.
Running it efficiently in production is the harder problem.
A team can successfully deploy a model and still waste significant cloud resources because of:
oversized GPU instances poor batching unnecessary context inefficient model selection idle GPU capacity lack of caching bad concurrency settings oversized output limits weak autoscaling missing observability
This is where BitHost can help organisations running private and self-hosted AI infrastructure.
The focus is not simply:
Put an LLM on a GPU.
The objective is to understand how the application, model, inference engine and cloud infrastructure interact, then optimise the complete inference path.
Cloud Cost Optimisation for AI Workloads
Traditional cloud optimisation often focuses on:
CPU RAM storage bandwidth instance uptime
AI infrastructure adds another layer.
You also need to understand:
GPU utilisation GPU memory model residency KV-cache consumption tokens/sec tokens/request batch efficiency queue depth model routing context length cache hit rate
A GPU showing 80% utilisation does not automatically mean the environment is cost-efficient.
The useful question is:
How much production value are we getting from each hour of GPU infrastructure?
BitHost can help evaluate this across the stack.
Right-Sizing AI Infrastructure
Instead of selecting infrastructure first and adapting the application around it, workload characteristics should drive infrastructure design.
For example:
Model size Context requirements Traffic Concurrency Latency SLA Output length Availability requirement
can be used to determine:
GPU class GPU count RAM CPU serving engine quantisation batch configuration scaling strategy
A 7B classification workload and a 70B reasoning workload should not automatically inherit the same deployment architecture.
Reducing Idle GPU Cost
GPU infrastructure is expensive when idle.
BitHost can help design environments around:
traffic patterns warm capacity autoscaling request queues batch workers scheduled workloads model loading strategy
For asynchronous workloads, requests may be grouped and processed efficiently rather than requiring low-latency GPU capacity to remain online continuously.
For interactive applications, minimum warm capacity can preserve responsiveness while additional workers scale with demand.
Optimising the Inference Stack
Cloud cost optimisation should also examine the software layer.
A deployment can be evaluated across:
Model ↓ Quantisation ↓ Inference Engine ↓ Batching ↓ KV Cache ↓ Concurrency ↓ GPU
The goal is to identify the configuration that meets quality and latency requirements at the lowest practical infrastructure cost.
That may involve comparing:
different model sizes different quantisation levels different serving engines different GPU types different batching configurations different context limits
The correct answer depends on the workload.
Benchmarking should decide it.
Building AI Infrastructure That Can Be Measured
Optimisation requires visibility.
A production AI environment should expose metrics such as:
Request volume Input tokens Output tokens Context length TTFT Generation speed Queue time Active requests GPU utilisation GPU memory KV-cache utilisation Cache hit rate Model selected Error rate Infrastructure cost
These metrics allow engineering teams to answer questions such as:
Why did AI infrastructure cost increase this week?
Instead of guessing, telemetry might show:
Average context: 4.2K → 9.8K tokens Average output: 320 → 610 tokens Cache hit rate: 41% → 19% GPU utilisation: 72% → 48%
Now the cost increase has technical explanations that can be investigated.
From "The Model Works" to "The System Is Efficient"
The first milestone in an AI project is often:
The model works.
The production milestone should be:
The model works reliably, at the required latency, under expected concurrency, with measurable infrastructure cost.
Those are very different standards.
Running your own model gives you control.
But control also means responsibility for:
model selection token usage context architecture inference optimisation GPU utilisation scaling monitoring cost
The most efficient AI platforms will not necessarily be the ones using the largest models or the most powerful GPUs.
They will be the ones that understand which model should perform which task, how much context it actually needs, how requests should be scheduled, and how much useful work each unit of compute produces.
