Running Local LLM Infrastructure

Calling a hosted LLM API is a single HTTP request. Running the model yourself means owning everything that request was hiding: GPU memory budgeting, batching, quantisation tradeoffs, and the process that keeps a model server alive under load.

The stack

Memory is the constraint that decides everything

A model's parameter count sets a hard floor on VRAM before you've served a single token — and that floor moves with precision. A 7B-parameter model at FP16 needs roughly 14GB just for weights, before KV-cache and activation memory for concurrent requests are added on top. Quantisation (INT8, INT4) trades some quality for headroom; batching trades latency for throughput.

# Serve a local model with an OpenAI-compatible API
python -m vllm.entrypoints.openai.api_server \
  --model /models/adg-local-7b \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

What you gain, and what you now own

• Data never leaves infrastructure you control — the tradeoff for that is you now own uptime, scaling and patching.
• Latency becomes a function of your hardware and batching strategy, not a third party's rate limits.
• Cost shifts from per-token billing to fixed GPU capacity — good at high, steady utilisation; wasteful at low utilisation.

None of this is a reason to avoid hosted APIs by default — it's the set of tradeoffs that matters once an application's data, latency or cost requirements make local inference the right engineering answer.