Why Your Local LLM Server Is Slow

You loaded a model that fits in your GPU, the first reply feels snappy, and then three people connect and everything crawls or the server dies with an out-of-memory error. If your local LLM server is slow, the model weights are rarely the villain. The real memory hog is the KV cache, and the real speed limit is memory bandwidth. This post shows how to size the cache with a few lines of arithmetic, why batching changes everything, and which knobs to turn in vLLM and llama.cpp.

What the KV cache actually is

A transformer generates one token at a time. To produce token N it needs the attention keys and values of every earlier token. Recomputing them each step would be absurdly wasteful, so the server stores them: that store is the KV cache. It grows by one entry per layer for every token of every active conversation, and it lives in GPU memory next to the weights.

The size per token is simple: 2 (keys and values) x layers x KV heads x head dimension x bytes per element. Grouped-query attention matters a lot here, because models share a small number of KV heads across many query heads. The published Llama 3 8B config has 32 layers, 32 attention heads and 8 KV heads, and its hidden size of 4096 divided by 32 heads gives a head dimension of 128.

def kv_bytes_per_token(layers, kv_heads, head_dim, dtype_bytes=2):
    # 2 = one key vector and one value vector per layer
    return 2 * layers * kv_heads * head_dim * dtype_bytes

# Llama 3 8B: 32 layers, 8 KV heads, head dim 4096 / 32 = 128, fp16
per_tok = kv_bytes_per_token(32, 8, 128)
print(per_tok, "bytes/token =", per_tok // 1024, "KiB/token")

ctx = 8192
per_seq = per_tok * ctx
print("one", ctx, "token sequence:", per_seq / 2**30, "GiB")

vram_gib = 24
weights_gib = 15      # about 8B params x 2 bytes in fp16
reserve_gib = 2       # activations, CUDA context, fragmentation
free = vram_gib - weights_gib - reserve_gib
print("sequences that fit:", int(free * 2**30 // per_seq))

That prints 131072 bytes (128 KiB) per token, 1 GiB for one 8K-token conversation, and 7 concurrent sequences on a 24 GiB card. Not 7 users with short chats, 7 users each using the whole context window. Double the context and you halve the concurrency. Quantise the cache to 8 bits and you double it back.

Why decoding is bandwidth-bound

Prompt processing (prefill) is compute-heavy: thousands of tokens go through the model in one big matrix multiply. Generation (decode) is different. Each new token needs a pass through every weight, but only one token's worth of arithmetic. The GPU spends its time waiting for weights to arrive from memory rather than computing.

A rough ceiling for one stream is memory bandwidth divided by bytes read per token. A 16 GB fp16 model on a card with 1 TB/s of bandwidth cannot decode much faster than about 60 tokens per second, whatever its compute rating says. Quantising the weights to 8 or 4 bits raises that ceiling because fewer bytes cross the bus. This is a back-of-envelope bound, not a benchmark, but it explains why a newer GPU with the same memory bandwidth feels no faster at chat.

Batching is the free lunch

Here is the trick that makes servers economical. If four requests decode together, the weights are read from memory once and used for four tokens. The bandwidth cost is shared, so total throughput climbs almost linearly until compute or the KV cache becomes the limit. Per-user speed stays similar while the box serves many people.

Static batching, waiting for a whole batch to finish, wastes time when one request is long. Continuous batching schedules at the token level: finished sequences leave and new ones join between steps. The vLLM paper adds the memory side with PagedAttention, which stores the cache in fixed-size blocks like virtual-memory pages, removing external fragmentation and allowing sharing between requests. The authors report 2-4x higher throughput than the state-of-the-art systems they compared against (FasterTransformer and Orca), with larger gains for longer sequences and bigger models.

Tuning vLLM

vLLM carves the GPU into weights, activations and a pre-allocated KV block pool. These engine arguments decide the split and the scheduling limits.

vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 16 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching

# the startup log reports how many KV cache tokens fit;
# compare it with the number your kv_budget.py predicted

• gpu-memory-utilization: the fraction of GPU memory vLLM may use for the model and KV cache. Raise it for more cache, lower it if other processes share the card.
• max-model-len: the longest context you will serve. Cutting it from the model maximum to what you really need is the cheapest way to fit more sequences.
• max-num-seqs: the cap on sequences processed together. Too low wastes the batching gain, too high just queues behind a full cache.
• kv-cache-dtype fp8: stores the cache in 8 bits for roughly twice the capacity, at some accuracy risk you should test on your own prompts.
• enable-prefix-caching: reuses cache for shared prefixes, a big win when every request starts with the same long system prompt.

Tuning llama.cpp

llama-server has the same ideas with different names. The gotcha: the context size is divided between parallel slots. Running with a 16384 context and 4 slots gives each conversation 4096 tokens, not 16384.

llama-server -m ./model-Q4_K_M.gguf \
  --ctx-size 32768 \
  --parallel 4 \
  --n-gpu-layers 99 \
  --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0
# 32768 / 4 slots = 8192 tokens per conversation

Measure instead of guessing

Both servers speak an OpenAI-compatible API, so a short script can tell you whether your tuning helped. Watch two numbers: time to first token (prefill plus queueing) and tokens per second per stream (decode).

N=8
for i in $(seq $N); do
  ( curl -s -o /dev/null -w "req $i: ttfb %{time_starttransfer}s total %{time_total}s\n" \
      http://localhost:8000/v1/chat/completions \
      -H 'Content-Type: application/json' \
      -d '{"model":"meta-llama/Meta-Llama-3-8B-Instruct","max_tokens":256,
           "messages":[{"role":"user","content":"Explain the KV cache in detail."}]}' ) &
done
wait

nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

The takeaway

Before blaming the GPU, do the sums. Weights decide whether the model loads, the KV cache decides how many people can talk to it, and memory bandwidth decides how fast each of them reads. Shrink the context to what you need, quantise the cache if your quality tests allow it, keep the batch full, and measure time to first token and per-stream speed after every change.