Data & ML · Pro
LLM Inference Server From Scratch
A GPU inference server that loads open weights and serves them with a KV cache, paged attention, and continuous batching behind an OpenAI-compatible API
You build a production-grade LLM inference server in Python and CUDA that loads Llama weights from disk, manages GPU memory with a paged KV cache, and serves concurrent requests through continuous batching. The server exposes an OpenAI-compatible REST API so any existing client can talk to it without modification. This project forces you to confront the real bottlenecks in transformer inference: memory bandwidth, KV cache fragmentation, and request scheduling. Understanding these mechanics is essential for anyone who works on ML systems at a company that hosts its own models.
What you build
- Loads Llama 2/3 or Mistral weights in fp16 or bfp16 directly from HuggingFace safetensors format
- Implements a paged KV cache that allocates fixed-size memory blocks to avoid fragmentation across variable-length sequences
- Runs continuous batching so new requests join in-flight batches without waiting for the current batch to finish
- Exposes a /v1/completions and /v1/chat/completions endpoint that is wire-compatible with the OpenAI API
- Uses Triton or CUDA kernels for fused attention with causal masking to maximize memory bandwidth utilization
- Tracks per-request token throughput and queue latency and exposes a /metrics endpoint in Prometheus format
What it teaches
- How transformer KV cache memory grows with sequence length and why naive allocation causes fragmentation
- Paged virtual memory techniques applied to GPU VRAM management via block tables
- Continuous batching scheduling and the tradeoff between latency and throughput for heterogeneous request lengths
- Writing custom Triton kernels for fused causal self-attention with non-contiguous memory access patterns
- OpenAI API streaming protocol using server-sent events and chunked transfer encoding
- Profiling GPU kernels with nsight-compute and identifying memory-bound vs compute-bound bottlenecks
How it works
- 1
API Layer
- POST /v1/chat
- SSE stream
- FastAPI
↓ enqueue
- 2
Scheduler
- waiting queue
- running queue
- preempt/swap
↓ batch
- 3
Block Manager
- block table
- paged KV slots
- GPU VRAM
↓ forward
- 4
Attention Kernel
- Triton fused attn
- causal mask
- GQA
↓ logits
- 5
Sampler
- top-p / top-k
- token ids
↓ stream
- 6
Token Stream
- SSE chunks
- finish_reason
Sign in to open the build guide
Free account. Get the step-by-step build and every resource link.
Take it further
- Add speculative decoding with a small draft model to reduce mean token latency by 2 to 3x on long outputs
- Implement tensor parallelism across two GPUs using NCCL all-reduce so you can serve 70B parameter models
- Build a prefix caching layer that reuses KV cache blocks for shared system prompts across requests
