Data & ML · Pro
Train a Small Language Model From Scratch
Pretrain a ~100M-parameter transformer on your own tokenized corpus, then instruction-tune and align it with DPO until it follows chat prompts
You implement a GPT-style decoder-only transformer from scratch in PyTorch, tokenize a raw text corpus with a BPE tokenizer, and run a full pretraining loop tracked by Weights and Biases. Once the base model converges, you apply supervised instruction fine-tuning followed by Direct Preference Optimization so the model learns to follow chat-style prompts rather than just predict next tokens. The project gives you end-to-end ownership of every stage that commercial LLM teams run, at a scale that fits on a single consumer GPU or a cheap cloud instance.
What you build
- Trains a ~100M-parameter transformer (configurable depth, heads, and width) on any plain-text corpus
- BPE tokenizer trained in-house via tiktoken or a hand-rolled encoder, producing a corpus-specific vocabulary
- FlashAttention-backed attention layers for memory-efficient O(N) GPU utilization
- Cosine LR schedule with warmup, gradient clipping, and mixed-precision (bf16/fp16) training
- Supervised instruction fine-tuning on a prompt/completion dataset to seed chat behavior
- DPO alignment stage that uses ranked preference pairs to push chosen completions above rejected ones without a reward model
- W&B experiment tracking: loss curves, perplexity, sample generations, and GPU utilization logged automatically
What it teaches
- Transformer architecture internals: attention, residual connections, RoPE or learned positional embeddings, and layer normalization placement
- Tokenization pipelines: BPE training, encoding/decoding, and why vocabulary size affects both memory and downstream quality
- Distributed and mixed-precision training: gradient accumulation, bf16 autocasting, and gradient checkpointing tradeoffs
- LR scheduling and optimization hygiene: warmup, cosine decay, gradient clipping, and weight decay for stable large-model training
- RLHF-free alignment via DPO: how the Bradley-Terry preference model translates into a classification loss over log-prob ratios
- Experiment tracking discipline: logging perplexity, sample quality, and hardware metrics to W&B to make training runs reproducible and comparable
How it works
- 1
Raw Corpus
- Plain text files
- Wikipedia / TinyStories
↓ tokenize
- 2
BPE Tokenizer
- tiktoken vocab
- Binary .bin shards
↓ pretrain
- 3
Transformer Pretraining
- ~100M params
- FlashAttention
- W&B logging
↓ SFT
- 4
Instruction Fine-Tuning
- Prompt/completion pairs
- Prompt tokens masked
↓ DPO
- 5
DPO Alignment
- Chosen / rejected pairs
- Frozen reference model
↓ generate
- 6
Chat-Capable Model
- Follows prompts
- GGUF / inference ready
Sign in to open the build guide
Free account. Get the step-by-step build and every resource link.
Take it further
- Add speculative decoding with a tiny draft model to cut inference latency by 2-3x during generation
- Implement model merging (SLERP or task arithmetic) to combine your DPO-aligned checkpoint with a separately fine-tuned coding checkpoint without retraining
- Export the final checkpoint to GGUF format and run it locally via llama.cpp to benchmark quantized (Q4_K_M) throughput on CPU
