Data & ML · Advanced

Fine-Tune and Serve a Domain LLM

Curate a domain dataset, fine-tune an open model with QLoRA, quantize it, serve it behind a fast inference endpoint, and prove the lift with an LLM-judge eval suite.

Advanced12-20 hoursPythonAI

You fine-tune an open-weight language model (such as Mistral 7B or Llama 3 8B) on a curated domain-specific dataset using QLoRA and Unsloth, then quantize the adapter-merged model to 4-bit and serve it via a vLLM OpenAI-compatible endpoint. The project closes the loop with an automated LLM-judge eval suite that scores the fine-tuned model against the base model on held-out domain prompts, giving you a reproducible number that proves the adaptation worked. This is the full MLOps cycle: data curation, parameter-efficient training, quantization, serving, and evaluation.

What you build

  • Curates and cleans a domain-specific instruction dataset in ShareGPT or Alpaca format using deduplication and quality filters
  • Fine-tunes an open-weight base model with QLoRA via Unsloth, reducing VRAM requirements to under 16 GB
  • Merges LoRA adapters into the base model and applies 4-bit GPTQ or AWQ quantization with bitsandbytes
  • Serves the quantized model behind a vLLM OpenAI-compatible REST endpoint with batched generation and streaming
  • Runs an LLM-judge eval suite that scores fine-tuned vs base model responses on domain holdout prompts
  • Tracks training loss, eval metrics, and judge scores in a Weights and Biases or local MLflow run
  • Publishes the adapter or merged model to the Hugging Face Hub with a model card

What it teaches

  • Parameter-efficient fine-tuning with LoRA and QLoRA, including rank selection and target module choice
  • 4-bit quantization trade-offs between GPTQ, AWQ, and bitsandbytes NF4 and when each is appropriate
  • High-throughput LLM serving with vLLM: continuous batching, paged attention, and the OpenAI-compatible API surface
  • Automated LLM-as-judge evaluation design: prompt construction, win-rate metrics, and guarding against position bias
  • Hugging Face Hub model card authoring and adapter vs merged-model publishing workflows
  • End-to-end MLOps instrumentation: experiment tracking, reproducible training configs, and holdout-set discipline

How it works

  1. 1

    Curate Dataset

    • domain docs
    • ShareGPT format
    • train/holdout split

    ↓ feed

  2. 2

    QLoRA Fine-tune

    • Unsloth + PEFT
    • 4-bit base model
    • W&B logging

    ↓ adapters

  3. 3

    Merge + Quantize

    • merge_and_unload
    • GPTQ / AWQ
    • safetensors save

    ↓ model

  4. 4

    Serve with vLLM

    • OpenAI-compat API
    • continuous batching

    ↓ responses

  5. 5

    LLM Judge Eval

    • holdout prompts
    • base vs fine-tuned
    • win-rate score
fig. 01 — data flows from raw domain text through fine-tuning and quantization into a served endpoint, with an llm judge closing the eval loop.

Sign in to open the build guide

Free account. Get the step-by-step build and every resource link.

Sign in to continue

Take it further

  • Add a retrieval-augmented generation (RAG) layer on top of the fine-tuned model using FAISS and sentence-transformers to handle long-tail domain facts without re-training
  • Implement continuous batched online serving with a simple FastAPI router that load-balances across two vLLM workers and exposes a /health endpoint for uptime monitoring
  • Build an active-learning feedback loop where low-confidence or judge-flagged outputs are queued for human review and folded into the next fine-tuning run automatically

More like this

All projects

Data & ML · Advanced

Real-Time Fraud Detection Pipeline

Data & ML · Advanced

Two-Tower Recommender Engine

Data & ML · Beginner

DataLab: Clean and Chart a Real Dataset