AI & Agents · Pro
Agent Evaluation & Observability Harness
A test rig that replays agent trajectories against graded task suites, scores them with an LLM judge, and tracks regressions across model and prompt versions
You build a structured evaluation harness that captures agent tool-call trajectories as OpenTelemetry spans, replays them against versioned task datasets, and scores each run with an LLM-as-judge rubric. The system stores per-run metrics in a SQLite or Postgres database so you can diff two prompt versions or model swaps and catch regressions before they reach production. It wires into CI so a failing eval blocks a merge, giving agent development the same safety net that unit tests give ordinary software.
What you build
- Captures full agent trajectories (tool calls, intermediate outputs, latency) as structured OpenTelemetry spans during any live or replayed run
- Stores versioned task datasets (input prompt, expected outcome, grading criteria) in a flat JSON or YAML corpus that is easy to extend
- Replays stored trajectories against a target agent without re-spending real API budget, using recorded tool responses as mocks
- Scores each trajectory with a configurable LLM-as-judge that returns a 0-1 score and a chain-of-thought explanation for each graded dimension
- Aggregates per-run metrics (pass rate, mean score, token cost, p95 latency) into a SQLite table and renders a diff table when comparing two runs
- Exposes a pytest plugin that fails the suite when aggregate score drops below a configurable threshold, enabling CI gating
What it teaches
- LLM-as-judge evaluation design: rubric authoring, grading dimensions, and how to reduce judge variance with structured output schemas
- OpenTelemetry instrumentation for non-HTTP workloads: spans, attributes, and exporters applied to agent tool-call pipelines
- Trajectory replay and deterministic testing: mocking external tool responses to get reproducible agent runs without live API calls
- Regression tracking with structured metrics: schema design for storing eval runs and writing SQL diffs to compare two versions
- CI gating for non-deterministic systems: choosing thresholds, handling flakiness, and surfacing actionable failure messages in pull requests
How it works
- 1
Instrument Agent
- OpenTelemetry spans
- tool call attributes
- latency + tokens
↓ record
- 2
Task Corpus
- YAML fixtures
- rubric strings
- expected tools
↓ load
- 3
Replay Engine
- mock tool responses
- deterministic run
- new span tree
↓ score
- 4
LLM Judge
- rubric + trajectories
- structured score
- CoT explanation
↓ store
- 5
Metrics DB
- SQLite run table
- score diff view
↓ gate
- 6
CI Assert
- pytest threshold
- PR block or pass
Sign in to open the build guide
Free account. Get the step-by-step build and every resource link.
Take it further
- Add a web dashboard (FastAPI + plain HTML) that renders score trends across git commits so you can visually spot the commit that caused a regression
- Implement a mutation tester that automatically perturbs prompt wording or tool descriptions and measures how much the judge score shifts, quantifying prompt brittleness
- Support multi-turn conversation trajectories and add a coherence dimension to the judge rubric that checks whether the agent correctly carries context across turns
