Project 06: AI Evaluation Framework

Objective

Build a reusable evaluation framework for prompts, RAG systems, agents, model comparisons, cost, latency, and regression reports.

Architecture

Golden Dataset -> System Under Test -> Evaluators -> Metrics Store -> Regression Report -> Release Decision

Tasks

• Define evaluation schema.
• Create sample datasets for RAG and agents.
• Implement exact-match and rubric-based checks.
• Implement retrieval metrics.
• Implement citation checks.
• Track latency and cost.
• Compare model versions.
• Generate Markdown reports.
• Add pass/fail thresholds.

Deliverables

• Evaluation dataset templates.
• Evaluation runner.
• Metrics summary.
• Regression report.
• Release decision checklist.

Tips

• Make evaluation easy to run locally.
• Keep evaluators deterministic when possible.
• Review sampled failures manually.
• Add every production issue to the test set.