Module 06: AI Evaluation, LLMOps, and MLOps

Goal

Create repeatable evaluation and monitoring systems so AI changes can be accepted or rejected with evidence.

Core Topics

• Golden datasets.
• Regression testing.
• Prompt evaluation.
• RAG evaluation.
• Agent evaluation.
• Human evaluation.
• Automated evaluation.
• Model comparison.
• Cost monitoring.
• Latency monitoring.
• Token monitoring.
• Error tracking.
• Observability.

Evaluation Pipeline

Test Dataset -> AI System -> Evaluation -> Metrics -> Regression Report -> Release or Reject

Study Tasks

• Create a golden dataset for one RAG system.
• Create a golden dataset for one agent system.
• Add automated evaluation scripts.
• Track cost per request.
• Track latency percentiles.
• Track model, prompt, and retrieval version.
• Create release gates based on quality thresholds.
• Write a regression report.

Project

Build projects/06-ai-evaluation-framework/.

Useful Metrics

• Retrieval precision.
• Retrieval recall.
• Answer correctness.
• Faithfulness.
• Citation correctness.
• Task success rate.
• Tool success rate.
• Latency p50, p95, p99.
• Cost per request.
• Error rate.

Tips

• Start with small, high-quality test sets.
• Include negative and adversarial examples.
• Store evaluation failures as future tests.
• Never change prompts without running regression checks.

Completion Checklist

• Can define a golden dataset.
• Can compare model versions.
• Can evaluate RAG and agents separately.
• Can produce a release decision report.
• Can monitor cost, latency, and errors.