Exercises: AI Evaluation and LLMOps

Concept Drills

• Explain golden datasets.
• Explain regression testing for prompts.
• Compare human and automated evaluation.
• Explain cost, latency, and quality trade-offs.

Build Drills

• Create a golden dataset CSV.
• Build an evaluation runner.
• Add retrieval metrics.
• Add answer correctness scoring.
• Add latency and cost tracking.
• Generate a Markdown regression report.

Failure Drills

• Change a prompt and detect regression.
• Change a model and compare results.
• Add adversarial examples.
• Add ambiguous examples.

Output

• Evaluation dataset.
• Evaluation script.
• Regression report.
• Release decision checklist.