Assessment: AI Evaluation and LLMOps

Explain

• What is a golden dataset?
• What is regression testing for AI?
• Why is answer quality not enough?
• How do you evaluate RAG separately from the generator?
• How do you compare models fairly?

Build

• Design an evaluation pipeline.
• Define release thresholds.
• Define monitoring metrics for an AI API.

Debug

• Quality dropped after a prompt change. What do you inspect?
• Cost doubled after a model change. What do you inspect?
• Latency increased. What do you measure?

Completion Gate

• Can create evaluation datasets.
• Can run regression checks.
• Can make release decisions from metrics.