Track 15: Evaluation, Observability & Safety

The track that makes everything before it production-grade rather than a demo. Evaluating a system whose output is free-form text is genuinely hard and needs its own methods — eval datasets, LLM-as-judge, regression testing prompts — plus hallucination detection, guardrails, agent tracing, and keeping token spend from quietly exploding.

Modules

1. Why GenAI Evaluation Is Hard
2. Building an Eval Dataset
3. Deterministic Evals and Assertions
4. LLM-as-Judge
5. Human Evaluation and Annotation
6. Benchmark Suites and Their Limits
7. Regression Testing Prompts
8. Hallucination: Causes
9. Hallucination: Detection Techniques
10. Grounding and Citation Verification
11. Guardrails: Input Filtering
12. Guardrails: Output Validation
13. PII and Data Leakage
14. Tracing Agent Execution
15. Metrics, Dashboards and Alerting
16. Cost Monitoring and Optimization
17. Red-Teaming and Adversarial Testing
18. Capstone Project