RAG evaluation engineering topics
12 short topics in AI Evaluation LLMOps and Governance. Each one is a single focused exercise with a clear piece of evidence at the end.
12 topics, one exercise each.
Work through them in order, or jump to the one you need.
These are draft topic outlines (draft; instructor review pending). Full lessons follow after review; ask Ajay if you need a topic sooner.
- 01
Evaluation dataset organization
Investigate evaluation dataset organization using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 02
Input and expected-output fixtures
Investigate input and expected-output fixtures using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 03
Metric and rubric definitions
Investigate metric and rubric definitions using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 04
Test case grouping
Investigate test case grouping using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 05
Baseline run capture
Investigate baseline run capture using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 06
Model or prompt comparisons
Investigate model or prompt comparisons using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 07
Human feedback integration
Investigate human feedback integration using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 08
Retrieval-specific evaluation
Investigate retrieval-specific evaluation using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 09
Adversarial and failure cases
Investigate adversarial and failure cases using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 10
Evaluation cost controls
Investigate evaluation cost controls using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 11
Result reporting
Investigate result reporting using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course → - 12
Release regression thresholds
Investigate release regression thresholds using a repeatable RAG evaluation engineering experiment; compare results with a baseline and record limitations. Keep the exercise bounded and record the behavior supported by the chosen version.
Open micro-course →
Learn the same ideas beyond RAG evaluation engineering.
These courses share topics with this one, so you can compare how each language or tool handles the same concept.
More in AI evaluation observability and protocols.
You have the curiosity.
Let’s turn it into capability.
Start with a practical workshop or bring a specific question to Ajay.