AI Evaluation LLMOps and Governance topics
10 short topics in AI Evaluation LLMOps and Governance. Each one is a single focused exercise with a clear piece of evidence at the end.
10 topics, one exercise each.
Work through them in order, or jump to the one you need.
These are draft topic outlines (draft; instructor review pending). Full lessons follow after review; ask Ajay if you need a topic sooner.
- 01
Evaluation datasets and rubrics
Run a controlled example of Evaluation datasets and rubrics for AI evaluation harness and release dashboard; record assumptions, baseline and measured results.
Open micro-course → - 02
Human and automated assessment
Run a controlled example of Human and automated assessment for AI evaluation harness and release dashboard; record assumptions, baseline and measured results.
Open micro-course → - 03
Prompt and model versioning
Run a controlled example of Prompt and model versioning for AI evaluation harness and release dashboard; record assumptions, baseline and measured results.
Open micro-course → - 04
AI application tracing
Run a controlled example of AI application tracing for AI evaluation harness and release dashboard; record assumptions, baseline and measured results.
Open micro-course → - 05
Provider routing and fallback
Run a controlled example of Provider routing and fallback for AI evaluation harness and release dashboard; record assumptions, baseline and measured results.
Open micro-course → - 06
AI cost budgets and usage metering
Run a controlled example of AI cost budgets and usage metering for AI evaluation harness and release dashboard; record assumptions, baseline and measured results.
Open micro-course → - 07
Prompt-injection resilience
Run a controlled example of Prompt-injection resilience for AI evaluation harness and release dashboard; record assumptions, baseline and measured results.
Open micro-course → - 08
Sensitive-data handling in AI workflows
Run a controlled example of Sensitive-data handling in AI workflows for AI evaluation harness and release dashboard; record assumptions, baseline and measured results.
Open micro-course → - 09
AI release regression gates
Run a controlled example of AI release regression gates for AI evaluation harness and release dashboard; record assumptions, baseline and measured results.
Open micro-course → - 10
AI governance and risk records
Run a controlled example of AI governance and risk records for AI evaluation harness and release dashboard; record assumptions, baseline and measured results.
Open micro-course →
More in Core IT concepts.
You have the curiosity.
Let’s turn it into capability.
Start with a practical workshop or bring a specific question to Ajay.