AI Agent Evals: Test Before You Trust
Measure whether an agent actually does the job before you trust it: test sets from real tasks, graders that use rules, people and AI judges, traces that show where it goes wrong, and release gates that keep quality up.
- Live class, online
- One 2 hour hands on live class
- Advanced level
- Certificate you can verify
- Recordings included
What you will be able to do
- Write test cases from real tasks and failures
- Grade with rules, people and AI judges
- Read traces to find where an agent goes wrong
- Set release gates and track quality over time
Hands on project
Build an eval suite and a release gate for a sample agent.
Curriculum
4 modules, 12 topics
01Part 1: Why evals (20 minutes)3 topics
- Why demos lie
- What to measure
- Offline and online evaluation
02Part 2: Test sets (35 minutes)3 topics
- Writing cases from real tasks and failures
- Covering edge cases
- Keeping the set fresh
03Part 3: Grading (35 minutes)3 topics
- Rule based checks
- Human review
- AI judges and their limits
04Part 4: Release gates (30 minutes)3 topics
- Reading traces to find failures
- Gates before every release
- Tracking quality over time
Tools you will use
- Python
- An eval framework
- Tracing tools
Who this course is for
- Engineers shipping agents to users
- QA and evaluation specialists
- AI product and delivery leads
What to know first
- Built or used an agent
- Basic Python
Upcoming schedule
Starts
Live sessions
Wednesday, 7:00 to 9:00 PM ET
Format
One 2 hour hands on live class
CA$99
A certificate employers can check
Every certificate has a code anyone can verify on getintech.ca. It also shows on your talent profile, so verified employers see it as a proven skill.
Taught live by practitioners who do this work. Meet the instructors.
Jobs that ask for these skills
Real AI and tech jobs in Canada, from verified employers and employers' own careers pages.
Questions about this course
Do I need an agent already?
No. We provide a sample agent to evaluate.
Is the class live or recorded?
Live, on a video call in Eastern time, with time for your questions. Every session is also recorded, and you keep the recordings for 30 days.
Do I get a certificate?
Yes. Finish the class and its exercises and you get a Get In Tech certificate with a code employers can check online.
Can my employer pay?
Yes. We invoice companies directly, and for three or more people from one team we can run a private session.
Are prices in Canadian dollars?
Yes. Prices are in CAD before tax. Sales tax is added at checkout based on your billing address.
Can I get a refund?
All sales are final, with no refunds. The one exception: if a PMP or BA cohort does not reach 10 students, you can move to the next cohort or get a full refund.