Free AI Evals Course : https://github.com/aishwaryanr/awesom...

🚀Get 3 months of Arize AX Pro Free: https://app.arize.com/auth/join?utm_s...



Building an agent is the easy part! Anyone can get one working in a day. The hard part is making it hold up in production, and the number one skill for that is AI evals. This applies whether you're an engineer, a PM, or a founder, and it's also one of the most misunderstood topics in AI interviews.

I've spent about a decade in AI, previously as an AI scientist and researcher at AWS, and now building an AI native startup in San Francisco, LevelUp Labs, where we help companies and teams go AI native through education, engineering, and strategy.

This episode breaks down what AI evals actually are and why they look so different from traditional software testing, the difference between model evals and product evals, the three ways to build evals and when to use each, and a full walkthrough of building your first eval from scratch using a fictional customer support agent (QuickCart) — including a real escalation check built with an LLM judge in Arize.

The core idea: evals aren't a build-once-and-forget system. You build them against simulated behavior before launch, then keep calibrating them against real user behavior once you're live — that loop never really stops.


By the end of this video you'll know:

00:00 – Why AI products can't be tested like normal software
04:55 – Model evals vs. product evals (and why benchmarks like MMLU Pro aren't enough)
07:56 – The three ways to build evals: human, code-based, and LLM judges
10:53 – Live walkthrough: building your first eval in Arize
13:31 – Finding the real problem through error analysis
15:30 – Writing rubrics and running your first LLM judge

My LinkedIn: LinkedIn: areganti
Website: https://levelup-labs.ai/
Free Newsletter: https://thenuancedperspective.substac...
Free Courses: https://levelup-labs.ai/education
Cohort Courses: https://maven.com/aishwarya-kiriti
Instagram: Instagram: aish_reganti