Experimentation in the AI Era: Lessons from the Trenches
Join this event
About this event
Your A/B tests were built for a different world.
GenAI features don't behave like traditional product changes. When model predictions constantly shape user behavior—and user feedback loops simultaneously alter the model—classical testing frameworks break down. SUTVA assumptions quietly fail, treatments spill between user groups, and traditional experimentation becomes a massive bottleneck to shipping.
To enable true AI-era delivery, engineering and data teams need ultra-high experimenting velocity. This session covers how to move past static frameworks and transition to continuous, intelligent optimization.
📌 WHAT YOU'LL LEARN
- ✔ Why traditional experimentation breaks in AI products: The exact structural failure mechanisms (user spillage, SUTVA violations, and rapidly evolving models) that invalidate standard frameworks.
- ✔ How to measure success in AI-powered experiences: Practical frameworks to design robust metrics and accurately track performance, even when treatments and feedback loops change dynamically.
- ✔ What the future of experimentation looks like: How to relieve the shipping bottleneck using multi-armed bandits, AI-assisted experiment design, continuous optimization loops, and autonomous experimentation systems.
🎯 WHO IS THIS FOR?
Data scientists, product managers, ML engineers, growth leaders, and analysts who are shipping AI/GenAI features and need their experimentation practices to keep pace. You should have some familiarity with basic A/B testing concepts. No prior AI experimentation experience is required.
👥 YOUR HOSTS
Mirza Rahim Baig , Founder of Baig Academy, Top 100 AI Leader,
author of 3 books on enterprise AI. 16+ years in AI & Data.
baigacademy.ai
Vishnukant Peddawad , Senior Product & Data Analytics Leader,
12+ years driving growth via CRM, A/B Testing & customer journey
insights. Ex-Idealo, Ex-CME Group.
Want to find AI jobs?
Join thousands of AI professionals finding their next opportunity