A/B-Test a Recommender Improvement Without Breaking Trust
Overview
What this challenge is about.
You’ll design an A/B test for a consumer AI recommender, define metrics and stopping rules, and pre-register the analysis — then get a verifiable certificate.
The scenario
The startup (around 70 staff, around USD 11M annual recurring revenue) has been burned twice by recommender changes that helped clicks but hurt save rate, and now requires written test plans before any production model swap.
The Brief
What you'll do, and what you'll demonstrate.
Design a trustworthy A/B test for a recommender upgrade with explicit guardrails and a pre-registered analysis plan.
Earning criteria — what you'll demonstrate
- Design a live A/B test with appropriate guardrails for ML deployments
- Compute required sample size for a target minimum detectable effect
- Pre-register an analysis plan to prevent post-hoc metric-hunting
- Translate offline ML metrics into product-grade success/guardrail metrics
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Machine Learning in Practice
Master · Machine Learning
Strong alignment
This challenge maps to Machine Learning in Practice at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Experiment Design
Apply experiment design to solve real industry problems and demonstrate production-level capability.
- Ab Testing
Apply ab testing to solve real industry problems and demonstrate production-level capability.
- Metric Design
Apply metric design to solve real industry problems and demonstrate production-level capability.
- Statistical Analysis
Apply statistical analysis to solve real industry problems and demonstrate production-level capability.
- Guardrail Metrics
Apply guardrail metrics to solve real industry problems and demonstrate production-level capability.
- Ml Problem Scoping
Apply ml problem scoping to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
AI Product Manager
Writing trustworthy test plans with guardrail metrics is the central AI-PM craft and is graded heavily in interviews at consumer-AI startups.
This challenge sharpens
- experiment-design
- metric-design
- guardrail-metrics
Data Scientist
Pre-registered analysis plans and sample-size discipline are exactly what hiring managers look for in data-scientist candidates joining experimentation platforms.
This challenge sharpens
- ab-testing
- statistical-analysis
- experiment-design
Applied AI Scientist
Bridging offline ML metrics to live product metrics with rigour is a defining applied-AI-scientist skill.
This challenge sharpens
- metric-design
- ml-problem-scoping
- experiment-design