Analysis
A/B Testing for a 40-Person SaaS Scale-up Moving to Enterprise
Overview
What this challenge is about.
You are a data analyst at TaskFlow. You are given the raw A/B test data (visitor logs, conversions, and downstream sales data). Your task is to perform a rigorous analysis: check for sample ratio mismatch, compute Bayesian and frequentist significance, segment by traffic source, and estimate the long-term revenue impact. Success means providing a clear go/no-go recommendation with quantified confidence, and a plan for follow-up tests if needed. Deliver a report that the VP of Product can use to make a decision.
The Brief
What you'll do, and what you'll demonstrate.
Determine whether the new 'Contact Sales' pricing page significantly improves key business metrics (conversion rate, lead quality, revenue per visitor) compared to the original self-serve pricing page.
Earning criteria — what you'll demonstrate
- Design and analyze A/B tests with proper statistical rigor
- Handle common pitfalls like peeking, multiple testing, and sample ratio mismatch
- Apply both frequentist (p-value, confidence intervals) and Bayesian (posterior distributions) methods
- Segment analysis by traffic source and user type to uncover heterogeneous effects
- Translate statistical results into business recommendations with quantified risk
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
Career mappings coming soon.