MCMC for Conversion-Funnel A/B Testing at a Marketplace
Overview
What this challenge is about.
You receive 6 weeks of per-visitor funnel data (visit, sign-up, trial start, trial-to-paid conversion) split by variant and by acquisition channel (organic, paid social, paid search). Fit a Bayesian hierarchical model with per-channel partial pooling on each funnel-stage conversion rate. Use the No-U-Turn Sampler (NUTS, a modern MCMC algorithm) in PyMC or NumPyro with 4 chains and clear convergence diagnostics (R-hat and effective sample size). Report per-stage and overall posterior probability of variant B winning, plus the 90 percent credible interval on the lift at each stage. Deliver a 2-page memo with a clear roll/kill recommendation.
The Brief
What you'll do, and what you'll demonstrate.
Reanalyze a funnel A/B test with hierarchical Bayesian MCMC to give growth a defensible per-stage roll/kill recommendation.
Earning criteria — what you'll demonstrate
- Build a Bayesian hierarchical model with partial pooling across groups
- Run NUTS MCMC and verify convergence with standard diagnostics
- Interpret posterior probabilities for business decisions
- Communicate Bayesian results to a frequentist-trained growth team
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
Data Scientist
Hierarchical Bayesian A/B reanalysis is exactly the work growth-focused data scientists do when frequentist tests stall.
This challenge sharpens
- mcmc
- bayesian-hierarchical-models
- ab-testing
Applied AI Scientist
Choosing priors, validating MCMC convergence, and translating posteriors into a roll/kill recommendation is core applied-research work.
This challenge sharpens
- mcmc
- convergence-diagnostics
- bayesian-hierarchical-models
ML Researcher
Defending hierarchical-model choices and prior sensitivity in writing prepares a student for research roles where methodological transparency matters.
This challenge sharpens
- bayesian-hierarchical-models
- mcmc
- convergence-diagnostics