Benchmark Conformal Prediction for a Healthcare Risk-Score
Overview
What this challenge is about.
Implement split, jackknife+, and Mondrian conformal prediction on a healthcare risk-score dataset, evaluate coverage per subgroup, and earn a verifiable certificate.
The scenario
The startup (around 50 staff, Series A, used by ~3,000 clinicians in the US) is preparing for an FDA Software-as-a-Medical-Device submission where calibrated probabilistic outputs strengthen the dossier.
The Brief
What you'll do, and what you'll demonstrate.
Pick the conformal-prediction variant that gives honest coverage with the tightest intervals across clinically meaningful subgroups.
Earning criteria — what you'll demonstrate
- Apply conformal prediction to a real classifier output
- Measure marginal and conditional coverage honestly
- Compare conformal variants on interval tightness + subgroup fairness
- Document methodology for regulatory-grade submissions
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Statistical Machine Learning
Master · Machine Learning
Strong alignment
This challenge maps to Statistical Machine Learning at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Conformal Prediction
Apply conformal prediction to solve real industry problems and demonstrate production-level capability.
- Uncertainty Quantification
Apply uncertainty quantification to solve real industry problems and demonstrate production-level capability.
- Calibration
Apply calibration to solve real industry problems and demonstrate production-level capability.
- Subgroup Analysis
Apply subgroup analysis to solve real industry problems and demonstrate production-level capability.
- Python
Write clean, efficient Python for data processing, automation, and backend services.
- Evaluation
Apply evaluation to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
Research Scientist
Applying conformal prediction with rigorous coverage analysis to a regulatory-bound product is the kind of work junior research scientists at clinical-ML startups own.
This challenge sharpens
- conformal-prediction
- uncertainty-quantification
- subgroup-analysis
ML Researcher
Benchmarking modern uncertainty methods against subgroup fairness is core ML-researcher craft at any safety-critical AI shop.
This challenge sharpens
- conformal-prediction
- calibration
- evaluation
AI Safety Researcher
Honest coverage analysis and dossier-grade documentation is exactly the AI safety researcher's contribution to medical-AI launches.
This challenge sharpens
- uncertainty-quantification
- subgroup-analysis
- calibration