Reproduce a Vision-Model Paper Under a Reproducibility Standard
Overview
What this challenge is about.
Pick a vision-model paper from CVPR or NeurIPS 2024-2025 with publicly available code and a manageable compute footprint (single-GPU under 24 hours). Reproduce the headline metric on a documented subset (full dataset is welcome but optional). Document every deviation from the paper. Score the paper against the Machine Learning Reproducibility Checklist (Pineau et al.). Produce a 6-page Reproducibility Report covering setup, deviations, results, and three honest takeaways on what worked and what did not. Include a final 'would I trust this method' assessment with reasoning.
The Brief
What you'll do, and what you'll demonstrate.
Reproduce a recent vision-model paper, score it against the Reproducibility Checklist, and publish a structured Reproducibility Report.
Earning criteria — what you'll demonstrate
- Reproduce a recent ML paper end-to-end on a controlled subset
- Apply a published reproducibility standard (Pineau checklist)
- Quantify and explain deviations from a paper's reported numbers
- Write an honest reproducibility report citable by the lab
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
Careers
Roles this prepares you for.
Real titles. Real skill bridges. Pick the one closest to your trajectory.
ML Researcher
Reproducing a recent paper and scoring it against a published standard is the literal first-week task of a new lab PhD.
This challenge sharpens
- reproducibility
- experimental-design
- research-writing
Research Scientist
Multi-seed discipline plus honest deviation logging is the rigor expected from a junior research scientist.
This challenge sharpens
- experimental-design
- model-evaluation
- reproducibility
Applied AI Scientist
Trust-assessment style reading of a paper is how applied AI scientists triage which methods to deploy.
This challenge sharpens
- model-evaluation
- computer-vision
- reproducibility