Evaluate a Generative AI Image Tool with a Within-Subjects Study
Overview
What this challenge is about.
Design a within-subjects study for an AI image tool, analyze paired data, and deliver a ship/kill recommendation to earn a verifiable certificate.
The scenario
The startup (~30 staff) has 1.2 million MAU and competes with several other selfie-AI apps; first-week retention is the single metric their next funding round hinges on.
The Brief
What you'll do, and what you'll demonstrate.
Decide whether guided-prompt onboarding genuinely improves first-week engagement using a clean within-subjects user study.
Earning criteria — what you'll demonstrate
- Design and run a within-subjects user study with counterbalancing
- Choose and report appropriate paired statistics for ordinal and continuous data
- Evaluate generative-AI tools with both subjective and behavioral measures
- Translate study results into a single ship/kill recommendation
Program Fit
Where this fits in your program.
Sharpens the same skills your degree expects you to demonstrate.
Human-Computer Interaction for AI Systems
Master · Applied Ai
Strong alignment
This challenge maps to Human-Computer Interaction for AI Systems at the Master level. It sharpens the same practical skills your coursework expects — but in a real industry context with actual constraints and deliverables.
Skills
Skills you'll demonstrate.
Each one shows up on your verified credential.
- Experiment Design
Apply experiment design to solve real industry problems and demonstrate production-level capability.
- User Study
Apply user study to solve real industry problems and demonstrate production-level capability.
- Within Subjects Design
Apply within subjects design to solve real industry problems and demonstrate production-level capability.
- Evaluation
Apply evaluation to solve real industry problems and demonstrate production-level capability.
- Statistical Analysis
Apply statistical analysis to solve real industry problems and demonstrate production-level capability.
- Human Ai Interaction
Apply human ai interaction to solve real industry problems and demonstrate production-level capability.
Careers
Career paths this challenge builds toward
Completing this challenge demonstrates skills that transfer directly to these roles:
AI Product Designer
Running a credible within-subjects study on a consumer AI surface plus shipping the recommendation is exactly the work AI product designers do for growth teams.
This challenge sharpens
- user-study
- within-subjects-design
- human-ai-interaction
AI Product Manager
Owning the ship/kill recommendation backed by paired statistics mirrors the AI PM's day-to-day at a feature-driven consumer-AI org.
This challenge sharpens
- experiment-design
- evaluation
- statistical-analysis
AI Safety Researcher
Pre-registered evaluations and honest reporting of effect sizes are the methodological core of AI safety research on real users.
This challenge sharpens
- experiment-design
- evaluation
- statistical-analysis