The situation
The Tel Aviv consumer AI startup (around 18 staff, post-seed) ships an iOS-first accessibility app with around 40,000 active users and treats caption usefulness as its core differentiation.
Your task
Fine-tune a vision-language model so its captions are actually useful for low-vision users, validated by a 30-user study.
The brief
What you do
- Take BLIP-2 or LLaVA-1.6 as the base.
- Fine-tune (LoRA is fine) on a 4,000-image accessibility-curated dataset where each image has a useful caption written by a low-vision-experienced annotator.
- Use an instruction-following caption style.
- Evaluate with both automated metrics (CIDEr, SPICE) and a 30-user study comparing fine-tuned vs base captions on perceived usefulness.
- Report a Likert-scale comparison.
- Write a 4-page memo with a go/no-go recommendation on shipping the fine-tune.
What you will practice
- Fine-tune a vision-language model with parameter-efficient methods
- Design a user study that measures real downstream usefulness
- Balance automated metrics with human judgment
- Make a ship/no-ship call on a model fine-tune
A suggested way in
- LoRA on the language head only is often enough for caption-style adaptation
- Automated caption metrics correlate poorly with usefulness — the user study is the source of truth
- Always include base-model captions in the user study unblinded to control for expectation bias
Skills you will practice
Vision language models, Lora fine tuning, Pytorch, User study design, Image captioning, Evaluation
Where it leads
Applied AI Scientist, ML Researcher, AI Product Designer
How it is assessed
- Fine-tune quality25%
Fine-tuned captions are visibly better than base on the sample gallery.
- Automated metrics20%
CIDEr/SPICE reported honestly, with caveats about their limits.
- User-study design30%
Study has a clear hypothesis, paired comparisons, honest sample-size caveats.
- Memo quality25%
Ship/no-ship recommendation is decisive and traceable to the data.
Resources
- PyTorchTool
Underlying framework for vision-language model fine-tuning.
- Hugging Face PEFTTool
LoRA implementation that keeps the fine-tune budget tractable.
- Weights & BiasesTool
Track training loss + automated metrics across fine-tuning runs.
- Google FormsTool
Run the 30-user paired-comparison study cheaply.
- Lambda LabsTool
Affordable per-hour GPU rental for the fine-tuning budget.
- NotionTool
Compose the 4-page memo with embedded sample gallery.
Finish it. Earn this.
A verified certificate with your name. Anyone can check it.
Time to practice.
Solve this challengeMore like this.
- AI Safety ResearcherRed-Team an Image-Classification Pipeline for a Banking KYC Workflow
- Research ScientistEmbodied Visual Reasoning for a Warehouse Pick Assistant
- Product ManagerOptimizing Onboarding for a Fintech App
- Market Research AnalystBusiness Cycle Positioning for a Renewable Energy Firm
-
Red-Team an Image-Classification Pipeline for a Banking KYC Workflow AI Safety Researcher You receive the production image classifier as a black-box API plus a labeled validation set of 5,000 ID images. Run untargeted FGSM and PGD attacks ( adversarial attacksrobust evaluationred teaming Open challenge → -
Embodied Visual Reasoning for a Warehouse Pick Assistant Research Scientist Use an embodied simulator (Habitat 3.0 or Isaac Sim — pick one and justify) to render 300 cluttered-bin scenarios with a target item label. For each s embodied visionvision language modelsvisual reasoning Open challenge → -
Optimizing Onboarding for a Fintech App Product Manager Your task is to analyze the current onboarding flow, identify drop-off points, and propose a redesigned onboarding experience. You will use user behav user researchux designdata analysis Open challenge → -
Business Cycle Positioning for a Renewable Energy Firm Market Research Analyst Your task is to analyze current business cycle indicators for Italy (GDP growth, unemployment, consumer confidence, interest rates) and the specific i business cycle analysisleading indicatorsdemand forecasting Open challenge →