The situation
The Munich scale-up (~110 staff, ~EUR 18 M ARR) serves ~60 consumer-packaged-goods brands across DACH; their detector-only product currently misses ~35% of the planogram-drift insights customers actually ask for.
Your task
Build a scene-graph generation pipeline for shelf photos and prove (or disprove) that scene graphs unlock insights a detector-only baseline cannot.
The brief
You start with
You receive 1,500 labeled shelf photos (anonymized product crops, bounding boxes, and ~12 relation types).
What you do
- Build a pipeline that, for a new shelf photo, outputs (a) detected products, (b) a scene graph of spatial + stock relations, and (c) a short structured insight summary ('Brand X has lost 2 facings vs the planogram').
- Use an off-the-shelf detector (YOLOv8 or DETR) and a lightweight relation predictor on top.
- Evaluate recall@K of the predicted relations against the labels and qualitatively review on a 50-photo holdout.
- Produce a written analysis: which of the 4 brand-team questions (planogram drift, share of shelf, adjacency to competitors, out-of-stock) does the scene-graph approach answer better than the current detector-only baseline?
What you will practice
- Implement a scene-graph generation pipeline combining detection and relation prediction
- Apply scene-understanding metrics (recall@K of relations) honestly
- Reason about when a richer representation actually unlocks downstream value
- Communicate computer-vision results to a product audience without jargon
A suggested way in
- Use a frozen YOLOv8 or DETR detector as the backbone — relation modeling is the new piece
- Recall@20 of triplets is the standard scene-graph metric; report at K=20 and K=50
- Visualize predicted graphs over the source photo before believing the metrics
Skills you will practice
Scene graph generation, Object detection, Relation prediction, Evaluation, Pytorch, Scene understanding
Where it leads
Computer Vision Engineer, Applied AI Scientist, Machine Learning Engineer
How it is assessed
- Pipeline quality30%
End-to-end pipeline runs on the holdout and reports per-relation recall@K.
- Relation accuracy25%
Recall@20 of spatial + stock relations beats a detector-only baseline by a documented margin.
- Insight comparison25%
Memo cleanly maps each of the 4 brand-team questions to a pipeline output.
- Reproducibility20%
Another engineer can rerun the evaluation from the notebook + saved checkpoints.
Resources
- PyTorchTool
Train the relation head and run the end-to-end pipeline.
- Ultralytics YOLOTool
Quick path to a strong product detector backbone.
- PythonTool
Glue the detection, relation, and insight layers into one pipeline.
- JupyterTool
Iterate on relation prediction and inspect graphs over real shelf photos.
- Weights & BiasesTool
Track relation-prediction experiments and visualize sample predictions.
- NotionTool
Author the insight-comparison memo for the product team.
Finish it. Earn this.
A verified certificate with your name. Anyone can check it.
Time to practice.
Solve this challengeMore like this.
- Software EngineerBuild a Multi-Level Feedback Queue Scheduler in xv6
- Software EngineerReverse-Engineer a Stripped Binary to Restore Linking Symbols
- Software EngineerImplement a Distance-Vector Routing Algorithm
- AI EngineerEnd-to-End Lane Following on a Donkeycar Platform
-
Build a Multi-Level Feedback Queue Scheduler in xv6 Software Engineer Fork the xv6-riscv repository and implement MLFQ in proc.c with 3 priority queues, per-queue time slices (10ms / 20ms / 40ms), priority demotion on sl operating systemsschedulingc programming Open challenge → -
Reverse-Engineer a Stripped Binary to Restore Linking Symbols Software Engineer Receive the stripped shared object (Linux ELF, x86-64, ~280KB), a crash dump with 12 frames of indecipherable addresses, and 4 years of partial git hi reverse engineeringlinking and loadingmachine code Open challenge → -
Implement a Distance-Vector Routing Algorithm Software Engineer Implement distance-vector routing in Python: each simulated router maintains a distance vector, exchanges vectors with neighbors every 1 second of sim routingdistance vectornetwork simulation Open challenge → -
End-to-End Lane Following on a Donkeycar Platform AI Engineer Use the public Donkeycar Tub dataset (or collect about 30 minutes of driving on the simulator). Train a CNN-policy baseline (the Donkeycar default arc end to end learningimitation learningpytorch Open challenge →