Ewance
← All challenges Computer Vision Engineer

Scene-Graph Generation for Retail Shelf Audits

Duration
3 weeks
Level
Advanced
Type
Code
Team
1 to 5 people

The situation

The Munich scale-up (~110 staff, ~EUR 18 M ARR) serves ~60 consumer-packaged-goods brands across DACH; their detector-only product currently misses ~35% of the planogram-drift insights customers actually ask for.

Your task

Build a scene-graph generation pipeline for shelf photos and prove (or disprove) that scene graphs unlock insights a detector-only baseline cannot.

The brief

You start with

You receive 1,500 labeled shelf photos (anonymized product crops, bounding boxes, and ~12 relation types).

What you do

  1. Build a pipeline that, for a new shelf photo, outputs (a) detected products, (b) a scene graph of spatial + stock relations, and (c) a short structured insight summary ('Brand X has lost 2 facings vs the planogram').
  2. Use an off-the-shelf detector (YOLOv8 or DETR) and a lightweight relation predictor on top.
  3. Evaluate recall@K of the predicted relations against the labels and qualitatively review on a 50-photo holdout.
  4. Produce a written analysis: which of the 4 brand-team questions (planogram drift, share of shelf, adjacency to competitors, out-of-stock) does the scene-graph approach answer better than the current detector-only baseline?

What you will practice

  • Implement a scene-graph generation pipeline combining detection and relation prediction
  • Apply scene-understanding metrics (recall@K of relations) honestly
  • Reason about when a richer representation actually unlocks downstream value
  • Communicate computer-vision results to a product audience without jargon

A suggested way in

  1. Use a frozen YOLOv8 or DETR detector as the backbone — relation modeling is the new piece
  2. Recall@20 of triplets is the standard scene-graph metric; report at K=20 and K=50
  3. Visualize predicted graphs over the source photo before believing the metrics

Skills you will practice

Scene graph generation, Object detection, Relation prediction, Evaluation, Pytorch, Scene understanding

Where it leads

Computer Vision Engineer, Applied AI Scientist, Machine Learning Engineer

How it is assessed

  • Pipeline quality30%

    End-to-end pipeline runs on the holdout and reports per-relation recall@K.

  • Relation accuracy25%

    Recall@20 of spatial + stock relations beats a detector-only baseline by a documented margin.

  • Insight comparison25%

    Memo cleanly maps each of the 4 brand-team questions to a pipeline output.

  • Reproducibility20%

    Another engineer can rerun the evaluation from the notebook + saved checkpoints.

Resources

  • PyTorchTool

    Train the relation head and run the end-to-end pipeline.

  • Ultralytics YOLOTool

    Quick path to a strong product detector backbone.

  • PythonTool

    Glue the detection, relation, and insight layers into one pipeline.

  • JupyterTool

    Iterate on relation prediction and inspect graphs over real shelf photos.

  • Weights & BiasesTool

    Track relation-prediction experiments and visualize sample predictions.

  • NotionTool

    Author the insight-comparison memo for the product team.

Sample Ewance certificate of completion, verified by LearnCoin
Sample certificate
Your proof

Finish it. Earn this.

A verified certificate with your name. Anyone can check it.

Time to practice.

Solve this challenge
Related challenges

More like this.