Ewance
← All challenges Research Scientist

Embodied Visual Reasoning for a Warehouse Pick Assistant

Duration
3 weeks
Level
Expert
Type
Research
Team
1 to 5 people

The situation

The Rotterdam startup (~18 staff, Series Seed, piloting in 3 Benelux third-party-logistics warehouses) treats high-level reasoning as the bottleneck on their first paid pilot — grasp execution already works, decision-making does not.

Your task

Show whether an open vision-language model can outperform a heuristic baseline at high-level pick-order reasoning in cluttered warehouse bins.

The brief

What you do

  1. Use an embodied simulator (Habitat 3.0 or Isaac Sim — pick one and justify) to render 300 cluttered-bin scenarios with a target item label.
  2. For each scenario, build two reasoning modules: (1) a heuristic baseline that picks the topmost unoccluded object closest to the target, (2) a vision-language reasoner that takes the RGB-D view + the target label and outputs the recommended pick with a short explanation.
  3. Use an open VLM (PaliGemma, Qwen-VL, or LLaVA) with no fine-tuning required.
  4. Evaluate: pick-correctness rate against simulator ground truth, explanation quality (3-point rubric judged by a peer), and latency.
  5. Document where each method fails.

Success looks like

Success: VLM pipeline beats heuristic by at least 15 percentage points on pick-correctness on a 100-scenario holdout.

What you will practice

  • Use an embodied simulator to construct a controlled visual-reasoning evaluation
  • Apply an open vision-language model as a high-level reasoning module
  • Compare an LLM-based reasoner against a strong heuristic baseline fairly
  • Translate research findings into a sprint-ready research question list

A suggested way in

  1. Use Habitat 3.0 for fast RGB-D scene generation if Isaac Sim is overkill for your hardware
  2. Cap the VLM's response format (target_object_id + one-sentence reason) to make scoring tractable
  3. Score explanations on a 3-point rubric (correct/partial/wrong) with two independent peers to estimate inter-rater agreement

Skills you will practice

Embodied vision, Vision language models, Visual reasoning, Simulation, Evaluation, Pytorch

Where it leads

Research Scientist, ML Researcher, Computer Vision Engineer

How it is assessed

  • Evaluation rigor30%

    VLM beats heuristic by ≥15 pp pick-correctness on the 100-scenario holdout, with confidence intervals.

  • Simulator setup25%

    Dataset is reproducible from the simulator config, with documented seeds.

  • Explanation quality25%

    Inter-rater agreement on the 3-point explanation rubric is reported.

  • Sprint-ready findings20%

    1-page research-question list is concrete enough for the next sprint's tickets.

Resources

  • HabitatTool

    Open-source embodied simulator with fast RGB-D rendering for cluttered scenes.

  • PyTorchTool

    Run inference for the open vision-language model.

  • Hugging Face TransformersTool

    Hosts PaliGemma, Qwen-VL, and LLaVA model variants.

  • PythonTool

    Glue the simulator, the VLM, the heuristic, and the evaluator.

  • Google SheetsTool

    Cross-tabulate pick-correctness and explanation scores per method.

  • NotionTool

    Author the evaluation report and the research-question list.

Sample Ewance certificate of completion, verified by LearnCoin
Sample certificate
Your proof

Finish it. Earn this.

A verified certificate with your name. Anyone can check it.

Time to practice.

Solve this challenge
Related challenges

More like this.