The situation
The Rotterdam startup (~18 staff, Series Seed, piloting in 3 Benelux third-party-logistics warehouses) treats high-level reasoning as the bottleneck on their first paid pilot — grasp execution already works, decision-making does not.
Your task
Show whether an open vision-language model can outperform a heuristic baseline at high-level pick-order reasoning in cluttered warehouse bins.
The brief
What you do
- Use an embodied simulator (Habitat 3.0 or Isaac Sim — pick one and justify) to render 300 cluttered-bin scenarios with a target item label.
- For each scenario, build two reasoning modules: (1) a heuristic baseline that picks the topmost unoccluded object closest to the target, (2) a vision-language reasoner that takes the RGB-D view + the target label and outputs the recommended pick with a short explanation.
- Use an open VLM (PaliGemma, Qwen-VL, or LLaVA) with no fine-tuning required.
- Evaluate: pick-correctness rate against simulator ground truth, explanation quality (3-point rubric judged by a peer), and latency.
- Document where each method fails.
Success looks like
Success: VLM pipeline beats heuristic by at least 15 percentage points on pick-correctness on a 100-scenario holdout.
What you will practice
- Use an embodied simulator to construct a controlled visual-reasoning evaluation
- Apply an open vision-language model as a high-level reasoning module
- Compare an LLM-based reasoner against a strong heuristic baseline fairly
- Translate research findings into a sprint-ready research question list
A suggested way in
- Use Habitat 3.0 for fast RGB-D scene generation if Isaac Sim is overkill for your hardware
- Cap the VLM's response format (target_object_id + one-sentence reason) to make scoring tractable
- Score explanations on a 3-point rubric (correct/partial/wrong) with two independent peers to estimate inter-rater agreement
Skills you will practice
Embodied vision, Vision language models, Visual reasoning, Simulation, Evaluation, Pytorch
Where it leads
Research Scientist, ML Researcher, Computer Vision Engineer
How it is assessed
- Evaluation rigor30%
VLM beats heuristic by ≥15 pp pick-correctness on the 100-scenario holdout, with confidence intervals.
- Simulator setup25%
Dataset is reproducible from the simulator config, with documented seeds.
- Explanation quality25%
Inter-rater agreement on the 3-point explanation rubric is reported.
- Sprint-ready findings20%
1-page research-question list is concrete enough for the next sprint's tickets.
Resources
- HabitatTool
Open-source embodied simulator with fast RGB-D rendering for cluttered scenes.
- PyTorchTool
Run inference for the open vision-language model.
- Hugging Face TransformersTool
Hosts PaliGemma, Qwen-VL, and LLaVA model variants.
- PythonTool
Glue the simulator, the VLM, the heuristic, and the evaluator.
- Google SheetsTool
Cross-tabulate pick-correctness and explanation scores per method.
- NotionTool
Author the evaluation report and the research-question list.
Finish it. Earn this.
A verified certificate with your name. Anyone can check it.
Time to practice.
Solve this challengeMore like this.
- Applied AI ScientistFine-Tune a Vision-Language Model for Image Captioning
- AI Safety ResearcherRed-Team an Image-Classification Pipeline for a Banking KYC Workflow
- Product ManagerOptimizing Onboarding for a Fintech App
- Market Research AnalystBusiness Cycle Positioning for a Renewable Energy Firm
-
Fine-Tune a Vision-Language Model for Image Captioning Applied AI Scientist Take BLIP-2 or LLaVA-1.6 as the base. Fine-tune (LoRA is fine) on a 4,000-image accessibility-curated dataset where each image has a useful caption wr vision language modelslora fine tuningpytorch Open challenge → -
Red-Team an Image-Classification Pipeline for a Banking KYC Workflow AI Safety Researcher You receive the production image classifier as a black-box API plus a labeled validation set of 5,000 ID images. Run untargeted FGSM and PGD attacks ( adversarial attacksrobust evaluationred teaming Open challenge → -
Optimizing Onboarding for a Fintech App Product Manager Your task is to analyze the current onboarding flow, identify drop-off points, and propose a redesigned onboarding experience. You will use user behav user researchux designdata analysis Open challenge → -
Business Cycle Positioning for a Renewable Energy Firm Market Research Analyst Your task is to analyze current business cycle indicators for Italy (GDP growth, unemployment, consumer confidence, interest rates) and the specific i business cycle analysisleading indicatorsdemand forecasting Open challenge →