Train a Manipulation Policy for Bin Picking with Imitation Learning
The situation
The supplier (around 4,500 staff, EUR 1.1B revenue, supplying door-handle assemblies to two German OEMs) currently does this picking manually at a station that's the takt-time bottleneck on one line.
Your task
Train an imitation-learning bin-picking policy that hits 85% success in simulation and degrades gracefully on novel parts.
The brief
You start with
You receive a dataset of 500 teleop trajectories on the in-distribution part plus a held-out simulation environment with a never-seen part.
What you do
- Train an imitation-learning policy (Diffusion Policy recommended; ACT acceptable) conditioned on RGB-D images and proprioception.
- Evaluate over 100 sim rollouts per condition: (a) in-distribution part, (b) novel part, (c) bin shifted by 5 cm.
- Report success rate, mean cycle time, and a failure-mode taxonomy.
- The memo recommends whether the team can move to a 50-pick real-robot pilot.
What you will practice
- Apply modern imitation-learning algorithms (Diffusion Policy / ACT) to manipulation
- Set up reproducible simulation evaluation for a manipulation policy
- Characterize policy generalization with structured rollout protocols
- Translate sim results into a defensible real-robot pilot recommendation
A suggested way in
- Use the official Diffusion Policy repo as the training baseline
- Use Isaac Sim's UR5e asset; randomize lighting + bin pose for the eval set
- Bucket failures: (a) grasp miss, (b) drop in transit, (c) collision — that's the readable taxonomy
Skills you will practice
Imitation learning, Manipulation, Diffusion policy, Pytorch, Robotics simulation, Evaluation
Where it leads
Machine Learning Engineer, ML Researcher
How it is assessed
- Policy performance35%
Hits 85%+ on in-distribution bin and reports honest novel-part rate.
- Evaluation rigor25%
100 rollouts per condition with seeds, CIs, and failure taxonomy.
- Generalization analysis20%
Novel-part and pose-shift conditions are clearly reported, not buried.
- Pilot-readiness memo20%
Memo names success criteria for the real-robot pilot, not just sim numbers.
Resources
- PyTorchTool
Standard for Diffusion Policy and ACT training.
- Isaac SimTool
Physics-accurate simulation with UR5e + bin assets out of the box.
- Weights & BiasesTool
Track training and per-condition eval metrics.
- PythonTool
Glue the eval harness.
- NotionTool
Deployment-readiness memo.
- MatplotlibTool
Failure-mode visualizations and success-rate plots.
Finish it. Earn this.
A verified certificate with your name. Anyone can check it.
Time to practice.
Solve this challengeMore like this.
- Software EngineerBuild a Multi-Level Feedback Queue Scheduler in xv6
- Software EngineerReverse-Engineer a Stripped Binary to Restore Linking Symbols
- Software EngineerImplement a Distance-Vector Routing Algorithm
- AI EngineerEnd-to-End Lane Following on a Donkeycar Platform
-
Build a Multi-Level Feedback Queue Scheduler in xv6 Software Engineer Fork the xv6-riscv repository and implement MLFQ in proc.c with 3 priority queues, per-queue time slices (10ms / 20ms / 40ms), priority demotion on sl operating systemsschedulingc programming Open challenge → -
Reverse-Engineer a Stripped Binary to Restore Linking Symbols Software Engineer Receive the stripped shared object (Linux ELF, x86-64, ~280KB), a crash dump with 12 frames of indecipherable addresses, and 4 years of partial git hi reverse engineeringlinking and loadingmachine code Open challenge → -
Implement a Distance-Vector Routing Algorithm Software Engineer Implement distance-vector routing in Python: each simulated router maintains a distance vector, exchanges vectors with neighbors every 1 second of sim routingdistance vectornetwork simulation Open challenge → -
End-to-End Lane Following on a Donkeycar Platform AI Engineer Use the public Donkeycar Tub dataset (or collect about 30 minutes of driving on the simulator). Train a CNN-policy baseline (the Donkeycar default arc end to end learningimitation learningpytorch Open challenge →