Ewance
← All challenges Machine Learning Engineer

Train a Manipulation Policy for Bin Picking with Imitation Learning

Duration
3 weeks
Level
Expert
Type
Code
Team
1 to 5 people

The situation

The supplier (around 4,500 staff, EUR 1.1B revenue, supplying door-handle assemblies to two German OEMs) currently does this picking manually at a station that's the takt-time bottleneck on one line.

Your task

Train an imitation-learning bin-picking policy that hits 85% success in simulation and degrades gracefully on novel parts.

The brief

You start with

You receive a dataset of 500 teleop trajectories on the in-distribution part plus a held-out simulation environment with a never-seen part.

What you do

  1. Train an imitation-learning policy (Diffusion Policy recommended; ACT acceptable) conditioned on RGB-D images and proprioception.
  2. Evaluate over 100 sim rollouts per condition: (a) in-distribution part, (b) novel part, (c) bin shifted by 5 cm.
  3. Report success rate, mean cycle time, and a failure-mode taxonomy.
  4. The memo recommends whether the team can move to a 50-pick real-robot pilot.

What you will practice

  • Apply modern imitation-learning algorithms (Diffusion Policy / ACT) to manipulation
  • Set up reproducible simulation evaluation for a manipulation policy
  • Characterize policy generalization with structured rollout protocols
  • Translate sim results into a defensible real-robot pilot recommendation

A suggested way in

  1. Use the official Diffusion Policy repo as the training baseline
  2. Use Isaac Sim's UR5e asset; randomize lighting + bin pose for the eval set
  3. Bucket failures: (a) grasp miss, (b) drop in transit, (c) collision — that's the readable taxonomy

Skills you will practice

Imitation learning, Manipulation, Diffusion policy, Pytorch, Robotics simulation, Evaluation

Where it leads

Machine Learning Engineer, ML Researcher

How it is assessed

  • Policy performance35%

    Hits 85%+ on in-distribution bin and reports honest novel-part rate.

  • Evaluation rigor25%

    100 rollouts per condition with seeds, CIs, and failure taxonomy.

  • Generalization analysis20%

    Novel-part and pose-shift conditions are clearly reported, not buried.

  • Pilot-readiness memo20%

    Memo names success criteria for the real-robot pilot, not just sim numbers.

Resources

  • PyTorchTool

    Standard for Diffusion Policy and ACT training.

  • Isaac SimTool

    Physics-accurate simulation with UR5e + bin assets out of the box.

  • Weights & BiasesTool

    Track training and per-condition eval metrics.

  • PythonTool

    Glue the eval harness.

  • NotionTool

    Deployment-readiness memo.

  • MatplotlibTool

    Failure-mode visualizations and success-rate plots.

Sample Ewance certificate of completion, verified by LearnCoin
Sample certificate
Your proof

Finish it. Earn this.

A verified certificate with your name. Anyone can check it.

Time to practice.

Solve this challenge
Related challenges

More like this.