MIT robot uses 3D keypoints for advanced coordination

Robots are extremely good at repetitive tasks with little to no variation, but struggle when dealing with added complexity or unfamiliar objects. To assess how objects should be picked up, robots tend to use either pose-based or geometry-based systems. Both methods have limitations, however, especially in the face of everyday tasks like picking up and placing a mug – a seemingly straightforward action that in fact requires advanced coordination and subtlety.

To equip its robot with that subtlety, the team from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) used an approach that identifies objects as a collection of 3D keypoints, providing a type of visual roadmap that allows more nuanced manipulation. Known as kPAM (Keypoint Affordance Manipulation), the technique gives the robot all the information it needs to pick up, move and place objects accurately, while also providing enough flexibility to deal with variation between different categories of object, such as different shaped mugs or different styles of shoe.