Reflex enables humanoid robots to catch thrown boxes using egocentric RGB-D vision

Humans can quickly react with their whole body to dynamic events like catching thrown objects, which requires predicting object trajectories and coordinating body movements within about a second. Achieving this on humanoid robots is difficult because it demands whole-body coordination, real-time reaction to fast object dynamics under sensor and processing delays, and grounding perception from egocentric vision where the camera moves with the robot and the object can be occluded.

Catching a thrown box is a representative task combining these challenges. Unlike catching balls with hands only, it requires repositioning the torso and arms to absorb impact and retain the box without losing balance. Existing robotic systems often rely on external motion capture or marked objects, or only control end-effectors, so they do not fully address whole-body reactive catching from onboard vision.

The problem is compounded by latency: the robot's camera-to-control latency is about 69 ms, while the box arrives roughly one second after release, requiring the robot to predict future object states rather than react to the latest observation. The egocentric camera view also mixes object and self-motion, and the robot's own arms occlude the object near contact, making perception and control tightly coupled and challenging.