EgoXperience enables joint 3D reconstruction of egocentric video components in a shared world frame

Recovering human activity from egocentric video requires reconstructing multiple tightly coupled 3D components: the observer's camera motion, the surrounding scene geometry, and the hands interacting with objects. These components must be consistent within a shared world coordinate frame to be useful for embodied AI applications.

Existing methods typically reconstruct only subsets of these components separately, leading to inconsistent scales and coordinate frames that complicate downstream use. A unified model that jointly reconstructs all components is desirable but requires large-scale egocentric data with aligned annotations of camera, scene, hands, and objects.

Obtaining such joint annotations is difficult due to challenges like hand-object occlusions, rapid head motion, and motion blur. Existing datasets often rely on specialized multi-camera or motion-capture setups that are hard to scale across diverse real-world environments. Thus, scalable collection and annotation pipelines are needed to provide consistent supervision for joint reconstruction.