ChronoReplay improves spatial VLMs by balancing and replaying diverse spatial tasks

Spatial vision-language models (VLMs) are trained to answer spatial queries such as categorical (e.g., which side an object lies on), metric (e.g., distance estimates), and temporal (e.g., when an event occurred) questions from streaming video observations. Answering these requires the model to accumulate observations over time, track object and observer motion, and maintain a coherent spatial state under changing reference frames.

Existing approaches often train on a single answer interface (e.g., categorical or metric) and report performance on benchmarks matching that interface. However, training on one interface can improve performance on that interface but transfer poorly or even degrade performance on others, a phenomenon the authors call interface lock-in.

Moreover, spatial targets like metric distances or timestamps depend on the specific causal history (the sequence of observations and actions leading to the query). Detaching targets from their valid history prefix leads to ambiguity, making it crucial to preserve the chronological order and context during training.