Monday, 26 October 2026 · 7 min read
UserArena benchmark and OPTD method improve long-horizon user simulation for shopping agents
UserArena provides a large-scale, real-world e-commerce benchmark with long user trajectories and production A/B validation. The proposed OPTD method uses trajectory-level intent and on-policy distillation to train simulators that better reproduce user behavior and policy effects over long interactions.
Based on the paper UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents by Jujia Zhao, Jiakai Tang, Jing Yu, Wen Chen and 5 others
Challenges in simulating long-horizon interactive shopping behavior
Agentic recommendation systems aim to assist shoppers by making sequences of context-dependent decisions that fulfill user goals. To develop and evaluate such systems, it is crucial to have recommendation world models that simulate both the shopping environment and realistic user responses over long interactions.
Existing datasets and benchmarks often lack long-horizon, fine-grained real-user trajectories aligned with visual context and full action spaces, limiting their ability to evaluate simulators on preserving user behavior across sessions.
Moreover, current evaluation methods mostly focus on next-action prediction accuracy, which does not capture whether simulators reproduce realistic sequential behavior or whether offline simulation outcomes align with real-world A/B test results, leaving a gap in validating long-term behavioral fidelity and policy effect realism.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- Three key formulas from the paper, explained symbol by symbol
- Three figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short