What the paper reports
> arXiv presents WebMRE, an offline benchmark of 541 tasks and 5,293 steps built from successful WebArena trajectories, with audited test labels and a deterministic protocol that scores a checkpoint identically on every run without an environment. Each step pairs a human-oriented guide sentence with a grounded action. Averaged over three seeds, the authors report the mutual reinforcement effect holds for both models in both decoding orders and grows with scale: jointly decoding a guide lifts element selection over an action-only reference by 0.9 and 0.2 points for Qwen3.5-4B and by 1.7 and 2.2 points for Qwen3.5-9B. All figures are the authors' own results on their benchmark, not independent evaluations. The abstract names GPT-5.5, Claude Opus 4.8, and Gemini 3.5 Flash as zero-shot models the fine-tuned models outperform on every offline metric, but gives no per-metric numbers for that comparison. [1]
Mediation and reward
> arXiv's mediation analysis describes the guide as a causal channel rather than commentary: forcing the gold guide as a decoding prefix lifts action accuracy from .422 to .684, another step's guide collapses it to .055, and a paraphrase renaming the target still recovers half the gain. The authors also report that the same channel yields an offline reward only a replayable protocol makes computable, though optimizing it from a strong checkpoint brings no gain yet — a stated negative result from the paper. [1]
Sources
- Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents
ArXiv CS.CL (Computation and Language) · Reporting ·