arXiv study finds residualization can decorrelate LLM eval scores without fixing them

Residualizing eval scores: gains are narrower than they look

A cross-listed arXiv CS.LG preprint (arXiv:2609.24194) argues that residualization — subtracting a score's predictable surface-form component — is not by itself a more valid measurement, because the removed component may carry construct-relevant signal and the method cannot tell which is which. The paper reports a public preference reward model choosing between a terse correct solution and a commented buggy solution for the same MBPP problem scored about like a coin flip (0.507). In designed interventions (unit-test labels with comment-only edits), residualization attenuated the reward model's format effects by about 0.12 on both correct and buggy code, while correct-versus-buggy margins moved by less than 0.01. In observational NLI and QA settings, the authors froze a held-out replication before scoring and re-evaluated it with labels from disjoint annotators; they say this supports only a narrower claim — better agreement with construct labels on a pre-declared slice where a surface-only predictor errs — not a repaired score. They add that full-population agreement fell in every observational setting with a reported positive slice gain, and within-question ranking fell in every such QA setting. The paper also reports that, in a controlled model, configurations just as damaging to construct alignment passed every pre-adjustment check, so no committed gate is a guarantee. Their proposed reporting protocol treats an adjusted score as an audit-time diagnostic reported alongside the construct-alignment cost it incurs, never a replacement for the raw score. These are the authors' own findings from a single preprint; independent replication is not reported in the supplied evidence. [1]