Question framing dominates model choice
A new arXiv preprint, Impact Is Not Invalidation: Ask About the Claim, Not the Diff, argues that coding-agent memory systems should test whether a specific stored claim still holds, rather than whether a commit preserves behavior. The authors report that content anchoring — invalidating a claim whenever the artifact it came from changes — fires constantly, and that semantic-equivalence classification asks about the diff instead of any stored claim. Asked whether a commit preserves behavior, five models spanning a 40x price range flagged 59-72% of real commits and reached precision of only 0.291 to 0.329 against a 0.25 base rate. Asked instead whether one specific claim still holds, the same models on the same diffs reached 0.705 to 0.974. A control that gave the behavior-preservation judge the claim text, changing only the question, moved precision by 0.010 and 0.016, while changing the question moved it by 0.49 and 0.65 — the authors' evidence that the gap is about the question, not model capability. [1][2]
A deployed selector doesn't close the gap
The paper compares against pytest-testmon, a deployed regression-test selector using coverage-derived dependency data. It reached 0.868 recall at 0.415 precision, which the authors say shows near-complete knowledge of what a change can touch does not identify what it falsifies. [1][2]
Ground truth and dataset
The authors define a claim as a test function passing at commit t, and say it has flipped if the same assertion text fails at t+1 — execution, not annotation, is the ground truth. They note a construction problem: on a CI-gated mainline, a commit that leaves a pre-existing test failing cannot merge, so the naive positive class is empty by design. They report 10,369 claims with 184 execution-verified flips mined from 23 Python libraries, with splits held out by repository, a post-knowledge-cutoff split, a shuffled-diff null, a paraphrase control, and a leave-one-repository-out analysis over 17 repositories. [1][2]
Sources
- Impact Is Not Invalidation: Ask About the Claim, Not the Diff
ArXiv CS.CL (Computation and Language) · Reporting ·
- Impact Is Not Invalidation: Ask About the Claim, Not the Diff