Audit: 99% fake-news detector accuracy mostly reflects source, not veracity

What the audit found

- An arXiv CS.CL preprint (2609.25006) audits the ISOT/Kaggle “Fake and Real News” corpus, where text classifiers routinely report accuracy and F1 above 0.98, using a transparent TF-IDF plus linear-classifier pipeline as its measurement instrument. Authors released code and derived numbers. [ArXiv CS.CL (Computation and Language)] What happened: researchers audited a widely used fake-news benchmark and found its headline scores mostly reflect source and topic separability rather than veracity. Why it matters: benchmark numbers like “99% accurate” can signal dataset artifacts instead of the skill being claimed, which matters for anyone citing such scores as evidence a system can judge truth. But the finding is corpus- and pipeline-specific: the authors measured leakage and transfer within this dataset and a set of related protocols, and independent replication is not reported. [ArXiv CS.CL] Per the preprint, leakage and shift: the two classes have disjoint subject metadata, so a classifier given only that field with article text discarded reaches F1 = 1.000; a newswire source tag appears in 99.2% of real articles, and 6,251 duplicate documents contaminate 19.4% of a naive test split. [ArXiv CS.CL] Per the preprint, removing all three leakage channels lowers F1 by only 1.21 points (0.9935 to 0.9814), and the residual signal looks like diffuse editorial style: deleting the 1,000 highest-weight unigrams still leaves F1 = 0.926. [ArXiv CS.CL] Per the preprint, under a topic-disjoint protocol average precision drops from 0.9995 to 0.9475 and deployed F1 from 0.9905 to 0.8067, with a prior-matched analysis confirming a genuine 5.2-point loss of discrimination; temporal transfer is nearly lossless. [ArXiv CS.CL] Per the preprint, a fine-tuned DistilBERT is stronger in-distribu [1]