Does Reasoning Make Search More Fair?

Reasoning models — the kind that generate intermediate “thinking” tokens before answering — have started showing up as rerankers in search pipelines, usually pitched as a strict upgrade: better relevance judgments, better ranking quality. What nobody had checked is whether that extra reasoning step changes how fair the resulting rankings are — for example, whether documents about different countries or demographic groups get systematically over- or under-ranked.

This project (Samuel et al., 2026) ran that comparison for the first time. Using the TREC 2022 Fair Ranking Track data, we evaluated six reranking models — a mix of reasoning and non-reasoning architectures — and measured fairness with Attention-Weighted Rank Fairness (AWRF), alongside standard relevance metrics.

The result was not what a “reasoning helps everything” narrative would predict. AWRF stayed in a tight 0.33-0.35 band across every model we tested, despite the models producing very different relevance quality. Reasoning rerankers didn’t come out ahead on fairness, but they didn’t come out behind either — they largely preserved whatever fairness characteristics were already present in the input ranking. And geographic attributes in particular showed consistent disparities no matter which architecture we used, reasoning or not.

The takeaway is a useful corrective: fairness isn’t a side effect of general capability improvements like reasoning. If you want a fairer reranker, you have to train and evaluate for that specifically — better reasoning about relevance doesn’t automatically translate into better reasoning about equitable exposure across groups.

This paper was accepted to ECIR 2026, joint work with Benjamin Van Durme and Eugene Yang at the Johns Hopkins HLTCOE. Read the paper on arXiv.

Reasoning rerankers are increasingly used in search, but it was unknown whether their extra reasoning steps help or hurt fairness. We ran the first systematic comparison of fairness in reasoning versus non-reasoning rerankers, evaluating six models on the TREC 2022 Fair Ranking Track data. Reasoning neither reliably improves nor harms fairness: Attention-Weighted Rank Fairness stayed within a narrow 0.33-0.35 band across all models despite large swings in relevance quality, and geographic-attribute disparities persisted regardless of architecture, suggesting fairness has to be specifically trained for rather than assumed to emerge from better reasoning.

References

2026