講演情報
[4E-01]A Comparative Study of Group Fairness Measures for Information Retrieval
*陶 思捷1、酒井 哲也1 (1. 早稲田大学酒井研究室)
発表者区分：学生
論文種別：ショートペーパー
インタラクティブ発表：あり
論文種別：ショートペーパー
インタラクティブ発表：あり
キーワード：
評価指標、グループフェアネス、情報検索、文書検索、ウェブ検索
In recent years, ensuring group fairness in Information Retrieval (IR) systems has become a growing concern. Shared tasks, such as Fair Ranking and FairWeb, have been proposed by IR researchers at TREC and NTCIR, to encourage people in the community to develop group-fairness-aware systems. However, these shared tasks used different evaluation measures. At TREC 2021 \& 2022 Fair Ranking Tracks, Attention-Weighted Rank Fairness (AWRF) was adopted and combined with nDCG, while in the NTCIR-17 Fairweb-1 task, the organizers utilized Group Fairness and Relevance (GFR) to evaluate the submitted runs. The evaluation measures have different strategies to compute group fairness, and the impact on the results by changing metrics remains unclear. In this paper, we re-evaluated the official results of the NTCIR-17 Fairweb-1 task with AWRF, to make a comparison of the two metrics. We further discuss how the metrics are affecting the evaluation results, and highlight key considerations for researchers to integrate group-fairness-aware evaluation into their work.