Klara Krieg, Emilia Parada-Cabaleiro, Markus Schedl, Navid Rekabsaz,
"Do Perceived Gender Biases in Retrieval Results Affect Relevance Judgements"
: Proceedings of the workshop on Algorithmic Bias in Search and Recommendation at the European Conference on Information Retrieval (ECIR 2022), 4-2022
Original Titel:
Do Perceived Gender Biases in Retrieval Results Affect Relevance Judgements
Sprache des Titels:
Englisch
Original Buchtitel:
Proceedings of the workshop on Algorithmic Bias in Search and Recommendation at the European Conference on Information Retrieval (ECIR 2022)
Original Kurzfassung:
This work investigates the effect of gender-stereotypical biases in the content of retrieved results on the relevance judgement of users/annotators. In
particular, since relevance in information retrieval (IR) is a multi-dimensional concept, we study whether the value and quality of the retrieved documents for
some bias-sensitive queries can be judged differently when the content of the documents represents different genders. To this aim, we conduct a set of experiments
where the genders of the participants are known as well as experiments where the participants? genders are not specified. The set of experiments comprise of
retrieval tasks, where participants perform a rated relevance judgement for different search query and search result document compilations. The shown documents
contain different gender indications and are either relevant or non-relevant to the query. The results show the differences between the average judged relevance
scores among documents with various gender contents. Our work initiates further research on the connection of the perception of gender stereotypes in users with
their judgements and effects on IR systems, and aim to raise awareness about the possible biases in this domain