Computational Versus Perceived Popularity Miscalibration in Recommender Systems
Sprache des Titels:
Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023)
Popularity bias in recommendation lists refers to over-representation of popular content and is a challenge for many recommendation algorithms. Previous research has suggested several offline metrics
to quantify popularity bias, which commonly relate the popularity of items in users? recommendation lists to the popularity of items in their interaction history. Discrepancies between these two factors
are referred to as popularity miscalibration. While popularity metrics provide a straightforward and well-defined means to measure popularity bias, it is unknown whether they actually reflect users? perception of popularity bias.
To address this research gap, we conduct a crowd-sourced user study on Prolific, involving 56 participants, to (1) investigate whether the level of perceived popularity miscalibration differs between common recommendation algorithms, (2) assess the correlation
between perceived popularity miscalibration and its corresponding quantification according to a common offline metric. We conduct our study in a well-defined and important domain, namely music recommendation using the standardized LFM-2b dataset, and
quantify popularity miscalibration of five recommendation algorithms by utilizing Jensen?Shannon distance (JSD). Challenging the findings of previous studies, we observe that users generally do perceive significant differences in terms of popularity bias between
algorithms if this bias is framed as popularity miscalibration. In addition, JSD correlates moderately with users? perception of popularity, but not with their perception of unpopularity.