J. Urbano, Markus Schedl,
"Towards Minimal Test Collections for Evaluation of Audio Music Similarity and Retrieval"
: Proceedings of the 21st International World Wide Web Conference (WWW 2012): 4th International Workshop on Advances in Music Information Research (AdMIRe 2012), 4-2012
Original Titel:
Towards Minimal Test Collections for Evaluation of Audio Music Similarity and Retrieval
Sprache des Titels:
Englisch
Original Buchtitel:
Proceedings of the 21st International World Wide Web Conference (WWW 2012): 4th International Workshop on Advances in Music Information Research (AdMIRe 2012)
Original Kurzfassung:
Reliable evaluation of Information Retrieval systems requires
large amounts of relevance judgments. Making these annotations
is quite complex and tedious for many Music Information
Retrieval tasks, so performing such evaluations requires too much
effort. A low-cost alternative is the application of Minimal Test
Collection algorithms, which offer quite reliable results while
significantly reducing the annotation effort. The idea is to
incrementally select what documents to judge so that we can
compute estimates of the effectiveness differences between
systems with a certain degree of confidence. In this paper we
show a first approach towards its application to the evaluation of
the Audio Music Similarity and Retrieval task, run by the annual
MIREX evaluation campaign. An analysis with the MIREX 2011
data shows that the judging effort can be reduced to about 35% to
obtain results with 95% confidence.
Forschungs-