Evaluation of retrieval effectiveness with incomplete relevance data: theoretical and experimental comparison of three measures
2008 (English)In: Information Processing & Management, ISSN 0306-4573, E-ISSN 1873-5371, Vol. 44, no 1, 212-225 p.Article in journal (Refereed)
This paper investigates two relatively new measures of retrieval effectiveness in relation to the problem of incomplete relevance data. The measures, Bpref and RankEff, which do not take into account documents that have not been relevance judged, are compared theoretically and experimentally. The experimental comparisons involve a third measure, the well-known mean uninterpolated average precision. The results indicate that RankEff is the most stable of the three measures when the amount of relevance data is reduced, with respect to system ranking and absolute values. In addition, RankEff has the lowest error-rate.
Place, publisher, year, edition, pages
Elsevier Ltd , 2008. Vol. 44, no 1, 212-225 p.
IdentifiersURN: urn:nbn:se:hb:diva-2387DOI: 10.1016/j.ipm.2007.01.011Local ID: 2320/3314OAI: oai:DiVA.org:hb-2387DiVA: diva2:870478