Thursday, December 13, 2018 9:00 AM EST
Haotian Zhang, PhD candidate
David R. Cheriton School of Computer Science
Dynamic sampling (DS) is applied to create a sampled set of relevance judgments in our participation of TREC Common Core Track 2018. One goal was to test the effectiveness and efficiency of this technique with a set of non-expert, secondary relevance assessors. We consider NIST assessors to be the experts and the primary assessors. Another goal was to make available to other researchers a sampled set of relevance judgments (prels) and thus allow the estimation of retrieval metrics that have the potential to be more robust than the standard NIST provided relevance judgments (qrels). In addition to creating the prels, we also submitted several runs based on our manual judging and the models produced by our HiCAL system.
