PhD Seminar • Dynamic Sampling used in TREC Core 2018

Haotian Zhang, PhD candidate
David R. Cheriton School of Computer Science

Dynamic sampling (DS) is applied to create a sampled set of relevance judgments in our participation of TREC Common Core Track 2018. One goal was to test the effectiveness and efficiency of this technique with a set of non-expert, secondary relevance assessors.  We consider NIST assessors to be the experts and the primary assessors. Another goal was to make available to other researchers a sampled set of relevance judgments (prels) and thus allow the estimation of retrieval metrics that have the potential to be more robust than the standard NIST provided relevance judgments (qrels). In addition to creating the prels, we also submitted several runs based on our manual judging and the models produced by our HiCAL system. 

The results show that the documents in the strata of dynamic sampling can achieve 92% system recall according to NIST qrels and one of our runs achieve the highest ranking score across all 72 submitted runs.