Please note: This PhD seminar will take place in DC 2585.
Xueguang
Ma,
PhD
candidate
David
R.
Cheriton
School
of
Computer
Science
Supervisor: Professor Jimmy Lin
While large language models (LLMs) have demonstrated impressive NLP capabilities, existing information retrieval applications mainly focus on prompting LLMs for query expansions or generating permutations for listwise reranking.
In this presentation, I will introduce our recent work on fine-tuning LLaMA for multi-stage text retrieval. Specifically, we fine-tune the open-source LLaMA-2 model as a dense retriever (repLLaMA) and a pointwise reranker (rankLLaMA). Our study shows that fine-tuned LLM retrieval models outperform smaller models, proving to be more effective and exhibiting greater generalizability with only a straightforward training strategy. Moreover, our pipeline allows for the fine-tuning of LLMs at each stage of a multi-stage retrieval pipeline, demonstrating the strong potential for optimizing LLMs to enhance a variety of retrieval tasks. Furthermore, as LLMs are naturally pre-trained with longer contexts, they can directly represent longer documents. This eliminates the need for heuristic segmenting and pooling strategies to rank long documents. Alongside our work, I will also introduce the most recent works on large embedding models that show potential for handling longer context windows and following user instructions.