PhD Seminar • Information Retrieval | Natural Language Processing • Nearest Neighbor Speculative Decoding for LLM Generation and AttributionExport this event to calendar

Thursday, June 20, 2024 — 2:00 PM to 3:00 PM EDT

Please note: This PhD seminar will take place online.

Minghan Li, PhD candidate
David R. Cheriton School of Compute Science

Supervisor: Professor Jimmy Lin

Large language models (LLMs) often hallucinate and lack the ability to provide attribution for their generations. Semi-parametric LMs, such as kNN-LM, approach these limitations by refining the output of an LM for a given prompt using its nearest neighbor matches in a non-parametric data store. However, these models often exhibit slow inference speeds and produce non-fluent texts. In this paper, we introduce Nearest Neighbor Speculative Decoding (Nest), a novel semi-parametric language modeling approach that is capable of incorporating real-world text spans of arbitrary length into the LM generations and providing attribution to their sources.

Nest performs token-level retrieval at each inference step to compute a semi-parametric mixture distribution and identify promising span continuations in a corpus. It then uses an approximate speculative decoding procedure that accepts a prefix of the retrieved span or generates a new token. Nest significantly enhances the generation quality and attribution rate of the base LM across a variety of knowledge-intensive tasks, surpassing the conventional kNN-LM method and performing competitively with in-context retrieval augmentation. In addition, Nest substantially improves the generation speed, achieving a 1.8× speedup in inference time when applied to Llama-2-Chat 70B.


To attend this PhD seminar on Zoom.

Location 
Online PhD seminar
200 University Ave West

Waterloo, ON N2L 3G1
Canada
Event tags 

S M T W T F S
30
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
1
2
3
  1. 2024 (184)
    1. September (1)
    2. August (5)
    3. July (20)
    4. June (17)
    5. May (23)
    6. April (41)
    7. March (27)
    8. February (25)
    9. January (25)
  2. 2023 (296)
    1. December (20)
    2. November (28)
    3. October (15)
    4. September (25)
    5. August (30)
    6. July (30)
    7. June (22)
    8. May (23)
    9. April (32)
    10. March (31)
    11. February (18)
    12. January (22)
  3. 2022 (245)
  4. 2021 (210)
  5. 2020 (217)
  6. 2019 (255)
  7. 2018 (217)
  8. 2017 (36)
  9. 2016 (21)
  10. 2015 (36)
  11. 2014 (33)
  12. 2013 (23)
  13. 2012 (4)
  14. 2011 (1)
  15. 2010 (1)
  16. 2009 (1)
  17. 2008 (1)