Please note: This master’s thesis presentation will take place online.
Sheng Yao, Master’s candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Freda Shi
Studies have shown that large language models (LLMs) have knowledge about English grapheme-to-phoneme conversion (G2P) and phonetics, but only to a moderate degree. They mainly use tasks such as rhyme detection, syllable counting, and G2P with words inside the vocabulary. We still believe that the acid test of such knowledge should involve pseudo-words — made-up orthographic words. If human participants all agree on a certain pronunciation for a given pseudo-word, it means they have used some common (implicit) knowledge about G2P and phonetics when making their prediction. Therefore, we aim to compare the predictions of the sound of pseudo-words made by LLMs to human predictions as an indicator of LLMs’ relevant knowledge.
It turns out that LLMs’ knowledge does have a remarkable degree of humanness, not only because 80% of LLMs’ predictions are the same as humans’ when there is zero inter-human divergence, also because LLMs’ bewilderment (measured by the how LLMs’ predictions vary across runs) correlates with humans’. Meanwhile, we also see substantial numbers of cases where LLMs’ predictions are far from humans’. More than half the cases suggest that LLMs actually oversimplify the matter of G2P, sticking to the most common mappings in the English vocabulary. These tendencies can be useful when we try to improve the performance of LLMs and even text-to-speech models on pronunciation tasks. We also included the text-to-speech component of SpeechT5, in order to compare the performance of text-only models and bi-modal ones. While there seems to be a bottleneck for LLMs, SpeechT5 is easily more human-like than all LLMs on several metrics.