Title
The limits of language model confidence
Abstract
The public has increasing interest in commercial Large Language Models and their ability (at a surface-level) to emulate human linguistic behaviours. Among linguists, particular interest is in the extent to which machine learning of language can approximate the learning patterns of humans, and further whether a ‘learned’ representation of language is reflective of what is encoded in our own mental representation of that same language. In this talk I will consider these questions with respect to underlying representations of language models and the robustness of state-of-the-art model representations to perform downstream tasks on imperfect language data. First, I compare the predictability (surprisal) of speech as indicated by a language model’s confidence in its own predictions, to evaluations of performance in a supervised setting. Second, I will present some case study results on the limits of model performance when applied to the clinical speech setting. Together, these lines of inquiry will lead us towards an answer to the question: How confident can we be about language model confidence?
Bio
Dr. Sara Ng‘s research focuses on issues in computational models of prosody and speech perception. Her primary research concerns how computational methods can be used to better understand the role of prosody in spoken language. She is interested in how humans use tune, rhythm, and pronunciation to convey pragmatic meaning such as discourse structure and conceptual pacts, and how this exchange of information is affected by hearing impairment and speech disorders. Her recent work also concerns how large commercial speech recognition systems such as OpenAI’s Whisper leverage prosody. She uses corpus-based, theory-agnostic, engineering approaches with grounding in attested generalities of language acoustics.
