51²è¹Ý

Abstract: We study the error rate of LLMs on tasks like arithmetic that require a deterministic output, and repetitive processing of tokens drawn from a small set of alternatives. By analyzing the accumulation of errors in the attention mechanism, we theoretically derive a quantitative two-parameter relationship between the accuracy and the complexity of the task. We empirically verify our formula across a range of tasks and state-of-the art LLMs find excellent agreement between the predicted and observed accuracy in many cases. We also identify deviations in some cases that lead us to interesting insights about the functioning of models. We show how this understanding helps to construct prompts to reduce the error rate.

About the speaker: Suvrat Raju is an Indian physicist whose research focuses on quantum gravity and quantum field theory. Suvrat has worked on black holes, focusing on the ‘information paradox’ around them. He has formulated the Papadodimas-Raju proposal for black holes. Suvrat studied physics at St. Stephen’s college in Delhi University and went on to complete his PhD at the Harvard University. He is currently a professor at the International Centre for Theoretical Sciences of the Tata Institute of Fundamental Research (TIFR).