Language allows people to communicate thoughts to each other, because everyone's brain responds to the meaning of words in the same way. In our recently published study, my colleagues and I developed a framework to model the brain activity of speakers as they engage in face-to-face conversations.
We recorded the electrical activity of the brains of two people while they were having unscripted conversations. Previous research has shown that when two people talk to each other, their brain activity becomes linked or aligned, and that the degree of neural coupling is associated with better understanding of the speaker's message.
A neural code refers to specific patterns of brain activity associated with different words in context. We found that speakers' brains were aligned to a shared neural code. Importantly, the brain's neural code resembled the artificial neural code of large language models, or LLMs.
The neural patterns of words
A large language model is a machine learning program that can generate text by predicting which words are likely to follow other words. Large language models are good at learning the structure of language, generating human-like text, and having conversations. They can even pass the Turing test, making it difficult for someone to tell whether they are interacting with a machine or a human. Like humans, LLMs learn to speak by reading or listening to text produced by other people.
By giving the LLM a transcript of the conversation, we were able to extract the "neural activations," or how it translates words into numbers as it "reads" the script. We then correlated the speaker's brain activity with both the LLM's activations and the listener's brain activity. We found that the LLM's activations could predict the shared brain activity of the speaker and listener.
In order to understand each other, humans share a common understanding of grammatical rules and the meaning of words in context. For example, we know that we should use the past tense of a verb to talk about past actions, as in the sentence "He visited the museum yesterday." Furthermore, we intuitively understand that the same word can have different meanings in different situations. For example, the word cold in the sentence "you are cold as ice" can refer to a person's body temperature or personality trait, depending on the context. Because of the complexity and richness of natural language, we lacked a precise mathematical model to describe it until the recent success of large language models.
Our research showed that large language models can predict how linguistic information is encoded in the human brain, providing a new tool for interpreting human brain activity. The similarity between the human brain's linguistic code and the large language model has allowed us for the first time to track how information is encoded into words in the speaker's brain and conveyed word by word to the listener's brain during face-to-face conversations. For example, we found that brain activity related to the meaning of a word emerges in the speaker's brain before a word is spoken, and that the same activity emerges again quickly in the listener's brain after the word is heard.
Powerful new tool
Our study has provided insights into the neural code for language processing in the human brain and how both humans and machines can use this code to communicate. We found that large language models were better at predicting shared brain activity compared to several features of language, such as syntax, or the order in which words are connected to form phrases and sentences. This is partly due to the LLM's ability to incorporate the contextual meaning of words and integrate multiple levels of the linguistic hierarchy into a single model: from words to sentences to conceptual meaning. This suggests important similarities between the brain and artificial neural networks.
A key aspect of our research is the use of everyday recordings of natural conversations to ensure that our findings capture real-life brain processing. This is called ecological validity. Unlike experiments where participants are told what to say, we give up control of the study and let participants converse as naturally as possible. This loss of control makes it difficult to analyze the data because each conversation is unique and involves two interacting individuals speaking spontaneously. Our ability to model neural activity as people engage in everyday conversations is a testament to the power of large language models.
Other dimensions
Now that we have developed a framework to assess the shared neural code between brains during everyday conversations, we are interested in what factors drive or hinder this coupling. For example, does linguistic coupling increase as a listener better understands the speaker's intent? Or perhaps complex language, such as jargon, can reduce neural coupling.
Another factor that can influence linguistic coupling is the relationship between the speakers. For example, you might be able to convey a lot of information in a few words to a close friend, but not to a stranger. Or you might be better neurally coupled to political allies than to rivals. This is because differences in the way we use words between groups can make it easier to adapt and become coupled with people inside rather than outside our social groups.
This article is republished from The Conversation, a nonprofit, independent news organization that brings you facts and reliable analysis to help you understand our complex world. It was written by: Zaid Zada, Princeton University Read more: Zaid Zada does not work for, consult, own stock in, or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond his academic appointment.