CC BY-NC-ND
23
Psychology. Journal of the Higher School of Economics.
2018. Vol. 15. N 2. P. 257-267. DOI: 10.17323/1813-8918-2018-2-257-267
THE ACQUISITION OF NEW WORD MEANING BY
AUDITORY-MOTOR ASSOCIATIONS IN A TRIAL-AND-ERROR LEARNING PARADIGM
B.V. CHERNYSHEVab, A.YU. NIKOLAEVAc, A.O. PROKOFYEVc, A.M. RAZORENOVAa, N.B. TYULENEVa, T.A. STROGANOVAc
"National Research University Higher School of Economics, 20 Myasnitskaya Str., Moscow, 101000, Russian Federation
b Lomonosov Moscow State University, 1 Leninskie Gory, Moscow, 119991, Russian Federation cMoscow State University of Psychology and Education, 29 Sretenka Str., Moscow, 127051, Russian Federation
Abstract
According to embodied cognition theory, speech is largely based on the body motor and sensory experience. The question that is crucial for our understanding of the origin of language is how our brain transforms sensory-motor experience into word meaning. We have developed an auditory-motor experimental procedure that allowed investigating neural underpinning of word meaning acquisition by way of associative "trial-and-error" learning that mimics important aspects of natural word learning. Participants were presented with eight pseudowords; four of them were assigned to specific body part movements during the course of learning — through commencing actions by one of a participant's left or right extremities and receiving a feedback. The other pseudowords did not require actions and thus were used as controls. Magnetoencephalogram was recorded during passive listening to the pseudowords before and after the learning. The cortical sources of the magnetic evoked responses were reconstructed using distributed source modeling. The learning of novel word meaning through word-action associations selectively increased neural specificity for these words in the auditory parabelt areas responsible for spectrotemporal analysis, as well as in articulatory areas, both located in the left hemisphere. The extent of neural changes was linked to the degree of language learning, specifically implicating the physiological contribution of the left perisylvian cortex in the speech learning success.
Keywords: embodied cognition, action words, learning, cortical plasticity, MEG, speech.
Introduction
One of the essential features of human language is that words are assigned meaning. Multiple evidence hints that natural language acquisition involves biological
This research is supported by the Russian Foundation for Basic Research, grant No 17-29-02168.
mechanisms of associative learning (Colunga & Smith, 2005; Preissler, 2008; Pulvermuller, 1999). Yet, considering an immense lexicon of natural human languages, such an amount of information stored in the brain poses a serious challenge for current learning and memory paradigms. The associative learning framework implies that the word referential meaning is implemented in the brain via associative Hebbian-type learning. So far, word-specific memory traces in the brain were formed during development in the process of mutual connection strengthening between different areas, as actions, objects or concepts are learnt when they are experienced in conjunction with the words used to describe them (Pulvermuller, 2005).
The ability to quickly acquire word-picture associations was shown to depend on the reorganization in neocortical networks including the left temporal area, especially the left temporal pole (Sharon, Moscovitch, & Gilboa, 2011), as well as temporoparietal, premotor, and prefrontal regions (Majerus et al., 2005; Mestres-Misse, Camara, Rodriguez-Fornells, Rotte, & Munte, 2008; Paulesu et al., 2009; Sharon et al., 2011).
Many psychophysiological studies of word meaning acquisition involved behavioral procedures such as word-picture associations (e.g. Sharon et al., 2011) — a procedure that has little resemblance to the circumstances of natural language acquisition. Moreover, it violates the situated nature of language processing (Myachykov, Scheepers, Fischer, & Kessler, 2014).
According to the embodied, or grounded, cognition theory, speech is largely based on the body motor and sensory experience (Barsalou, 2008). Additionally, action words proved to be a useful tool for psychophysiological study of word meaning (Pulvermuller, 2005; Shtyrov, Butorina, Nikolaeva, & Stroganova, 2014). Proceeding from this, we designed a behavioral paradigm that involves rapid word meaning acquisition. Specifically, the participants were presented with pseudo-words that acquired meaning of actions by way of associative "trial-and-error" learning. This procedure mimics important aspects of natural word learning and is most relevant to associative biological interpretation of word meaning acquisition (Colunga & Smith, 2005; Preissler, 2008; Pulvermuller, 1999). Using data-driven approach, we attempted to find time and location of significant events in the brain linked to acquisition of word meaning. We expected to find the effects in temporal and prefrontal areas of the left hemisphere (MacGregor, Pulvermuller, van Casteren, & Shtyrov, 2012; Sharon et al., 2011).
Methods
Twenty-eight adult Russian-speaking right-handed participants took part in the experiment.
Participants were presented binaurally with eight two-syllable pseudowords (Table 1); four of them were assigned to specific body part movements during the course of learning — through commencing actions by any of participant's left or right extremities and receiving an auditory feedback. The pseudowords that acquired meaning are referred below as "words". The other pseudowords — referred
below as "distractors" — did not require actions, and were used as controls to account for repetition suppression, which is typically observed when spoken words or pseudowords are presented repeatedly (Majerus et al., 2005; Paulesu et al., 2009). Experimental procedure in-
volved four sessions administered within one experimental day in the following order: Passive session 1, Active session 1 (learning), Active session 2 (stable performance), Passive session 2 (Figure 1).
Magnetoencephalogram was recorded using "VectorView" (Elekta Oy, Finland) during both passive sessions.
Root mean square (RMS) signal over gradiometers was separately analyzed for the left and the right sides of the head (Figure 2), for the two sessions (Passive session 1 vs. Passive session 2, i.e. before and after learning), for the two types of stimuli ("words" vs. "dis-
Table 1
Stimuli used in the experiment and the stimulus-to-response mapping
Stimulus Meaning (action)
"hicha" left hand
"hichu" no action (distractor)
"hisha" no action (distractor)
"hishu" left foot
"hisa" right foot
"hisu" no action (distractor)
"hiva" no action (distractor)
"hivu" right hand
Figure 1
Schematic illustration of the experimental procedure. Top panel - passive sessions, bottom panel - active sessions. ISI - interstimulus interval
tractors"). Statistical significance of the RMS response was assessed for the double difference:
double difference = (W1 - D1) - (W2 - D2),
where W1 and W2 stand for magnetic evoked responses to "words", and D1 and D2 — for magnetic evoked responses to "distractors" in Passive session 1 and Passive session 2, correspondingly. Since we expected that repetition suppression (Majerus et al., 2005; Paulesu et al., 2009) will be diminished for "words" compared with "distractors", we considered only positive significant values. Using t-tests with max-cluster-size permutation procedure (1000 permutation repetitions) to account for multiple comparisons, we determined the time intervals during which the double difference was statistically significant.
The cortical sources of the magnetic evoked responses were reconstructed using distributed source modeling (MNE software). We used unsigned source signal amplitude. Cortical areas were obtained as clusters of significant voxels (p < .05) exhibiting the same (positive) sign of effect as revealed by RMS analysis within specific time windows revealed by RMS analysis. For the clusters obtained, we calculated timecourses of brain activity averaged over all voxels of each cluster. We calculated the time intervals during which the double difference was significant, using the same permutational procedure as described above.
Additionally, we averaged magnetic evoked responses over space and time within clusters (over voxels and over significant time points), and calculated Pearson correlation with the behavioral measure of learning progress (the number of trials needed to acquire the word meaning).
Results
Behavioral data
All participants were successful on the task: average hit rate during the Active session 2 was 96.3 ± 3.8% (mean ± standard deviation).
Sensor level analysis
Brain responses to both types of stimuli underwent repetition suppression, yet the repetition suppression was stronger for "distractors" compared to "words" (Figure 3). Analysis of the RMS signal in the left hemisphere produced two statistically significant intervals: 230-280 ms and 465-515 ms after the disambiguation
Figure 2
Left and right sensors used for RMS analysis (grey and black asterisks, correspondingly)
point (Figure 3). Analysis of the RMS signal in the right hemisphere produced no statistically significant intervals (Figure 3).
Source level analysis
For the time interval 230-280 ms, the effect was found in two clusters of voxels, both in the left hemisphere, in perisylvian cortical regions (Figure 4). One was located in the superior temporal sulcus (STS) (extending dorsally into the superior temporal gyrus, STG). The other one included the frontal operculum and the anterior insula. As can be seen in Figure 4, the effect in both clusters was longer than the interval revealed by RMS analysis: the effect started around 200 ms after the disambiguation point and lasted up to 500 ms or longer.
For the time interval 465-515 ms, no significant clusters of voxels were found in any hemisphere.
Correlation analysis revealed that learning-related cortical activation in the temporal cluster was inversely related to the number of trials needed to acquire the word meaning (r = —.39, p = .04).
Discussion
The data obtained in the current study is compatible with the notion that natural language acquisition may involve biological mechanisms of associative learning (Colunga & Smith, 2005; Preissler, 2008; Pulvermuller, 1999).
Figure 3
Timecourses of the grand-average RMS signal. Vertical line denotes word disambiguation point ("0"), after which "words" differ from "distractors"
Note. Left panel — left hemisphere, right panel — right hemisphere. Black bars beneath timecourses indicate the time windows for which the double difference was significant (permutation statistics, р < 0.05). W1 and W2 — magnetic evoked responses to "words", and D1 and D2 — magnetic evoked responses to "distractors" in Passive session 1 and Passive session 2, correspondingly.
Figure 4
Significant spatial clusters on the left cortical surface (left panels) and timecourses (right panels) of learning-related activation within these clusters
Note. Top panels - superior temporal sulcus (STS), bottom panels - frontal operculum and insula. Conventions as in Figure 3.
Moreover, in the current study, acquisition of word meaning occurred during a single experiment within a time span of approximately two hours. Although there is evidence that acquisition of meaning of novel words requires at least one night of sleep needed for consolidation to occur (Davis & Gaskell, 2009; Gaskell & Dumay, 2003), some current behavioral and psychophysiological evidence suggests that meaning can be assigned to novel words very quickly - this phenomenon is often referred to as "fast mapping" (Borovsky, Kutas, & Elman, 2010; Dollaghan, 1985; Mestres-Misse et al., 2008; Mestres-Misse, Rodriguez-Fornells, & Munte, 2007; Shtyrov, 2012; Shtyrov, Nikulin, & Pulvermuller, 2010).
Our data demonstrated involvement of the auditory areas in the STS/STG (mostly parabelt areas), as well as the frontal operculum and the insula. Importantly, in the current study, the effect of word meaning acquisition was evident only in the left hemisphere.
Correlation between the brain activity and the learning rate in participants proves that the effect observed is indeed relevant to formation of a memory trace linking the acoustic pattern of the pseudowords to their meaning (i.e. actions).
Importantly, brain activity was measured before and after the learning, in passive conditions, when participants did not perform any actions in response to the pseudoword stimuli.
Neuroimaging research has demonstrated that comprehension of word meaning relies upon a widely distributed cortical network comprising temporo-parietal, pre-motor, and prefrontal regions of both hemispheres, as well as the left temporal pole (Majerus et al., 2005; Mestres-Misse et al., 2008; Paulesu et al., 2009; Sharon et al., 2011). Our findings differ from those mentioned above, and they form a very distinct pattern.
We found a significant effect in the middle part of the STS/STG that mostly includes the auditory parabelt areas responsible for spectrotemporal analysis (Hickok & Poeppel, 2007) and initial steps of word recognition (Scott & Johnsrude, 2003; Scott & Wise, 2004). Processing of a new word also activated the posterior opercular part of the inferior frontal gyrus (IFG) that is involved in subvocal rehearsal and articulatory coding of the perceived speech sounds (Hickok & Poeppel, 2007), this fact emphasizing the role of articulatory sensory-motor experience in acquisition of word meaning.
Thus, in sharp contrast to previously reported broad brain networks supporting natural word semantics, acquisition of an artificial new word was related to a rather selective activation of perisylvian structures of the left hemisphere. We did not detect any effects in the associative areas of the temporal and frontal cortices that are thought to mediate the stored word representations through their meaningful relations with the corresponding words (Pulvermuller, Moseley, Egorova, Shebani, & Boulenger, 2014).
The timing of a new-word effect on the evoked responses was also different from the earliest values reported for the well-learnt words of natural language. While some recent studies demonstrated that the brain discriminates real words from meaningless pseudowords as early as 50 ms (MacGregor et al., 2012), we found the new-word-pseudoword differential brain activity to occur at least 200 ms later in the course of word processing.
Spatial-temporal pattern of a new-word effect in the current study indicates that the neural processes that the brain engages to learn the unknown word are quite different from those involved in the decoding of well-known word semantics. Taken together, our findings imply that long-term effects of natural language usage may involve multiple consolidation/reconsolidation phases, and rooting the word meaning into one's sensory-motor experience is a necessary but not a sufficient prerequisite for its embedding into the associative structure of semantic memory.
Conclusions
Taken together, our findings show that learning of novel word meaning through word-action association selectively increased neural specificity for these words in the auditory parabelt areas responsible for spectrotemporal analysis, as well as in articulatory areas located in the left hemisphere. Importantly, this effect was detected in passive conditions after active learning, evidencing formation of a
robust memory trace. The extent of neural changes was linked to the degree of language learning, specifically implicating the physiological contribution of the left perisylvian cortex in the learning success.
Acknowledgements
We gratefully appreciate the invaluable contribution of A.A. Zhukova to data analysis.
References
Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617-645.
doi: 10.1146 / annurev.psych.59.103006.093639 Borovsky, A., Kutas, M., & Elman, J. (2010). Learning to use words: Event-related potentials index singleshot contextual word learning. Cognition, 116(2), 289-296. doi:10.1016/j.cognition.2010.05.004 Colunga, E., & Smith, L. B. (2005). From the lexicon to expectations about kinds: A role for associative
learning. Psychological Review, 112(2), 347-382. doi:10.1037/0033-295x.112.2.347 Davis, M. H., & Gaskell, M. G. (2009). A complementary systems account of word learning: neural and behavioural evidence. Philosophical Transactions of the Royal Society B-Biological Sciences, 364(1536), 3773-3800. doi:10.1098/rstb.2009.0111 Dollaghan, C. (1985). Child meets word: "fast mapping" in preschool children. Journal of Speech and
Hearing Research, 28(3), 449-454. Gaskell, M. G., & Dumay, N. (2003). Lexical competition and the acquisition of novel words.
Cognition, 89(2), 105-132. doi:10.1016/s0010-0277(03)00070-2 Hickok, G., & Poeppel, D. (2007). Opinion - The cortical organization of speech processing. Nature
Reviews Neuroscience, 8(5), 393-402. doi:10.1038/nrn2113 MacGregor, L. J., Pulvermuller, F., van Casteren, M., & Shtyrov, Y. (2012). Ultra-rapid access to words in the brain. Nature Communications, 3, 711. doi:10.1038/ncomms1715
Majerus, S., Van der Linden, M., Collette, F., Laureys, S., Poncelet, M., Degueldre, C.....Salmon, E.
(2005). Modulation of brain activity during phonological familiarization. Brain and Language, 92(3), 320-331. doi: 10.1016 /j.bandl.2004.07.003 Mestres-Misse, A., Camara, E., Rodriguez-Fornells, A., Rotte, M., & Munte, T. F. (2008). Functional neuroanatomy of meaning acquisition from context. Journal of Cognitive Neuroscience, 20(12), 2153-2166. doi:10.1162/jocn.2008.20150 Mestres-Misse, A., Rodriguez-Fornells, A., & Munte, T. F. (2007). Watching the brain during meaning
acquisition. Cerebral Cortex, 17(8), 1858-1866. doi:10.1093/cercor/bhl094 Myachykov, A., Scheepers, C., Fischer, M. H., & Kessler, K. (2014). TEST: A Tropic, Embodied, and Situated Theory of Cognition. Topics in Cognitive Science, 6(3), 442-460. doi:10.1111/tops.12024 Paulesu, E., Vallar, G., Berlingeri, M., Signorini, M., Vitali, P., Burani, C., ... Fazio, F. (2009). Supercalifragilisticexpialidocious: How the brain learns words never heard before. Neurolmage, 45(4), 1368-1377. doi:10.1016/j.neuroimage.2008.12.043 Preissler, M. A. (2008). Associative learning of pictures and words by low-functioning children with autism. Autism, 12(3), 231-248. doi:10.1177/1362361307088753
Pulvermuller, F. (1999). Words in the brain's language. Behavioral and Brain Sciences, 22(2), 253-279.
doi:10.1017/s0140525x9900182x Pulvermuller, F. (2005). Brain mechanisms linking language and action. Nature Reviews Neuroscience,
6(7), 576-582. doi:10.1038/nrn1706 Pulvermuller, F., Moseley, R. L., Egorova, N., Shebani, Z., & Boulenger, V. (2014). Motor cognition-motor semantics: Action perception theory of cognition and communication. Neuropsychologia, 55, 71-84. doi:10.1016/j.neuropsychologia.2013.12.002 Scott, S. K., & Johnsrude, I. S. (2003). The neuroanatomical and functional organization of speech
perception. Trends in Neurosciences, 26(2), 100-107. doi:10.1016/S0166-2236(02)00037-1 Scott, S. K., & Wise, R. J. S. (2004). The functional neuroanatomy of prelexical processing in speech
perception. Cognition, 92(1-2), 13-45. doi:10.1016/j.cognition.2002.12.002 Sharon, T., Moscovitch, M., & Gilboa, A. (2011). Rapid neocortical acquisition of long-term arbitrary associations independent of the hippocampus. Proceedings of the National Academy of Sciences of the United States of America, 108(3), 1146-1151. doi:10.1073/pnas.1005238108 Shtyrov, Y. (2012). Neural bases of rapid word learning. Neuroscientist, 18(4), 312-319.
doi:10.1177/1073858411420299 Shtyrov, Y., Butorina, A., Nikolaeva, A., & Stroganova, T. (2014). Automatic ultrarapid activation and inhibition of cortical motor systems in spoken word comprehension. Proceedings of the National Academy of Sciences of the United States of America, 111(18), E1918-E1923. doi:10.1073/pnas.1323158111 Shtyrov, Y., Nikulin, V. V., & Pulvermuller, F. (2010). Rapid cortical plasticity underlying novel word learning. Journal of Neuroscience, 30(50), 16864-16867. doi:10.1523/jneurosci.1376-10.2010
Boris V. Chernyshev — head of laboratory, Cognitive Psychophysiology Laboratory; associate professor, School of Psychology, National Research University Higher School of Economics; associate professor, Lomonosov Moscow State University, Ph.D.
Research area: neurophysiology, psychophysiology, attention, cognitive control, learning, reinforcement, speech. E-mail: bchernyshev@hse.ru
Anastasia Yu. Nikolaeva — research fellow, MEG-Center, Moscow State University of Psychology and Education.
Research area: developmental psychology, psychophysiology of speech perception. E-mail: anastasia.y.nikolaeva@gmail.com
Andrey O. Prokofyev — senior research fellow, MEG-Center, Moscow State University of Psychology and Education, Ph.D.
Research area: developmental psychology, neurophysiology, psychophysiology. E-mail: nzay2008@yandex.ru
Alexandra M. Razorenova — researcher assistant, Cognitive Psychophysiology Laboratory,
National Research University Higher School of Economics.
Research area: psychophysiology, cognitive control, speech, medical equipment.
E-mail: razoral@ya.ru
Nikita B. Tyulenev — researcher assistant, Cognitive Psychophysiology Laboratory, National Research University Higher School of Economics.
Research area: psychophysiology, cognitive control, speech, medical equipment. E-mail: tnb6@yandex.ru
Tatiana A. Stroganova — head of the MEG-Center, Moscow State University of Psychology and Education, D.Sc.
Research area: autism, MEG, speech perception, speech generation, visual perception. E-mail: stroganova56@mail.ru
Научение смыслу новых слов через слуховые-моторные ассоциации при реализации обучающей парадигмы проб-и-ошибок
Б.В. Чернышев",ь, А.Ю. Николаева0, А.О. Прокофьев0, А.М. Разоренова", Н.Б. Тюленев", Т.А. Строганова0
'Национальный исследовательский университет «Высшая школа экономики», 101000, Россия, Москва, ул. Мясницкая, д. 20
ЬМГУ имени М.В. Ломоносова, 119991, Россия, Москва, Ленинские горы, 1
сМосковский государственный психолого-педагогический университет, 127051, Россия, Москва, ул. Сретенка, д. 29
Резюме
В соответствии с теорией телесно-воплощенного познания, речь в значительной степени базируется на моторном и сенсорном опыте. Вопрос, критически важный для понимания природы речи, состоит в том, как наш мозг трансформирует сенсорно-моторный опыт в смысл, присваиваемый словам. Мы разработали слуховую моторную экспериментальную задачу, позволяющую исследовать мозговые механизмы научения смыслу слов путем ассоциативного научения по типу «проб-и-ошибок», имитирующего важные аспекты естественного научения речи. Участникам предъявляли восемь псевдослов; четыре псевдослова в ходе научения приобретали смысл и обозначали движения конкретными частями тела; процедура включала в себя выполнение испытуемыми движений правыми или левыми конечностями и получение ими обратной связи. В ответ на остальные псевдослова никаких действий выполнять не требовалось, и они служили в качестве контрольных стимулов. Магнитоэнцефалограмму регистрировали во время пассивного прослушивания слов до и после научения. Кортикальные источники магнитных вызванных ответов реконструировали с помощью модели распределенных источников. Научение смыслу новых слов путем ассоциаций между словами и действиями селективным образом усилило специфическое сродство к этим словам в области слухового парапояса, ответственной за спектрально-временной анализ, а также в артикуляционных областях, причем обе области были локализованы в левом полушарии. Выраженность изменений в мозговой активности коррелировала со скоростью речевого научения, что подчеркивает физиологический вклад левой перисильвиллярной коры в успешность речевого научения.
Ключевые слова: телесно-воплощенное познание; слова, обозначающие действия; научение, кортикальная пластичность, МЭГ, научение речи.
Чернышев Борис Владимирович — заведующий лабораторией, лаборатория когнитивной психофизиологии; доцент, департамент психологии, Национальный исследовательский университет «Высшая школа экономики»; доцент, кафедра высшей нервной деятельности, Московский государственный университет имени М.В. Ломоносова, кандидат биологических наук.
Сфера научных интересов: нейрофизиология, психофизиология, внимание, когнитивный контроль, обучение, подкрепление, речь. Контакты: bchernyshev@hse.ru
Николаева Анастасия Юрьевна — научный сотрудник, Центр нейрокогнитивных исследований «МЭГ-Центр», Московский государственный психолого-педагогический университет.
Сфера научных интересов: психология развития, физиологические основы восприятия речи. Контакты: anastasia.ynikolaeva@gmail.com
Прокофьев Андрей Олегович — старший научный сотрудник, Центр нейрокогнитивных исследований «МЭГ-Центр», Московский государственный психолого-педагогический университет, кандидат психологических наук.
Сфера научных интересов: психология развития, нейрофизиология, психофизиология. Контакты: nzay2008@yandex.ru
Разоренова Александра Михайловна — стажер-исследователь, лаборатория когнитивной психофизиологии, Национальный исследовательский университет «Высшая школа экономики».
Сфера научных интересов: психофизиология, когнитивный контроль, речь, медицинская техника.
Контакты: razoral@ya.ru
Тюленев Никита Борисович — стажер-исследователь, лаборатория когнитивной психофизиологии, Национальный исследовательский университет «Высшая школа экономики». Сфера научных интересов: психофизиология, когнитивный контроль, речь, медицинская техника.
Контакты: tnb6@yandex.ru
Строганова Татьяна Александровна — руководитель, Центр нейрокогнитивных исследований «МЭГ-Центр», Московский государственный психолого-педагогический университет, доктор биологических наук, профессор.
Сфера научных интересов: аутизм, МЭГ, восприятие речи, генерация речи, зрительное восприятие..
Контакты: stroganova56@mail.ru
