Prof. Judith Holler

Principal Investigator and group leader

I am Full Professor and Chair for Multimodal Communication in Social Interaction at the Donders Institute for Brain, Cognition & Behaviour, Radboud University, and affiliated researcher at Max Planck Institute for Psycholinguistics. My research program investigates human language in the very environment in which it has evolved, is acquired, and used most: face-to-face interaction. Within this context, my focus is on the semantics and pragmatics of human communication from a multimodal perspective considering spoken language within the rich, visual infrastructure that embeds it, such as manual gestures, head movements, facial signals, and gaze. The core question my research focuses on is how we encode, decode and align on meaning in such a multimodal environment, and how social cognition and interactional processes (e.g. pragmatic inferencing, communicative intentions, common ground, turn-taking, grounding, recipient design) shape multimodal language and the cognitive processes that underpin it. This will allow us to understand how meaning (from individual semantic constituents to speech acts) is produced and understood within the design space in which human communication typically operates and the constraints and affordances that result for psycholinguistic processing.

My research focus on situated psycholinguistics is based on an interdisciplinary approach combining the micro-analysis of multimodal language, CA-informed analyses of conversational interaction, psycholinguistics and neuroscience. In terms of methodology, it uses cutting-edge techniques (e.g. motion capture, Virtual Reality, dual-EEG, mobile eye-tracking), and is enriched by combining qualitative and quantitative language corpus analyses with controlled experimentation, including dialogic experimental paradigms for both production and comprehension.

Recently, I was awarded a European Research Council consolidator grant funding the current CoAct (Communication in Action) project which investigates the multimodal architecture of speech acts (in terms of the form and timing of the visual and verbal signals that constitute them) and their cognitive processing, the role of visual communicative signals in predictive language processing and turn-taking, as well as the cognitive and neural mechanisms that govern the binding and segregation of multimodal signals and gestalts.

Other current research lines focus on alignment in dialogue (including the recipient-designed signalling and collaborative construction of meaning, as well as the marking of information status) and on the role of the addressee in achieving mutual understanding and in shaping the speaker’s multimodal utterances. More recently, I have also begun new research lines that compare multimodal communication across different languages and species, as well as applying AI techniques to develop multimodal communication analysis tools, at the same time as applying insights from human-human interaction to interactions between humans and artificial agents.

Over the years, my research has been supported by a range of funders in addition to the European Research Council, including European Union Horizon Actions (Marie Curie Fellowship), Economic & Social Research Council (UK), Parkinson UK, The Leverhulme Trust (UK), the British Academy (UK), Volkswagen Stiftung (Germany) and the German Science Foundation (DFG, Mercator Fellowships).

My research group Communication in Social Interaction is based at the Max Planck Institute (https://www.mpi.nl/department/communication-social-interaction/18) as well as the Donders Institute for Brain, Cognition & Behaviour (https://www.ru.nl/donders/research/theme-1-language-communication/resea…).

team

The CoSI lab consists of post-docs, PhD and Masters students, interns, as well as research and technical assistants, all of whom are vital to the research we do.

Post-doc

Marlijn ter Bekke

My research focuses on how visual signals facilitate language processing during face-to-face conversation and intention communication in interaction with prosodic and linguistic cues and how this may differ across people. I use corpus analyses, behavioural experiments, electroencephalography, and virtual animation.

Affiliated researcher

Alexandra Emmendorfer

I worked as a post-doc in the CoSI lab until recently, investigating the role of visual signals (gaze and interactive gestures) in language comprehension and utterance planning with corpus studies, behavioural experiments and EEG studies, and am still collaborating on bringing the on-going studies to completion.

Masters student

Fay Han

I am a 2nd year CNS Masters student and am completing my thesis project in the CoSI lab, investigating the role of gaze in language processing during live conversational interaction.

Masters student

Erica Hsieh

I am a CNS Masters student and am currently doing a research internship in the CoSI lab, helping out with taking EEG measurements from people engaged in conversational interaction.

ViCom_17.11.2212414_Kadava-Sarka_edited.jpg

PhD student

Šárka Kadavá

My PhD is financed through the DFG Priority Programme "ViCom" and supervised by Susanne Fuchs (ZAS Berlin), Alexandra Ćwiek (ZAS Berlin), Wim Pouw (Radboud University and Judith Holler. My research focuses on how different modalities contribute to the creation of shared meaning, and on how language-like structures emerge in interaction.

PhD student

Emma Loefgren

In my thesis, I study the role of prosodic cues and visual signals in turn-taking during French face-to-face conversation. My PhD is supervised by Judith Holler, Francisco Torreira and Stephen Levinson and funded by the Max Planck Society.

PhD student

Olesya Moiseenko

I study how social interactions help shaping infants' attention. My PhD is supervised by Marlene Meyer, Sabine Hunnius, Max Hinne and Judith Holler and is funded by the Donders Institute for Brain, Cognition & Behaviour.

Affiliated researcher

Dan Mu

I am a PhD student at the University of Groningen, but still collaborating on an expansion of the study from my Maters thesis which I did in the CoSI lab, looking at the influence of eye gaze direction on language processing.

Masters student

Ella Nieuwenhuijsen

I am a CNS Masters student and I am completing my Capita Selecta project in the CoSI lab, looking at multimodal communication and physiological reactions.

PhD student

Milo Reinmöller

I investigate how interlocutors convey and process communicative intentions in face-to-face interactions, focusing on visual body signals (e.g., hand gestures, facial expressions) and their interaction with prosodic features. My PhD is funded by the NWO and supervised by Judith Holler and Marlijn ter Bekke and embedded in the Donders Centre for Cognition.

PhD student

Koen de Reus

I study communication in harbour seals, with a special focus on rhythm and vocal learning, and also visual signalling behaviours. My PhD is supervised by Andrea Ravignani, Bart de Boer and Judith Holler and is funded by the Belgian Science Foundation (FWO).

Student assistant

Karlijn van der Vleut

I am a CNS Master student and a student assistant in the CoSI lab, where I assist with a study investigating the influence of gaze direction in face-to-face conversation, as well as with annotating multimodal corpora.

Bachelor student

Lorin Wolf

I am a Bachelor student in Psychology and am completing my Honours thesis in the CoSI lab, investigating the influence of hand gestures on response planning.

Student assistant

Wout Zeelte

I am a Bachelor student in Linguistics and a student assistant on a NWO-funded project investigating the interplay between visual bodily signals, prosody and conversational structures. I assist with annotating multimodal corpora, especially for prosodic patterns, and will also be helping out with experiments.