eye think lab

gaze coordination
When two people have a conversation about a picture, map, diagram or movie, how are their eye movements linked? We are interested in this question from a theoretical standpoint because it can tell us about the role of visual context in verbal communication. It has practical implications for how we coordinate visual attention while teaching, during joint activities in the workplace and in virtual collaboration across the internet.

The first paper laid the groundwork for the project by quantifying the gaze coordination between a speaker and a listener (Richardson & Dale, 2005). Subsequent papers looked at two people engaged in a live conversation, manipulating the degree of information that they shared (Richardson, Dale & Kirkham, 2007), or the degree to which they believed they were looking at the same scenes (Richardson, Dale, Tomlinson, 2009).
In the our first experiment, we recorded speakers talking about TV shows, and then played back their speech to listeners. Above you can see a short example clip of a speaker's eye movements (the crosshair) and a listener's eye movements (the blob). You will notice that the listener's eye movements appear to follow the speaker's.

Does the link between eye movements tell us anything about language? We tested listeners' memory for what the speaker had said. Watch the eye movements of a speaker, a listener with good recall and a listener with poor recall (from Experiment 1, Richardson & Dale, 2005)

What is the precise nature of the link between speaker and listener eye movements, and how does it change between listeners? We used a technique called cross recurrence analysis to answer this question (for an introduction, see Dale & Spivey 2005; Dale, Warlaumont & Richardson, 2011 ). The diagram to the left shows the recurrence, or overlap, between the speaker and listener's gaze at different time lags.

From the moment a speaker looked at a picture of a person, and for the following six seconds, a listener was more likely than chance to be looking at that same picture. The overlap between speaker and listener eye movements peaked at about two seconds. In other words, two seconds after the speaker looked at a cast member, the listener was most likely to be looking at the same cast member. The eye movement coupling between each speaker listener pair. This measure was correlated with the number of comprehension questions the listener answered correctly. In other words, there was a link between coordinating visual attention and understanding spoken language.

In our second experiment, we sought to show that there was a causal link between eye movements and language comprehension. A second set of listeners heard the speeches and watched the displays. In the simultaneous condition, the pictures of the cast members flashed when the speaker had looked at them. This flashing caused the listeners' eye movements to follow the speakers more closely . In the shuffled condition, the pictures flashed in a randomized order. Listeners in the simultaneous condition had eye movements that more closely followed the speakers. As a result, they answered the comprehension questions almost 40% faster. On the left you can see an excerpts the simultaneous condition condition used in Experiment 2, Richardson & Dale (2005).
We uncovered a close coupling between speaker and listener eye movements that plays a role in the success of the communication. But does this only hold for when a listener is passively attending to a speaker's monologue? What about a real, interactive conversation, where both participants can speak, interrupt each other, and disagree. Will there still be a close coupling of eye movements?

n the next set of studies we recorded the gaze of two participants engaged in a live, spontaneous dialog. As before, participants talked about Friends or The Simpsons while looking at pictures of the cast. Cross-recurrence analysis revealed that once more, the participants' eye movements were coupled. As you can see in the figure below, when both participants can speak, the recurrence peaks a 0ms, with no log between them. This means that in a dialogue, conversants are most likely to be looking at the same thing at the same time.
How do conversants synchronize their gaze in this manner? One factor could the the 'common ground' (Clark, 1996) that they share. Common ground is the knowledge, beliefs and assumptions that are shared between people. We investigated whether increasing the amount of common ground between conversants would bring their gaze patterns closer together. We showed people a painting by Dali, and asked them to discuss it. Prior to their conversation, they either heard two different encyclopaedia entries about Dali (one concerning his painting, one concerning his life) or the both heard the same entry. When it came it discussing the painting, we found that eye movement couplings increased by over a third when the conversants had the same knowledge in common ground.

In later work, we found that conversants are also sensitive to what they believe each can see. In one experiment we manipulated the fact that a visual scene was shared or not and the belief that a visual scene was shared or not. Participants watched videos of actors talking about a controversial topic, then discussed their own views while looking at either a blank screen or the actors. Each believed (correctly or not) that their partner was either looking at a blank screen or the same images. We recorded conversants’ eye movements, quantified how they were coordinated, and analysed their speech patterns. We found that both the presence of the visual scene, and beliefs about its presence for another, influenced language use and gaze coordination.
In our most recent work, we examined the gaze coordination between two people playing puzzle game: the tangram task. They are each presented with the same set of abstract shapes, laid out in different orders on a computer screen. One person instructs the other to arrange their shapes so that the orders match. To do this, they must find a way to refer to the abstract shapes.Typically they say things like, 'the guy who looks like a skier". You can watch a split screen video of two people playing the tangram game. After playing several rounds of this game, we found that people became faster at completing the task. Also, their eye-movements synchronized over time. Cross-recurrence analysis was used to quantify this coordination, and showed that as participants’ words coalesced, their actions approximated a single coordinated system.
Read a full account of our gaze coordination experiments so far in:

Richardson, D.C & Dale, R. (2005). Looking to understand: The coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension Cognitive Science, 29, 1045–1060.

Richardson, D.C., Dale, R. & Kirkham, N.Z. (2007) The art of conversation is coordination: Common ground and the coupling of eye movements during dialogue Psychological Science, 18 (5), 407-413
(This was an Editor's Choice in the May 23rd edition of Science)

Richardson, D.C., Dale, R. & Tomlinson Jr, J.M. (2009) Conversation, gaze coordination and beliefs about visual context. Cognitive Science

Dale, R., Kirkham. N.Z., & Richardson D.C. (2011). The Dynamics of Reference and Shared Visual Attention, Frontiers in Cognition.