Reference Material

Research in the MACLab makes use of a lot of advanced techniques, and interesting linguistic findings. This page provides background information on many of these things.

Eye-tracking


The Visual World Paradigm

Language unfolds over time--it can take a couple hundred millesecond to hear a complete word, and a couple of seconds to hear all of the material in a sentence. But listeners don't wait! As they are hearing words and sentences they are already forming inferences about what they are hearing. These inferences are not always right (e.g. the famous garden-path sentences) and they're typically only partial committments. But understanding these inferences as they form moment-by-moment provides a critical window on language processing.

However, doing so is hard. You need to somehow measure what people are thinking at a millisecond time-scale, without breaking up the input, or interfering with their normal processing. The Visual World Paradigm allows you to do exactly that. The VWP (Cooper, 1974; Tanenhaus et al., 1995) has been in use for the last 10 years as an important tool for studying language comprehension as it happens. This paradigm combines natural, meaningful language with a simple task in a visual context.

Subjects are presented with auditory instructions to manipulate one or more objects or images on a computer screen. The set of objects represents possible momentary interpretations of the stimulus at referential (Chambers, Tanenhaus, Eberhard, Filip & Carlson, 2002), syntactic (Tanenhaus et al., 1995), lexical (Allopenna et al., 1998) or sublexical (McMurray et al, 2003) levels of analysis. Eye-movements are monitored while subjects follow the instructions and reveal partial interpretations of the stimulus before they make their ultimate response (the action). For example, Allopenna et al. (1998) presented subjects with screens containing a target (e.g. sandal), cohort (sandwich), rhyme (candle), and unrelated object (necklace). Subjects heard an auditory target, and selected it with a mouse while fixations to each object were recorded. Initial fixations were equally likely to be directed to the target and cohort (since disambiguating information had not yet been heard). Later fixations were more likely to the rhyme than unrelated items (by then, subjects had heard material compatible with the rhyme). Importantly, 97% of the variance in the pattern of fixations was accounted for by dynamic activation from TRACE (McClelland & Elman, 1986), when activation was converted to fixation probability by a simple linking hypothesis (converting activation of all of the words in the lexicon to fixations to one of four objects on the screen). Subsequent work demonstrates that the VWP replicates basic findings in word recognition including effects of frequency (Dahan, Magnuson & Tanenhaus, 2001), neighborhood density, (Magnuson, Tanenhaus, Aslin & Dahan, 2003), and coarticulatory mismatch (Dahan et al., 2001b). Thus, fixations to visual competitors provide a real-time window on the dynamics of lexical activation.

The VWP offers some advantages over other methods. The task is easy and based on meaning. Subjects don't have to think about language--they just have to use it. Subjects are not typically aware of their eye-movements, so it's largely implicit. Eye-movements are fast and often initiated prior to the end of the auditory stimuli, allowing us to look at early points in processing. Moreover, eye-movements are roughly time-locked to language processing—an eye-movement only reflects the processing that has occurred up to that point. The VWP also provides a secondary measure: the ultimate response, (analogous to word or phoneme identification) which can be used to validate stimuli, as an exclusionary measure for subjects, and when combined with the fixations, to provide more conservative tests of hypotheses. Thus, while there are serious critiques of this paradigm (which will be addressed shortly), the VWP is ideal for answering the questions posed here in that it provides an implicit, graded measure of real-time lexical dynamics that is tied to a natural, meaning-based task.

Three features of the VWP (the implicit measurements of lexical activation; the fine-grained measurements of temporal dynamics; and the accessibility of the paradigm for special populations.) are fundamental to work in the MACLab. Virtually every theoretically important effect in spoken word recognition has been demonstrated with the VWP, including cohort effects (Allopenna et al, 1998; Dahan et al, 2001); frequency effects (Dahan et al, 2001; Magnuson et al, 2003); neighborhood density (Magnuson et al, 2003); subphonetic sensitivity (McMurray et al, 2002) and coarticulatory mismatch (Dahan et al, 2001b), and fits between eye-movement data and the TRACE model of word recognition (McClelland & Elman, 1986) are quite good (Allopenna et al, 1998; Dahan et al, 2001). This is an excellent way to measure word-recognition. Moreover, it is also an implicit, meaning-based measure. McMurray et al, (submitted) directly assess this, comparing sensitivity to VOT across a lexical (picture identification) task and an explicit phoneme decision task with the same stimuli. Here, much more sensitivity to VOT was seen in the more natural picture identification task suggesting that there are important differences between meaning-based word recognition and explicit phoneme judgments. Likewise, Spivey and Marian (1999) report that Russian/English bilinguals briefly fixate a cross-language cohort (e.g. the Russian fishku [gamepiece] after hearing fish)—it is unlikely that this is an explicit, conscious process. Finally, Yee and Sedivy’s (2006) demonstration that subjects fixate semantic competitors during online word recognition, cements the VWP as a task that is sensitive to meaning and interpretation.

Temporal sensitivity has been apparent in work using the VWP from the beginning. For example, Allopenna et al, show that fixations to the target and cohort diverge from unrelated objects approximately 200 ms after the onset of the word (it takes approximately 200 ms to plan an launch an eye-movement), and the target deviates from the cohort approximately 200 ms after the point of disambiguation. Ongoing work in the MACLab has shown an even finer grain of temporal sensitivity. Effects of VOT and vowel length are found to appear approximately 200 ms apart in the fixation record (the vowel was 200 ms long) (Clayards & McMurray, 2006), and reliable effects of assimilated consonants can be seen in the eye-movement record, despite these cues lasting for less than 100 ms (Gow & McMurray, in press).

Finally, the VWP offers a simple, natural task that avoids metalinguistic decisions and has low memory demands. Thus, it has been used with a variety of populations including children as young as 3 (Arnold, Novick, Brown-Schmidt & Trueswell, 2001), dyslexics (Desroches et al., 2006), autistic children (Campana, Silverman, Tanenhaus, Bennetto & Packard, 2005), and aphasics (Yee, Blumstein & Sedivy, 2004). Thus, it is ideal for use with SLI. Moreover, our own work with with SLI subjects suggests it is an easy task for them: in a task based on Allopenna et al (1998), the impaired subjects (SLI, SCI, NLI) averaged more than 98% correct and RTs differed from normals by only 100 ms (Mnor-mal=1429 ms; Mimpaired=1526 ms). Thus, the VWP is an excellent task for use with impaired subjects. Moreover, by offering a task that they do well at (while providing a measure of how the accomplish the task), it may offer a complementary window to other measures.

References

Eye-tracking with Infants


We can and have learned a lot of information by observing how long babies look at and listen to stimuli over the years. However, these studies have largely been based on gross measures of looking: is the infant looking at the stimulus at all (and for how long). In recent years, however, remote eye trackers allow us to assess eye movements while infants are looking at stimuli. This allows us to not only learn how long they are looking at the stimuli, but also to learn about what particular things they are looking at. We can also ask when they make these fixations which can tell us if they are anticipating events or reacting to them.

The eye tracker consists of a small infrared camera that obtains an image of the eye and locates the pupil and corneal reflection. This works in the same way as the adult eye-tracker, only the camera is mounted remotely from the subject. The camera is located directly under a large monitor which shows the stimuli the baby will be looking at. Of course, babies can't be counted on to keep their heads still. Thus, the camera is mounted on small servo-motors which allow it to move around to keep track of the eye. In addition, infants wear a cotton hat or headband on their head with a small sensor attached to it.This sensor is part of a magnetic head tracker which aids the eye-tracker by automatically finding the eye after it has moved out of the camera’s field of view. After going through all of this, we get a detailed video showing us exactly what the infant is looking at 30 times / second.

Participants are tested in a small experimental room and are seated on their parents’ lap a few feet away from the plasma monitor. A computer presents the stimuli on the large monitor. The experimenter then begins a short process of calibrating the eye-tracker to the participant. To calibrate each participant’s eye movements, the infants are shown various stimuli such as movies or colorful shapes in order to keep them interested while the experimenter manually moves the camera to locate the participant’s eye. Next, the head tracker is calibrated at this time to store the distance between the eye and the head tracker sensor on the sweatband. This allows it to take over control of the camera. Finally, the infant is directed to look at a number of locations on the monitor while the camera to takes a snapshot of the eye at each known location. It can then use this data to determine where it is looking throughout the session. This entire calibration process generally takes less than one minute to complete. Once that is complete, we move onto the particular experiment that we are working on!

Click here to see an example of an infant’s eye movements while participating in Comparison and Animal Categories.

Papers

Aslin, R.N., and McMurray, B. (2004). Where babies look: An overview of methods for assessing visual fixations and eye movements in young infants. Infancy, 6(2), 155-163.


The Head Turn Preference Procedure


The head turn preference procedure is a conjugate reinforcement technique used to collect behavioral data from infant subjects. It involves teaching the infant that when they turn their head in a certain way, usually to face a visual stimuli, an auditory stimulus will place. This way the infant controls what he/she listens too.

In our lab we use two types of the head turn preference procedure. The first involves the infant seated on their parents lap in front of a large TV display. When the experiment begins a large visual stimulus is presented on the TV display (in our case a bulls eye) as long as the infant is looking at the screen an auditory stimulus will play, but if the infant looks away from the screen for more than two seconds the auditory stimuli stops, and the screen goes blank.

The other type of head turn procedure we use involves one white blinking light situated in front of the infant and two red blinking lights situated on either side of the infant. In this procedure the white light will blink until the infant orients his or her head toward it, then extinguish and one of the red lights will begin to blink. Once the infant orients to this red blinking light the auditory stimulus will begin to play. Like before if the infants look away from the red blinking light for more than 2 seconds the auditory stimulus stops and the white light begins to blink starting a new trial.

Habituation Techniques


The Head turn preference procedure is a great way to assess infants preferences for differing stimuli, their ability to categorize, and what they can and can not discriminate. However this procedure assumes that they have some pre-existing preference or knowledge about stimuli or a category of stimuli. In order to teach them something brand new another technique is required.

Habituation is a form of learning which can be conscious or unconscious, and is measured by the gradual reduction in an observed behavioral response. Once the behavioral response has reached a certain level, usually some percentage of the average response to the first three habituation trials, the subject is said to have habituated. For example, say we placed an infant in front of a television screen and display on that screen images of different trees over and over again. By recording the amount of time the infant looks at the screen while the trees are present we would observe that over time the infant looks less and less at the trees, presumably getting bored. Once their looking time has reached threshold they have habituated, and we have taught them a category for trees! Now we can show them something else, like a bush and see if their looking time increases. If it does, then the infant has dis-habituated and thinks that this bush is something new and different, if the infant doesn’t perk up then they are grouping bushes in with boring old trees.

The Anticipatory Eye Movement Paradigm


Techniques like habituation and the head turn preference procedure have been extremely successful over the last few years. We've learned an immense amount of knowledge about infants perceptual, cognitive, and language abilities using them and it's hard to believe how much researchers have managed to glean with these simple techniques alone. However, ultimately they are both one-alternative tasks. As an experimenter, you can ask whether a stimulus was like the training/familiarization-set or not. Of course, people have been pretty clever in designing both the test stimulus and the familiarization-set and we've learned a lot. But sometimes, a one-alternative task is not enough.

Consider, for example, an experiment you might do to understand infants concepts about dogs. Using habituation, it would be easy to ask if the picture on the left is a member of the the dog on the right's category. You just habituate the baby to the poodle and then present the pug. If the baby dishabituates (perks-up) it must be novel, if not it must be a member of the category.


However, babies don't necessarily know all that much about dogs yet (if they did why do the study?). A baby could continue to habituate (in effect, saying they are the same category) because they are both dogs, or because they are both mammals, or because they both have lots of curvy edges. On the other hand, it might dihabituate (say that they are different) because they are different breeds, different colors, or different aspect ratios. How can the experimenter set the criteria for what counts as different, if the baby doesn't know anything?


The answer is to use a two alternative task. In the example above, it's quite clear that the correct criteria is to use the breed of dog and the correct answer is that the dog is a pug. Habituation and the head-turn procedure simply can't give us this type of answer.

The anticipatory eye-movement procedure (or AntiEM), on the other hand, was designed to do exactly that. In this procedure infants are trained to make any eye-movement to the left in response to one category and an eye-movement to the right in response to the other category. After training, we can assess how infants categorize a variety of stimuli by presenting something new and monitoring where they look.

Training is based on infants' natural proclivities to anticipate where an object is going to be. The work of Claes Von Hofsten, Marshall Haith, and Scott Johnson has all demonstrated that by around 2-4 months, infants are quite good at anticipating objects behaviors. For example, if an object moves behind an occludor, infants will make an eye-movement to the other side before it comes out. The AntiEM procedure makes use of this natural behavior by training infants that an object's trajectory (under an occludor) is related to its category.

In the example on the right, the infant sees a cross appear and then move up and under the blue occludor. Shortly thereafter it emerges on the left side of the screen (click here for an example). On other trials, the infant may see a circle appear and move up and under the occludor. It will reappear on the right. After about 20-30 trials, many infants catch on (adults do so in about 10-15) and will begin making eye-movements to the correct location before the object reappears. Of course these eye-movements are quick and hard to catch, so automated infant eye-tracking is essential.

Once the babies have learned to categorize these initial stimuli, we can present new stimuli and see how they are categorized. We might ask, for example, if color is an integral part of shape categories by presenting them with an orange cross and seeing if they still anticipate in the correct location. We can ask if infants prefer to make shape or color categories by presenting them with a red circle and seeing if they go left (color) or right (shape). The range of questions are endless.

Using this technique we have examined basic visual properties like shape, color and orientations. We've also looked at face perception and are even adapting it to look at short-term memory. It can also be used to look at speech perception (click here for an example. To do this, a small circle (or smiley face--babies love that) comes out on the screen and the infant hears a word. In this case, the identity of the word predicts the side that the object will emerge on.

The AEM paradigm has a number of advantages beyond the ability to ask two-alternative questions. First, you can get a lot of data for each subject (some subjects will give you 20-30 datapoints), compared to the habituation technique (where 3 is the average). This allows you to ask questions that require many different stimulus variations (e.g. to look at two or more factors simultaneously). Second, its flexible and can be adapted to ask just about any questions. Third, you can ask questions about the categories that infants have without having to train them on the entire category. For example, if you wanted to ask questions about an infants dog category using habituation you would have to habituate them to a large number of dogs in order to see any category-like behavior (e.g. generalization). This of course, may teach infants what a dog is in the course of the experiment. However, with AntiEM, you can train infants on a single dog (and a single cat) and potentially see interesting generalization, thus avoiding this problem.

Habituation and the head-turn technique have gotten us along way in the last 50 years and they are still important tools in the MACLab for understanding development. AntiEM gives us one more way to a glimpse at the strange and fascinating mind of the infant.

Papers

The Switch Task


The switch task is a technique used in conjunction with habituation. In this paradigm the infant is firt presented with numerous pairings of different stimuli until they habituate. Once habituated the subject enters the test phase where they recieve three trials.

One trial consists of a stimulus pairing they were exposed to during habituation, this is called the "same trial". If the subject did indeed learn this pairing they should continue to habituate. Another trial consists of two stimuli that the subject was exposed to during habituation but which have never been paired together, this is called the "switch" task. If they infant had successfully learned the pairings during habituation, this new pairing should be suprising and the infant should dishabituate.

The last type of test trial is the "novel" trial, in this trial infants are presented with two stimuli that they did not encounter during habituation. This trial is a control trial, the infants should dishabituate to this trial since it is completely novel, if they do not something may be wrong with the experiment.




Site maintained by Bob McMurray, Cheyenne Munson, and Joe Toscano.