We know from our experience of looking into each other's eyes that the appearance of an eye conveys important visual information. Research scientist Ko Nishino and Professor Shree K. Nayar at Columbia University have developed a novel computer algorithm for extracting detailed visual information from the appearance of an eye in an image.

In their work, Nishino and Nayar have shown that, given a picture of a person, one can determine from an eye of the person a wide-angle view of the world around the person. Even more interesting is that one can determine exactly what the person is looking at. In other words, one can obtain an estimate of the image falling on the retina of the person.

This new result has broad implications. For instance, the image of the world computed from an eye tells us the location and the details of how the picture was taken. Such information can be of great value to historians and journalists who can go back to high quality archived photographs to know more about the circumstances under which pictures of important people were captured. When the quality of the images is very high (many pixels in an eye), one may be able to use the same tools for security applications, such as determining from a picture of a sought-after individual their whereabouts.

From a technological viewpoint, the method developed by Nishino and Nayar can also lead to the development of advanced human-machine interfaces. Knowing where a person is looking and what they are looking at can make it easier to interact with a desktop PC or to teach a robot to perform complex tasks without the need for cumbersome programming. In some cases of paralysis, the only control the patient has is the motion of their eyes. In such cases, the newly developed tools can make it easier for the patient to communicate in order to use a variety of devices.

In related work, Nishino and Nayar have shown how the information within an eye can be used in computer graphics. The extracted wide-angle view of the person's environment also tells us the complex lighting conditions of the scene within which the person's picture was taken. Therefore, the eye can be used as a lighting probe. Nishino and Nayar have used lighting determined in this manner for various computer graphics applications. For instance, they have been able to take images from old movies, "recover" the lighting from the eyes of the actors in the images, and use this information about the lighting to replace the faces of the original actors with other actors, while keeping the appearances of the newly inserted actors consistent with the rest of the image.

The information produced by the new techniques of Nishino and Nayar can also be used in human affect studies, where researchers are interested in understanding the relationship between a person's reaction and what they are reacting to. The developed algorithms enable one to take a normal photograph of a person and view the world around the person from his/her point of view. Knowing what a person is looking at may bring us one step closer to understanding what the person may be feeling or thinking. >from *The World In Eyes*. COMPUTER VISION LABORATORY DEPT. OF COMPUTER SCIENCE
COLUMBIA UNIVERSITY

related context
> vision and art: how artists can manipulate the human visual system. 'neuroscientist revealed some of the science behind human visual perception of art.' february 20, 2003
> eye gaze: implications for new-age technology. 'noting that the eyes have long been described as mirrors of the soul, a computer scientist is studying the effect of eye gaze on conversation and the implications for new-age technologies.' december 4, 2002

imago
> everything her eyes see