Wearable Computing Project

Our vision of a wearable computing device is an intelligent assistant which is always with you and helps you to solve your every day tasks. An important aspect of such a device is that it can perceive the world from a first-person perspective: a wearable computer can see what you see and hear what you hear in order to analyze, model and recognize things and people which are around you. As an example we present a system which can recognize objects in your visual field of view in realtime and display information which you have associated with it.
To date, personal computers have not lived up to their name. Most machines sit on the desk and interact with their owners for only a small fraction of the day. Smaller and faster notebook computers have made mobility less of an issue, but the same staid user paradigm persists. Wearable computing hopes to shatter this myth of how a computer should be used. A person's computer should be worn, much as eyeglasses or clothing are worn, and interact with the user based on the context of the situation. With heads-up displays, unobtrusive input devices, personal wireless local area networks, and a host of other context sensing and communication tools, the wearable computer can act as an intelligent assistant.

Sensor Augmented Computing

A major challenge for wearable computing is to model and recognize the context of the user and the situation. This context information is a key aspect in order to achieve seamless interaction with the user. At Orbit 2000 we present a system which uses a head-mounted camera to record and analyze the visual environment of the user. In particular we present a computer vision program which recognizes objects in the visual field of view of the user in real-time. The system is based on a sound statistical Bayesian framework for object modeling and recognition.

An example application: Museum-Gallery's Guide - or which painting was this again?

An application of the sensory augmented computing device is the museum-gallery guide. A Museum is a rich visual environment and is often accompanied with facts and details (from a guide, text or web-page) to be associated with the paintings. For example, as you walk around in a museum you can record video clips of a guide's explanation of the paintings. Such video clips can then be associated with the painting itself so that every time you and the wearable system see the painting again the associated video-clip is replayed.

Since the sensory augmented system recognizes the paintings one can use the system also to index into the database of the museum for example by retrieving more information about the particular painting, finding similar paintings in the gallery or showing other paintings from the same painter.

Publications:

• An Interactive Computer Vision System - DyPERS: Dynamic Personal Enhanced Reality System. Bernt Schiele, Nuria Oliver, Tony Jebara and Alex Pentland. In ICVS'99 International Conference on Vision Systems.
• Recognition without Correspondence using Multidimensional Receptive Field Histograms. Bernt Schiele and James L. Crowley. IJCV - International Journal of Computer Vision. 36(1), p 31-50, January 2000
• Situation Aware Computing with Wearable Computers. Bernt Schiele, Thad Starner, Brad Rhodes, Brian Clarkson and Alex Pentland. Chapter in forthcoming book Augmented Reality and Wearable Computers , W. Barfield and T. Caudell (editors), Lawrence Erlbaum Press.
• Visual Context Awareness via Wearable Computing. Thad Starner, Bernt Schiele and Alex Pentland. In ISWC'98 International Symposium on Wearable Computers.
• Recognizing Places using Image Sequences. Hisashi Aoki, Bernt Schiele and Alex Pentland. In PUI'98 Perceptual User Interfaces.
• Augmented Realities Integrating User and Physical Model. Thad Starner, Bernt Schiele, Bradley Rhodes, Tony Jebara, Nuria Oliver, Josh Weaver and Alex Pentland. In IWAR'98 International Workshop on Augmented Reality.