CogVis

Cognitive Vision Systems

The objective of this project is to provide the methods and techniques that enable construction of vision systems that can perform task oriented categorization and recognition of objects and events in the context of an embodied agent. The functionality will enable construction of mobile agents that can interpret the action of humans and interact with the environment for tasks such as fetch and delivery of objects in a realistic domestic setting.

Our group will be responsible for the development of robust and reliable object recognition and categorization strategies that work under real-world conditions. In particular we will investigate the following topics:

• Dynamic multi-cue recognition
• Recognition of spatio-temporal structures and relations
• Learning the categorization of natural objects
• Statistical modelling of objects and events

Partners of the project are Henrik Christensen & Jan-Olof Eklundh (KTH, Sweden), Heinrich Buelthoff (MPI, Germany), David Hogg (Leeds, UK), Giulio Sandini (DIST, Italy), and Bernd Neumann (Hamburg, Germany). See the proposal for a more detailed description of the project vision and of all the project parts.

Here at PCCV, we will focus on the following project parts:

Categorization & Recognition of Structures, Events, and Objects

Recognition has traditionally been based on highly specific models, and in this context recognition is truly RE-cognition, i.e. priming of a specific prior defined model. Today, recognition systems have been developed and tested mainly with very specific and restricted object sets and very stereotyped dynamic events, and even then the robustness is often challenged when recognition of these objects is tested in a natural environment. There is thus a need for new methods that enable recognition in the context of unrestricted sets of objects and events. This holds even more for object categorization (is this an: animal, vehicle, furniture, ... ) for which virtually no working computational approaches exist, but which can easily be performed by humans. The same holds for dynamic events where a sufficiently general computational model of gesture recognition is still lacking.

In order to build a cognitive vision system that can successfully perform recognition and categorization, it is important to have a firm understanding of how these processes are carried out by the human visual system. Based on psychophysical experiments performed by our partners at MPIK, we will design an active recognition and categorization system based on a dynamic multi-cue strategy and compare human and machine recognition and categorization performance. In addition, we will investigate how spatial and temporal relations that form the basis features of the early stages of the cognitive system can be extracted from images.

Learning and Adaptation

In order to categorize objects or reason about events and scenes, the ability to learn from experience and adapt must be at the core of any cognitive vision system. We require that learning in a CogVis system be:

• Continuous. The system must be "on" and learning all the time. There should be no artificial distinction between "training" and "testing" modes. This requirement forces a rethinking of the organization of memory and the process that extracts "regularities" from ongoing experience.
  
• Unsupervised/Explaratory. The agent must pro-actively explore the environment and learn in an unsupervised mode. Even when humans are engaging the system, it is unrealistic to expect a group of examples pre-tagged with True/False labels for every concept that is taught.
  
• Staggered (or phased). There is an enormous amount of implicit/explicit knowledge (in very different representations) to be learned. One way to handle such complexity is to divide the things that need to be learned into different stages, with each stage enabling the learning of the next stage.

Building on the cognitive research described in the previous section, we will develop novel algorithms that are capable of extracting categories out of a large and potentially growing number of natural objects. For this, efficient learning methods will play a crucial role. The activity will be focused on a set of common household objects, and different learning algorithms will be evaluated using the same set of objects and the same categories.

more

Publications:

• Analyzing Appearance and Contour Based Methods for Object Categorization. Bastian Leibe and Bernt Schiele, In International Conference on Computer Vision and Pattern Recognition (CVPR'03), Madison, Wisconsin, June 2003.
• Saliency of Interest Points under Scale Changes. Daniela Hall, Bastian Leibe, and Bernt Schiele, In British Machine Vision Conference 2002, Cardiff, Wales, September 2002.
• How Many Classifiers Do I Need? Bernt Schiele. In International Conference on Pattern Recognition ICPR 2002, Quebec, Canada, August 2002.

Contact:

Bastian Leibe (leibe@inf.ethz.ch)

Bernt Schiele (schiele@inf.ethz.ch)

---