Vision for robotics

Foundations and Trends R in Robotics Vol. 1, No. 1 (2010) 1–78 c 2009 D. Kragic and M. Vincze DOI: 10.1561/2300000001

Vision for Robotics
By Danica Kragic and Markus Vincze

Contents

1 Introduction 1.1 Scope and Outline

2 4 7 7 9 12 17 18 27 32 35 42 44 48 49 52

2 Historical Perspective 2.1 2.2 2.3 Early Start and Industrial Applications Biological Influences and Affordances VisionSystems

3 What Works 3.1 3.2 3.3 3.4 3.5 3.6 Object Tracking and Pose Estimation Visual Servoing–Arms and Platforms Reconstruction, Localization, Navigation, and Visual SLAM Object Recognition Action Recognition, Detecting, and Tracking Humans Search and Attention

4 Open Challenges 4.1 4.2 Shape and Structure for Object Detection Object Categorization

4.3 4.4

Semantics and SymbolGrounding: From Robot Task to Grasping and HRI Competitions and Benchmarking

54 56 59 64 65

5 Discussion and Conclusion Acknowledgments References

Foundations and Trends R in Robotics Vol. 1, No. 1 (2010) 1–78 c 2009 D. Kragic and M. Vincze DOI: 10.1561/2300000001

Vision for Robotics
Danica Kragic1 and Markus Vincze2
1

2

Centre for Autonomous Systems, Computational Vision andActive Perception Lab, School of Computer Science and Communication, KTH, Stockholm, 10044, Sweden, dani@kth.se Vision for Robotics Lab, Automation and Control Institute, Technische Universitat Wien, Vienna, Austria, vincze@acin.tuwien.ac.at

Abstract
Robot vision refers to the capability of a robot to visually perceive the environment and use this information for execution of various tasks. Visualfeedback has been used extensively for robot navigation and obstacle avoidance. In the recent years, there are also examples that include interaction with people and manipulation of objects. In this paper, we review some of the work that goes beyond of using artificial landmarks and fiducial markers for the purpose of implementing visionbased control in robots. We discuss different application areas,both from the systems perspective and individual problems such as object tracking and recognition.

1
Introduction

For many living species, not least in the case of humans, visual perception plays a key role in their behavior. Hand–eye coordination ability gives us flexibility, dexterity, and robustness of movement that no machine can match yet. To locate and identify static, as well asmoving objects, to determine how to grasp and handle them, we often rely strongly on our visual sense. One of the important factors is our ability to track objects, that is, to maintain an object in the field of view for a period of time using our oculomotor system as well as head and body motions. Humans are able to do this quickly and reliably without much effort. It is therefore natural to expectthat the artificial cognitive systems we aim at developing will, to a certain extent, be able to demonstrate similar capabilities. Robot vision refers to the capability of a robot to visually perceive the environment and interact with it. Robot vision extends methods of computer vision to fulfill the tasks given to robots and robotic systems. Typical tasks are to navigate toward a given targetlocation while avoiding obstacles, to find a person and react to the person‘s commands, or to detect, recognize, grasp and deliver objects. Thus, the goal of robot vision is to exploit the power of visual sensing to observe and perceive the environment and react to it. This follows
2

3 the example of humans. It has been found that more than half of the human sensory cortex is attributed to seeing.Computer vision attempts to achieve the function of understanding the scene and the objects of the environment. With the increasing speed of processing power and progress in computer vision methods, making robots see became a main trend in robotics. There, however, remains a fundamental difference between computer vision and robot vision. Computer vision targets the understanding of a scene mostly...