NSF/NRI-sponsored Workshop Report:

Robot Planning in the Real World:

Research Challenges and Opportunities

Organizers: Ron Alterovitza, Sven Koenigb, and Maxim Likhachevc,1

aDepartment of Computer Science, University of North Carolina at Chapel HillbDepartment of Computer Science, University of Southern California

cRobotics Institute, Carnegie Mellon University

January 2014

Abstract: Endowing robots with the ability to plan their actions and motions willenable robots to assist people with a wider range of tasks in the real world. Robotplanning can improve a robot’s usability and capabilities for a variety of applications,from manufacturing to medicine to home assistance to transportation to disasterresponse. Recent years have seen significant technical progress that has resultedin planners for such challenging tasks as driving, flying, walking, and manipulatingobjects. However, robots that have been commercially deployed in the real worldtypically have no or minimal planning capability. In October 2013, a workshopwas held with support from the National Science Foundation (NSF) and NationalRobotics Initiative (NRI) to discuss how the research community can help robotplanning mature to enable wider real-world deployment. In this report, we highlightkey conclusions from the workshop and also include an appendix with individualposition statements from the attendees of the workshop. This report summarizesopportunities and key challenges in robot planning and includes challenge problemsidentified in the workshop that can help guide future research towards making robotplanning deployable in the real world.

1 Introduction

Since the first industrial robot began service in a car assembly line just over 50 years ago, roboticshas grown into a multi-billion dollar worldwide industry. Robotics is having a significant impactin multiple domains, from manufacturing to warehouse automation to medicine to home assis-tance. U.S.-based companies such as Google, Intuitive Surgical, Amazon, iRobot, and Apple areinvesting heavily in robotics for their products and supply chains2,3,4. With advances in research,the next generation of robots have the potential to improve performance in established domainsand create entirely new applications. Improvements in robot capabilities and usability could re-sult in commercial deployment of self-driving vehicles, disaster-response robots, snake-like robotsfor minimally-invasive surgery, assistive robots capable of helping the elderly in their homes, and

1The organizers would like to thank the workshop participants (listed in Appendix A) for their contributions.2http://www.nytimes.com/2013/12/04/technology/google-puts-money-on-robots-using-the-man-behind-android.

html3http://www.bloomberg.com/news/2013-11-13/apple-s-10-5b-on-robots-to-lasers-shores-up-supply-chain.html4Amazon Prime Air, http://www.amazon.com/b?node=8037720011

many other robot types. Achieving the full potential of robotics will require improving the abilityof robots to reason about how to accomplish a task, process sensor data in real time, utilize avail-able resources effectively, cooperate with humans, and adapt to changes in the environment. A keybuilding block of robots with these desirable capabilities is robot planning: computing actions andmotions for robots to achieve their objectives.

Recent years have seen significant technical progress on robot planning, including new algo-rithms, methods, and software. Exciting research has resulted in planners for such challengingtasks as manipulating objects, driving, flying, and walking. Robot planning algorithms can beused in a variety of contexts in robotics, including task planning, mixed-initiative planning, pathplanning, motion planning, and grasp planning. Robot planning is a key enabler of multiple corerobot capabilities, including navigation through the environment, manipulating tools and objects,maintaining safety around humans, gathering information necessary to complete a task, and co-ordinating teams or groups of multiple robots and humans. For each of these capabilities, robotplanning can enable full robot autonomy or can facilitate shared autonomy in which the capabilityis achieved by shared human-robot control.

Although substantial progress has been made in robot planning, robots that have been commer-cially deployed in the real world typically have no or minimal planning capability. These robots aretypically manually programmed, tele-operated, or programmed to follow simple rules, as is the casefor most manufacturing robots (such as automobile assembly manipulators), special-purpose homeassistance robots (such as the iRobot Roomba), and medical robots (such as Intuitive Surgical’sda Vinci System). Although these robots are highly successful in their respective niches, a lack ofplanning capabilities limits the range of tasks for which currently-deployed robots can be used.

There is currently a substantial gap between the potential of robot planning to enable excitingrobotics applications and the reality of the limited deployment of robot planning in the real world.This gap introduces many research challenges, and filling this gap will create new opportunities forrobots to assist humans in the real world.

Workshop on Research Challenges and Opportunities. To discuss how the research com-munity can help robot planning mature to enable wider real-world deployment, a workshop washeld with support from the National Science Foundation (NSF) and National Robotics Initiative(NRI)5. This workshop, held October 28–29, 2013 in Arlington, Virginia, brought together re-searchers and practitioners to discuss the state of the art in robot planning, its use in variousrobotic applications, current research challenges, and opportunities for the future. The workshopincluded 37 participants, including 23 participants from academic institutions and industry and14 representatives from government agencies. By bringing together experts with diverse expertise,the workshop spanned technical areas that cover the many aspects of robot planning, from motionplanning to task planning to human-aware planning. The diversity of the workshop participants’interests also enabled discussions that covered many potential real-world application areas of robotplanning, including personal assistance, medicine, manufacturing, warehouse automation, defense,and transportation. Information about the workshop, including the program and slides from thepresentations, can be found at the workshop web site:

http://robotics.cs.unc.edu/PlanningWorkshop2013/.

Report overview. This report follows up on the workshop by summarizing the discussions andpresenting some of the identified opportunities and key challenges in robot planning. In Section

5This workshop was supported by NSF under award #1349355. Any opinions, findings, and conclusions orrecommendations expressed in this report do not necessarily reflect the views of the NSF.

2

2, we highlight the variety of applications discussed by workshop participants, which motivate theneed for better robot planning. In Section 3, we highlight research challenges that, if addressed, canhelp make robots operate more robustly and efficiently. These research challenges were identifiedin the workshop by studying robot planning research across different applications, analyzing robotplanning as part of complete robot architectures, and exploring the interaction of planning withother robot modules (such as perception, control, and user interfaces). In Section 4, we highlightchallenge problems identified in the workshop, which the planning research community shouldtarget. The challenge problems are specific problems that, if studied, will likely move researchin a direction that makes planning even more relevant for real-world robotics applications. Theopportunities and challenges highlighted in this report are based on discussions at the workshopand should be refined further based on the participation from the broader community in order toenable deployment of robot planning in a broad class of real-world problems.

2 Applications and Opportunities for Robot Planning

Robot planning can help improve the capabilities of currently deployed robots and create oppor-tunities for new robotics applications. Below is a survey of some of these real-world roboticsapplications and how robot planning can help.

Image courtesy of Rethink Robotics

Manufacturing. The needs of the manufactur-ing industry led to the birth of modern robotics;the first industrial robot entered service in a Gen-eral Motors assembly line in New Jersey in 1961.Despite declines, manufacturing remains a signifi-cant part of the U.S. economy. Manufacturing sup-ports over 10 million U.S. employees6 and is cur-rently seeing growth in certain sectors with compa-nies like Apple and Lenovo in-sourcing portions oftheir manufacturing operations to the United States.Most robots used in manufacturing are manually pre-programmed to rapidly and independently performrepetitive tasks for a large volume of goods in fenced-off spaces. Robot planning can help enable a newgeneration of manufacturing robots that operate cooperatively with humans, can be used in nimblefactories with rapidly changing products and needs, facilitate personalized manufacturing (in com-bination with 3D printing), and offer precision and reliability beyond the skills of human workers.Such robots will require planning to decrease the burden on users to pre-program the robot, tofacilitate quick adaptation to new tasks, and to enable cooperative task completion with humans.Creating robots with these capabilities will require research on manipulation planning, efficientuser interfaces for conveying how tasks should be performed, human-robot cooperation, enablingsituational awareness, compensating for environmental and operational uncertainty, and assuringperformance.

Warehouse automation. Most warehouses today are labor-intensive, and robotics has the po-tential to make warehouses operate more efficiently. Kiva Systems, recently bought by Amazon.com

6U.S. Bureau of Labor Statistics, 2012

3

for over $700 million7, uses fleets of small robots to move inventory shelves (pods) around in ware-houses. The robots carry inventory shelves to people on the perimeter of the warehouse whomanually complete tasks (such as placing items in boxes and/or replenishing the shelves). Kiva’srobots make the workforce 2–3x more efficient by eliminating walking on the warehouse floor butmore sophisticated planning technology could reduce the number of robots needed and thus resultin further cost savings. Robot planning is already used for coordinating the motions of the manysmall robots. Advances in robot planning could enable robots to autonomously place items inboxes and replenish shelves as well as integrate task planning with path planning, enabling fullyautomated, highly efficient warehouse operations.

Medicine. Medical robots have the potential toaugment the capabilities of physicians and enablenew medical procedures with fewer negative side ef-fects. Intuitive Surgical’s commercially successfulda Vinci system allows surgeons to tele-operate en-doscopic instruments with improved accuracy andprecision. New snake-like and tentacle-like medicalrobots could maneuver along curved, winding pathsto reach anatomical targets in highly constrainedspaces, enabling minimally-invasive access to previ-ously unreachable sites. Robot planning can helpmedical robots reach their full potential by facilitat-ing intuitive operation of complex robots, for exam-ple, by passively suggesting a path for the roboticmechanism to follow, by actively guiding the sur-geon’s motion to respect motion constraints, and/or by guaranteeing safety by automaticallyavoiding anatomical obstacles and sensitive structures. Robot planning for medical applications ischallenging because of complex constraints on robot motion, large robot configuration spaces, thecommon need to pass through highly constrained spaces, the need to reason about uncertainty anddeformable environments, and the need for fast, high quality plans with safety guarantees.

Personal assistance. Personal robots have thepotential to assist people with a variety of tasks inhomes and workplaces. Assisting people with activi-ties of daily living (such as eating and cleaning) coststhe U.S. economy over $350 billion each year8, andthese costs will continue to grow as America’s agingand disabled population increases. Personal robotswith manipulation capabilities could assist peoplewith activities of daily living, thus enabling the el-derly and people with disabilities to remain indepen-dent in their homes longer without needing to moveto expensive institutions. Personal robots with theability to navigate in human environments (without

7Kiva Systems Press Release, http://www.kivasystems.com/amazon-press/8E. Kassner, S. Reinhard, W. Fox-Grage, A. Houser, J. Accius, B. Coleman, and D. Milne, “A balancing act:

State long-term care reform,” Technical report, AARP Public Policy Institute, Washington, DC, July 2008.

4

necessarily needing arms for manipulation) could be used in workplaces, museums, and publicspaces as guides, escorts, or automatic transport (e.g., robotic wheelchair). Robot planning canenable personal robots to efficiently, autonomously navigate and manipulate objects in people’shomes and other environments designed for humans. Navigating and manipulating objects in en-vironments designed for humans raises numerous challenges for robot planning. The robot plannermust process relevant sensor data in real time, must be fast and reactive, and must consider thepresence of humans and animals in the environment. The robot planner must also handle unstruc-tured and dynamic environments and consider uncertainty. Furthermore, the robot should generateconsistent, intuitive plans such that humans in the environment can safely anticipate the robot’smotions. The robot planner must also take as input a vague description of a task and create a planthat satisfies the user’s intent in a manner that is flexible and robust.

Image courtesy of Google

Transportation. Car accidents kill more than30,000 Americans each year9. Robotic, driverlessvehicles have the potential to reduce death and in-jury due to car accidents and to increase the effi-ciency of our road network, particularly in highlycongested urban areas. Autonomous ground, wa-ter, and air vehicles (e.g., quadrotors) can also beused in other transportation-related contexts, includ-ing package delivery (as proposed by Amazon10), ex-ploration, security patrols, and industrial tasks (e.g., object transport with a forklift). Excitingprogress has been made in recent years. The DARPA Grand and Urban Challenges were com-pleted successfully by multiple entrants and the Google driverless cars have already completed over300,000 miles of autonomous driving11. But key challenges remain before self-driving cars andother autonomous vehicles will be widely adopted. Robot planning is a necessary component ofan autonomous vehicle, and the integration of perception and planning needs to be improved. Au-tonomous vehicles need to better understand real-world uncertainties, and planners must be robustto uncertainties arising from limitations in robot perception capabilities.

Disaster response. Robots have the potential toassist in a variety of emergency response situations,including search and rescue operations, firefighting,bomb diffusion, and surveillance. Although robotscan already be used in many of these situations un-der tele-operation, robot planning could enable widerdeployment in the real world by requiring less humaneffort to make robots accomplish their tasks and bymaking the robots easier to use in high-stress, dy-namically evolving disaster response situations. Avariety of robot architectures are relevant to disasterresponse, including ground vehicles, aerial vehicles,underwater vehicles, humanoid robots, and snake-like robots. Key challenges for robot planning include the ability to operate in a team with other

9U.S. Department of Transportation, http://www-nrd.nhtsa.dot.gov/Pubs/811630.pdf10Amazon Prime Air, http://www.amazon.com/b?node=803772001111Google Official Blog, http://googleblog.blogspot.com/2012/08/the-self-driving-car-logs-more-miles-on.html

5

robots and/or humans, handling large degrees of freedom with significant constraints on motion(for example, for humanoids and snake robots), situational awareness, integrating perception withplanning, real-time performance, and the need to operate at a tempo beyond the capabilities ofmost current robotics systems.

Surveillance and monitoring. Robots have the potential to assist with tasks such as surveil-lance, inspection of structures (both on the ground and underwater), and environmental monitoring.Aerial vehicles such as drones and quadrotors are ideally suited for above-ground surveillance andmonitoring tasks. Key challenges for robot planning are similar to the challenges for applicationssuch as transportation and disaster response. The planning algorithms will need to enable a robotto operate in a team with other robots and/or humans, to integrate perception with planning, tocompute and execute motions in real-time, and to assess the current situation and request helpfrom humans when necessary.

Emerging and non-robotics applications. Robot planning is likely to be used in many ad-ditional robotics applications, some of which have not yet been thought of. Emerging roboticsapplications that could benefit from enhanced robot planning capabilities include construction ofstructures and automated farming. These applications combine the needs of other applications,including transportation, personal assistance, and disaster response. They also introduce newchallenges, including direct interaction with nature and coordination of large teams of robots andhumans. Advances in robot planning can also be integrated with education; robots with integratedplanning can help inspire interest in computer science and engineering in children. Planning algo-rithms are also used for applications beyond robotics. Robot planning algorithms have made theirway into diverse applications, such as modeling protein folding, animating agents for games andvirtual environments, and simulating large crowds for optimizing security and emergency evacua-tion procedures. Thus, addressing the research challenges in real-world robotics applications willlikely benefit other domains as well.

Enhancing and expanding robot capabilities. Advances in robot planning will have signifi-cant impact on a variety of robot capabilities which span multiple existing and emerging roboticsapplications. Planning is critical for robust robot autonomy, which – depending on the application– may require manipulation of objects and tools, navigation, maintaining safety around humans,information gathering, multi-robot coordination, and human-robot teaming (for example, in thecontext of sliding autonomy, offering and requesting advice and help, and providing information).The research community has investigated planners that enable these core capabilities, but thereremains a large gap with respect to applicability in real-world conditions. For example, significantprogress has been made on handling single rigid objects but less progress has been made on han-dling collections of rigid objects (such as needed for packing boxes) or handling deformable objects.Similarly, significant progress has been made on robot navigation for ground, aerial, and marinevehicles, but less progress has been made on navigation in dynamic and unstructured environmentswith time pressure. Examples of challenging scenarios that require these capabilities include navi-gating in the presence of people (e.g., a mobile robot navigating through a crowd of people), underextreme conditions (e.g., docking a sea-surface vehicles in a rain storm), and under time constraints(e.g., an aerial vehicle performing complex maneuvers). Moving in a simple, elegant, and agile styleby effectively utilizing dynamics is a robot capability that has been not been studied sufficiently.Additionally, robots should be able to explain their behavior, the plans need to be understandableby humans, and the planner should be able to quantify how likely it is to succeed and communicate

6

this information to the human user if necessary. Although the subareas of robot planning havebeen studied by the research community to varying degrees, each subarea still includes unsolvedproblems that are important for broadening robot deployment in the real world.

3 Research Challenges for Robot Planning

Achieving the full potential of robotics in the applications discussed above will require addressingmultiple research challenges related to robot planning. In this section, we list research challengesthat were found by the workshop participants to be the most important and most likely to enablethe deployment of robot planning in the real world. In general, planning methodologies differdepending on the type of robot and the type of environment the robot operates in. For example,wheeled and flying robots typically require substantially lower dimensional planning than mobilemanipulators, humanoids, and snake-like robots. Similarly, outdoor environments typically are notas complex and cluttered as indoor environments, although indoor environments are sometimesmore structured. These differences lead to different approaches to robot planning. However, manyrobot planning research challenges (including many outlined below) are shared across multiple robottypes and environments.

In addition to the list of research challenges in this section, additional research challenges canbe found in the appendix which contains the individual position statements of the participants. Wenote that the topics discussed in the workshop are a subset of the important research challengesthat must be addressed for widespread deployment of planning in real-world robots.

Tight integration of planning with perception. In many domains, the bottleneck to robotautonomy lies with perception. For example, the participants discussing unmanned ground vehiclesagreed that image understanding will not be fully solved anytime soon. Instead, planning shouldexplicitly deal with the uncertainty an autonomous vehicle has in its perception of the world. Sim-ilarly, lightweight micro-aerial vehicles and surgical robots have poor or limited sensing capabilitiesand planning must compensate for this. A challenge is how to plan with uncertainty in perceptionin a way that scales, especially when it is hard to quantify the uncertainty. Are there planning rep-resentations that are amenable to real-time requirements on planning yet capture critical elementsof uncertainty?

Modern robots are sometimes equipped with an array of sensors, many of which are controllableeither directly (e.g., controlling a servo) or by repositioning or reconfiguring the robot itself (e.g.,moving an arm equipped with a camera or tactile skin). This leads to massive amounts of incomingsensory data. Some of the data may even be contradictory due to noise in sensing. To helpwith uncertainty in perception, planning should reason about when and how the robot can controlits sensors in order to obtain information that disambiguates the uncertainty that jeopardizes therobustness of the robot completing its task.

The world has infinite dimensionality. How should planning represent it? As a robotmoves in the real world, the planner faces the challenges of what it should model in its environmentas well as when and how. For example, a typical kitchen may contain hundreds of relevant objects,such as pots, dishes, and utensils. A personal assistance robot operating in the kitchen should nothave to model all of these objects for planning a specific task. Furthermore, even if the robot couldmodel everything computationally, the question is how these objects should be modeled in the firstplace. Geometric information about the world is relatively easy to obtain and represent but thephysical behavior of objects – for example, articulated, deformable objects – is much harder to

7

represent and estimate. In medical robotics applications as well as cooking applications for homeassistance, understanding the deformation of objects such as tissues or foods is often critical totask success. The brittleness of autonomy in the real world often comes from failures to accountfor certain factors or from errors in the model. On the other hand, much of information about theworld may be completely irrelevant to the task that the robot tries to achieve. A challenge thenis to infer a planning representation that is reasonable and useful for a given task. This inferenceprocess may also be combined with robot actions that explore the world and lead to better modelestimates. The planner can aid in this exploration given its knowledge of the task and the potentialsolutions it considers.

For a robot to come up with a compact representation for planning without any prior experi-ences or human input is challenging, if not impossible. Exploring the role of human demonstrationsfor planning could help with this potential avenue of research. Can a planner utilize human demon-strations in building a compact planning representation for the task at hand? Can the planner figureout when to ask for demonstrations and then learn from them the “right” planning representation?In what form should these representations be given (for example, tele-operated, simulated, or kines-thetic demonstrations of the full task, or only advice on what factors the planner should considerin its planning and how)?

Another important research direction is to explore the benefits of experiences. Can planninglearn from prior planning and execution episodes what the “right” planning representation is for agiven task? Past failures in execution may suggest the necessity for additional factors in planning,whereas the analysis of successful plans that do not exercise certain degrees of freedom in the worldmay allow the planner to construct a more compact representation for the given task.

Consistency, predictability, and understandability of robot behavior. It was brought uprepeatedly during the workshop that the behavior of robots needs to be consistent, predictable, andunderstandable. This is especially the case in manufacturing where an operator needs to be ableto anticipate what action the robot is going to perform next and how it will perform the action sothat the operator can intervene when necessary. Predictability also simplifies the operator’s taskof coordinating multiple robots. The same holds in defense applications where a soldier needs topredict and understand the behavior of the robot in order to trust it and plan his/her own actions.In the domain of household assistance, predictability of robot behavior helps a human trust therobot and simplifies the coordination of the human’s own actions.

Consequently, a challenge is to generate and re-generate plans that are consistent (for example,similar for similar scenarios) and easy to understand by a human. These plans need to be generatedand re-generated in real time despite the fact that many robotic systems (such as mobile manipu-lation platforms) are many degrees of freedom (DOFs). Computing feasible and optimal plans isalready computationally challenging for high-DOF systems. Being able to compute, in real time,consistent plans for high-DOF systems is therefore a considerable research challenge. In additionto pure computational challenges, there is a question of the understandability of robot behaviorand motion, i.e., what behaviors and motions are understandable and how can understandability bemaximized.

Human-aware planning. Autonomous robots working alongside humans face an additional setof unique challenges. The robot should behave in a way which is safe and consistent with thebehavior of humans in the environment and which helps humans accomplish their tasks withoutbecoming a nuisance. To achieve this, the robot needs to infer human intentions and goals andincorporate them into its plans so that the robot helps the humans without causing delay, danger,

8

or confusion. However, human intentions are typically impossible to predict perfectly. Instead,based on the context and prior observations, intentions are typically inferred probabilistically.A challenge for planning is, therefore, to compute safe plans that account for the uncertainty inhuman intentions as well as utilize actions that disambiguate this uncertainty (e.g., asking clarifyingquestions) when necessary and possible. Robots also need to be able to explain their behavior tohumans on request. A challenge is therefore to create planning techniques with the ability to provideexplanations.

On the other hand, the presence of humans presents not just challenges but opportunities forrobots to improve their reliability. Robots can ask humans for help in accomplishing tasks that arehard to complete autonomously and when perception fails or is not sufficiently accurate. A challengefor planning is to reason about the chances of successfully accomplishing a task without human helpand the possibility, cost, and utility of asking humans for help. Furthermore, humans can alsobe asked to provide demonstrations. Planning should reason about when demonstrations should beprovided and how demonstrations can be used to infer what behaviors and motions are expected fromthe robot, what planning representation is best suited for planning, and what constraints need to beobeyed during task execution. This can be even more challenging if human inputs are only partialdemonstrations or advice as opposed to full demonstrations of how a task can be accomplished.

Robotic systems with guarantees on performance. Robots for many applications are be-coming more and more complex, with higher degrees of freedom and/or massive arrays of sensors.As a consequence, the software architectures of robots are also becoming more and more complex,incorporating numerous distinct software modules. Given such complexity, it becomes difficult toassure that the behavior of the robot is going to be correct under different conditions, and the lackof such assurance jeopardizes the employment of autonomous robots in many domains. For exam-ple, workshop participants repeatedly mentioned that, in domains such as defense, transportation,medicine, and manufacturing, the operators of the robots and human co-workers expect reliableand repeatable behavior from the robots.

Consequently, the software modules of robots need to be designed in a way that the reliability,the repeatability, and the performance of the overall system can be analyzed. Since planning isresponsible for decision making, this places a significant burden on the planner. That is, in addi-tion to the requirement that the planner itself have guarantees on its performance and generatesconsistent solutions, we need to reason about its interaction with other components. More specifi-cally, it brings up several challenges for the design of planning architectures. How should differentlevels of planning (such as task-level planning, motion-level planning, and low-level controls) becombined in a principled way? What properties does each of these modules need to satisfy in orderto maintain guarantees on the performance of the overall system? How should planning interactwith non-planning modules (such as perception) in order to provide guarantees on performance?

Planning that utilizes the availability of massive amounts of data. Much of the brittlenessof current autonomous robots comes from the fact that they lack a deeper understanding of theworld. It is much easier to plan motions for simple tasks (such as pick-and-place tasks) than togenerate plans to accomplish more complex tasks (such as cooking or washing clothes in a washingmachine). Geometric information about the world can be relatively easily perceived by a robot,but the semantics of perceived objects are much harder to derive. A robot often has a goodunderstanding of how its own body moves, but knowledge about how other objects, especiallyarticulated or deformable objects, can be manipulated is difficult and sometimes impossible toencode beforehand and is often impractical to try to estimate online. Finally, many tasks require

9

a prior knowledge of “recipes” for how they can be achieved. These “recipes” provide an abstractand potentially partial specification for how to achieve a task. It is impractical to pre-program the“recipes” for all tasks the robot may encounter during its lifetime.

On the other hand, modern robots typically have access to the Internet and consequently massiveamounts of data available on the web. Can this data be utilized to empower robots with a deeperunderstanding of the world and to improve their robustness? For example, can planning utilizepartial “recipes” on the web for how tasks should be accomplished? When planning to manipulate anon-rigid object, can a planner collect data from the web about how this object can be manipulatedand utilize the data to build an effective planning representation and guide the search for a plan?

Furthermore, given the network connectivity of robots, the knowledge and experiences gatheredby one robot can and should be shared among other robots when possible. Sharing informationamong robots has the potential to accelerate their understanding of the world. With this in mind,the question is how to build a common shared database of knowledge and experiences for robots andwhat information should go into the database given the vast differences in modern robotic systems.

Open-source planning libraries. The development of the Robotic Operating System (ROS)12

has had an enormous effect on sharing research results between academic groups and transitioningrobot technologies into the commercial world. ROS is now being used by numerous companiesand nearly every university that does research in robotics. Part of this success can be attributedto the fact that many ROS components were built in joint efforts between researchers at WillowGarage and in academia. Equally important is the fact that ROS and its components are under anopen-source license that allows for the unrestricted use of the software.

While there are several planning libraries (such as OMPL13 and SBPL14) available under ROS,the workshop participants have agreed on the importance of developing more open-source planningtools that are interoperable with commonly-used robotic software infrastructures (such as ROS)and are available to the robotics community without any restrictions. While the development ofthese tools requires significant resources and efforts, especially to achieve a form that is useablein industry, they can dramatically help with making joint progress towards full autonomy and itscommercialization. Government research agencies and industrial collaborators should recognize theimportance of such efforts and support them.

4 Challenge Problems for Robot Planning

We present a set of challenge problems, which are problems that, if studied, will likely move researchin a direction that makes planning even more relevant for real-world robotics applications. Below,we provide a high-level summary of the challenge problems identified by the workshop participantsduring workshop discussions. Workshop participants also identified challenge problems in theirposition statements, which can be found in Appendix B.

Desirable properties of challenge problems include the following. Challenge problems shouldspell out possible evaluation metrics. They should have a low barrier to entry, i.e., researchersshould be able to tackle them without necessarily having access to specialized equipment andwithout having perfectly working low-level robot functionality (such as control and perception).

12http://www.ros.org13I. A. Sucan, M. Moll, and L. E. Kavraki, “The Open Motion Planning Library,” IEEE Robotics & Automation

Magazine, 19(4):72–82, December 2012.14M. Likhachev, Search-based Planning Library (SBPL): Open-source library of graph searches and their applica-

tions to robotics. Available at http://wiki.ros.org/sbpl.

10

This can be achieved by starting with robot simulations or experimental datasets on one hand andcarefully crafted problems on the other hand. For example, having people in the loop can providesome required capability that is missing, e.g., the challenge problem of helping a blind personcook requires perception capabilities but no manipulation capabilities. Larger challenge problemsshould span multiple robot capabilities and involve researchers from different disciplines, such asfrom artificial intelligence and robotics.

Challenge problems can be created around the applications or robot capabilities describedearlier. Below, we describe several challenge problems identified in the workshop. Each of thechallenge problems requires enabling multiple robot capabilities using planning.

Challenge Problem: Box and bin handling. Creating robots that can handle boxes, bins,and the small rigid and flexible items in them is a challenge problem that has implications formultiple applications, including warehousing, manufacturing, and home assistance. Specific tasksinclude opening and unpacking boxes, placing items into bins, finding items in bins, and packingitems into boxes. This challenge problem requires planners that advance the state-of-the-art inmultiple subareas of planning, including grasp planning, manipulation planning, motion planning,and task planning.

Challenge Problem: Warehousing for manufacturing-on-demand. Manufacturing-on-demand allows a company (especially a small business) to manufacture customized products insmall batches as they are purchased. Warehousing includes the close coordination of multiple robotsthat navigate in tight spaces to transport objects between different locations in a warehouse. In thecurrent state-of-the-art, planners are typically provided with the start and goal locations for eachrobot. It is an open problem how to effectively integrate low-level path planning with high-leveltask planning, which is critical for effective automated manufacturing-on-demand. In this challengeproblem, because products can be customized, it is necessary to determine sequences of goal loca-tions for the robots that not only achieve the task-planning objective but consider the impact of theselected sequences on path planning (e.g., to keep the resulting paths short and prevent congestionof the robots).

Challenge Problem: Fetching and cleaning in home environments. Personal robots inhomes, assisted living centers, and nursing homes must operate in human spaces, which are typicallycluttered, unstructured, and include humans and pets. A challenge problem in this domain is tofetch items for a person with a disability and clean up a cluttered room. Planning in this domainrequires awareness of humans, which raises multiple challenges as discussed in the prior section.For example, the motions of robots need to feel natural to humans so that they are predictable andenable cooperation. This challenge problem can be extended to substantially more complex tasks.Possible extensions include building a maid, butler, nurse, or cook robot. A home assistant robot,for example, could be required to perform tasks such as delivering daily medication, doing laundry,changing linens, cooking, serving food, and helping with personal hygiene, eating, and dressing. Asubset of these tasks is currently covered by the RoboCup@Home15 league, which defines specificscenarios for competitions.

Challenge Problem: Surgical manipulation. Many surgical tasks require manipulating de-formable tissues inside the human body. Planning motions for surgical robots that account fordeformation could enable surgeons to perform safer and more efficient surgery. A representative

15http://www.robocupathome.org

11

challenge problem is retraction and exposure: the objective is for a laparoscopic surgical robot tograsp a flap or section of tissue and lift it to expose tissue underneath. This challenge problemrequires robust manipulation of deformable objects as well as an appropriate level of perception ofthe objects in the scene in real-time as they deform. This challenge problem can be made morerealistic (and more difficult) by considering constraints on visibility of tissues and high levels ofuncertainty in the motion of instruments and tissue. Furthermore, some surgical sites may onlybe safely accessed by maneuvering along curved trajectories through constrained cavities, whichwould require planning motions for snake-like or tentacle-like robots with many degrees of freedomto bend around anatomical obstacles.

Challenge Problem: Search and rescue. Searching for and rescuing a human or animal inan unstructured environment raises numerous planning challenges for a robot. A representativechallenge problem in search and rescue is for a mobile manipulator robot to navigate over rubble,search for an object, grasp the object, and then bring it to a new location. Many aspects of thetasks in search and rescue scenarios are included in the RoboCup Rescue16 league competitions.This challenge problem can be extended to consider situations with large crowds of humans, whichintroduces large numbers of degrees of freedom that must be considered during planning. Other ex-tensions include using ground, marine, and aerial vehicles and considering larger and more complexenvironments and tighter time constraints.

5 Conclusion

Although robots are increasingly being used in a variety applications, the deployment of advancedplanning capabilities in real-world robots has thus far been limited. The NSF/NRI-sponsoredworkshop on robot planning in October 2013 highlighted the potential of research on planning toaid significantly in bringing a broad range of autonomous robots closer to real world deployment.The workshop identified specific research challenges that researchers working in the field of planningshould explore. The workshop attendees also suggested several challenge problems that encompasskey research issues and can be used by planning researchers as test domains. The opportunities andchallenges highlighted in this report are based on discussions at the workshop and should be refinedfurther based on the participation from the broader community. We hope that the outcomes of theworkshop will help guide researchers, inspire new research directions, and lead to new programsthat stimulate research on robot planning with the goal of making robots with advanced planningcapabilities ready for real-world deployment.

16http://www.robocuprescue.org

12

Appendix A: Workshop Participants

• Ron Alterovitz, University of North Carolina at Chapel Hill

• Nancy Amato, Texas A&M University

• Ronald Arkin, Georgia Institute of Technology

• Jennifer Barry, Rethink Robotics

• Michael Beetz, Universitat Bremen, Germany

• Kostas Bekris, Rutgers University

• Ronald Boenau, U.S. Department of Transportation

• Sachin Chitta, SRI International

• Howie Choset, Carnegie Mellon University

• James Donlon, National Science Foundation (NSF)

• David Ferguson, Google

• Gary Gilbert, Telemedicine and Advanced Technology Research Center (TATRC)

• S.K. Gupta, National Science Foundation (NSF)

• Kris Hauser, Indiana University at Bloomington

• Joe Hays, Naval Research Laboratory (NRL)

• Geoffrey Hollinger, Oregon State University

• Wes Huang, iRobot

• Purush Iyer, U.S. Army Research Office (ARO)

• Leslie Kaelbling, Massachusetts Institute of Technology

• Subbarao Kambhampati, Arizona State University

• Behzad Kamgar-Parsi, Office of Naval Research (ONR)

• Dai Hyun Kim, Office of the Assistant Secretary of Defense for Research and Engineering

• Sven Koenig, University of Southern California

• Hadas Kress-Gazit, Cornell University

• Vijay Kumar, University of Pennsylvania

• Maxim Likhachev, Carnegie Mellon University

• Kevin Lynch, Northwestern University

• Dinesh Manocha, University of North Carolina at Chapel Hill

• Jeremy Marvel, National Institute of Standards and Technology (NIST)

• Brett Piekarski, U.S. Army Research Laboratory (ARL)

• Fred Proctor, National Institute of Standards and Technology (NIST)

• Craig Schlenoff, National Institute of Standards and Technology (NIST)

• Don Sofge, Naval Research Laboratory (NRL)

• Mike Stilman, Georgia Institute of Technology

• Manuela Veloso, Carnegie Mellon University

• Richard Voyles, National Science Foundation (NSF)

• Peter Wurman, Kiva Systems

13

Appendix B: Position Statements

Ron Alterovitz, University of North Carolina at Chapel HillMotion Planning for Medical and Assistive Robots

Emerging robots have the potential to improve healthcare, from enabling new surgical procedures toautonomously assisting people with daily tasks in their homes. Tentacle-like medical robots couldmaneuver along curved, winding paths to reach anatomical targets in highly constrained spaces,enabling minimally-invasive access to previously unreachable sites. Personal robots in people’shomes could assist people with activities of daily living, such as eating and cleaning, thus enablingthe elderly and people with disabilities to remain independent in their homes without needing tomove to expensive institutions. To reach their full potential in real-world environments, these newmedical and assistive robots will need planning algorithms to enable them to operate in a safe andintuitive manner.

However, most robots currently deployed in the real world have no or limited planning capa-bilities. Current medical robots, such as the Intuitive Surgical da Vinci system, are tele-operated,which limits their capabilities to low-degree-of-freedom motions that can be directly controlled bya human expert. Commercial robots designed for the home, such as the iRobot Roomba, are cur-rently limited to specific tasks and follow simple rules rather than reason about optimal motionsor human preferences.

To be successful in real-world applications, planning algorithms for medical and assistive robotsmust guarantee safety, facilitate intuitive operation, and enable successful performance of complextasks. Key challenges to be successful in the real world include:

• Compensating for uncertainty : Uncertainty is an inevitable implication of medical robotsbecoming smaller and gaining degrees of freedom and assistive robots using less precise ac-tuators and sensors to gain compliance and decrease cost. To ensure safety and task success,planning algorithms will need to explicitly consider uncertainty in the robot’s shape. Plan-ning algorithms also need to consider uncertainty in sensing (e.g., noisy point cloud data,noisy medical imaging, and partial information).

• Real-time, near-optimal planning : For intuitive and safe operation in dynamic and possiblydeforming environments, planners will need to be responsive by computing high-quality plansat interactive, real-time rates.

• Integrating human expertise into planning : Many surgical and assistive tasks involve con-straints on motion that humans are aware of from context and intuition. For example, whencarrying a glass of water, the glass must be kept level to avoid spills. New algorithms areneeded to automatically learn such constraints from human-generated data and efficientlyintegrate this learned information with planning algorithms to enable successful performanceof a wide variety of tasks.

Nancy M. Amato (with Jyh-Ming Lien, Marco Morales, Samuel Rodriguez,Lydia Tapia, and Shawna Thomas), Texas A&M University

Motion Planning

A primary challenge for robotics is to perform complex, unsafe, or difficult tasks. For example,robotics has tremendous potential to increase quality of life and also decrease costs by enablingindividuals to live at home. To do this, robots must: use sensors, be reactive to changing conditions,

14

intelligently learn, interpret high-level task instructions, cooperate with humans and other agents,and control complex bodies and dynamics. Robotics techniques also have potential to have atransformative impact in domains outside of traditional robotics. For example, robotics motionplanning methods for articulated systems can be extended and adapted to model and plan formolecular motions which has the potential to provide insight into causes and potentially cures forsuch devastating diseases such as Alzheimer’s that are related to protein misfolding.

While tremendous advances in motion planning methods have been made, there are still somemajor issues that must be addressed to develop robust and reliable methods for such scenarios. Forinstance, although motion planning has been applied with success to increasingly complex problems,planning coordinated interactions of large numbers of agents or motions for very flexible systemswith physical constraints results in high-dimensional configuration spaces which are challenging forbest methods today. Also, while recent advances in planning in belief space have made progress inextending the successful sampling-based motion planning strategies to scenarios with localizationerrors and incomplete or noisy models of the obstacles, additional work is required to handledynamic environments. Motion planning methods need to be extended and adapted to addressthese types of challenges.

Possible research directions include enriching the data structures used to represent and querythe planning space to enable them to scale to complex systems, large numbers of agents, anddynamic environments. Another challenge for motion planning is to find appropriate mechanismsto include human agents in the what has traditionally been a fully automatic planning process.This will require methods for formalizing the interaction with and information provided by thehuman agents, and strategies for integrating it into the planning process.

Benchmark applications for such scenarios include assisted living (a robotic assistant aids a hu-man with everyday tasks in the home), city-level evacuation planning (multi-agent systems used byemergency responders), or molecular motion and interaction (reactions involving multiple moleculesin crowded environments with complicated dynamics and incomplete, approximate information).While not all traditional robotics applications, they will spur advances in motion planning.

Ronald Arkin, Georgia Institute of TechnologyPlanning in the Presence of Ethical Requirements

As robots are moving out of the laboratory and into the real world at an ever increasing pace, it isimportant to consider not simply how they interact with humans, but how to maintain the quality,dignity, and legality of that interaction. Machine ethics is a relatively young community that hasbegun to address these issues, more and more in a robotics context.

Problems facing generating plans that conform to ethical constraints include17:

• The ethical laws, codes, or principles are almost always provided in a highly conceptual,abstract level.

• Their conditions, premises or clauses are not precise, are subject to interpretation, and mayhave different meanings in different contexts.

• The actions or conclusions in the rules are often abstract as well, so even if the rule is knownto apply, the ethically appropriate action may be difficult to execute due to its vagueness.

• These abstract rules often conflict with each other in specific situations. If more than onerule applies it is not often clear how to resolve the conflict.

17B. McLaren, “Lessons in Machine Ethics from the Perspective of Two Computational Models of Ethical Reason-ing,” AAAI Fall Symposium on Machine Ethics, AAAI TR FS-05-06, 2005.

15

Research in our laboratory in this domain at Georgia Institute of Technology focuses on twomachine ethics application areas: lethal autonomous robots (ARO)18 and healthcare (NSF), bothserving as possible benchmark problems. The first addresses the requirement for robots possessinglethal force to conform to international humanitarian law, while the second application, reusing ar-chitectural components derived from the military application, involves preserving dignity in patient-caregiver interactions in early stage Parkinson’s disease. How moral emotions affect action selectionand modulate ongoing robot behavior plays a role in both applications. The preservation of dignityin healthcare relationships incorporates partner modeling (theory of mind) to assist in achievingcongruence between the human parties by means of a mediating robot.

The application of deontic logic in high-level abstract moral reasoning19 is another importantarea that others have studied and warrants further investigation for guiding and planning appro-priate, ethical, and dignified human-robot interaction.

Jennifer Barry, Rethink RoboticsIndustrial Manipulation Planning

We have made great strides in the last two decades in real-time and near real-time manipula-tion planning20,21. More recently, the development of integration tools22 has significantly loweredthe barriers to implementing state-of-the-art planning algorithms on industrial arms. Neverthe-less most industrial manipulators do almost no planning, despite the growing movement towardsflexible industrial robots23,24. Industrial robots are either pre-programmed or trained on-site bydemonstration; a robot that could plan its own tasks would remove a burden on the user. Moreover,the more a robot can reliably do on its own, the fewer people are needed to keep it operating. Thisis especially valuable in a factory where robotics experts may not be on-site. However, flexibleindustrial robots almost exclusively follow user-demonstrated or programmed paths. Manipulationplanning is used only as a last resort.

In an industrial setting, speed and predictability of motion is paramount. Non-deterministicplanning algorithms are not well-suited to this environment. Moreover, unlike in a research labo-ratory, an industrial robot has almost no information about its environment. These robots do nothave RGBD sensors or laser scanners. It is usually easier to show the robot the path it shouldfollow than to give the robot a description of free space or trust it to acquire this information onits own.

One method for overcoming this lack of information is to combine human demonstration, learn-ing, and inference to form an understanding of the free space. A human-taught path provides anexample of a collision-free trajectory. A robot-planned and then human-modified path provides evenmore information. Ideally the amount of necessary demonstration decreases significantly during asingle task and across tasks.

18R. Arkin, Governing Lethal Behavior in Autonomous Robots, Taylor-Francis, 2009.19K. Arkoudas, S. Bringsjord, and P. Bello, “Toward Ethical Robots via Mechanized Deontic Logic,” AAAI Fall

Symposium on Machine Ethics, AAAI TR FS-05-06, 2005.20L. Kavraki, P. Svestka, J-C Latombe, and M. Overmars, “Probabilistic Roadmaps for Path Planning in High-

Dimensional Configuration Spaces,” IEEE Transactions on Robotics and Automation, August 1996.21S. LaValle and J. Kuffner Jr., “Rapidly-Exploring Random Trees: Progress and Prospects,” in Algorithmic and

Computational Robotics: New Directions, 2000.22S. Chitta and I. Sucan, “Introduction to ROS and MoveIt!,” ROS-Industrial Consortium Open House, March

2013.23Rethink Robotics. http://www.rethinkrobotics.com.24Universal Robots. http://www.universal-robots.com.

16

There is room for planning in industrial robotics. We need methods for inferring free spacewithout dedicated sensors, and planners that reliably return predictable paths that can be executedquickly and smoothly.

Michael Beetz, Universitat Bremen (Germany)Everyday Manipulation Planning

The field of autonomous robot manipulation experiences tremendous progress: the cost of robotplatforms is decreasing substantially, sensor technology and perceptual capabilities are advancingrapidly, and we see an increasing sophistication of control mechanisms for manipulators. Re-searchers have also recently implemented robots that autonomously perform challenging manipu-lation tasks, such as making pancakes, folding clothes, baking cookies, and cutting salad. Thesedevelopments lead us to the next big challenge: the investigation of control systems for roboticagents, such as robot co-workers and assistants, that are capable of mastering human-scale every-day manipulation tasks.

Robots mastering everyday manipulation tasks will have to perform tasks as general as “cleanup,” “set the table,” and “put the bottle away/on the table.” Although such tasks are vaguelyformulated the persons stating them have detailed expectations of how the robot should performthem. I believe that an essential planning capability of robotic agents mastering everyday activitywill be their capability to reason about and predictively transform incomplete and ambiguousdescriptions of various aspects of manipulation activities: the objects to be manipulated, the toolsto be used, the locations where objects can be manipulated from, the motions and the grasps tobe performed, etc. Vague descriptions of tasks and activities are not only a key challenge for robotplanning but also an opportunity for more flexibility, robustness, generality, and robustness of robotcontrol systems.

A promising approach for realizing plan-based control systems for everyday manipulation tasksare knowledge enabled transformational planning techniques for concurrent reactive robot plans.

Kostas Bekris, Rutgers UniversityMotion Planning with Guarantees

A grand challenge for robotics is the development of systems that safely operate next to humansin unstructured environments and effectively construct structures given access to individual parts,tools and high-level instructions. Examples include teams of robots and people deployed aftera natural disaster to set up temporary housing or deployed in small-business factories to buildcustom-made products.

The above challenges require the generation of high-quality and safe robot motions in a com-putationally efficient manner, while addressing the following complexities:

• real-time response in partially-observable, dynamic scenes;

• uncertainty;

• high-dimensional spaces with complex constraints, e.g., closed chains, deformations, dynam-ics;

• integration with control and task-planning;

• rearrangement of manipulable objects in cluttered scenes;

• interaction with people and other robots;

17

Desirable solutions should go beyond heuristic methods that empirically work well in small-scaleexperiments. They should provide formal guarantees and reproducible results. Even basic motionplanning, however, is computationally hard. Thus, an important tradeoff between performanceguarantees and computational cost arises.

This realization has led in the past to practical sampling-based planners that construct roadmapsand sacrifice completeness for quickly computing solutions. These methods were recently shown tobe asymptotically optimal for kinematic challenges given dense enough roadmaps. We need to buildon top of this progress and utilize advances, such as cloud computing, to produce powerful plannersfor important applications. Towards this objective, we are considering the following strategies:

• Finite-time computation properties: We have recently shown that sampling-based planners are“probably near-optimal” after finite computation, which is useful to practitioners, replanningtasks and guarantees for task-planning.

• Compact roadmaps: Effective representations that can be queried fast, have small memoryfootprint, provide communication benefits when using vast but remote computing resourceswhile still providing formal guarantees.

• Complex systems: Providing formal guarantees for systems without a steering function, whichcan also have an impact on planning under uncertainty.

• Multi-robot planning : We work towards bridging the gap between coupled and decoupledplanners in discrete domains by achieving completeness and polynomial complexity at thecost of suboptimal solutions. These results can benefit object rearrangement challenges.

• Interactive planning : Planning robot motion among other agents, and potentially people,brings additional considerations for planners, such as information requirements, deadlockavoidance, and game-theoretic optimality notions.

In all of the above research efforts, the key issue relates to the impact that computationallimitations have in providing performance guarantees.

Sachin Chitta, SRI InternationalPlanning for Manipulation and Navigation in Unstructured Environments

A current challenge for robots in the real world is fast reactive manipulation and navigation inunstructured environments. An example of such a task is an assistive robot working in elder-carefacilities where it needs to be especially alert and reactive to the presence of older adults. Robotsoperating in such environments will face a plethora of issues including dealing with dynamic chang-ing environments, lack of adequate sensor coverage of the environment and the constant presenceof people. A big challenge in human-aware path planning is also the generation of consistent paths,which allow the humans working with the robots to anticipate its motions better.

Motion planning plays a big role in such environments, ranging from planning long paths fornavigation, reactive planning of paths in the presence of people, human-aware path planning toaccount for the motion of people. Motion planners will need to be faster than they currently are ingenerating high-quality, consistent paths in such scenarios, especially in manipulation tasks wheredynamics start to play a bigger role. Optimality is also important since the robots will need to beefficient in carrying out their tasks to provide a good return on investment. Dealing with flexibleobjects is another area where motion planning needs to make better strides to address some of theissues in elder-care applications.

A good benchmark challenge would have to address several types of issues to make a significantimpact. One possible problem would be to load several items in a stockroom onto a delivery cart

18

(including several flexible items like towels or linen) and then stock some of these items in theresident’s rooms. The benchmark here would be human speed in currently achieving this task.This task will cover a range of issues including navigation, mobile manipulation, manipulation inconstrained semi-structured environments, planning for sensing and human-aware planning.

Howie Choset, Carnegie Mellon UniversityPlanning for Medical Robots

Successful realization of medical robots requires a number of challenges to be addressed. Thesechallenges can be characterized as environmental, task-related, robot-intrinsic and user-based. Allof these challenges interact with each other. For example, environmental challenges, such as manyand complex constraints, make planning for a given task difficult. Many tasks in medicine require arobot to pass through highly convoluted spaces, requiring new mechanisms and hence new plannersfor them. These planners must reason about large configuration spaces, uncertainty, and soft anddeformable environments. Moreover, new approaches for optimal and dynamic planning need tobe developed to handle the sheer complexity of the planning problem in the medical context.

Planning for medical applications should not occur in isolation of other tasks commonly faced inrobotics. For example, new and novel robot designs present new challenges in terms of kinematics,sensing and control, and new planners must also accommodate for these tasks. Medical imagingtechniques (CT-scans, X-Rays) are used to develop static pre-operative geometric models but donot account for physiological motion (rigid and deformation) of anatomical features leading todiscrepancies between the pre-operative model and the real environment. Planners need to workin conjunction with estimation and control techniques to handle such discrepancies.

Planning must not be relegated to moving an autonomous robot. Many planners can serve asassistive tools to surgeons by either passively suggesting a path for the robotic mechanism to followor by actively guiding the surgeon’s motion to respect the path constraints defined by the planner.Recently, we have seen the use of 3-d printed artifacts being used in medical interventions, whichrequire planning for creation of the artifact. Planning has also been used to inform the design offlexible robots where the design of the robot influences the feasible obstacle free paths of the robotin the environment.

Dave Ferguson, GooglePlanning with Uncertainty

Currently, there are few robots operating in the real world. After decades of seemingly fruitfulresearch, robotic systems are still predominantly relegated to the factory floor.

Why? Why don’t we have personal robots yet? Why don’t we have autonomous roboticvehicles?

There are several challenges in planning for robotic systems operating in real environments,and I’m sure we will touch upon many of these in this workshop. But I would argue that we arenot held back by the complexity of motion planning or task planning or interaction planning. Theplanning community has made great strides in these areas and we can now plan for mechanismswith ridiculously high degrees of freedom in cluttered environments, string together sophisticatedaction sequences, and understand and respond to natural language interactions.

We have shown that if we have perfect information about the world, we can do fantastic things.But it is operating within the confines of the information that we actually have that is hard and, Ibelieve, the biggest challenge we need to tackle in the planning community. Like any experienced

19

planning researcher knows, it is always perception’s fault. But the only way we will get real systemsto work is if we pick up the slack in the planner. Because perception is too hard.

We all have our favorite planning areas and interesting problems we like to mull over and writeneat papers about. But as a community it is up to us to focus on what is most important. Wedo ourselves no favors by setting up toy examples (no matter how complex they are) and tacklingthe wrong challenges. Instead, we need to collaborate closely with the perception community (andbetter yet, to work on fielded systems) to understand the nature of the uncertainty we need toincorporate in our planning approaches. This uncertainty will reduce as perception improves, andthe nature of its distribution may evolve, but it will always be important to reason about it and berobust to it. And when our robots can handle real world uncertainty, they will be able to handlethe real world.

Kris Hauser, Indiana University at BloomingtonMotion Planning

Motion planning has matured significantly in the last decade, with enormous progress being made intechniques to handle high-dimensional systems in complex environments, large teams of cooperativerobots, and systems with uncertainty. Yet despite this success, and the promise of planning togenerate intelligent behaviors, planners are still not widely deployed on real-world robots. Why isthis so?

Along a similar timeframe, spurred by the development of cheap hardware and widely availablesoftware for gathering and processing 3D sensor data, robotics was transformed by a perceptionrevolution. Now, perception is a widely available tool: nearly every robot now has a Kinect orsome other depth sensor. I hold that every robot should have a planner. To achieve this goal, themotion planning field must shift its perspective to become a provider of useful tools for robotics.

The field has failed to deliver on two fronts. First, it is tedious and difficult to integrate plannersinto robots. Planners must deliver “out of the box” operation with state-of-the-art objectives,dynamic models, and perceptual data, and must also integrate into robot development and testingworkflows. Second, planners are too slow. Planners must be fast enough to deliver high-qualitysolutions in a responsive manner. Overcoming these challenges will likely require insights into thefundamental nature of planning problems.

Based on my recent experience on using motion planning on humanoid robot locomotion prob-lems in the DARPA Robotics Challenge, and on human-operated semiautonomous robots, I suggesta few promising research directions:

• Real-time optimal replanning. Approach the goal of computing high-quality solutions withtime constraints.

• Structured motion planning. Planners should accept a larger variety of dynamic models, ob-jective functions, and constraints, including hybrid models and semantic knowledge. Plannerswill need to examine problem structure to devise good solution strategies.

• Black-box motion planning. Abstract the choice of planning algorithm from the API. Provideknobs for controlling solution time/quality/repeatability tradeoffs.

• Large models from perception. Plan quickly in the face of large, possibly dynamic point cloudmodels (e.g., tens or hundreds of millions of points). Incorporate uncertain initial states andpredictive models from state estimators (e.g., particle filters).

• Use real dynamics. Cope with the nuanced behavior of low-level controllers and actuators,e.g., motor controllers, passive damping, friction, back-EMF, backlash, drivetrains, etc.

20

• Debugging. Setting up large planning problems is error-prone, as a single typo can cause anincorrect planner failure. Provide a method for explaining / debugging planner reasoning.

Geoffrey Hollinger, Oregon State UniversityPlanning for Autonomous Information Gathering

Planning and coordination of autonomous vehicles for information gathering in unstructured out-door environments is a key future research challenge. Emerging application domains include envi-ronmental monitoring, aerial surveillance, emergency response, and agricultural sensing. Teams ofautonomous vehicles are uniquely suited for performing these large-scale monitoring and surveil-lance tasks because of their potential for collecting, processing, and reacting to incoming sensordata. In this sense, multi-vehicle planning is effectively a big data problem where vast quantitiesof sensor data are streaming continuously during operation.

Existing planning algorithms are insufficient for coordinating teams of vehicles in these large-scale monitoring domains. Key advancements must be made in (1) online adaptation and learningwith changing objectives, (2) scalability to large environments and increasing team sizes, and (3)real-time reactive decision making during in-the-loop planning. Effective surveillance and monitor-ing platforms will need to sense and react to incoming information in real time and also communicateinformation back to human operators. Planning and decision making in these scenarios will allowfor supervised autonomy during which autonomous vehicles act to assist scientists, first responders,and intelligence officers.

Potential avenues for solving these research challenges can be found in combining statisticalmachine learning methods with motion planning techniques. Recent advances in data processingand machine learning have allowed for adaptation in real time with impressive accuracy. Simi-larly, motion planning algorithms are utilizing sampling-based techniques to scale up and generatecomplex coordination of multi-robot teams with performance guarantees. By combining scalablemachine learning algorithms with sampling-based motion planning techniques, we can potentiallyimprove the adaptability and scalability of robot planning methods.

One key benchmarking problem in autonomous surveillance and monitoring is the effectivereconstruction of a time-varying scalar field over a large space. For example, a team of autonomousunderwater vehicles may be deployed to determine the dissolved oxygen content in a bay duringan ecological event. Given some underlying model of the scalar field and a team of assets withlimited speed, maneuverability, and sensing radius, the planning problem is to choose the optimaltrajectories for the entire team that minimize the reconstruction error over the field. Prior solutionshave employed techniques like gradient descent, submodular optimization, sampling-based planning,and model predictive control to optimize trajectories in these scenarios. However, the developmentof a general solution that is broadly applicable and can effectively adapt to changing objectivesremains an open problem.

Wes Huang, iRobotPlanning in Human Environments

There are many planning challenges as robots increasingly operate in human environments, fromgracefully navigating around those environments to assisting people with various tasks in thoseenvironments.

Robots need to navigate in shared human environments without causing any disruption. Al-though the problem of getting from point A to point B is well understood, doing so gracefully

21

in a busy dynamic environment is still a challenge – people are not ”just” obstacles. Aside fromtraffic-like conventions of right-of-way and turn-taking, nonverbal social cues and sometimes ex-plicit verbal communication are used. No robot currently has a human adult level understandingof these issues. While there are perception problems to be solved here, there are plenty of planningproblems. For example, how should a robot use its gaze to naturally or more effectively navigate?When should a robot ask people to make way for it to pass versus take another route?

Assisting seniors at home or in long-term care facilities will be an increasingly relevant source ofplanning problems. Most seniors prefer to stay in their own homes instead of moving to a long-termcare facility; however, people’s abilities decline with age. Formal and informal care and supportcan help seniors continue to live independently, but there can be significant cost for formal careand a significant burden for families and friends. Robots will eventually help seniors perform manypersonal and household tasks, but there are many planning challenges. How does a robot help aperson get dressed? How does a robot prepare, serve, and clean up after a meal? How can a robotoffer assistance before it is requested? To identify some general themes, these are tasks in which:a robot needs to cooperate with a human; the environment or objects involved in a task may beill-defined or not easily modeled; or the robot needs to observe a situation and decide where it canor should play a role.

Purush Iyer, U.S. Army Research Office (ARO)Planning for Dealing with Unawareness

Traditional planning is considered a separate module, in autonomous systems, that employs pow-erful scheduling algorithms to meet a set of constraints. In a future where autonomous systems areexpected to rely less on detailed input from the operator and where the need to operate as part ofa human-machine team will be greatest, automated planning systems could play a major role inadvancing learning and reflection to improve the skill set of a robot. In particular, the automatedplanning component could help in staging experiments to learn and discover, as part of its currenttask set, the boundaries of the autonomous system’s physical capabilities (say, ability to jump) andalso user preferences (is its operator left-handed, right-handed or ambidextrous). The latter (forinstance) would be needed in collaborative activities as, for instance, in a situation where a robotand a human need to clear debris. Thus, the desiderata are:

• Planning systems should be layered throughout the architecture of autonomous systems;in particular, they cannot be independent modules that have a fixed functionality or API.Furthermore, planning systems should be able to infer the needs of the various componentsof a system rather than be explicitly given a task list/ goal to schedule.

• Planning systems should improve with time, by learning, from past experience.

• Planning systems should be actively involved in the learning mechanism of the autonomoussystem. In particular, planning needs to account for reducing unawareness, or increasingawareness, of the system with respect to its abilities that have not been programmed explicitly.

Leslie Pack Kaelbling, Massachusetts Institute of TechnologyPlanning in Unstructured Environments

One critical set of challenges faced by robots come about in highly unstructured environments,such as unmodified private homes or disaster areas. The problems in these domains are numerous:the dimensionality is high, the horizon for planning is long, and the uncertainty is significant.

22

One important challenge is representational: how can we represent the robot’s state of knowledgeabout its environment? It is especially difficult when the input data is multi-modal, including,for example, 3d and 2d vision, touch information, natural-language or other knowledge-base input,tactile or auditory information, etc. High-dimensional spaces must ultimately be represented withfactored representations. I would suggest that a collection of heterogeneous representations, whichare combined at query time, may be the most effective solution to this problem. It is commonwisdom that some form of hierarchical solution is appropriate for addressing long planning horizon,but there are as yet very few workable concrete suggestions about how to do this. I believe that theonly way to effectively handle uncertainty is to model it explicitly and consider it during planning.Decision theory provides a solution to optimal sequential decision-making uncertainty, but it iswildly computationally intractable. We should aggressively seek efficient alternatives, but do so ina way that allows us to understand trade-offs between computation and utility.

Household robots are a great challenge domain, because it is relatively easy to set up test-bedsin regular lab space. It is important to note that the fundamental insights gained and methodsdeveloped to solve problems in the household robotics domain will generalize to a wide variety ofmobile manipulation domains.

Subbarao Kambhampati, Arizona State UniversityPlanning for Human-Robot Teaming

Most current applications of robots involve viewing them as sensors and effectors that can beremotely operated by humans. Much of the role for planning in robotics has thus been limitedto path and manipulation planning (and in the case of multiple robots, task assignment). Anincreasing number of applications, including search and rescue, however demand that robots befull-fledged members of human-robot teams. Such teaming scenarios present a significantly richerset of challenges and opportunities for planning that go beyond path planning and task assignment.In particular, robot team members operating in such human-robot teams need task planning andreplanning to engage in goal-directed reasoning, and plan and intention recognition techniques toanticipate the actions of their human team members, and even dialog planning to effectively com-municate with the human team members. The teaming context poses significant novel challenges toeach of these planning roles, thus precluding the direct adoption of techniques developed for theirtraditional settings. Our main position with respect to this workshop is to advocate investigationof the challenges faced by each of these roles of planning in human-robot teaming.

Our own ongoing work, supported in part by ONR and ARO, has been addressing several of thechallenges of planning for human-robot teaming, with particular focus on task planning/replanningin the context of teaming. The planning challenges here stem both from the long-term nature ofteaming tasks, distribution of world model across team-members, the open-world nature of theenvironment, as well as the need for supporting effective communication between the human andthe robot. These in turn demand that the planner guiding the robot must deal with incompletelyspecified models, uncertain objectives, open and dynamically changing worlds, as well as the abilityto take continual human instructions (including those that change and/or modify goals and actionmodels), and return meaningful status reports.

We are working towards handling incomplete models through generation of robust plans; han-dling uncertain objectives and partial preferences through diverse plans and open world conditionalgoals; and handling continual state, goal and model-updates with the help of commitment and op-portunity sensitive replanning. We are also collaborating with human-factors researchers to betterunderstand planning challenges arising from the communication with the humans in the team. Aninteresting problem here is the automatic generation of excuses as a way of explaining planning

23

failures to the humans in the loop, which in turn helps elicit additional domain knowledge fromthe humans. Finally, we are adapting plan/intent recognition techniques to help the robot bettercoordinate with the human team members.

Behzad Kamgar-Parsi, Office of Naval Research (ONR)Planning for Area Surveillance

A particularly important and challenging problem is area surveillance. Currently area surveillanceis performed by one or a few agents (e.g. mobile and stationary cameras) without coordination.Progress is being made in wide area surveillance with many agents, where planning and task as-signments are done in a centralized manner. This requires robust communications with a lot ofband-width, a powerful central processing station, and typically involves delays. Future trend inwide area surveillance is to use many autonomous heterogeneous agents that plan coordinatedsurveillance in a decentralized manner. Usual challenges for planning algorithms include the fol-lowing: (a) information is often uncertain and incomplete, and sometimes contradictory, (b) inchanging environments new information is collected that may necessitate changing goals and pri-orities, which may require rapid re-planning, and (c) the space in which agents make decisions (beit state space, information space, belief space) grows rapidly hence algorithms become computa-tionally intractable. Therefore we want to develop approaches to planning that shrink the spacewhile preserving its essential features, can deal with uncertainties and changing information andpriorities, and can assess their own performance and provide a measure of optimality.

Sven Koenig, University of Southern CaliforniaCombining Task and Motion Planning for Multi-Robot Systems

Robots are becoming better at solving low-level planning tasks (such as navigation and manipula-tion planning). At the same time, robots also have to solve high-level planning tasks (such as taskplanning). It is an open problem how to best combine low-level and high-level planning since theyoften work on different data structures (with no clear correspondence) and according to differentprinciples (for example, due to the continuous nature of low-level planning and the symbolic na-ture of high-level planning). Low-level and high-level planning are also predominantly studied bydifferent research communities, namely robotics and artificial intelligence.

We speculate that progress on heuristic search (in particular, the study of suitable variantsof the heuristic search method A*) has good chances to result in a systematic interface betweenlow-level and high-level planning, for the following reasons: Heuristic search-based planning is nowpredominant for high-level planning, and there has been progress on using heuristic search for low-level planning (for example, in the context of lattice-based planning). Hierarchical search couldthen be used to combine low-level and high-level planning. It is very encouraging that there hasbeen recent progress on variants of A* that are helpful in this regard, such as

• hierarchical versions of A* (that search on different abstraction levels),

• any-angle versions of A* (that do not restrict the resulting paths to lie on given graphs),

• any-time versions of A* (that produce paths quickly and then improve them over time), and

• incremental versions of A* (that use experience with previous similar search problems tore-plan faster).

24

However, a lot more research is required on both the individual techniques in isolation andtheir integration (including understanding their behavior and taming combinatorial state-spaceexplosions) with an eye on achieving real-time planning with performance guarantees. Facilitatingadditional interaction between the robotics and artificial intelligence research communities wouldlikely help to accelerate progress.

Finally, planning becomes more complicated as teams of robots need to solve tasks cooperatively,for example, because single robots are unable to solve them in isolation or because one wantsto achieve robustness or parallelism. More research is required on both low-level and high-levelplanning for multi-robot systems in isolation and their integration.

A good benchmark problem for combining task and motion planning for multi-robot systems isconstruction, where a team of robots cooperatively needs to assemble a structure.

Hadas Kress-Gazit, Cornell UniversityCombining Task and Motion Planning with Behavioral Guarantees

Most people cannot program and everyone expects machines to perform their tasks predictably,successfully and safely. These facts lead to two major challenges in robotics that must be overcomein order to transition robots from niche application to wide impact in many domains; people mustbe able to instruct robots in an abstract and intuitive way, and robots must be “well behaved” atall times and in all situations. More specifically, this means that robots must a) be able to interactat a high-level, at a specification (the “what”) and not at an implementation (the “how”) leveland b) must provide guarantees for safe and reasonable behavior most of the time, and gracefuldegradation when failure is inevitable.

To address these two challenges, planning algorithms must span different abstraction layers;they must seamlessly integrate low-level control designed for specific robot platforms with high-level reasoning targeted at creating complex behaviors as specified by the task. There must beformal mechanisms allowing the planning problem to be refined and abstracted while preservingthe correctness of the solution at the different layers.

Since robots are physical systems with noisy sensor and imperfect actuators operating in com-plex environments around humans, providing absolute guarantees that a robot will never fail isimpossible. To prove correctness properties, allow for graceful degradation and to inform users ofpossible failure modes, planning algorithms must reason about their own robustness; they must ex-amine the assumptions that they make about the problem, they must reason about failure situationsand they must provide meaningful feedback when tasks cannot be done or cannot be completed.

Vijay Kumar, University of PennsylvaniaMotion Planning

Most real world applications of robots require an operational tempo that is currently beyond thestate of the art in robotics. The two exceptions to this are in semi-structured or completelystructured environments such as warehouses (the Kiva systems example) or carefully mapped roads(the Google car). The central challenge for us is to be able to design planners, and more broadlyperception-action loops, that can exhibit the level of performance seen in the Google car andthe Kiva robots, but in less structured settings while being able to reason about uncertainty.There is a second set of challenges surrounding manipulation and other tasks in which robots mustexhibit controlled contact interactions with a non-cooperative, physical environment. The DARPARobotics Challenge will hopefully address this set of problems.

25

There are three directions of research that need to come together in order address the mainchallenges. First, in order to reason about uncertainty, efficient probabilistic approaches thatsuccessfully solve the dual integration problem - integrating over the belief space and integratingover the set of measurements that are possible at future steps - need to be developed. Second, thebranch of control theory dealing with model-predictive control or receding-horizon control needsto be developed further to deal with completeness and convergence guarantees in settings withuncertainty and non-trivial dynamics. Finally, as we start building complex systems (flying andrunning robots) and systems that physically interact with the real world, it is necessary to cometo grips with nested perception-action loops. Traditional approaches where a planner provides afeedforward or reference trajectory for the entire system do not scale well for complex systems.

Possible benchmarks include

• 3-D Kiva: Imagine a three-dimensional indoor environment where you want to send 100 fly-ing robots to respond to requests by humans by flying to goals safely and efficiently. Whatarchitecture (decentralized versus centralized or hybrid) must we consider for solving thisproblem? Does the cloud change the fundamental multi-agent coordination problem space?What if the environment were not fully known? How do robots adapt to unstructured en-vironments and how do they share information to build a model that others can leverage?Does the cloud lend itself to new real-time, planning and control algorithms?

• Humanoid : DRC but real-time performance at a tempo that is significant and meaningful forsearch and rescue.

Maxim Likhachev, Carnegie Mellon UniversityIntegrated Planning in Complex Worlds

Real-world in its full glory contains too many factors and unknowns to account for all of themupfront while developing an autonomous robot. The first challenge for planning therefore is tooperate on a “deeper” level than just a static representation of the world. Planning representationsneed to “adapt” to the tasks the robot executes, the environment it works in, the experiences itgathers and demonstrations it receives. For example, human demonstrations for how to open doors,for how to sort items arriving on a conveyor or for how to assemble furniture shouldn’t be usedas simple re-play motions but instead should be used by the robot to infer how relevant objectsbehave and how to construct planning representations that are effective for given tasks.

The second challenge is in developing planning frameworks that integrate tighter with othermodules in the system such as perception, control and task planning. The planners need to reasonabout the weaknesses and strengths of these modules and generate plans that avoid their weak spotsand capitalize on their strengths. Furthermore, these weaknesses and strengths should be revisedbased on the robot experiences. For example, a micro-aerial vehicle has limited payload leadingto noisy sensing and actuation. Generated plans should therefore exercise as much as possible thecontrollers that are robust such as visual servoing towards easily detectable landmarks, and avoidgenerating segments of the paths that require the use of fragile controllers. Similarly, a motionplanner for a manipulator needs to reason about the limitations of grasping and the constraints ofhigher-level task planning. To address these challenges without the explosion of the state-space,we need to research planning representations that vary in their dimensionalities, model limitationsand strengths of modules and revise all of it based on experience.

Third, for robots to operate alongside humans, we need to make them more consistent andpredictable. Consider, factory workers working side-by-side with industrial robots, surgeons gettinghelp from robots bringing medical tools and soldiers working with robots manipulating IEDs. In all

26

of these scenarios, humans need to be able to predict what action the robot is going to execute nextand how. This allows humans to intervene when necessary as well as plan their own actions. Thepredictability of motions is also beneficial to task-level planning and remote execution monitorswhich need to predict the behavior of the robot and re-plan or intervene as necessary.

Kevin Lynch, Northwestern UniversityReal-Time Motion Planning for Uncertain Hybrid Mechanical Systems

As robots become increasingly dynamic (examples include the early hoppers and runners fromRaibert’s group, Boston Dynamics’ Atlas, RHex, Cheetah, and the ParkourBot), extreme locomo-tion becomes possible. Methods for real-time motion planning, estimation, and control are requiredto take advantage of these robots’ capabilities.

As our challenge problem we consider motion planning for robot parkour. Given a dynamicmodel of the robot and a well characterized environment (including possible footholds and hand-holds), the problem is to plan a sequence of runs, jumps, vaults, swings, etc., to traverse a complexenvironment by taking advantage of the various constraints present.

This motion planning problem has four important characteristics:

• Hybrid : The equations of motion of the system change depending on the contact constraintscurrently active (e.g., which hands or feet are in contact, and whether these contacts aresliding or sticking). Transitions between different regimes may be characterized by impacts.

• Mechanical : The equations of motion in each regime are not arbitrary, but are characterizedby an inertia tensor, Coriolis terms due to the Christoffel symbols of the inertia, potentialforces, etc.

• Uncertain: Uncertainty is introduced due to errors in the robot and environment model aswell as a lack of controllability to recover from all perturbations within a particular regime.

• Real time: The usual decoupling of “planning a trajectory” and “controlling to follow theplan” does not suffice due to lack of controllability within each regime separately. Thereforereal-time multi-step replanning is required.

A successful motion planning framework will account for the several possible dynamic regimesand their transitions; make use of the structure of the equations of motion for increased efficiency;explicitly model the propagation of uncertainty and its mitigation, shaping, or increase by variousdynamic primitives; and fine tune the nominal plan during execution.

Such a framework is likely to include a non-real-time component that addresses the combinato-rial problem of searching for a sequence of maneuvers and generating a nominal motion plan with ahigh likelihood of success (e.g., modeled on traceurs’ careful assessment of an environment) as wellas gradient-based real-time tuning of the nominal plan to account for the evolving state uncertainty(belief distribution) maintained by a belief filter during execution.

The basic framework is likely to apply to hybrid robot manipulation as well.

Dinesh Manocha, University of North Carolina at Chapel HillMotion Planning

Algorithmic motion planning has been actively studied in robotics and related areas for more thanthree decades. There is a rich collection of motion planning algorithms based on computationalgeometry and algebraic methods, local or potential field techniques, randomized sampling, handling

27

kinodynamic or non-holonomic constraints, optimization methods, etc. Most of these algorithmshave been successfully used for CAD/CAM, bioinformatics, computer gaming and other applica-tions. At the same time, advances in manufacturing technologies, sensing, and actuator deviceshave led to the development of powerful robots, including humanoid robots and general-purposeand programmable mobile manipulators. However, there has been limited use or application of mo-tion planning algorithms on these “physical robots” to perform various autonomous or navigationtasks. This is due to issues related to dynamic constraints, modeling uncertainty, perception, aswell as real-time computation of motion strategies on the robotic platform.

Recent developments in programmable mobile manipulators, along with open-source operatingsystems and environments (e.g. ROS), seem to open up many new possibilities to bridge this gap.Furthermore, the availability of better sensors (depth sensors, for example) and high computingpower (multi-core CPUs and many-core GPUs) makes real-time planning algorithms for high-DOFphysical robots feasible. However, there is relatively less work in terms of developing fast planningcapabilities that can take into account model and sensor uncertainty.

We need to develop motion planning algorithms for physical robots that cannot be separatedfrom the underlying application or context. These include necessary interfaces with low sensory-motor control and symbolic reasoning than to explore geometric issues that do not correspond tomajor bottlenecks in the physical world. In many ways, we have reached a level of maturity in termsof classic geometric motion-planning problems, like the ones related to the curse of dimensionalityor planning in “narrow passages”. However, this maturity can be considered at a conceptual leveland we need to extend that along other dimensions. For example, some of the widely used motionplanning techniques are based on probabilistic approaches that perform reasoning not only in robotconfiguration spaces but can also be extended in “augmented” spaces, like state spaces that areused to model control issues or belief spaces used to account for decisional issues. Moreover, weneed to continue development of good software tools, such as FCL, OMPL, MoveIT, OpenRAVE,to integrate these algorithms into new applications.

Jeremy Marvel, National Institute of Standards and Technology (NIST)Planning for Robot Collaboration in Manufacturing

From my perspective, one of the largest challenges facing current and next-generation robotics inmanufacturing is that of intrinsic collaboration. Tasking and re-tasking robots for manufacturingapplications is already quite difficult due to limitations stemming from proprietary hardware andinterfaces. Getting these robots to also collaborate intelligently requires considerably more effort.Currently, robot-robot collaboration is achieved by explicitly coordinating motions based on a setof event-based constraints. In contrast, human-robot collaboration is limited to instances of simplecolocation, rule-based reactive actions, and safety-related mechanisms. These, however, are nottrue collaborations in the sense of robots and humans coopering with a shared understanding ofthe manufacturing process to achieve some desired end goal. Getting robots to collaborate withhumans or other robots requires a significant advancement in autonomy and performance assurance.This necessitates improvements in situational awareness, real-time coordination of motions based onsensor feedback, dynamic planning and re-tasking to compensate for environmental and operationaluncertainty, and task decompositions and representations to enable more accurate mappings ofobservations to responses. Combined with correlated test methods and metrics, advances in thesefields will enable greater confidence in the capability of collaborative robots to complete theirassigned tasks correctly while meeting their assembly performance objectives.

Don Sofge, Naval Research Laboratory (NRL)

28

Safe Navigation Planning in Crowded Environments

A key planning challenge faced by mobile robots in the real world is the need to operate safelyamongst people, and other robots, while avoiding collisions. People possess an innate skill formoving through crowds and passing one another (both on foot and while driving) while generallyavoiding collisions. This skill relies on understanding how we move through space, observing others,predicting others’ paths, and automatically adjusting our own motion to avoid collisions. Onepossible approach to a solution would be for robots to have a different (i.e. better) representationof space with direct feedback to motion control parameters (e.g., swerve left, or veer right). In orderto be successful the robot should not only sense the presence of others (both human and robot),but also anticipate where they will move next. A benchmark problem that would spur research inthis area might involve having a mobile robot navigate through a crowded mall setting, or perhapsa crowded conference or an outdoor concert, while avoiding collisions with others.

Mike Stilman, Georgia Institute of TechnologyPlanning for MacGyver Robots

Future robots should have the capability to use their entire bodies and environments in order tosolve complex tasks. Previously, we have developed planning algorithms for Navigation AmongMovable Obstacles (NAMO) where robots could move objects out of the way in order to achievenavigation or manipulation tasks.

The MacGyver Concept : Our group is moving for-ward from the initial ability to interact with environ-ment objects. We believe that robots should not onlyremove objects, but also use them to help in achievingtasks by building simple machines. For instance, con-sider robots that will use a board to build a bridge oruse a broom as a lever to create additional force fromdistance. Such static and dynamic concepts should beincorporated into autonomous planning by humanoidrobots and mobile manipulators. We believe that uti-lizing the entire environment and the robot’s entirebody is critical to future advances in robot autonomy.

Planning in Constraint Space: In order to make it possible for robots to utilize their environ-ments we propose to plan in the space of constraints. Our preliminary work, presented at ICRA2013, shows that a robot can create structures from environment objects without explicitly decidingtheir locations. At each step of the plan the robot solves an optimization problem to verify thevalidity of the plan step. Once a plan is determined, the constraint satisfaction problem is fullysolved and the plans are finalized.

Planning with Uncertainty : It is challenging for a robot to work with environment objects thathave unknown or uncertain properties. We are building on our work from the NAMO domain toincorporate uncertainty through decision theory as well as hierarchical planning in the now. Weaim to maximize the robot’s capacity to execute actions and quickly identify novel circumstancesand plans that can overcome them.

Manuela Veloso, Carnegie Mellon UniversityPlanning for Symbiotic Robot Autonomy

29

Actual robots in the real world are mostly used in targeted industrial, in space, and militaryapplications. They execute complete planned motions, motion plan for given high-level goals,or are remote operated. Besides the Roomba robots, and instances of service robots in hospitalenvironments, there are not many autonomous robots in the real world. There are the Google andother self-driving cars, which are becoming a reality.

The real world offers challenges to robot autonomy, at all levels, namely perceptual, cognitive,and actuation. Focusing on the real world where humans habitate, robots will further concretelyface navigation, timing, quality, and interaction challenges. Planning involves having models of theworld, and models of actions. It is challenging, or most probably impossible, to find appropriaterepresentations for general-purpose models, independent of the task and goals. Furthermore, thereal world is dynamic, includes lots of uncertainty, unpredictability, and other external, at bestpoorly modeled, actuators, such as other robots and people.

I believe that the challenges offered by the real world towards our real autonomous robotsmay be unsurmountable in the close future, unless the robot limitations themselves are explicitlyincluded in the planning, so that the robot can ask for help from available external resources, suchas humans, other robots, and the web for information and crowdsourcing.

Planning can then include the representation of world features (e.g., location of objects, under-standing of language), and actions (e.g., going up a floor, picking up any object) that the robotmay not be able to assess or execute, but for which the robot will be able to ask for help.

Planning can also help attack autonomy problems at different levels of abstraction and differentlevels of uncertainty, such that robots can evaluate alternatives courses of action, and select onesthat address complex joint objectives.

The science of “planning for robots in the real world” is an integrated approach for planning,execution, replanning, and continuous multi-model learning, driven by experience.

The RoboCup@Home tasks are successful examples in home environments. Mobile servicerobots, like the CoBot robots at Carnegie Mellon University, are also good examples of tasks thatinclude several levels of planning can be benchmarked. A benchmark restaurant servicing taskwould be able to include multiple types of robots with different capabilities, such as non-movableBaxter-like robots with arms and non-armed movable CoBot-like robots. We could also have mobilerobots collect situated, localized environment data, e.g., temperature, wifi, noise, pollution levels.

Peter Wurman, Kiva SystemsMulti-Robot Path Planning

Kiva Systems use fleets of small robots to move inventory shelves (pods) around in warehouses.The inventory shelves are carried to replenishment or pick workers who stand at stations along theperimeter. By eliminating the walking that these people would do in a traditional warehouse, theworkforce becomes 2-3x more efficient.

The largest facilities that use Kiva have over a thousand robots on a single, continuous floor,typically laid out as a grid of cells. The robot density can reach as high as one robot per every6 cells of floor space. One of Kiva’s key challenges is what is generally known as multi-vehiclerouting, however there are aspect of Kiva’s problems that are not well-captured in typical researchtreatments. Thus, the Kiva scenario represents a multi-vehicle path planning problem of enormoussize that is ideally suited to a challenge problem.

A Kiva floor is generally broken up into three regions. In the storage area, pods are arrangedin dense, city-like blocks. The perimeter is usually lined with queuing areas for each station whererobots with the next deliveries for that station can line up. Between the station queues and thestorage areas is an interchange area.

30

A typical robot mission starts with the robot having just set down an inventory pod. Havingcompleted one mission, it immediately becomes available for a new mission. It is assigned a new goalof fetching a specific inventory pod and bringing it to a specific station. At the station, the humanwill complete the pick or replenishment task. The duration of the human’s task is imperfectlypredictable. Once the human is done interacting with the pod, the robot must return the pod tothe storage area.

Three things distinguish the Kiva multi-vehicle path planning problem:

• The number and density of robots.

• The delivery pattern in which a continuous stream of robots descends on a small number ofperimeter stations.

• The variable human task at the station that causes robots to release from stations as astochastic process.

A challenge problem that presented an abstract version of a Kiva floor with its three keydistinguishing features would spur research into multi-vehicle coordinated path planning. A break-through in this area has the potential to significantly reduce the number of robots needed to runthese facilities.

31