Reactive and Hybrid AgentsBased on �An Introduction to MultiAgent Systems� and slides

by Michael Wooldridge

The Subsumption (Reactive) Architecture

Problems with symbolic/logical approaches (transduction,computational complexity) resulted in the reactive paradigm

The best-known reactive agent architecture based on thefollowing theses

Intelligent behavior can be generated without explicitrepresentations

Intelligent behavior can be generated without explicitabstract reasoning

Intelligence is an emergent property of certain complexsystems

Characteristics of the Subsumption Architecture

1 An agent's decision making is realized through a set oftask-accomplishing behaviors

1 Each behavior can be seen as an individual selection function,which continually maps perceptual input to an action toperform

2 Behaviors (behavior modules) are implemented as �nite-statemachines/rules and include no complex symbolicrepresentation (situation → action rules)

2 Many behaviors can �re simultaneously

1 Modules are arranged into a subsumption hierarchy, with thebehaviors arranged into layers

2 Lower layers in the hierarchy can inhibit higher layers (thelower a layer is, the higher is its priority)

Action Selection in Layered Architectures

Raw sensor input is notprocessed or transformed much

Action selection is realizedthrough a set of behaviorstogether with an inhibitionrelation

We write b1 ≺ b2, and read thisas 'b1 inhibits b2' � b1 is lowerin the hierarchy than b2, andwill hence get priority over b2

Mars Explorer

The objective is to explore a distant planet, and in particular, tocollect sample of a precious rock. The location of the samples is notknown in advance, but it is known that they tend to be clustered.A number of autonomous vehicles are available that can drivearound the planet collecting samples and later re-enter a mothership spacecraft to go back to Earth. There is no detailed map ofthe planet, though it is known that the terrain is full of obstacleswhich prevent vehicles from exchanging any communication

Mechanisms used in the Explorer

1 A gradient �eld

The mother ship generates a radio signal so that agents canknow in which direction the mother ship liesAn agent needs to travel 'up the gradient' of signal strengthThe signal need not carry any information

2 Indirect communicationAgents will carry 'radioactive crumbs', which can be dropped,picked up and detected by passing robots

Individual Behaviors

b0: if detect an obstacle then change direction

b1: if carrying a sample and at the base then drop sample

b2: if carrying a sample and not at the base then travel upgradient

b3: if detect a sample and not at the base then pick upsample

b4: if true then move randomly

The above behaviors are arranged into the hierarchy:b0 ≺ b1 ≺ b2 ≺ b3 ≺ b4

Cooperative Behaviors

b5: if carrying a sample and not at the base then drop 2crumbs and travel up gradient

b6: if sense crumbs then pick 1 crumb and travel downgradient

The modi�ed subsumption hierarchy is the following:b0 ≺ b1 ≺ b5 ≺ b3 ≺ b6 ≺ b4

Limitations of Reactive Agents

1 If agents do not employ models of their environment, thenthey must have su�cient information available in their localenvironment to determine an acceptable action

2 It is di�cult to see how decision-making could take intoaccount non-local information (a 'shot-term' view)

3 The relationship between individual behaviors, environmentand overall behavior is not understandable. It is di�cult toengineer agents for speci�c tasks and there is no methodologyfor building such agents

4 It is di�cult to build agents that contain many layers (> 10)due to dynamics and complexity of interactions between thedi�erent behaviors

Showtime

Now a bit of a show... :)

Hybrid Agents

It is claimed that neither a completely deliberative (pro-active)nor completely reactive approach is suitable for building agents

An obvious approach is to build an agent out of two (or more)subsystems

a deliberative one, containing a symbolic world model, whichdevelops plans and makes decisions in the way proposed bysymbolic AIa reactive one, which is capable of reacting to events withoutcomplex reasoning

Subsystems are arranged into hierarchy of interacting layers

(→ layered architectures)

Types of Layered Architectures

Horizontal layeringLayers are each directly connected to the sensory input andaction output. In e�ect, each layer itself acts like an agent,producing suggestions as to what action to perform

Vertical layeringSensory input and action output are each dealt with by atmost one layer each

Types of Layered Architectures

TouringMachines

TouringMachines consists of three horizontalactivity-producing layers

Each layer continually produces suggestions for what actionsthe agent should perform

Demonstration scenario � autonomous vehicles driving betweenlocations through streets populated by other similar agents

Architecture of TouringMachines

Layers in TouringMachines

Reactive layer

provides immediate response to changes that occur in theenvironmentimplemented as a set of situation-action rules, similarly tosubsumption architecture

r u l e −1: kerb−avo i dancei f

i s − in− f r o n t ( Kerb , Obse rve r ) andspeed ( Obse rve r ) > 0 ands e p a r a t i o n ( Kerb , Obse rve r ) < KerbThreshHold

thenchange−o r i e n t a t i o n ( KerbAvo idanceAng le )

Layers in TouringMachines

Planning layer

achieves the agent's proactive behavior ('day-to-day' runningof the agent)constructs plans employing a library of 'skeletons' calledschemas (hierarchically structured plans)elaborates at run-time in order to decide which plan to follow

Modeling layer

represents various entities in the world (including the agentitself)predicts con�icts between agents and generates new goals toresolve these con�ictsthese goals are posted down to the planning layer

Control Subsystem in TouringMachines

It is e�ectively responsible for deciding which of the layersshould take control over the agent

It is implemented as a set of control rules

Control rules can either suppress sensor information, or censoraction outputs from the control layers

censor − r u l e −1:i f

e n t i t y ( ob s t a c l e −6) i n p e r c ep t i o n −b u f f e rthen

remove−s en so r y − r e c o r d ( l a y e r −R, e n t i t y ( ob s t a c l e −6))

InteRRaP

InterRRaP uses a vertically layered two-pass architecture

It contains three layers

behavior-based layer deals with reactive behaviorlocal planning layer deals with everyday planning to achievethe agent's goalscooperative planning layer deals with social interactions

Each layer has an associated knowledge base (i.e.,representation of the world appropriate for this layer) � from'raw' information to complex models

InterRRaP Architecture

Layer Interactions in InterRRaP

Bottom-up activationa lower layer passes control to a higher layer because it is notcompetent to deal with the current situation

Top-down executiona higher layer makes use of the facilities provided by a lowerlayer to achieve it goals

The basic �ow of control begins when perceptual input arrivesat the lowest layer, and control may �ow to higher layers, andthen back again

DARPA Challenge and Stanley

�Robot� (?) that won the DARPA Grand Challenge in 2005

Developed at Stanford by the team of Sebastian Thrun (→Google driverless car)

Complex software system (+ sensors) �tted in a standard VWTouareg R5 expanded with drive-by-wire capabilities

Software Architecture

Computer Hardware

6 servers (Pentium M) running Linux

3 for running the software, 1 for logging, 2 idle

1 server responsible for video processing2 servers responsible for remaining acitivities (e.g., planning)

Planning

No need for long-term planning

organizers supplied a �le with the routethe �le included waypoints, corridor widths, speed limits

Short-term planning (→ search with cost functions)

driving close to the center of the road (corridor)avoiding obstaclesmaximizing speed (within limits) and minimizing driving time

Planning

Google Car

Cost of additionalequipment ~ 150,000 $

The Future is Now... (?)

Operation of driverless carspermitted by several U.S. states

1 Nevada (2011)

2 Florida (2012 � �rstregistration of Google Car inMay 2012)

3 California (2012)

4 Michigan (2013, but with ahuman in the driver seat)

Commercial availabilitybetween 2017-2020

More than 700 000 miles ofaccident-free tests (April2014)