The Pragmatic Web: Agent-Based Multimodal WebInteraction with no Browser in Sight

Alexander Repenning & James Sullivan

University of Colorado, Computer Science Department, Boulder, USA

ralex@cs.colorado.edu

Abstract: To a large degree information has become accessible – anytime, anywhere – but not necessarilyuseful. Unless the right information is presented at the right time, in the right place, and the right wayinformation may simply become unwelcome noise. The Pragmatic Web is about giving control to informationconsumers so they can customize how to use information. The original Web – The Syntactic Web – has givenmost of the control to the information producers but, aside of trivial font and plug-in options, has provided littlecontrol to information consumers. The Semantic Web better separates content from presentation but has not yetreceived general acceptance and lacks end-user customization tools. The Pragmatic Web – presented here – isabout providing control to information consuming end-users by enabling them to express computationally howto turn existing information into personally relevant information.

Keywords: auditory input and output, context aware computing, end-user programmable agents, end-userdevelopment and adaptation, sensors and actuators, virtual reality and 3D interfaces.

1 IntroductionInformation is no longer a scarce resource. However,this achievement is largely useless if information isnot provided in a format tailored to the user. Mostpeople in the United States, for instance, now haveaccess to online information thanks to inexpensivecomputers and information appliances (Fox et al.,2000), and public access to computing resources inlibraries and other institutions. Web-basedinformation can be accessed – through wires orwirelessly – from desktop computers, laptops,PDAs, cell phones, and specialized informationappliances. However, ubiquitous information accessdoes not imply universal information access. Therepresentational formats chosen by informationproducers often do not match the needs ofinformation consumers. For instance, a Web pagemay contain highly relevant information to a user,but this information cannot be accessed if the usercannot see or cannot read. Reasons forproducer/consumer information representationmismatch include:

• Wrong Modality: Blind users cannot readtextual descriptions. Automatic text-to-speechinterfaces may be able to verbally convey thetextual contents of a Web page to users, but ifthe Web page is formatted for visual access, the

sequential presentation of information as speechmay be unintelligible or inefficient.

• Wrong Language: Crucial explanatory textmay be provided in the wrong language. Lessthan 15% of U.S. Web sites contain Spanishtranslations (Lamb, 2001).

• Wrong Nomenclature: Information may beexpressed in an unfamiliar measurement system.The translation of Celsius to Fahrenheit orkilometers to miles, while scientifically trivial,may present a serious impediment to many users.

• Wrong Time: Information may be correct,relevant and readable, but presented at the wrongtime. Stock information, for instance, is mostuseful when presented in real time.

• Wrong Format: Information can look great ona large computer monitor, but be completelyunsuitable for small information devices such asPDAs and cell phones.

A mismatch between information presentation andthe formats required by an information consumer canbe difficult to address with a traditional Webbrowser. For economic reasons, a producer maychoose to use a single representation scheme thataddresses only the needs of an anticipated majorityof information consumers. Because creating andmaintaining multilingual Web sites can be costly,

Figure 1: Distribution of Control over Representationshared by Information Producer and Information

Consumer.

information consumers who are not proficientEnglish readers have few options with this model.

In the case of a person with a cognitive disabilityplanning to use the public transportation system,there is a good chance that essential informationexists on the Web but cannot be accessedmeaningfully with conventional informationtechnology. The goal of this proposal is to extendcontrol to information consumers and let them useinformation in fundamentally new ways that mightnot be anticipated by information producers.

The ultimate questions involve who controlsinformation representation and how the informationis processed. The Control over Representationdiagram (Figure 1) illustrates a continuum of controlthat ranges between two extreme positions. Fromleft to right, it identifies conceptual as well astechnical frameworks in order of increasinginformation consumer control: the Syntactic Web,the Semantic Web and the Pragmatic Web.

The Syntactic Web. In this first generation of Webtechnology, a markup language (HTML) is used todefine content at a high level of detail. Thissyntactic level controls the appearance ofinformation. Information producers define content,font selection, layout, and colors. Informationconsumers have limited control over representationsin their browser, including adjusting the size offonts, and enabling/disabling animations and plug-ins.

The Semantic Web. According to Tim Berners-Lee,the Semantic Web (W3C, 2001, Berners-Lee et al.,2001, Berners-Lee, 1999) will “radically change thenature of the Web.” (Berners-Lee et al., 2001) Theformal nature of representation languages such as theeXtensible Markup Language (XML) and theResource Description Framework (RDF) make Web-based information readable not only to humans, butalso to computers. For instance, semantic-enabledsearch agents will be able to collect machine-readable

data from diverse sources, process it and infer newfacts. Other research projects, such as the Avantiproject, have studied how to separate web contentfrom display modality to better serve the sensoryand perceptive abilities needs of users (Stephanidisand Savidis, 2001).

Unfortunately, the full benefits of the SemanticWeb may be years away and will be reached onlywhen a critical mass of semantic information isavailable. Critics of the Semantic Web (Frauenfelder,2001) point out the enormous undertaking ofcreating the necessary standardized informationontologies to make information universallyprocessable.

The Pragmatic Web. In contrast to the Syntacticand Semantic Web the Pragmatic Web is not aboutform or meaning of information but about howinformation is used.

The Pragmatic Web’s mission is to provideinformation consumers with computational agents totransform exis t ing in format ion into relevantinformation of practical consequences. Thistransformation may be as simple as extracting anumber out of a table from a single Web page ormay be as complex as intelligently fusing theinformation from many different Web pages intonew aggregated representations.

This agent-based transformation needs to beextremely flexible to deal with a variety of contextsand user requirements. An agent running on adesktop computer with a large display may utilizerich graphical representation versus an agent runningon a cell phone with a small display may have toresort to synthesized text information to convey thesame information.

The Pragmatic Web research explores the practiceof using information and the design of toolssupporting this process.

Instead of the traditional “click the link” browser-based interfaces, agents capable of multimodalcommunication will provide access to Web-basedinformation. Agent communication methods includefacial animation, speech synthesis, and speechrecognition and understanding. End-users orcaretakers will instruct agents to transforminformation in highly customized ways. Agents willwork together to combine information from multipleweb pages, access information autonomously ortriggered by voice commands, and representsynthesized information through multimodalchannels.

End-User Customization (Nardi, 1993, Jones,1995) will be an integral part of the Pragmatic Web.Successful development of end-user customizationwill make giant steps toward an Every Citizen

Interface (ECI) (Committee et al., 1997) by lettingminority groups of information consumers, possiblydown to the single individual level, obtain ways tocontrol information representations in accordancewith their specific needs. This computer-supportedInformation Processing is a form of knowledgemanagement (Sumner, 1999, Murray, 1999) thatturns raw data into information. End-usercustomization will let users specify w h e r einformation is accessed (e.g. part of an existing Webpage), how it is accessed (e.g., voice activated), andhow information is further processed. For instance, aPragmatic Web application could run on a wirelessPDA equipped with GPS to help a person with acognitive disability navigate through town using thelocal public transportation system.

The Pragmatic Web does not intend to subsumethe Syntactic Web or the Semantic Web. On thecontrary, the Pragmatic Web will initially work withthe Syntactic Web by letting end-user customizableagents extract information out of existing (HTML)Web pages (Inmon, 1996). When the Semantic Webreaches a minimal critical mass, the Pragmatic Webwill then utilize the Semantic Web with agents thataccess ontologies and make inferences based on theserepresentations.

This article describes how end-user programmableagents allow users to change modalities to makeinformation show up at the right time, and to fuseinformation from multiple sources into new formats.To illustrate the Pragmatic Web framework, twoexample applications are presented. The MountainBike Advisor keeps track of current weatherconditions and personal biking preferences. A voiceinterface recognises requests, sends out agents toaccess remote sensor information and applies user-defined rules to interpret information pragmatically.The Boulder Transportation system tracks GPS-equipped busses in real time, renders a 3Dvisualization, and interprets bus location informationto make navigational recommendations for personswith cognitive disabilities.

2 Example ApplicationsTwo early Pragmatic Web applications are presentedhere to illustrate how agents are being programmedby end-users, how these agents access information inexisting Web pages, how they process thatinformation, and finally how the interact with theusers by interpreting the information found. Bothapplications are working prototypes.

2.1 Boulder Mountain Bike AdvisorThis AgentSheets-based application connects real-time Web information with speech recognition. Auser asks, “Where should I go mountain biking?”

Several agents located on a map of Boulder Countyreact to this voice command (Figure 2). These agentsare representing locations that are possible candidatesfor biking and also feature real time, Web accessibleweather information sensors.

Figure 2: Boulder mountain bikeadvisor.

Rules, previously defined by the users, capturepragmatic interpretations. For instance, an agent mayreply (using speech output): “It’s really nice up hereat Betasso but you should bring a jacket because it’sa little windy.”

Behind the ScenesHow does all of this work? The information used bythe Bike Advisor originates in a number of Weatherstations featuring a large array of sensors accessiblevia Web pages. The network of weather stations inBoulder County, Colorado is relatively dense so thatfor most bike trail locations a weather station can befound to sufficiently well estimate current weatherconditions. The C1 Niwot Ridge Weather station isclosest to the Sourdough Bike Trail and features awell-organized Web page (Figure 3).

Figure 3: NCAR’s Foothill Weather Sensor Web Pageserves as input for agents to make recommendations.

Our scenario begins with the user speaking the word“Biking.” All the agents, represented as icons on themap (Figure 3), listen to voice commands. Agentsin AgentSheets are programmed by end-users inVisual AgenTalk (Repenning and Ambach, 1996).Visual AgenTalk is a rule-based language. The firstrule of the Sourdough biking advisor agent has aspeech recognition condition becoming true as theresult of what the user just said. The agent nowtriggers a second rule group called “check” whichincludes two conditions accessing the C1 weatherstation Web page to extract the current temperatureand wind speed information. The Fahrenheitinformation is numerically converted into Celsiusinformation and, using text to speech, announced tothe user: “Temperature at Sourdough is currently–3.2 degrees Celsius.”

Figure 4: AgentSheets end-user programmableauthoring environment illustrating if-then rules.

Temperature and wind speed are further interpretedby calling a third rule group. This is the pragmaticpart of the interpretation using temperature and windthresholds that are only relevant to the user who hasexpressed these rules. Unlike the objective part ofthe rule which merely communicated the numericalvalue of the temperature, the pragmatic part isdirectly employed to reach a decision. In our case

since the temperature (in Fahrenheit) is less than 40degrees the agent advises against a bike ride atSourdough: “It's bitter cold up there. Don't comehere!” In more moderate cases the agent would haverecommended to bring additional clothing such aswind stoppers in case the wind speed exceeds adifferent threshold.

There are two important things to note here. First,the information returned by the agent is of a highlypragmatic nature that is directly relevant to the userand to the goal of mountain biking. These rules areeasily created and modified by the user using ahighly visual programming environment (Figure 4).

Second, the interface, input and output, can betailored to suit user needs and delivered in multi-modal forms, including visual and speech. Forexample, the map display while helpful to locate aspecific trail, is not necessary to interact with thesystem. Indeed the shift in modality from text tospeech allows the entire interaction to take place overa cell phone without the need for any display or atraditional browser, nor any need to modify theoriginal Web page in any way.

2.2 Mobility AgentsNavigating through a city public transportationsystem can be a daunting challenge for a person withmemory and attention problems due to cognitivedisabilities. The Mobility-for-All research projectsponsored by the Coleman Institute for CognitiveDisabilities is studying how personalized mobileinformation technologies can assist such travelers byeliminating information overload from traditionalnavigational artifacts (Sullivan et al., 2002).

artifact Purpose

maps spatial relationships between one’scurrent location and destination; identifyrouting options; provide an abstractmeans to assess overall trip progress.

schedules temporal information about routeavailability at a given day and time.

landmarks to confirm global progress and anticipateimportant events or tasks that will comenext, such as prepare to get off, etc.

labels andsigns

to understand the local environment,including: current location, where to meettransportation vehicles; identify the“right” vehicle; where to get on and off;where to pay; etc.

clocks to synchronize schedules with physicalevents, including transportation vehiclearrivals and departures.

Table 1: Essential navigation artifacts commonly foundin public transportation systems

As buses travel on Boulder Colorado city streets,they report their Global Positioning System (GPS)locations on a wireless network. Agents track buslocations, and use this information to generatepersonally relevant “just-in-time” attention andmemory prompts in a visual and/or auditory formthat can be tailored to the mobile user’s needs,abilities, and preferences.

ArchitectureOur current prototype consists of a number ofconnected components (Figure 5). The existinginfrastructure includes t h e GPS-equippedtransportation system of 27 busses, transceivers thatconnect the GPS sensors (one per bus) with a centralGPS information receiver, and a Web server thatputs the positional information onto the Web. Everytwo seconds each bus sends an update of GPSposition, heading, identity and speed. Currently,this information is gathered on an operations consolewhere it is both archived and visualized as dotsmoving on a map. These dots are observed by ahuman dispatcher, who informs drivers of bunchingand other problems. To a caretaker or our traveler, aperson with a cognitive disability, thisrepresentation of information (at the syntactic level)is without meaning or use.

Mobility AgentsThe Mobility Agent server is the central architecturalcomponent that mediates all communications.Mobility Agents running on this server can read theGPS information (bus ID and location) from theprovider server and track buses currently in the

transportation system. The server in turn feeds realtime bus and traveler information to the 3DTransportation Situation Viewer.

When a traveler makes a choice, such as picking adestination, the server receives that choice alongwith the traveler’s location. Mobility agents generateappropriate responses, which the server thentransmits to the traveler via events that containmultimodal instructions, reminders, or prompts. Theserver can also deal with traveler confirmations andpanic alarms.

The responses and information provided to thetraveler vary with the user profile, and depend onmobility agents with custom settings for thatspecific traveler. User profiles for multiple travelersare maintained on the server and contain personalizedinformation such as schedules, typical itineraries anddestinations, and contact information. The behaviorof mobility agents can be readily modified, so theserver also provides caregivers customizationcapabilities. The current version of the Mobilityagents uses Visual AgenTalk rules similar to theones shown previously to map data at the syntacticlevel (the GPS location strings) to pragmatic level(e.g., “your bus is here”).

Mobility agents produce signals that: 1) defineevents available to the traveler; 2) create responses indifferent modalities; and 3) provide exceptionhandling and error resolution mechanisms. Acaregiver then personalizes the mobility agents forparticular travelers by selecting settings appropriateto travelers’ needs and abilities.

Figure 5: Mobility Agent Architecture

Real-Time Transportation ViewerCaretakers see a real-time visualization of thetransportation state via the Real-Time TransportationSituation Viewer. The client tracks the buses in thetransportation system as they run their routes. Thebuses are depicted as moving 3D objects on a maprendered in OpenGL. Bus stops are clearly markedwith superimposed bus stop signs. The travelers’locations are tracked and represented on the 3D mapin real-time. The Mobility Agent server provides busand traveler location information.

Figure 6: A mobile agent-based prototype that providesjust-in-time prompts for people with cognitive

disabilities as they use public transportation systems.

Transportation Visualization lets caregivers monitortravelers, assess a traveler’s ability to effectively usethe transportation system, detect difficulties in dailyroutines, and receive emergency notifications.Viewers can assume different perspectives in the 3Dword including fixed camera positions (e.g., at the abus stop), and cameras tracking objects (e.g., busdriver, or traffic helicopter perspective).

Mobility Agent Customization ClientThe Mobility Agent Customization Client would letcaregivers customize user profiles and mobilityagents through a browser interface. User profiles cancontain personal information such as typicalitineraries; such information can be used to detectdeviations from a normal route and evoke error-handling procedures. The Customization clientwould provide an interface for customizing thebehavior of mobility agents to map specificsituations to appropriate actions. The client wouldallow the specification of normal as well asexceptional responses to traveler actions and choices.For example, in a normal situation when everythinggoes according to plan, the caregiver can define thesequence of actions and prompts to be given toMelanie, the cognitive disabled teen, after shechooses her destination (home, in the case illustrated

by the scenario). This might include: 1) finding thelocation of the right bus; 2) informing Melanie howclose it is to the bus stop she is at; 3) warning herwhen it is arriving; 4) guiding her to board the bus;5) informing her when her destination isapproaching; and 6) reminding her to gather herbelongings before leaving the bus. In an exception-handling situation, such as Melanie boarding thewrong bus, the caregiver can customize increasinglyintrusive prompts. These might include: firstwarning her that she is on the wrong bus with subtlesound notifications; then advising her to talk to thedriver; and then, if the situation is not resolved,having the client application on the cell-phone senda panic alarm to the caregiver. The caregiver couldthen contact the traveler directly by phone. Eventsand reactions are tailored to travelers with differentcognitive abilities. For example, an individual withthe ability to follow more complex instructionsmight be advised to get off the bus, cross the street,and catch a bus going in the other direction.

Wireless deviceThe client application running on a wireless deviceaccepts a traveler’s input choices and providesinstructions, reminders and prompts to the travelerusing multimodal means of communication, such asvoice input and output, sound, images, and movies.

Our current prototype is based on a cell phoneemulator running on a laptop equipped with awireless network. The prototype is functional in thesense that it is connected to the Mobility agents,receives information based on the actual buslocations and allows users to define their goals.

The prototype illustrates agent-based componentsof a mobile architecture (Figure 7). The personsimulated in this prototype (Melanie) is assumed tobe a teen with developmental disabilities includingattention and memory deficits. Melanie can be“directed” to a bus stop where a bus is approaching,and the prompting sequence is “triggered” byselecting a destination option on her phone.

As the simulation runs, Melanie’s mobile phonegenerates visual and auditory prompts triggered byreal world events. Prompts are generated to “getready” for her approaching bus, “please board now”when the bus stops at her location, “please pull thestop cord and prepare to get off” as the busapproaches the destination stop, “please get off here”at the destination stop, and finally, “don’t forgetyour backpack.” As Melanie performs these tasks,she is also rewarded with reinforcing praise.

The 3D visualization component could alsoprovide bus system operators or waiting passengerswith an overview of bus locations and status.Observers can watch real time traffic, play backrecorded data or assume different camera perspectives

(e.g., birds-eye, bus stop perspective, bus driverperspective). Bus users can locate relevant bussesbased on their current position and bus identificationinformation. End-user development tools allows caregivers or service-providers, to specify rules that turnthe general bus information space into personallyrelevant, pragmatic information communicatedthrough cell phones.

3 DiscussionA big concern with parsing general Web pages isthat the location or the format of a Web page maychange. Indeed this is a valid problem, which,depending on the severity of the change and therobustness of the parsing approach used, may resultin information that no longer can be accessed, or,worse, may result in reporting the wronginformation. One way to think about the problem isto assume that changes to Web information whichwould confuse agents are also likely to confuse andeven upset real users. For instance, these days Webdesigner are more careful with the creation of URLssince they know that people share them in emailsand collect bookmarks. A Web site design strategythat frequently changes the location of essential Webpages is extremely likely to upset users.

A different approach to make parsing of Webpages more robust is to factor out Web pagedepending parts, which can be maintained separately.AgentSheets agents can be freely distributed throughthe Behavior Exchange (Repenning and Ambach,1997). This way a community of service providerscan take over the responsibility of centrallymaintaining Web page depending agents. End-userdevelopers of agent-based applications may chooseto download the latest version of agents just in time.In AgentSheets, for example, this is simple sinceagents can download other agents.

The Semantic Web with its formalized interfacecould provide a more robust approach. ThePragmatic Web framework can and should use theSemantic Web but at least for now may need to fallback in most cases onto the Syntactic Web and moreinformal parsing simply because the majority ofWeb information is not yet available in semanticform.

4 ConclusionsThe goal of The Pragmatic Web is to provide morecontrol to end-users with respect to usinginformation. Relatively simple end-userprogramming techniques allow end-users to

Figure 7: A mobile architecture for locating and delivering traveler information – and finding help if something goeswrong.

transform general information available on the Webinto personally relevant information accessible viawireless devices. End-user control - where to accessinformation, how to process information, how toinvoke information access, when to presentinformation, how to process information and how topresent information.

We have built a number of applications that,using relatively simple end-user programs, havewrapped up existing Web information in radicallydifferent interfaces. The resulting shift is not about aquantitative information access improvement butabout a qualitative shift in affordances. Theapplications built are encouraging but should just beconsidered simple instances of the much largerPragmatic Web framework.

5 AcknowledgementsThis research has been supported by the NationalScience Foundation (EIA 0205625, DMI 0233028)and by the Coleman Initiative.

6 ReferencesBerners-Lee, T. (1999) Weaving the Web: The

Original Design and Ultimate Destiny ofthe World Wide Web by its Inventor,Harper, San Francisco, CA.

Berners-Lee, T., Hendler, J. and Lassila, O. (2001)The Semantic Web, Scientific American.

Commission on Physical Sciences. (1997) MoreThan Screen Deep: Toward Every-CitizenInterfaces to the Nation's InformationInfrastructure, National Academy Press,Washington, D.C.

Fox, A., Johanson, B., Hanrahan, P. and Winograd,T. (2000) Integrating InformationAppliances into an Interactive Workspace ,IEEE Computer Graphics andApplications, 30, 54-65.

Frauenfelder, M. (2001) A Smarter Web, TechnologyReview.

Inmon, W. H. (1996) The Data Warehouse and DataMining, Communications of the ACM, 39,49-50.

Jones, C. (1995) End-User Programming, IEEEComputer, 28, 68-70.

Lamb, E. (2001) Web content struggles to goworldwide , In Red Herring, Vol. 91, pp.38-39.

Murray, A. J. (1999) Knowledge management andconsciousness, Advances in Mind–BodyMedicine, 16, 233-237.

Nardi, B. (1993) A Small Matter of Programming,MIT Press, Cambridge, MA.

Repenning, A. and Ambach, J. (1996) TactileProgramming: A Unified ManipulationParadigm Supporting ProgramComprehension, Composition and Sharing,Proceedings of the 1996 IEEE Symposiumof Visual Languages, Boulder, Colorado,pp. 102-109

Repenning, A. and Ambach, J. (1997) TheAgentsheets Behavior Exchange:Supporting Social Behavior Processing, InCHI 97, Conference on Human Factors inComputing Systems, ExtendedAbstractsACM Press, Atlanta, Georgia, pp.26-27.

Stephanidis, C. and Savidis, A. (2001) UniversalAccess in the Information Society:Methods, Tools, and InteractionTechnologies , International Journal ofUniversal Access in the InformationSociety, 1, 40-55.

Sullivan, J., Fischer, G., Binder, T. and Gregory, J.(2002) Human-Centered PublicTransportation Systems for Persons withCognitive Disabilities - Challenges andInsights for Participatory Design, In 7thParticipatory Design Conference (Ed,Wagner, I.) Malmö, Sweden, pp. 194-198.

Sumner, M. (1999) Proceedings of the 1999 ACMSIGCPR conference on Computer personnelresearch, In Proceedings of the 1999 ACMSIGCPR conference on Computer personnelresearch New Orleans, LA USA.

W3C (2001), http://www.w3.org/