© w. wahlster, dfki easss 2000 2nd european agent systems summer school monday, august 14 2000...

EASSS 20002nd European Agent Systems Summer School

Monday, August 14 2000

German Research Center for Artificial Intelligence, DFKI GmbH

Stuhlsatzenhausweg 366123 Saarbruecken, Germany

phone: (+49 681) 302-5252/4162fax: (+49 681) 302-5341e-mail: wahlster@dfki.de

WWW:http://www.dfki.de/~wahlster

Wolfgang Wahlster

Generating Virtual Webpages

First Generation Second Generation Third Generation

Static Web Sites

Fossils cast in HTML

Interactive Web Sites

JavaScripts and Applets

Database Access andTemplate-based Generation

Dynamic Web Sites

Virtual Webpages

Netbots,Information Extraction,Presentation Planners

Adaptive Web Sites

User Modeling,Machine Learning,

Online Layout

Three Generations of Web Sites

A Virtual Web Page (Wahlster 1998)

is generated on the fly as a combination of various media objects frommultiple web sites or as a transformation of a real web page.

looks like a real web page, but is not persistently stored.

integrates generated and retrieved material in a coordinated way.

can be tailored to a particular user profile and adapted to a particularinteraction context.

has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components.

Virtual Memory, Virtual Relation, Virtual Reality...

What is a Virtual Web Page?

Hotel Guide

Yahoo News Server

Yahoo Weather Server

PAN Travel Agent Andi Car Route Planner

Gault Millau Restaurant

AiA: Information Integration for Virtual Webpages

Virtual Webpage Retrieved from 5 Different Servers

Multi-DomainProblem Specs

NETBOT

RetrievedResults

Information Structures Relations, Lists KR Terms

Media Objects Texts, Sounds, Videos Pictures, Maps,

Animations

DistributedInformation

MultipleData Sources

The Combination of Retrieved and Generated Media Objects for Virtual Webpages

RetrievedResults

Select & Design

Select CannedMedia Objects

Design NewMedia Objects

Information Structures Relations, Lists KR Terms

Graphics,Animation

Text, Speech,Mimic

Icons, Clip Art

Frames, Sounds

Reuse & Transform

CoordinateMedia Objects

TransformMedia Objects

Temporal Synchroni-zation

SpatialLayout

Clip, Convert,Abstract

Zoom, Pan,TransitionEffects

Media Objects Texts, Sounds, Videos Pictures, Maps,

Animations

The Combination of Retrieved and Generated Media Objects for Virtual Webpages

Operational Models of Referential Semantics for Robots and Netbots (Wahlster 1999)

Robot Netbot

“Screw”

PhysicalObjects

Screw 1 Screw N...

Set of RecognizersSet of Recognizers

Set of SubsumptionRelations in an Ontology

“Departure Time”

Set of SubsumptionRelations in an Ontology

WWWObjects

DT 1 DT N

Set of WrappersSet of Wrappers

System is able to flexibly tailor presentations to the individual user and the current situation.

Enhancement of User Interfaces through Personalization

An animated characterserves as “Alter Ego”

of the presentation system.

Personalized Presenters at DFKI

The Role of Ontological Annotations for the Generation and Analysis of Virtual Webpages (Wahlster 1999)

Information Extraction Agent

PresentationPlanner

Webpageswith Ontological

Annotations

Webpageswith Ontological

Annotations

Webpageswithout Ontological

Annotations

Webpageswithout Ontological

Annotations

Virtual WebpageVirtual Webpage

PresentationAgent

Persona

InformationExtraction

AgentsTriAS

With Ontological Annotations in: SHOE, OML, XOL,OIL, DAML

and Persona Annotation in PML

Towards Plan-based Multimodal Interface Agents

Action Planningin Robotics

Artificial Intelligence

Speech Act Theory

Philosophy of Language/Linguistcs

Plan-basedNL Dialog Systems

Computational Linguistics

Plan-basedMultimodal Interface Agents

Artificial Intelligence

Graphical Acts

Semiotics

Gestural Actsand Mimics

Semiotics

NetbotPAN

Trip Data

Pictures andGraphics

Pieces of Text

Coordinates forPointing Gestures

Input for SpeechSynthesis

Icons forHyperlinks

HotelAgent

MapAgent

Address

WeatherAgent

Train & FlightScheduling

Major EventAgent

VirtualWeb

Presentation

Constraint-basedOnlineLayout

PresentationPlanner

PersonaServer

Componentsof virtual

Webpages

The Generation of Virtual Webpages with PAN and AiA

Persona as a Personal Travel Consultant

Browser MS-Agent

Controller

WMLSMILAgent Script

PET- PML

PET Persona

Player SMIL

Player

AIA’s presentation planner (Andre) has been extended to accommodate for various target platforms through the

introduction of a mark-up language layer

Presentation Planner

PersonaServer

Behaviors

Presentation GesturesReactive BehaviorsIdle-time actionsNavigation actions

Auditory Characteristics

Sound effects, auditory iconsVoice: male, female

Visual Appearances

Hand-drawnCartoonBitmaps

GeneratedBitmaps from 3D-Models

Video Bitmaps

PPP’s Persona Server implements a generic Presentation Agent that can be easily adapted to various applications

Classification of Persona Gestures(Andre, Baldes, Rist)

Talking Posture 1• cautious, hesitant• appeal for compliance • avoids body-gestures

Talking Posture 2• active, attentive• self-confident• uses body-gestures

Gesture Catalogue

take-position (t1 t2 ) point-to (t3 t4 )

move-to (t1 t2 ) r-stick-pointing (t3 t4 )

High-LevelPersona Actions

Context-SensitiveExpansion(including Navigation Actions)

Decomposition into Uninterruptable Basic Postures

r-turn (t1 t21 )

r-step (t21 t22 )

f-turn (t22 t2 )

r-hand-lift (t3 t31)

r-stick-expose (t31 t4 )

Bitmaps... ... ... ...

Context-Sensitive Decomposition of Persona Actions

Production ActPresentation Act

Introduce

Create-Graphics

S-Show

S-Wait

S-Position Elaborate-Parts

S-Create-Window

S-Depict

Label Label

S-Point S-Speak S-SpeakS-Point

Qualitative constraints: Create-Graphics meets S-Show, ...

Metric constraints: 1 <= Duration S-Wait <= 1, ...

Distinction between production and presentation acts

(i.e. Persona- or display acts)

Explicit representation of qualitative and quantitative constraints

Extensions of the Representation Formalism

Persona Presents an Automatically DesignedBusiness Chart

Virtual Webpage with Animation Effects Based on a Single GIF Image

Transition Effects in a Series of Retrieved Pictures

Objective:

Enable non-professional computer users to populate their web pages with lifelike characters

PET comes with:

a set of characters and basic gestures

an easy-to-learn Persona markup language

Developer’s PET will include:

a character design tool which enables users to build their own characters

Technical Realization:

Based on XML and Java

PET: Persona-Enabling Toolkit (Müller, Neurohr)

Specification of the character to be used

Specification of Persona actions

<html><head><title> Persona Test </title></head><body><persona bitmap=“cartoon” ...><uselib url= .../><do name=“greet”/><do name =“speak” args=“hello”/></persona></body></html>

Features:– XML-based

– easy to learn

The Persona Markup Language

PET-ParserPreprocessing (required for non-standard HTML pages) Identification of Persona Tags

PET-Generator

Replacement of Persona Tag with Java Applet

Determination of parameters for Java Applet

Realization of Persona active elements as JavaScript

Generation of DHTML layers (transparency of Applet box)

Generation of data, e.g. audio files for speech output

Generation of scripts

Tasks of PET

URL of Webpage with Persona Tag

Persona Engine

Behavior Monitor

Character Composer

Event Handler

<html><head><title> Persona Test </title></head><body><persona bitmap=“cartoon” ...><uselib url= .../><do name=“greet”/><do name=“standard”/><do name =“speak” args=“hello”/></persona></body></html>

Persona Scriptswaitscreen 4gesture greet 0 0 nullgesture laugh 0 0 null...

Audio Data

Bitmaps

PET Application Server

PET Parser

PET Generator

Webpage with Reference to Java Applet

Functional View of PET

Text Input

Speech Input

Menu Input

Direct Manipulation

Web Persona

Triggers

actions of the Persona

Triggers

operations on elements

of the webpage

Mouse Clicks

Mouse Movements

The Bidirectional Control Flow onPersona-Enabled Webpages

<PDIV DIVNAME=“name of the element” DVFRAME=“frame name” TOP=“anchor-y” LEFT=“anchor-x”>some HTML elements</PDIV>

Active Images An active image starts a persona action when clicked.

Addressable ObjectsAn addressable object is an object which can be addressed and manipulated by Persona via its name and its position.

Persona Active Elements (PAE)

Spatial and Temporal Coordination of Multimedia Presentations

• Multimedia coordination in previous systems:– no declarative representation of spatial and temporal

layout– no synchronization of Persona actions with other

dynamic multimedia objects

• Solution: Use PrePlan for automated generation of SMIL expressions – Synchronized Multimedia Integration Language– officially recommended by W3 consortium

Approach

• High-level specification of temporal and spatial constraints within the operators of the Presentation Planner PrePlan

• Use of the incremental constraint solving toolkit Cassowary– Uniform treatment of spatial and temporal constraints– Transformation of qualitative constraints into metric

constraints– Integration of a backtracking mechanism to handle

disjunctions

Processing Steps

Decomposition of presentation goals into elementary acts and collection of spatial and temporal constraints

Constraint propagation using extended Cassowary constraint solver

Representation of spatial and temporal layout as SMIL constructs

Example of a Plan Operator

(define-plan-operator

:header (A0 (ShowPresentation ?topic))

:constraints

(*and* (BELP (Illustrates ?video ?topic)) (BELP (Video ?video))

(BELP (Sets-to-Music ?audio ?topic)) (BELP (Audio ?audio))

(BELP (Summarizes ?title ?topic)))

:inferiors

(A1 (SAddSmilCode (?video)))

(A2 (SAddSmilCode (?audio)))

(A3 (SAddSmilCode (?title)))

:temporal ((A1 (d) A3) (2 <= begin A1 - begin A2))

:spatial

((aligntop A1) (centerh A1) (centerh A3)

(20 <= top A1 - bottom A3 <= 20))

Representation of Spatial and Temporal Constraints

• Spatial constraints– Qualitative constraints

(Image1 LeftOf Image2), (Image1 TopAlign Image2)– Quantitative constraints

(4 < Top Image1 - Top Image2)

• Temporal constraints– Qualitative constraints

Allen constraints: (meets, before, overlaps, ...)– Quantitative constraints

Linear inequalities: (Begin Audio1 - End Audio2 < 3),

(2 <= Duration Audio1 <= 6)

Resulting SMIL-Document

<smil>

<head>

<root-layout height="50" width="300"/>

</layout>

</head>

<body>

</par>

</body>

</smil>

Synchronization of Persona Actions with other Media Objects (Andre, Kleinbauer)

Some Examples:

• point to an object when it appears in a video

• comment on a video 3 seconds after it has started

• Smile when video appears on the screen

• repeat lip movements until audio stops

Synchronization of Persona Actions(define-plan-operator

:header (A0 (ShowPresentation ?topic))

:constraints

(*and* (BELP (Describes ?audio ?topic)) (BELP (Audio ?audio))

(BELP (TalkingGesture ?video)) (BELP (RepeatGesture ?video))

:inferiors

(A1 (PresentPictureSequence (?topic)))

(A2 (SAddSmileCode (?audio)))

(A3 (SAddSmilCode (?video)))

:temporal ((A2 (d) A1) (2 <= begin A2 - begin A1) (A2 (e) A3))

:spatial

( (aligntop A1) (alignleft A1)

(1 <= bottom A0 - bottom A3 <= 1)

(1 <= right A0 - right A3 <= 1))

SMIL Specification for Persona Presentation

<root-layout width="650" height="385" background-color="#9898F8"/>

</layout> </head> <body>

<animation begin= "2.0s" end="15.1" region="reg471101"

src= "talking-gst.rp"/>

</par> </body> </smil>

Resulting Timeline Diagram

At archeological finds in Schwarzenacker, ...

......

Using SMIL to Synchronize Persona Actions

Information Extraction Agents

• Information Filtering• Information Retrieval• Information Integration

identify relevant documents

wrappers

– ...– identify and extract relevant pieces of information – transform them into canonical form

wrappers• operational descriptions of a target concept• abstract from concrete occurrence within document• robust against modifications

The Trainable Information Agents Framework (Bauer, Dengler)

Browser

Application

InfoBroker

Info ExtractionTrainer

planning knowledgeuser preferences

domain ontology

Web site annotations

Userrequests

training

specifications

results

info requests info

info requests info or script

PBD dialog

preferences/heuristics

site info/update

site information

combination of "classical" problem-solving methods and information agents

query planning, optimization, and execution

improved dialog guidance

High Degree of Parallelism of Queries

Knowledge about a Webpage Shared by User and Agent

structural

visual/semantic

procedural

NaiveUser

Learning AnnotationAgent

common part (usable for communication)

Example - Ontology

Train_Connection [ from =>> Location;

to =>> Location;

travel_date =>> Date;

time =>> Time;

depart_time =>> Time;

arrive_time =>> Time;

cost =>> Price;

travel_duration =>> Duration;

info_url =>> URL;

Query Planning - I

• states: information states– concepts / attributes and instantiations

• operators: querying schemes– preconditions (´+´) and effects (´-´)

to time arrive_time travel_duration

< +, +, +, +, -, -, -, u, - >

from travel_date depart_time cost info_url

Query Planning - IV

City.value = München

City.language = German

CityName1

value language

Language

String

Ontology

< String, Language, String, Language > + + – –

babelfish...

Operators

opprec cc ciSi ::: 0 )(0 opIntS

Query Plan Visualization

• Features– alternative queries– past states– future states– state descriptions– PBD requests– accept / reject PBD

request– assessment of plans– expected completion time

Using the presentation planner to serve mobile users (Rist)

accommodate for device-specific display and interaction limitations (e.g. 60*90 pixel displays)

The new challenge

- develop new designs for presenting information on mobile devices

- identify elementary design elements

- define new presentation strategies for the automated composition of designs

Approach

Application example: Route Descriptions over the Mobile Phone

Planner

“runway view” “isometric view”

“vertical bar view”

a selected path gets translated by the planner into a sequence of WML pages to be displayed on a mobile phone

Use of a Life-like Character for Electronic Commerce

Digital Assistant Selector

Simulated Dialogues as a Novel Presentation Technique

• Presentation teams convey certain rhetorical relationships in a more canonical way

– Provide pros and cons

• The single presenters can serve as indices which help the user to classify information.

– Provide information from different points of view, e.g. businessman versus tourist

• Presentation teams can serve as rhetorical devices that allow for a continuous reinforcement of beliefs

– involve pseudo-experts to increase evidence

Presentation Teams (Andre, Rist, Klesen)

I recommend you this SLX limousine.

Underlying Knowledge Base

• Representation of domain

– FACT attribute car_1 consumption_car_1

• Value dimensions for cars adopted from a study of the German car market

– safety, economy, comfort, sportiness, prestige, family and environmental friendliness

– FACT polarity consumption_car_1 economy negative

• Difficulty to infer implication of dimension on attribute

– FACT difficulty consumption_car_1 economy low

Example of a Dialogue Strategy

Question: How much gas does it consume?

Answer: It consumes 8l per 100 km.

Negative Response: I’m worrying about the running

costs.

Dampening Counter: Forget about the costs. Think of the prestige!

Header: (dampening_counter ?agent ?prop

Constraints: (*and*

(positive ?agent) (pol ?prop ?other_dim positive))

Inferiors: (Speak ?agent

(“Forget about the ” ?dim “!”))(Speak ?agent

(“Think of the ” ?other_dim “!”))

Plan Generation and Plan Recognition as Dual Processes

Plan Generation

Given:

Result:

Initial State Goal State

Sequence of actions to reach the goal state from the initial state

How can I reach my dialog goal?

Plan Recognition

Given:

Result:

Initial State Sequence of

observed actions

Goal State

What is the communication goalof my dialog partner?

Operator-based Methods Deductive Methods Graph-based Methods

The generation of virtual webpages is can be achieved by plan-based internet agents.

Ontological annotations are needed not only for information extraction agents but also for presentation agents

Realization procedures and wrappers form an important part ofthe referential semantics of objects on the web

Using presentation planning with temporal and spatial constraintprocessing the low-level media synchronization can be done automatically using SMIL annotations

see www.dfki.de/~wahlster/easss

Conclusion

Computer Science:

Elisabeth AndréStephan BaldesMathias BauerDietmar DenglerMartin KlesenThomas KleinbauerAlexander Kröner Marcus MeyerJochen MüllerStephan NeurohrGaby PaulThomas RistWolfgang Wahlster

Graphics Design:

Bernhard KirschRenato OrsiniPeter Rist

Cognitive Psychology:

Susanne van Mulken

The WebPersona and Virtual WebpagesTeam at DFKI

© w. wahlster, dfki easss 2000 2nd european agent systems summer school monday, august 14 2000...

virtual web page wahlster

generated media objects

dfki virtual webpage

netbots wahlster

wolfgang wahlster

dfki system

multiple web sites

dfki easss

Documents

+ interactive text exploration günter neumann, dfki,...

►thierry declerck (dfki gmbh, lt lab. saarbrücken,...

http:// mobile© 2004 dfki ein fußgängerleitsystem für...

prof. wolfgang wahlster german research center for...

implementing spmd control flow in llvm using ... › devmtg...

agent-based modeling and simulation tutorial - easss 2009 -...

prof. dr. wolfgang wahlster ceo and scientific director of...

© w. wahlster - dfki · international financial reporting...

part ii chap 10-final-code - dfki

user models in dialog systems - dfki › wwdata ›...

the xml3d architecture - dfki

algorithms for matching - dfki

mobile speech-to-speech translation of …...mobile...

wieviel musik- empfehlungen braucht der mensch | stephan...

© 2001 dfki gmbh -1- ontoviz tab; xml tab; rdf backend;...

case-based reasoning and expert systems - dfki

dfki/im&da-00 s. 1 the clockwork project* overview of the...

ehrenvortrag technische univesität darmstadt wolfgang...

partner presentation: saarland university & dfki

innovation und der richtige zeitpunkt - vortrag reinhard...