© w. wahlster, dfki mexican international conference on artificial intelligence 2000 wednesday,...

© W. Wahlster, DFKI

Mexican International Conference on Artificial Intelligence 2000 Wednesday, April 12

German Research Center for Artificial Intelligence, DFKI GmbH

Stuhlsatzenhausweg 366123 Saarbruecken, Germany

phone: (+49 681) 302-5252/4162fax: (+49 681) 302-5341e-mail: [email protected]

WWW:http://www.dfki.de/~wahlster

Wolfgang Wahlster

Personalised E-Commerce Agents for Virtual Webpages


First Generation Second Generation Third Generation

Static Web Sites

Fossils cast in HTML

Interactive Web Sites

JavaScripts and Applets

Database Access andTemplate-based Generation

Dynamic Web Sites

Virtual Webpages

Netbots,Information Extraction,Presentation Planners

Adaptive Web Sites

User Modeling,Machine Learning,

Online Layout

Three Generations of Web Sites


A Virtual Web Page

is generated on the fly as a combination of various media objects frommultiple web sites or as a transformation of a real web page.

looks like a real web page, but is not persistently stored.

integrates generated and retrieved material in a coordinated way.

can be tailored to a particular user profile and adapted to a particularinteraction context.

has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components.

Virtual Memory, Virtual Relation, Virtual Reality...

What is a Virtual Web Page?


Yahoo Weather Server

PAN Travel Agent Andi Car Route Planner

Yahoo News Server

Gault Millau Restaurant

Guide

Hotel Guide

AiA: Information Integration for Virtual Webpages


Virtual Webpage Retrieved from 5 Different Servers


Multi-DomainProblem Specs

NETBOT

RetrievedResults

Information Structures Relations, Lists KR Terms

Media Objects Texts, Sounds, Videos Pictures, Maps,

Animations

DistributedInformation

MultipleData Sources

The Combination of Retrieved and Generated Media Objects for Virtual Webpages


RetrievedResults

Select & Design

Select CannedMedia Objects

Design NewMedia Objects

Information Structures Relations, Lists KR Terms

Graphics,Animation

Text, Speech,Mimic

Icons, Clip Art

Frames, Sounds

Reuse & Transform

CoordinateMedia Objects

TransformMedia Objects

Temporal Synchroni-zation

SpatialLayout

Clip, Convert,Abstract

Zoom, Pan,TransitionEffects

Media Objects Texts, Sounds, Videos Pictures, Maps,

Animations

The Combination of Retrieved and Generated Media Objects for Virtual Webpages


Operational Models of Referential Semantics for Robots and Netbots

Robot Netbot

“Screw”

PhysicalObjects

Screw 1 Screw N...

Set of RecognizersSet of Recognizers

Set of SubsumptionRelations in an Ontology


“Departure Time”



WWWObjects

DT 1 DT N

Set of WrappersSet of Wrappers

...


System is able to flexibly tailor presentations to the individual user and the current situation.

Enhancement of User Interfaces through Personalization

An animated characterserves as “Alter Ego”

of the presentation system.

Personalized Presenters at DFKI


The Role of Ontological Annotations for the Generation and Analysis of virtual Webpage

Information Extraction Agent

PresentationPlanner

Webpageswith Ontological

Annotations

Webpageswith Ontological

Annotations

Webpageswithout Ontological

Annotations

Webpageswithout Ontological

Annotations

Virtual WebpageVirtual Webpage

PresentationAgent

Persona

InformationExtraction

AgentsTriAS

With Ontological Annotations in: SHOE, OML, XOL, OIL

and Persona Annotation in PML


Video Dresdner Bank


NetbotPAN

Trip Data

Pictures andGraphics

Pieces of Text

Coordinates forPointing Gestures

Input for SpeechSynthesis

Icons forHyperlinks

HotelAgent

MapAgent

Address

WeatherAgent

Train & FlightScheduling

Agent

Major EventAgent

VirtualWeb

Presentation

Constraint-basedOnlineLayout

PresentationPlanner

PersonaServer

Componentsof virtual

Webpages

AiA

The Generation of Virtual Webpages with PAN and AiA


Persona as a Personal Travel Consultant


WML-

Browser MS-Agent

Controller

WMLSMILAgent Script

PET- PML

PET Persona

Player SMIL

Player

Presentation Planner

AIA’s presentation planner has been extended to accommodate for various target platforms through the introduction of a mark-up language layer


PersonaServer

Behaviors

Presentation GesturesReactive BehaviorsIdle-time actionsNavigation actions

Auditory Characteristics

Sound effects, auditory iconsVoice: male, female

Visual Appearances

Hand-drawnCartoonBitmaps

GeneratedBitmaps from 3D-Models

Video Bitmaps

PPP’s Persona Server implements a generic Presentation Agent that can be easily adapted to various applications


Classification of Persona Gestures

Talking Posture 1• cautious, hesitant• appeal for compliance • avoids body-gestures

Talking Posture 2• active, attentive• self-confident• uses body-gestures

Gesture Catalogue


take-position (t1 t2 ) point-to (t3 t4 )

move-to (t1 t2 ) r-stick-pointing (t3 t4 )

High-LevelPersona Actions

Context-SensitiveExpansion(including Navigation Actions)

Decomposition into Uninterruptable Basic Postures

r-turn (t1 t21 )

r-step (t21 t22 )

f-turn (t22 t2 )

r-hand-lift (t3 t31)

r-stick-expose (t31 t4 )

Bitmaps... ... ... ...

Context-Sensitive Decomposition of Persona Actions


Production ActPresentation Act

Introduce

Create-Graphics

S-Show

S-Wait

S-Position Elaborate-Parts

S-Create-Window

S-Depict

Label Label

S-Point S-Speak S-SpeakS-Point

Qualitative constraints: Create-Graphics meets S-Show, ...

Metric constraints: 1 <= Duration S-Wait <= 1, ...

Distinction between production and presentation acts

(i.e. Persona- or display acts)

Explicit representation of qualitative and quantitative constraints

Extensions of the Representation Formalism


Persona Presents an Automatically DesignedBusiness Chart


Virtual Webpage with Animation Effects Based on a Single GIF Image


Transition Effects in a Series of Retrieved Pictures


Objective:

Enable non-professional computer users to populate their web pages with lifelike characters

PET comes with:

a set of characters and basic gestures

an easy-to-learn Persona markup language

Developer’s PET will include:

a character design tool which enables users to build their own characters

Technical Realization:

Based on XML and Java

PET: Persona-Enabling Toolkit


Specification of the character to be used

Specification of Persona actions

<html><head><title> Persona Test </title></head><body><persona bitmap=“cartoon” ...><uselib url= .../><do name=“greet”/><do name =“speak” args=“hello”/></persona></body></html>

Features:– XML-based

– easy to learn

The Persona Markup Language


PET-ParserPreprocessing (required for non-standard HTML pages) Identification of Persona Tags

PET-Generator

Replacement of Persona Tag with Java Applet

Determination of parameters for Java Applet

Realization of Persona active elements as JavaScript

Generation of DHTML layers (transparency of Applet box)

Generation of data, e.g. audio files for speech output

Generation of scripts

Tasks of PET


URL of Webpage with Persona Tag

Persona Engine

Behavior Monitor

Character Composer

Event Handler

<html><head><title> Persona Test </title></head><body><persona bitmap=“cartoon” ...><uselib url= .../><do name=“greet”/><do name=“standard”/><do name =“speak” args=“hello”/></persona></body></html>

Persona Scriptswaitscreen 4gesture greet 0 0 nullgesture laugh 0 0 null...

Audio Data

Bitmaps

PET Application Server

PET Parser

PET Generator

Webpage with Reference to Java Applet

<html> ...<APPLET archive=“personaplayer.jar”...</APPLET>...</html>

Functional View of PET


Text Input

Speech Input

Menu Input

Direct Manipulation

Input

Web Persona

Triggers

actions of the Persona

Triggers

operations on elements

of the webpage

Mouse Clicks

Mouse Movements

The Bidirectional Control Flow onPersona-Enabled Webpages


Video 911 & Boxter


<PDIV DIVNAME=“name of the element” DVFRAME=“frame name” TOP=“anchor-y” LEFT=“anchor-x”>some HTML elements</PDIV>

<ACTIVEIMAGE SRC=“image” HREF=“url” NAME=“image name” STATUS=“status message” ALT=“tooltip” CACTION=“persona action onClick” OACTION=“persona action on MouseOver” ...>

Active Images An active image starts a persona action when clicked.

Addressable ObjectsAn addressable object is an object which can be addressed and manipulated by Persona via its name and its position.

Persona Active Elements (PAE)


Spatial and Temporal Coordination of Multimedia Presentations

• Multimedia coordination in previous systems:– no declarative representation of spatial and temporal

layout– no synchronization of Persona actions with other

dynamic multimedia objects

• Solution: Use PrePlan for automated generation of SMIL expressions – Synchronized Multimedia Integration Language– officially recommended by W3 consortium


Approach

• High-level specification of temporal and spatial constraints within the operators of the Presentation Planner PrePlan

• Use of the incremental constraint solving toolkit Cassowary– Uniform treatment of spatial and temporal constraints– Transformation of qualitative constraints into metric

constraints– Integration of a backtracking mechanism to handle

disjunctions


Processing Steps

Decomposition of presentation goals into elementary acts and collection of spatial and temporal constraints

Constraint propagation using extended Cassowary constraint solver

Representation of spatial and temporal layout as SMIL constructs


Example of a Plan Operator

(define-plan-operator

:header (A0 (ShowPresentation ?topic))

:constraints

(*and* (BELP (Illustrates ?video ?topic)) (BELP (Video ?video))

(BELP (Sets-to-Music ?audio ?topic)) (BELP (Audio ?audio))

(BELP (Summarizes ?title ?topic)))

:inferiors

(A1 (SAddSmilCode (?video)))

(A2 (SAddSmilCode (?audio)))

(A3 (SAddSmilCode (?title)))

:temporal ((A1 (d) A3) (2 <= begin A1 - begin A2))

:spatial

((aligntop A1) (centerh A1) (centerh A3)

(20 <= top A1 - bottom A3 <= 20))


Representation of Spatial and Temporal Constraints

• Spatial constraints– Qualitative constraints

(Image1 LeftOf Image2), (Image1 TopAlign Image2)– Quantitative constraints

(4 < Top Image1 - Top Image2)

• Temporal constraints– Qualitative constraints

Allen constraints: (meets, before, overlaps, ...)– Quantitative constraints

Linear inequalities: (Begin Audio1 - End Audio2 < 3),

(2 <= Duration Audio1 <= 6)


Resulting SMIL-Document

<smil>

<head>

<meta content="Smiling Saarland" name="title"/>

<layout>

<root-layout height="50" width="300"/>

<region height="30" id="reg43463" left="0" top="0" width="300"/>

</layout>

</head>

<body>

<par>

<audio begin="1.0s" end="21.807s" src="snowman.wav"/>

<textstream begin="0.0s" end="22.807s" region="reg43463" src="snowman.rt"/>

<video begin="3.0s" end="21.807s" src="snowman.avi"/>

</par>

</body>

</smil>


Interactive Web Demo


Synchronization of Persona Actions with other Media Objects

Some Examples:

• point to an object when it appears in a video

• comment on a video 3 seconds after it has started

• Smile when video appears on the screen

• repeat lip movements until audio stops


Synchronization of Persona Actions

(define-plan-operator

:header (A0 (ShowPresentation ?topic))

:constraints

(*and* (BELP (Describes ?audio ?topic)) (BELP (Audio ?audio))

(BELP (TalkingGesture ?video)) (BELP (RepeatGesture ?video))

:inferiors

(A1 (PresentPictureSequence (?topic)))

(A2 (SAddSmileCode (?audio)))

(A3 (SAddSmilCode (?video)))

:temporal ((A2 (d) A1) (2 <= begin A2 - begin A1) (A2 (e) A3))

:spatial

( (aligntop A1) (alignleft A1)

(1 <= bottom A0 - bottom A3 <= 1)

(1 <= right A0 - right A3 <= 1))


SMIL Specification for Persona Presentation

<smil> <head>

<meta name="title" content="Cyberella in RealPix Demo" />

<layout>

<root-layout width="650" height="385" background-color="#9898F8"/>

<region id=“reg471101" top="0" left="12" width="385" height="284" background-color="#9898F8"/>

<region id=“reg081591" top="95" left=“354" width="295" height="385" background-color="#9898F8"/>

</layout> </head> <body>

<par>

<img region= " reg471101" begin= " 0.0" src= "schwarzenacker.rp" fill="freeze"/>

<animation begin= "2.0s" end="15.1" region="reg471101"

src= "talking-gst.rp"/>

<audio begin= "2.0s" end="15.1" src="schwarzenacker.rm" />

</par> </body> </smil>


Resulting Timeline Diagram

At archeological finds in Schwarzenacker, ...

...

...

......

...


Using SMIL to Synchronize Persona Actions


Video Otto 1


Information Extraction Agents

• Information Filtering• Information Retrieval• Information Integration

identify relevant documents

wrappers

– ...– identify and extract relevant pieces of information – transform them into canonical form

wrappers• operational descriptions of a target concept• abstract from concrete occurrence within document• robust against modifications

wrappers• operational descriptions of a target concept• abstract from concrete occurrence within document• robust against modifications


The Trainable Information Agents Framework

Browser

Application

InfoBroker

Info ExtractionTrainer

planning knowledgeuser preferences

domain ontology

Web site annotations

Userrequests

training

specifications

results

info requests info

info requests info or script

PBD dialog

preferences/heuristics

site info/update

site information

combination of "classical" problem-solving methods and information agents

query planning, optimization, and execution

improved dialog guidance


High Degree of Parallelism of Queries


Knowledge about a Webpage Shared by User and Agent

structural

visual/semantic

procedural

NaiveUser

Learning AnnotationAgent

common part (usable for communication)


Video PAN


Example - Ontology

Train_Connection [ from =>> Location;

to =>> Location;

travel_date =>> Date;

time =>> Time;

depart_time =>> Time;

arrive_time =>> Time;

cost =>> Price;

travel_duration =>> Duration;

info_url =>> URL;

... ]


Query Planning - I

• states: information states– concepts / attributes and instantiations

• operators: querying schemes– preconditions (´+´) and effects (´-´)

to time arrive_time travel_duration

< +, +, +, +, -, -, -, u, - >

from travel_date depart_time cost info_url


Query Planning - II

• merging of alternative plans with identical intermediate states

••

••

••

••

••

••

••

(a)(a)

(a,b)(a,b) (a,c)(a,c)

(a,b,c,d)(a,b,c,d)

(a,b,c,d,e)(a,b,c,d,e)

(a,b,c,d)(a,b,c,d)

op1op1 op2op2

op3op3 op4op4

op5op5

(a,b,c,d,f)(a,b,c,d,f)

op6op6

•• (a,b,c,d,f)(a,b,c,d,f)

••

••

••

••

(a)(a)

op1op1 op2op2

(a,b)(a,b) (a,c)(a,c)

(a,b,c,d)(a,b,c,d)

op3op3 op4op4

op5op5

mergingmerging

•• (a,b,c,d,e)(a,b,c,d,e)

op6op6


Query Planning - III

City.value = München

City.language = German

...

State

CityName1

City

value language

Language

Top

String

...

Ontology

< String, Language, String, Language > + + – –

babelfish...

Operators

opprec cc ciSi ::: 0 )(0 opIntS


Query Plan Visualization

• Features– alternative queries– past states– future states– state descriptions– PBD requests– accept / reject PBD

request– assessment of plans– expected completion time


Using the presentation planner to serve mobile users

accommodate for device-specific display and interaction limitations (e.g. 60*90 pixel displays)

The new challenge

- develop new designs for presenting information on mobile devices

- identify elementary design elements

- define new presentation strategies for the automated composition of designs

Approach


Planner

“runway view” “isometric view”

“vertical bar view”

a selected path gets translated by the planner into a sequence of WML pages to be displayed on a mobile phone

Application example: Route Descriptions over the Mobile Phone


Video Otto 2


Use of a Life-like Character for Electronic Commerce

Digital Assistant Selector


Simulated Dialogues as a Novel Presentation Technique

• Presentation teams convey certain rhetorical relationships in a more canonical way

– Provide pros and cons

• The single presenters can serve as indices which help the user to classify information.

– Provide information from different points of view, e.g. businessman versus tourist

• Presentation teams can serve as rhetorical devices that allow for a continuous reinforcement of beliefs

– involve pseudo-experts to increase evidence


Presentation Teams

I recommend you this SLX limousine.


Underlying Knowledge Base

• Representation of domain

– FACT attribute car_1 consumption_car_1

• Value dimensions for cars adopted from a study of the German car market

– safety, economy, comfort, sportiness, prestige, family and environmental friendliness

– FACT polarity consumption_car_1 economy negative

• Difficulty to infer implication of dimension on attribute

– FACT difficulty consumption_car_1 economy low


Example of a Dialogue Strategy

Question: How much gas does it consume?

Answer: It consumes 8l per 100 km.

Negative Response: I’m worrying about the running

costs.

Dampening Counter: Forget about the costs. Think of the prestige!

Header: (dampening_counter ?agent ?prop

?dim)

Constraints: (*and*

(positive ?agent) (pol ?prop ?other_dim positive))

Inferiors: (Speak ?agent

(“Forget about the ” ?dim “!”))(Speak ?agent

(“Think of the ” ?other_dim “!”))


But, it’s fast!

Multiple Decision Support Agents

... ...weighted propositions

User-Adaptive Search Planning

Spare parts for this car are rather

expensive!

Current and Future Work: Multiple Interface Agents for User-adaptive Decision Support


The generation of virtual webpages is heavily based on ontological annotation

The plan-based approach to generating virtual webpages guarantees high-level ontological annotations

Ontological annotations are needed not only for information extraction agents but also for presentation agents

Realization procedures and wrappers form an important part of the referential semantics of objects on the web

Using presentation planning with temporal and spatial constraint processing the low-level media synchronization can be done

automatically using SMIL annotations

Conclusion


Computer Science:

Elisabeth AndréMathias BauerDietmar DenglerAlexander Kröner Marcus MeyerJochen MüllerStephan NeurohrGaby PaulThomas RistWolfgang Wahlster

Graphics Design:

Bernhard KirschRenato OrsiniPeter Rist

Cognitive Psychology:

Susanne van Mulken

The WebPersona and Virtual WebpagesTeam at DFKI


URL of this Presentation:http://www.dfki.de/~wahlster/micai_talk/

© w. wahlster, dfki mexican international conference on artificial intelligence 2000 wednesday,...

Documents