Multi-modal e-BusinessMulti-modal e-Business

Stéphane H. MaesStéphane H. Maessmaes@us.ibm.com

Human Language Technologies

Mobile Speech Solutions &

Conversational AdTech

OUTLINEOUTLINEMobile e-business and m-Commerce

WAP and WMLi-Mode

Voice e-Business: VoiceXMLA new Middleware for Mobile e-BusinessMulti-channel and Multi-modal applications

ScenariosPain PointsInhibitorsTranscoding

MVC architecture and use of the DOMKey principles: the DOM (and DOM remote control)Steps through a simple application scenarioMulti-modal browser ArchitectureProtocol discussionProgramming models:

tiersAuthoring methodologies

Standard landscapeConclusions: summary of Multi-modal landscape

OBJECTIVESOBJECTIVES

Understand the issues with Mobile e-BusinessUnderstand the directions of Mobile e-BusinessUnderstand the associated time-frame and infrastructure changes

Mobile e-Business and Mobile e-Business and M-CommerceM-Commerce

WAP & WML - The WAP & WML - The birth of M-Commercebirth of M-CommerceSee WAP Forum: http://www.wapforum.org

Web Server

Content

CGIScripts

etc.

WM

L D

ecks

with

WM

L-Sc

ript

WAP Gateway

WML Encoder

WMLScriptCompiler

Protocol Adapters

Client

WML

WML-Script

WTAI

Etc.

HTTPWSP/WTP

Web Server

ContentContent

CGIScripts

etc.

CGIScripts

etc.

WM

L D

ecks

with

WM

L-Sc

ript

WAP Gateway

WML Encoder

WMLScriptCompiler

Protocol Adapters

WAP Gateway

WML Encoder

WMLScriptCompiler

Protocol Adapters

WML Encoder

WMLScriptCompiler

Protocol Adapters

Client

WML

WML-Script

WTAI

Etc.

ClientClient

WML

WML-Script

WTAI

Etc.

WML

WML-Script

WTAI

Etc.

HTTPWSP/WTP

The WAP Architecture

Internet

Comparison between Internet and WAP Technologies

HTMLJavaScript

HTTP

TLS - SSL

TCP/IPUDP/IP

Wireless Application Protocol

Wireless ApplicationEnvironment (WAE)

Session Layer (WSP)

Security Layer (WTLS)

Transport Layer (WDP)

Other Services andApplications

Transaction Layer (WTP)

SMS USSD CSD IS-136 CDMA CDPD PDC-P Etc..Bearers:

HTMLJavaScript

HTTP

TLS - SSLTLS - SSL

TCP/IPUDP/IP

Wireless Application Protocol

Wireless ApplicationEnvironment (WAE)

Session Layer (WSP)

Security Layer (WTLS)

Transport Layer (WDP)

Other Services andApplications

Transaction Layer (WTP)

Wireless ApplicationEnvironment (WAE)

Session Layer (WSP)Session Layer (WSP)

Security Layer (WTLS)Security Layer (WTLS)

Transport Layer (WDP)

Other Services andApplications

Transaction Layer (WTP)Transaction Layer (WTP)

SMS USSD CSD IS-136 CDMA CDPD PDC-P Etc..Bearers:

SMS USSD CSD IS-136 CDMA CDPD PDC-P Etc..Bearers:

WAP specifies…

❚ Wireless Application Environment● WML Microbrowser● WMLScript Virtual Machine● WMLScript Standard Library● Wireless Telephony Application Interface● WAP Content Types

❚ Wireless Protocols● Wireless Session Protocol (WSP)● Wireless Transport Layer Security (WTLS)● Wireless Transaction Protocol (WTP)● Wireless Datagram Protocol (WDP)● Wireless network interface definitions

For a detailed discussion: seehttp://www1.wapforum.org/what/docs/WD99Technical990506.ppt

Other players: iMode Other players: iMode

iMode: http://www.nttdocomo.com/top.shtmlC-HTML

Voice e-BusinessVoice e-Business

Authoring of speech applications in C/C++ (SRAPI, SAPI etc...)Authoring of speech application in Java (JSAPI)Authoring of Speech application in state tables (DirectTalk - IVRs) and other proprietary IVR scriptsUse of prepackaged libraries:

Java beans (IBM DT Java)Speech objects / Dialog components (Nuance Speech Objects)

Conventional MethodologiesConventional Methodologies

Desktop - Enterprise - Telephony - Network - Embedded

Automatic Speech

Recognition(ASR)

Text toSpeech(TTS)

Names etc.Commands

Natural Language

Understanding(NLU)

Natural LanguageGeneration

(NLG)

DialogManagement

LanguageTranslation

InformationRetrieval/

Management

DictationE-mail

etc.

SpeakerIdentification &

Verification

Speakersimulation &voice fonts

Face & GestureAnimation

Audio-visualRecognition

Conversational Engines Core Components

Desktop - Enterprise - Telephony - Network - Embedded

Java Server Pages

Servlets

EnterpriseJava Beans

HTTPServer

Connectors Data

Markup + Java Java

Web Application Server

- Runs as extension of HTTP Server- Emphasizes separation of roles: - JSPs handle presentation logic - Servlets, EJBs handle business logic- Connectors for access to data & content- Load balancing, distributed processing- Cross-platform portability

ApacheVisual Browser - HTML - WML (thorugh gateway) - ...

DB2, SAP, CICS ...

IBM WebSphereBEA WeblogicSun NetDynamics...

Web Programming Application ModelWeb Programming Application Model

Web-based IVR - Voice Access to Web Web-based IVR - Voice Access to Web ApplicationsApplications

HTTPServer

DataWeb App

ServerVisual

Browser

VoiceBrowser

Web IVR- Common application and business logic- Separate presentation logic for visual & speech channels- Common development tools & skills

HTMLPages

VXMLPages

Voice Access- Voice & data channels over IP- IVR integration

Voice Browser- Manages audio, call control, recognition, text-to-speech, dialog with user

MenuFormBrowser

...

JavaSpeech API

HTTP / Distributed Conversational Protocols

SpeechServer

Java ServletEnterprise

Data

CGI ScriptWeb

Server

Speech Engine

JavaSpeech

TranscodingProxy

HTML Web Server

DominoServer Notes

DataCGI Program

Local Application

Local database

Local service

VoiceXML / HTTP

Speech Engine

Audio:

Client

Application Server

Engine Server

Voice Application Server

VoiceXML browsersVoiceXML browsers

VoiceXMLVoiceXML

IBM: SpeechMLMotorola: VoxMLAT&T / Lucent: PMLVoiceXML Forum: http:www.voicexml.orgSpecifications:

VoiceXML 1.0: http://www.w3.org/Submission/2000/04/W3C Voice activity: http://www.w3.org/Voice/

Speech Synthesis Mark-upMark-up for speech grammarsDialog Mark-up language: VoiceXML 2.0Reusable Dialog ComponentsNatural Language Semantics Multi-modal working group

New Middleware for Mobile New Middleware for Mobile e-businesse-business

MAINFRAME/ TEXT-BASED PARADIGM

PC / GUI PARADIGM

Infrastructure Change: Mainframe to PC

WEB / DECLARATIVE GUI PARADIGM

MOBILE INTERNET / CCI PARADIGM

Date Type Amount04/01 1539 $ 19.5004/02 ATM $ 200.0004/03 1548 $ 250.0004/03 1541 $ 12.5004/04 1533 $8500.0004/05 ATM $ 200.0004/10 ATM $ 200.0004/11 Deposit $ 250.0004/12 1540 $ 979.7704/13 1542 $ 98.50

SERVER

CLIENT

Network

Infrastructure Change: Network and Pervasive Clients

Conversational Computing

Multi-channelMulti-modal: Synchronized channelsDistributedConversational / Free Flow / Mixed

initiative

CLIENT

DefinitionsDefinitions

Channel: a particular browser, device, or a particular modality. Multi-channel applications: applications designed for ubiquitous access through different channels, one channel at a time. No particular attention is paid to synchronization or coordination across different channels. Multi-modal applications: multi-channel applications, where multiple channels are simultaneously available and synchronized.

Note that according to this definition, there are no fundamental differences between multiple devices (multi-device browsing) and multiple modalities.

Note: Multi-media players (e.g. Flash) can be considered as a different type of channel just as today GUI is actually multi-modal:

multi-modality is changing over time. For example, large screen graphical output, keyboard and mouse input constitute the GUI user interface while being clearly multi-modal. As platforms and tools are designed to seamless design applications that integrate these different modalities and applications become widely available, we do not distinguish them any more as multi-modal. They rather become a well

understood and well accepted composite modality.

Multi-channel scenario: travel Multi-channel scenario: travel reservations todayreservations today

Multiple access devicesOne interaction mode per device

PC

Standardized rich visual interfaceNot suitable for mobile use

I need a direct flight from New York to San Francisco after

7:30pm today

There are five direct flights from New York's LaGuardia airport to San Francisco after 7:30pm today: Delta flight nnn...

Book me on the United flight

Access from any telephoneOutput is inherently sequential

Voice

Flights Hotels Cars Packages Cruises Maps EXPRESS SEARCHDeparting from: Going to:[__________________] [__________________]

When are you leaving? When are you returning?[Dec] [31] [Noon] [Jan] [ 1] [Noon]

Tip: We have many more flight, hotel, and car options.

WHAT'S NEWSki Travel: Choose from more than 80 ski destinationsCruise Travel: Take a virtual tour of select cruise ships

WAP or i-modeFROM LGATO SFONON-STOPFLIGHT DEP ARR DLnnn 7:30p 10:55pAAnnn 7:55p 10:55pUAnnn 8:30p 11:55pTOnnn 8:45p 0:10pMWnnn 9:15p 0:45p

FROM LGATO SFONON-STOPFLIGHT DEP ARR DLnnn 7:30p 10:55pAAnnn 7:55p 10:55pUAnnn 8:30p 11:55pTOnnn 8:45p 0:10pMWnnn 9:15p 0:45p

FROM LGATO SFONON-STOPFLIGHT DEP ARR DLnnn 7:30p 10:55pAAnnn 7:55p 10:55pUAnnn 8:30p 11:55pTOnnn 8:45p 0:10pMWnnn 9:15p 0:45p

From: LGA__To: ______Date: ______

From: LGA__To: SFO__Date: ______

From: LGA__To: SFO__Date: 00/12/11

Mobile and becoming ubiquitousHard to enter data

Pain points in multi-channel e-businessPain points in multi-channel e-business

Pain pointshard to enter and access data using small devices

tiny keypads and screens

one interaction mode does not suit all circumstanceseach mode has its pros and cons

all-in-one devices are no panaceabulky and expensive

Most mobile device usage today is not for e-business applications

No immediate relief is in sight:Devices are getting smaller, not largerDevices and applications are becoming more complexAdding color, animation, camera, etc. does not simplify or contribute to e-business CRMs / IVRs are mostly not yet web-centric

Different applications maintained by different IT organizations!

TranscodingTranscoding

XSL stylesheets

Transcoder

Developer pain pointsDeveloper pain points

Multiple Authoring: Manual or automated through transcodingTranscodingChallengess with multiple authoring:

M pages for N channels requires M x N stylesheets or final form pagesTo be developedTo be maintained

How to support new modalities?Each supported device characteristics must be known to the application provider

Authoring of the synchronization for multi-modal applications: Merged languages or synchronization tags

How to support different levels of synchronization granularity?Combinatorial problem

Different channel to synchronizeDifferent synchronization levels

Mobile e-Business requires a new programming models

[ ][ ] [**][**] [ ][ ][**][**] [**][**] [**][**][ ][ ] [**][**] [**][**][**][**] [**][**] [**][**][_][_][_] [_][_][_] [_][_][_][_][_][_] [_][_][_] [_][_][_][_][_][_] [_][_][_] [_][_][_][_][_][_] [_][_][_] [_][_][_][_][_][_] [_][_][_] [_][_][_][_][_][_] [_][_][_] [_][_][_][_][_][_] [_][_][_] [_][_][_][_][_][_] [_][_][_] [_][_][_][_][_][_] [_][_][_] [_][_][_]

Select seat 3A or

Multi-modal scenario: travel Multi-modal scenario: travel reservations tomorrowreservations tomorrow

or

Multiple interaction modes per deviceInteraction mode depends on task and situation and can change at any timeMulti-device browsingEarlier versions: transactional persistence and notificationsLater versions: Free flow - Conversational

I need a direct flight from New York to San Francisco

after 7:30pm today

Book me on the United

flight

Example: mobile GUI + voice

We have many more flight, hotel, and car

options.

Additional examples:

display seat selection chart (not simply "window or aisle")

use voice or keys to enter PIN code and perorms speaker verification

use audio or voice for notifications

information can be saved for later use

FROM LGATO SFONON-STOPFLIGHT DEP ARR DLnnn 7:30p 10:55pAAnnn 7:55p 10:55pUAnnn 8:30p 11:55pTOnnn 8:45p 0:10pMWnnn 9:15p 0:45p

Would you like a window or an aisle seat?

If you know your desired itinerary you can use a shortcut...

Express reservations.From which

airport do you wish to depart?

Multi-modal e-business value Multi-modal e-business value propositionproposition

Multi-modal e-businessvalue proposition

easily enter and access data using small devicesby combining multiple input & output modes

choose the interaction mode that suits the task and circumstancesinput: key, touch, stylus, voice...output: display, tactile, audio...

use several devices in combinationby exploiting the resources of multiple devices

Conversational extensionInteracts with and across application as in an every day dialog: free form, mixed initiative and context management

Context:Devices are getting smaller, not largerDevices and applications are becoming more complexAdding color, animation, camera, etc. does not simplify e-business

User can access a given enterprise via

multiple channels

Common DataCommon Business Logic

Common Presentation Design

MULTIMODAL

Multi-modal Space DescriptionMulti-modal Space Description

User perspective: Integration of multiple modes

Developer perspective:Integration of e-business channels

MULTICHANNEL

MULTIPLEAUTHORING

User can access a given application via

multiple channels

Developer uses several modes to implement a

given transaction

User can choose what mode to use for each of

the interactions that make up a transaction

Common Interaction Design SINGLE

AUTHORING

Nature of Nature of Nature of Nature of the the the the

InteractionInteractionInteractionInteraction

EnvironmentalEnvironmentalEnvironmentalEnvironmental

ConsiderationsConsiderationsConsiderationsConsiderations

Free Flow

:

Conve

rsatio

nal

Customer group Customer need Status

In-vehicle information services

Create portal for car buyers that can produce an annuity revenue stream

Early adopters are purchasing "hard-coded" solutions (not based on middleware)

Financial servicesTravel services

Improve mobile access for high-value transactions Requires 3G wireless networks

Mobile information services Build mindshare and customer base for future ubiquitous services Requires 3G wireless networks

Wireless carriers Grow user base and minutes of airtime by improving mobile ease of use Requires 3G wireless networks

PDA buyers Enable use with one hand, hands-free, or eyes-free

Limited by impact on device cost, size and battery life

Logistics, healthcare, insurance, manufacturing

Improve productivity of mobile employees with hands-busy or eyes-busy transaction tasks

Limited by time and cost to develop a solution

Demand DriversDemand Drivers

Numbers of quotes:

We'll figure out a way to integrate speech recognition with [cell phone] displays ... the answer lies in using the modes of interaction together"

Thomas S. Tullis, VP in charge of human interface design, Fidelity, 11/00 (New York Times)

Mobile devices [have] the possibility of having visual as well as aural information. So you might see something that combines the two"

Chan Suh, CEO, Agency.com, 11/00 (Internet World)

Inhibitors

Presentation gateways-Synchronized voice browser-Mode synchronization

Device capabilities-Audio subsystem and SVD / DSR (Distributed Speech

Recognition)-Synchronized GUI browser

Application development-Developer tools-Standards

Distributed architectureenables 'thin' clientrequires network connection

.. or ..

Device capabilties-Audio subsystem -Multimodal browser-CPU/RAM

Application development

-Developer tools-Standards

Client-centric architectureenables disconnected userequires 'fat' client

Carrier infrastructure-Simultaneous voice &

data (SVD) connection or support for DSR

Model View Controller Principle Model View Controller Principle (MVC)(MVC)

Model

View1

View 2

Controller1

Controller2

Controller3

Dialog State

repository

GUI Renderer

SpeechRenderer

GUIInteraction

Multi-modalInteraction

SpeechInteraction

Modality-independentrepresentation

Invertible mappings from model to views

User must be able to switch channel at any unpredictable moment while interacting with the application and seamlessly continue to interact.

Each piece is distributable

GUI Browser

DOM

Wrapper

Voice Browser

DOM

Wrapper

MM Shell

Model View Controller Architecture for Multi-modal Model View Controller Architecture for Multi-modal BrowserBrowser

Steps through a simple Steps through a simple application scenarioapplication scenario

What is your name?

Please select your drink

GUI Browser Speech Browser

What is your name?

Please select your drink

User enters his nameDependingon the granularity (event level or slot level), each character is transmitted to MM shell or buffered until focuus leavs the slot (DOM events)Shell is informed of the new data model valueViews are updated: corresponding VoiceXML guard variable is updated: DOM update (set variable)Speech browser pompts: Please select your drink

Other scenarios can be considered: e.g. name provided by voice.

CommunicationManager

VoiceTransportInterface

Dat

aTr

ansp

ort

Inte

rface

Sync

hron

izat

ion

Inte

rface

GUI Browser DOM Wrapper

Voice Browser

DOM

Wrapper

MM Shell

ConversationalEngines

DSR encoder

GUI drivers

GUI I/O

Audio drivers

Audio I/O

ContentServer

HTTP

SynchronizationInterface

NetworkEdge Server

Audio Codec (s)

DSR decoder

Net

wor

k Tr

ansp

ort L

ayer

Net

wor

k Tr

ansp

ort L

ayer

Gat

eway

and

rout

er w

ith V

oice

tran

spor

t and

Sy

nchr

oniz

atio

n Su

ppor

t

Multi-modal Browser Architecture: Thin Client Multi-modal Browser Architecture: Thin Client

NetworkEdge Server

GUI Browser

DOM

Wrapper

Voice Browser

DOM

Wrapper

MM Shell

Engines

Audio drivers

Audio I/O

GUI drivers

GUI I/O

ContentServer

HTTP

Local Engines

CommunicationManager

VoiceTransportInterface

Dat

aTr

ansp

ort

Inte

rface

Sync

hron

izat

ion

Inte

rface

Audio Codec (s)

DSR decoder

Net

wor

k Tr

ansp

ort L

ayer

Net

wor

k Tr

ansp

ort L

ayer

Gat

eway

and

rout

er w

ith V

oice

tran

spor

t and

Sy

nchr

oniz

atio

n Su

ppor

t

DSR encoder

A Fat Client Variation of the Architecture A Fat Client Variation of the Architecture

Network(s)Edge Server

GUI Browser

DOM

Wrapper

Voice Browser

DOM

Wrapper

MM Shell

EnginesAudio drivers

Audio I/O

GUI drivers

GUI I/O

ContentServerHTTP

Local Engines

CommunicationManager

VoiceTransportInterface

Dat

aTr

ansp

ort

Inte

rface

Sync

hron

izat

ion

Inte

rface

DSR decoder

Voice Browser

WAPWMLGUI

Browser

PSTN

HTTP

Multi-channel /Multi-modal

Configuration

Net

wor

k Tr

ansp

ort L

ayer

Net

wor

k Tr

ansp

ort L

ayer

Gat

eway

and

rout

er w

ith V

oice

tran

spor

t and

Sy

nchr

oniz

atio

n Su

ppor

t

Audio Codec (s)

DSR encoder

Multi-channel and Multi-modal server Multi-channel and Multi-modal server

CommunicationManager

VoiceTransportInterface

Dat

aTr

ansp

ort

Inte

rface

Sync

hron

izat

ion

Inte

rface

GUI Browser DOM Wrapper

Voice Browser

DOM

Wrapper

MM Shell

ConversationalEngines

DSR encoder

GUI drivers

GUI I/O

Audio drivers

Audio I/O

ContentServer

HTTP

SynchronizationInterface

NetworkEdge Server

Audio Codec (s)

DSR decoder

1

2 2

3

4

5

6

7

7

8

Net

wor

k Tr

ansp

ort L

ayer

Net

wor

k Tr

ansp

ort L

ayer

Gat

eway

and

rout

er w

ith V

oice

tran

spor

t and

Syn

chro

niza

tion

Supp

ort

9

Components, protocols and interfaces to specify and / or standardize:1.Voice transport protocol, in particular RT streaming DSR protocols. This includes: Interface between the DSR codec and the communication manager; Protocol between network edge server and DSR decoder; Interface between DSR decoder and conversational engines2. Synchronization protocols between multi-modal shell and DOM wrapper filters (e.g. SOAP, UDDI)3. Interface and protocol for transmission of the synchronization protocols, conversational distributed protocols, data transport protocols and remote conversational engine control protocols between the communication manager and the network edge server.4. Conversational engine remote control protocols5. VoiceXML DOM. Events and interfaces must be specified.6. WML DOM. This exists as XML DOM, but it has never been officially adopted for WML.7. DOM wrappers8. Authoring frameworks, programming models and language specifications: multi-channel and multi-modal9. Multi-modal shell

Standard, Protocols and Interfaces to specifyStandard, Protocols and Interfaces to specify

Synchronization ProtocolsSynchronization Protocols

DOM filtered events:DOM Level 2 UI events (and higher): XHTML generalized UI events, VoiceXML events, Etc…HTTP (or other protocols) requests:URI requests

DOM commands:Page pushOutput eventsSet eventsGet/set variablesDOM tree manipulation

Blocking messagesConfirmation messagesEvents are systematically time stamped. A clock synchronization protocol - see Network Time Protocol (NTP ?)

SOAP- XML RPCIndependent of Transport layer

SOAP v1.1 Spec Defines ..SOAP v1.1 Spec Defines ..

A transport-independent messaging model

Transport bindings for HTTP

An encoding model for a type system

RPC over HTTP

+ Today: SOAP Attachments (MIME type for binaries)

See http://www.w3.org/2000/xp/

Service-Oriented ArchitectureService-Oriented Architecture

Service Instance

Formats & Protocol

Disc

over

y

Service Types

Service QoS

Service FlowsWSFL

WSDL

WSDL

WSEL

SOAP

UDDI

UDDIADS

XML ProtocolsSOAP

GUI Browser

DOM

Wrapper

Voice Browser

DOM

Wrapper

MM Shell

The DOM MVC Multi-modal Browser architecture is an example of web intermediary / service composition

WSFL

WSDL

Service Broker

Service Requester

Service Provider

Other / New ProtocolsOther / New Protocols

Synchronization granularity negotiationDiscovery (UDDI / ADS)Registration of the viewsDescription of the view characteristics / capabilities (WSDL & CC/PP)Spontaneous connection of new views DisconnectHandshake to confirm active connection

Engine remote control protocol:RSTP extensions?etc...

Voice Transport ProtocolsVoice Transport Protocols

Voice and data.Voice over IP: RTP DSR: RT streaming version?

MVC supports all the MVC supports all the authoring authoring methodologiesmethodologies

Multiple authoringMultiple authoring

explicit synchronization tags: indicate when a channel must be synchronized to another and how. This includes co-visit tags that indicate:

to a channel what the other channel should load and present (hence the name co-visit) - Co-browser approaches

Authoring method primarily designed for co-browsers Indicates that a notification must be given to the MM shell to update the other views

HTML VoiceXML

Global Cafe Block Global Cafe Block

Name block Name block

List Sel block List sel block

Submit "Submit"

Multiple authoringMultiple authoringnaming conventions:

In each channel, fields or items that should be synchronized have a same name or portion of name. Modality specific blocks have no corresponding blockMM shell stores a pseudo DOM tree of nodes

HTML VoiceXML

Global Cafe Block Global Cafe Block

Name block Name block

List Sel block List sel block

Submit "Submit"

<name = node 1/>

<name = node 1/>

<name = node 2/>

<name = node 3/>

<name = node 4/>

<name = node 2/>

<name = node 3/>

<name = node 4/>

Multiple authoringMultiple authoringmerged pages:

An application is described by pages that consists of merge portions associated to each target channel; e.g. a series if snippets of HTML and VoiceXML. Also encompasses annotations in one channel page to support other channels or modalities, e.g. to indicate the role of an element or provide associated data file information like the grammar to associated to a form field. (includes traditional command and control)MM shell can produce naming convention or synchronization tags

Global Cafe Block

Global Cafe Block

Name block

Name block

List Sel block

List sel block

Submit

"Submit"

Single authoringSingle authoringOne common modality independent representation of the application (interaction and data model) + specializationTransformation to different views with for example naming conventionMVC

Global Cafe Block

Global Cafe Block

Name block

Name block

List Sel block

List sel block

Submit

"Submit"

Message to user

Message to user& Free form input$name

Message to user

Selection from list$list

Action submit

Data model to transmit

<name = node 1/>

<name = node 2/>

<name = node 3/>

<name = node 4/>

<name = node 5/>

<name = node 6/>

Channel-independent description of the application

Modality/channel Specific Specialization

Modality Specific Presentation

Data

Business Logic

HTML VoiceXML

Global Cafe Block Global Cafe Block

Name block Name block

List Sel block List sel block

Submit "Submit"

<name = node 1/> <name = node 1/>

<name = node 2/>

<name = node 4/>

<name = node 6/>

<name = node 2/>

<name = node 4/>

<name = node 6/>

<name = node 3/> <name = node 3/>

<name = node 5/> <name = node 5/>

Channel-independent description of the application

Modality/channel Specific Specialization

Modality Specific Presentation

Data

Business Logic

Tier 1 & Tier 0

Tier 2

Tier 3

Conventional Factorization

in Tiers

Multi-channel / Multi-modalFactorization

Multi-modalMVC mapping

Views (Acted upon by

controllers)

Model: Channel independent

Representation

Business and Backend

Multi-modalBrowser

MVC Multi-modal

Browser(Local or

Distributed)

Web Application

Server, Web Server and

backend

Tiers positioning with respect to the MVC paradigmTiers positioning with respect to the MVC paradigm

Tiers IllustrationTiers Illustration

What is your name?

Please select your drink

GUI Browser Speech Browser

What is your name?

Please select your drink

Tier 0/1 - Presentation layer: Dialog interaction with the user: What is your name, please select your drink

Tier 2 - Business login:Flow of application: collect user name, provide list of drinks, collect selection, transfer selection to bar for preparation, compute bill, ask the user if he wants to pay or keep a tab; add to restaurant books

Tier 3 - Data base:List of customers, List of drinks, "business ledger/books" etc...

Single Authoring Programming ModelSingle Authoring Programming Model

Data

Business Logic

Data Models

Interaction

Modality Specific Presentation

Specialization

Programming Model for:Multi-channel ApplicationMulti-modal ApplicationMulti-device Applications

Programming Model:1. Programming by interaction (channel/modality independent):

Interaction and data Model layer2. Specialize for class of channel3. Specialize for specific channel4. Support different navigation flows

Single Authoring is the key programming model for Mobile e-Business

Web Application

Server

iML________

Multi-channel Architecture - Single Multi-channel Architecture - Single AuthoringAuthoring

Transcoder

Network Edge

Server:Gateways

Voice Application

Server

HTML

WML

VoiceXML

HTMLBrowser

WMLBrowser

Voice

Application

Backend

XML________

XSLTs

ClientCharacteristics

Data Files

Support Servlets, JSPs, beans etc...

HTTP

WSP

ETSI DSR Application and Protocols WG

W3CDevice-Independent

Authoring Activity

W3CMulti-modal WG

Multi-modalSub-group X

WAP Forum

Multi-modal Web Workshop

Other W3CActivities

Device Indep AuthWeb Workshop

Other W3CActivities

Call forreview

Call forindustry proposal

Multi-modalRequirements& case scenarios

- Authoring (single and multiple authoring)- Multi-modal architecture and interfaces

Single authoring

CC/PP & Other specs

VoiceXMLForum

W3C Voice Activity

~7/01

2/26/01

11/00

10/00

9/00

12/00

1/01

1Q00

Multi-Modal Standard Multi-Modal Standard LandscapeLandscape WAP Forum

3GPP

Other W3CActivities

?

Multi-modal Protocols

specification and recommended

architecture

MC Single authoringmethodologies requirements

andrecommendations

Multi-modal authoring specifications,

Architecture& Interfaces

2Q00 3Q00 4Q00 1Q01 2Q01 3Q01 4Q014Q00 1Q022Q00 3Q00 4Q00 1Q01 2Q01 3Q01 4Q014Q00 1Q02

(Thin client) Multi-modal browser architecture recommendation

Protocols specs+ other TBD

EnablementOpportunity

Summary of multimodal e-business landscape

Customer groupsIn-vehicle information servicesFinancial and travel servicesMobile information servicesWireless carriersPDA buyersLogistics, healthcare, insurance, manufacturing

Pain pointshard to enter and access data using small devicesone interaction mode does not suit all circumstancesall-in-one devices are no panacea

3GStandards

Device capabilities

Multimodal 'thin client'2001 - Japan2002 - Europe2003 - Americas

Multimodal 'fat client'2000 - Cars2001 - PDAs2008? - Feature phones

Key inhibitors

Trends show that all inhibitors will dissapear in the next 2 to 3 years