recent work in model-based user interfaces jeffrey nichols lecture #13 05-830: advanced user...
TRANSCRIPT
![Page 1: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/1.jpg)
Recent Work in Model-Based User Interfaces
Jeffrey Nichols
Lecture #13
05-830: Advanced User Interface Software
![Page 2: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/2.jpg)
Last time…
Model-based User Interfaces Automatic generation of the user interface so the
programmer won’t do a bad job. Dialog boxes are relatively easy to generate The full application interface is hard to generate Abstract descriptions of the interface can be
longer and harder to generate than implementing the interface itself.
Interface builders turned out to be easier…
![Page 3: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/3.jpg)
But work continued…
Focus Changed Task models were leveraged more Design assistant aspect emphasized
A Couple Projects of Interest: Trident Mecano & Mobi-D FUSE AIDE
![Page 4: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/4.jpg)
TRIDENT
Vanderdonckt, J., Knowledge-Based Systems for Automated User Interface Generation: the TRIDENT Experience, Technical Report RP-95-010, Facultes Universitaires Notre-Dame de la Paix, Institut d’Informatique, Namur, 1995.
An interface design assistant Interesting features:
Knowledge-based approach (i.e. expert system) Choosing Widgets Doing Layout
Use of Task Models Decides where separate windows are needed
![Page 5: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/5.jpg)
Choosing Widgets
Used a decision tree Chose abstract
interaction objects (AIO)
Similar to Brad’s Interactor Model
Lots of parameters Continuous? Capacity Etc.
![Page 6: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/6.jpg)
Choosing Layout
Uses Right/Bottom Strategy Next component is placed to the right or below
the current component Decision made by heuristics or designer
![Page 7: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/7.jpg)
Right/Bottom Strategy
![Page 8: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/8.jpg)
Windows from Task Models
Basically used for constructing wizard-like interfaces What information should be on the first screen, etc.
![Page 9: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/9.jpg)
What are task models, anyway?
Description of the process a user takes to reach a goal in a specific domain
Typically have hierarchical structure Introduced by GOMS
Number of different task modeling languages GOMS UAN ConcurTaskTrees
![Page 10: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/10.jpg)
ConcurTaskTrees
Developed by Fabio Paterno et al. for the design of user interfaces
Goals Graphical for easy interpretation Concurrent model for
representing UI tasks Different task types
Represent all tasks, including those performed by the system
![Page 11: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/11.jpg)
Task Building Process
Three phases Hierarchically decompose
the tasks Identify the temporal
relationships among tasks at same level
Identify what objects are manipulated and what actions can be performed on them, and assign these to the tasks as appropriate.
Temporal Relationships T1 [] T2 - Choice T1 ||| T2 - Interleaving T1 |[]| T2 - Synchronization T1 >> T2 - Enabling T1 []>> T2 - Enabling with
Information Passing T1 [> T2 - Deactivation T1* - Iteration T1(n) - Finite Iteration [T1] - Optional T – Recursion
![Page 12: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/12.jpg)
Example
Note: First example is ambiguous
![Page 13: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/13.jpg)
Another Example
![Page 14: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/14.jpg)
Building/Editing Task Models
Tools are available ConcurTaskTrees
Environment
http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”
![Page 15: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/15.jpg)
Recent Systems
XIML – eXtensible Interface Markup Language Developed by the makers of Mecano/Mobi-D and Trident Kitchen-sink language for modeling any part of the
interface design process XWeb
Now known as ICE – Interactive Computing Everywhere ICrafter
A system for integrating user interfaces from multiple devices
Personal Universal Controller My research…
![Page 16: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/16.jpg)
XIML
eXtensible Interface Markup Language Designed by RedWhale Software Intended to support the
full lifecycle of interfacebuilding
![Page 17: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/17.jpg)
XIML Requirements
Central Repository of Data For one user interface or many
Comprehensive Lifecycle Support Abstract and Concrete Elements Relational Support Underlying Technology
XIML must be independent of particular tools
![Page 18: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/18.jpg)
Models in XIML
An XIML document can contain any type of model Task Domain User Presentation Dialog
![Page 19: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/19.jpg)
Example Use for XIML
Multi-platform interface development
![Page 20: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/20.jpg)
Status of XIML
Used by RedWhale Software to drive their interface consultant business They have developed many tools
move interaction data to/from XIML Leverage data in XIML to better understand various
interfaces Automate parts of the interface design process
![Page 21: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/21.jpg)
Model-Based Interfaces for Control XWeb ICrafter PUC
![Page 22: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/22.jpg)
XWeb
Work by Dan Olsen and group at BYU Premise:
“Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service.”
The web comes close to solving this problem, but is interactively insufficient.
![Page 23: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/23.jpg)
XWeb Protocols
Based upon the architecture of the web XTP Interaction Protocol Server-side data has a tree structure Structured Data in XML URLs for location of objects
xweb://my.site/games/chess/3/@winner xweb://automate.home/lights/livingroom/ xweb://automate.home/lights/familyroom/-1
![Page 24: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/24.jpg)
XWeb & XTP
CHANGE message (similar to GET in HTTP) Sequence of editing operations to apply to a sub-tree
Set an attribute’s value Delete an attribute Change some child object to a new value Insert a new child object Move a subtree to a new location Copy a subtree to a new location
![Page 25: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/25.jpg)
Platform Independent Interfaces
Two models are specified DataView – The attributes of the service XView – A mapping of the attributes into high-level “interactors” Interactors are somewhat like abstract interaction objects
Atomic Numeric Time Date Enumeration Text Links
Aggregation Group List
![Page 26: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/26.jpg)
XWeb Example
DataView
![Page 27: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/27.jpg)
XWeb Example
XView
![Page 28: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/28.jpg)
XWeb Example
Interface
![Page 29: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/29.jpg)
Other XWeb Details
Has simple approach for adjusting to different screen sizes Shrink portions of the
interface Add additional columns of
widgets Also capable of generating
speech interfaces Based on a tree traversal
approach like Universal Speech Interfaces
![Page 30: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/30.jpg)
ICrafter
Part of the Interactive Workspaces research project at Stanford
Main objective: “to allow users of interactive workspaces to flexibly interact
with services” Contribution
An intelligent infrastructure to find services, aggregate them into a single interface, and generate an interface for the aggregate service.
In practice, much of the interface generation is done by hand though automatic generation is supported.
![Page 31: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/31.jpg)
ICrafter Architecture
![Page 32: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/32.jpg)
How is aggregation accomplished? High-level service interfaces (programmatic)
Data Producer Data Consumer
The Interface Manager has pattern generators Recognize patterns in the services used Generate interfaces for these patterns
This means that unique functionality will not be available in the aggregate interface
![Page 33: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/33.jpg)
Automatic Generation in ICrafter
![Page 34: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/34.jpg)
Manual Generation in ICrafter
![Page 35: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/35.jpg)
Personal Universal Controller
My work with Brad Problem:
Appliance interfaces are too complex and too idiosyncratic.
Solution: Separate the interface from the appliance and use
a device with a richer interface to control the appliance: PDA, mobile phone, etc.
![Page 36: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/36.jpg)
Idea
Specifications
Control
Feedback
Control existing appliances Generate multi-modal interfaces
![Page 37: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/37.jpg)
- Comm. Protocol- Interface Generators- Specification Lang.Architecture
XML-based
- Appliance Adaptors
![Page 38: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/38.jpg)
Language Design
Approach Create reference interfaces
AIWA Shelf Stereo AT&T Telephone/Answering
Machine
Test interfaces with subjects Users twice as fast and made half
the errors with reference interfaces as compared to manufacturers’ interfaces
Analyze interfaces for functional information
![Page 39: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/39.jpg)
Language Elements
State Variables and Commands Represent functions of appliance State variables have types
Boolean, Enumeration, Integer, String, etc. Variables sufficient for most functions but not all
“seek” button on a Radio
Label Information One label not suitable everywhere
The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and
text-to-speech information
![Page 40: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/40.jpg)
Language Elements, cont.
Group Tree Specify organization
of functions We use n-ary tree
with variables or commands at leaves
![Page 41: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/41.jpg)
Language Elements, cont.
Dependency Information Formulas that specify when a variable or
command is active in terms of other state variables
Equals, Greater Than, Less Than Linked with logical operators (AND, OR)
For example, <and>
<equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals></and>
![Page 42: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/42.jpg)
Interface Generators
Generators for Two Modalities Graphical
Implemented for PocketPC in Java 1.1 Uses dependency information to generate panel structure of
interface
Speech Implemented using Universal Speech Interface (USI)
techniques [Rosenfeld 2001] Uses dependency information to disambiguate shortcut
words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)
![Page 43: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/43.jpg)
Graphical Interface Generator
Focuses on panel structure of user interface
Small groups of controls have basic layouts
Complexity comes from structure of groups
Structure can be inferred from dependency info!
![Page 44: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/44.jpg)
Inferring Structure
Find sets of variables that are “mutually exclusive” Every variable in a set will
never be active at the same time as a variable in another set
Create structure with sets, using overlapping panels
![Page 45: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/45.jpg)
Choosing Panel Types
a)
b)
c)
full screen
tabbed
partial screen
![Page 46: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/46.jpg)
Making the Interface ConcreteFinish conceptual layout
Choose controls (decision tree)
Choose row layouts(one column, two column, etc.)
Allocate space Examine panel contents and
choose sizes
Instantiate and place controls
![Page 47: Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software](https://reader036.vdocuments.mx/reader036/viewer/2022062320/56649c725503460f94924613/html5/thumbnails/47.jpg)
Generating Speech InterfacesAutomatically build USI tree from dependencies
Allows verbal navigation of functional groups
Automatically generate grammar for parser Phrases for query and control
“What is playmode?”“Set playmode to play”“play”
Automatically generate language model and pronunciation for recognizer