1 EMODE workshop – 27-28 September 2007
Automatic Usability Assessment of Multimodal User Interfaces
Based on Ergonomic Rules
Adrian StanciulescuJean Vanderdonckt
Benoit Macq
Université catholique de Louvain (UCL)http://www.isys.ucl.ac.be/bchi
http://www.similar.cchttp://www.openinterface.org
2 EMODE workshop – 27-28 September 2007
Outline
• Introduction• Context
– OpenInterface platform– UsiXML language– UsiXML for OpenInterface
• Navigation in large scale images– Existing solutions– Proposed solution
• Structure of the navigational tasks– Reaching the structure– Working hypopothesis
• Expanded UsiXML model for vocal interaction• Conclusions & future work
3 EMODE workshop – 27-28 September 2007
Introduction
• Information vizualization: succesful paradigm in HCI
• Zoomable user interfaces– Organize information in space and scale– Interaction techniques: translation and scale– E.g.: storytelling, web browsing, browsing of maps– Few usability studies, inconclusive results
• Interaction devices– Traditional interaction: mouse and keyboard– New emergent interaction: vocal and tactile– The problem: optimize the interaction– Combining vocal and tactile vs. Vocal-only: faster, less error
prone, less linguistic complexity, less time to complete the task
4 EMODE workshop – 27-28 September 2007
Introduction
• Our objectives:– Automatic usability assessment of MM navigational tasks over
large scale images– Interactions types: translation and zooming– Interaction modalities: graphical (keyboard, mouse), vocal(speech
input), tactile (finger, stylus pen)– Specification of composing elements: sequential or order-
independent interaction
5 EMODE workshop – 27-28 September 2007
OpenInterface platform
• Provide an open source platform for the design and rapid development of multimodal prototyped applications – Components: pieces of software that provide services (CIDL)– Pipelines: inter-connections between components (PDCL)
• Advantages:– Allow seamless integration of heterogeneous software (C/C++,
Java, Matlab)– Allows rapid prototyping of multimodal applications (easy
software connection)– Reusability of delivered software
• To address:– No OI module for usability guidance at design time– A knowledge base of multimodal ergonomic rules
6 EMODE workshop – 27-28 September 2007
UsiXML language
• Structured according to four levels of abstraction (Cameleon framework)
• Advantages:– Support for multimodal I/O– Separation of modalities– Support for CARE properties– Extendibility to new modalities
• Shortcomings:– Non-hierarchical structure of the grammars– Undefined order of specification for grammar elements– No means to specify the cardinality of grammar elements
7 EMODE workshop – 27-28 September 2007
UsiXML for OpenInterface
Usability adviser tool
Multimodal OpenInterface Applications
UsiXML Specification
Usability Evaluation
Ergonomic rules
Abstracti
on
Abstra
ction
8 EMODE workshop – 27-28 September 2007
Usability adviser tool
• Shortcomings of the existing tools:– No tool to assess the usability of both web and non-web UIs– Formalized language to specify the ergonomic rules: limitations
with respect to the quality of the evaluation
• Our objectives:– Real time guidance during the development life cycle– Evolutional knowledge base– Multiplatform ergonomic rule validation (UsiXML specifications)– Ergonomic rules expressed in a formal language close to natural
language– No need to learn the formal language (rule specification assistant)– Portability of the knowledge base (single file storing the rules)
9 EMODE workshop – 27-28 September 2007
Navigation in large scale images
• Existing solutions– Combine multiple smaller displays
into a single one– Multiple LCDs butted together
into a single one– Cost-intensive, space-intensive, or both
• Our proposal:– Navigational techniques are required– 3 X 3 overlaying grid– Meaningful symetrical pairs
(Top/Bottom, Left/Right)
10 EMODE workshop – 27-28 September 2007
Taxonomy of the navigational tasks
• Translation: top / bottom / left / right / main diagonal up (Bottom Left) / main diagonal down (Top Right)/ second diagonal up (Bottom Right)/ second diagonal down (Top Left)– Translate without specifying a value: the user decides when to stop the
translation – Translate by specifying the number of grid units
• Zooming: zoom in / zoom out – Zooming without specifying the zoom factor: it is inferred from the
selected zoomed area and the size of the display window– Zooming with a zoom factor:
• Fixed zoom factor: zoom by steps• Specified zoom factor: the value is specified by the user
– Zooming in a single selected grid unit (e.g., zoom in Top Left / zoom in Centre, zoom in Bottom Right)
– Zooming in multiple selected grid units: • Adjacent horizontal grid unit selection: zoom in Horizontal Top/ zoom
in Horizontal Centre / zoom in Horizontal Bottom• Adjacent vertical grid unit selection: zoom in Vertical Left / zoom in
Vertical Centre / zoom in Vertical Right• Diagonal grid unit selection: (e.g., zoom in Left Centre to Bottom
Right / zoom in Centre to Top Right.
11 EMODE workshop – 27-28 September 2007
Reaching the structure
• General structure:Instruction:= {Action, Object, Parameter}
• EBNF specific format:Instruction:= {{Action}- [Object] [{Parameter}]}-
• Instanciation of the instruction:
12 EMODE workshop – 27-28 September 2007
Working hypothesis
A. Hypothesis 1: Action applied over 1 optional Object which has 2 Parameters
B. Hypothesis 2: Action-Parameter order• Ordered instruction: Instruction: = Action >> Parameters• Order-independent instruction: Instruction: = Action |=|
Parameters
13 EMODE workshop – 27-28 September 2007
Working hypothesis
C. Hypothesis 3: Equivalent navigational instructions:
{ Sub-instruction} :• groups one or more elements
of an instruction• specified by employing a
monomodal/multimodal interaction
14 EMODE workshop – 27-28 September 2007
Expanded UsiXML model
• VUI support added:– Grammar– Part– Item
• Advantages:– Flexibility
– Reusability
15 EMODE workshop – 27-28 September 2007
Conclusions & future work
• Conclusions:– Large number of choices when deciding in the favour of a
particular set of navigational tasks and the employed modality– Usability studies required: speed, error rates and subjective
satisfaction– Usability guideline are limited and no yet validated– Automatic usability assesment based on a knowledge base of
ergonomic rules
• Future work:– Address the tactile interaction– Expand UsiXML to support tactile interaction– Adding new ergonomic rules related to MM interaction
16 EMODE workshop – 27-28 September 2007
Thank you very much for your attention
http://www.isys.ucl.ac.be/bchi BCHI Lab
http://www.similar.ccSimilar netork of Execellence
http://www.usixml.orgUser Interface eXtensible Markup Language
http://www.openinterface.org OpenInterface