evaluation adam bodnar cpsc 533c monday, april 5, 2004
Post on 19-Dec-2015
223 views
TRANSCRIPT
![Page 1: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/1.jpg)
Evaluation
Adam Bodnar
CPSC 533CMonday, April 5, 2004
![Page 2: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/2.jpg)
Motivation So many techniques, so little
evaluation Are they really effective?
How effective? When are they effective? Why are they effective?
![Page 3: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/3.jpg)
Papers User Studies: Why, How and When?
(Kosara et al., 2003) Navigation Patterns and Usability of
Zoomable User Interfaces with and without an Overview (Hornbaek et al., 2002)
An Evaluation of Information Visualization in Attention-Limited Environments (Somervell et al., 2002)
![Page 4: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/4.jpg)
From Theory to Practice Can we design an effective colour
sequence to illustrate features? Chromatic sequence reveals categories (a) Luminant sequence reveals form (b)
![Page 5: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/5.jpg)
Comparison of Techniques Can we design an
effective texture that conveys 3D shape information better than the current method? Phong shading is default
(a) One principal direction
texture mapping (b)
![Page 6: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/6.jpg)
Study Within Context Can we effectively
integrate semantic depth of field into an application? Multi-layer map
viewer Layers can be opaque,
semi-transparent, or SDOF
No significant results
![Page 7: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/7.jpg)
Other Techniques User studies aren’t always the best
choice Time consuming, difficult to run, answer
only small questions Field study
Observe the user in their native setting Visual designers
Replace part of user test with an expert
![Page 8: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/8.jpg)
What to take away… Good experiments are difficult to
design but are worth the effort
User studies aren’t always the most appropriate method of evaluation
We need to establish evaluation as a standard InfoVis practice
![Page 9: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/9.jpg)
Critique Strengths
Promotes evaluation through example Accessible to those without a
background in HCI Weaknesses
Only good points of studies presented No critique of alternative evaluation
techniques
![Page 10: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/10.jpg)
Papers User Studies: Why, How and When?
(Kosara et al., 2003) Navigation Patterns and Usability
of Zoomable User Interfaces with and without an Overview (Hornbaek et al., 2002)
An Evaluation of Information Visualization in Attention-Limited Environments (Somervell et al., 2002)
![Page 11: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/11.jpg)
Experimental Background Interfaces with an overview
Details of information space together with an overview of the entire information space
Established usability in literature Zoomable user interfaces
Organize information in space and scale, and use panning and zooming to navigate
Mixed results for usability in literature The usability of overviews for zoomable
user interfaces has not been studied
![Page 12: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/12.jpg)
What to Investigate? Question
How does the presence or absence of an overview in a zoomable interface affect usability?
Hypotheses Subjects will prefer the overview
interface The overview interface will be faster for
comparison and browsing based tasks
![Page 13: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/13.jpg)
Dataset and Tasks Dataset
Two maps based on census data
Differ in levels (single vs. multi-level)
Tasks Navigation and
browsing
![Page 14: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/14.jpg)
Study Design Experimental Design
Within 2 x 2 x 2 (interface, task, map) Counterbalanced conditions 32 subjects
Measures Quantitative
Accuracy, recall, speed, navigation actions
Qualitative Preference, satisfaction
![Page 15: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/15.jpg)
Results Significant Effects
Subjects preferred interface with an overview (H1)
Subjects faster with interface without an overview for multi-layer map (H2)
Other No difference between interfaces in
subjects’ ability to correctly solve tasks
![Page 16: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/16.jpg)
Study Implications Consider the trade off between
satisfaction and task completion time
Unify overview with detail window
Consider how map design influences usability
![Page 17: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/17.jpg)
Critique Strengths
Detailed methodology Real dataset and real test subjects Strong statistical analysis and
discussion Weaknesses
Investigators created the maps No explanation for small display used
in experiment
![Page 18: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/18.jpg)
Papers User Studies: Why, How and When?
(Kosara et al., 2003) Navigation Patterns and Usability of
Zoomable User Interfaces with and without an Overview (Hornbaek et al., 2002)
An Evaluation of Information Visualization in Attention-Limited Environments (Somervell et al., 2002)
![Page 19: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/19.jpg)
What to Investigate? Motivation
InfoVis as a secondary display is a practical application but has not been evaluated
Questions How quickly and effectively can people
interpret information visualization while busily performing other tasks?
What are the issues we must consider?
![Page 20: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/20.jpg)
Experimental Setup Primary task
Video game Secondary task
Multiple choice questions about visualization target
Target could be single item or cluster
![Page 21: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/21.jpg)
Study Design Experimental Design
Between/Within 2 x 2 x 2 (time, info density, task)
Counterbalanced conditions 28 subjects
Measures Quantitative
Performance, correctness Qualitative
None
![Page 22: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/22.jpg)
Results Significant Effects
Subjects performed as good or better on low density visualizations vs. high density visualizations
Subjects achieved greater correctness (answering questions) when time = 8sec
Other No difference in primary task
performance before or after the visualization appeared
![Page 23: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/23.jpg)
Study Implications Peripheral visualizations can be
introduced without hindering primary task performance
Effective interpretation in a duel-task scenario requires more than one second
Low information density displays result in performance that is as good as high density displays in a duel-task scenario
![Page 24: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/24.jpg)
Critique Strengths
Ground experiment in previous work Strong statistical analysis and
discussion Weaknesses
Lack of real underlying data Only focused on one type of primary
task
![Page 25: Evaluation Adam Bodnar CPSC 533C Monday, April 5, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d405503460f94a1a27d/html5/thumbnails/25.jpg)
Conclusion Empirical evaluation can lead to
improvements in the design of information visualization
Questions?