mutually linked studies - balancing threats to internal and ecological validity in infovis...
DESCRIPTION
Presenter: Niklas ElmqvistBELIV 2010 Workshophttp://www.beliv.org/beliv2010/TRANSCRIPT
![Page 1: Mutually Linked Studies - Balancing Threats to Internal and Ecological Validity in InfoVis Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062616/54b4853d4a7959fd348b45c6/html5/thumbnails/1.jpg)
MUTUALLY LINKED STUDIESBalancing Threats to Internal and
Ecological Validity in InfoVis Evaluation
NIKLAS ELMQVISTSchool of Electrical & Computer Engineering
Purdue University
![Page 2: Mutually Linked Studies - Balancing Threats to Internal and Ecological Validity in InfoVis Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062616/54b4853d4a7959fd348b45c6/html5/thumbnails/2.jpg)
2
The Problem of Validity
• InfoVis evaluation often comes in two flavors:– Rigorous + unrealistic• Scientific rigor• Toy dataset• => Lacking in ecological validity
– Realistic + ad hoc• Real dataset• Imperfect experimental design• => Lacking in internal validity
![Page 3: Mutually Linked Studies - Balancing Threats to Internal and Ecological Validity in InfoVis Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062616/54b4853d4a7959fd348b45c6/html5/thumbnails/3.jpg)
3
Example: The ColorLens[McDonnel & Elmqvist 2009]
![Page 4: Mutually Linked Studies - Balancing Threats to Internal and Ecological Validity in InfoVis Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062616/54b4853d4a7959fd348b45c6/html5/thumbnails/4.jpg)
4
Example: Evaluating the ColorLens
Realistic photographs:• Ecologically valid (+)• Learning effects (-)• Cannot control images (-)• Does not “feel” canonical! (-)
Perlin Noise:• Full control over dataset (+)• Eliminate learning (+)• Canonical and abstract (+)• Not realistic! (-)
?
![Page 5: Mutually Linked Studies - Balancing Threats to Internal and Ecological Validity in InfoVis Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062616/54b4853d4a7959fd348b45c6/html5/thumbnails/5.jpg)
5
Mutually Linked Studies
• Why choose one type of study? Do both! • Idea: Mutually linked studies– 1 x toy study – canonical and exhaustive– 1 x realistic study – prove it works– Complement each other– Recycle much of the mechanics for both
• Experimental design• Tasks• Analysis• Participants (!)
Reduces time and paper space requirements for conducting two studies (they are not separate)!
![Page 6: Mutually Linked Studies - Balancing Threats to Internal and Ecological Validity in InfoVis Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062616/54b4853d4a7959fd348b45c6/html5/thumbnails/6.jpg)
6
Questions?
• Acknowledgments: Color Lens co-authors(coming soon to a journal near you)Pierre DragicevicHM Jean-Daniel Fekete
![Page 7: Mutually Linked Studies - Balancing Threats to Internal and Ecological Validity in InfoVis Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062616/54b4853d4a7959fd348b45c6/html5/thumbnails/7.jpg)
7
Validity Primer
• Several different types of validity…• I will discuss the following here:– Internal validity• Degree to which the outcome is a function of the
controlled parameters of experiment
– Ecological validity• Degree to which the results of the experiment can be
applied to realistic situations