evaluating vr systems. scenario you determine that while looking around virtual worlds is natural...
TRANSCRIPT
![Page 1: Evaluating VR Systems. Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult](https://reader036.vdocuments.mx/reader036/viewer/2022082821/5697c0131a28abf838ccca13/html5/thumbnails/1.jpg)
Evaluating VR Systems
![Page 2: Evaluating VR Systems. Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult](https://reader036.vdocuments.mx/reader036/viewer/2022082821/5697c0131a28abf838ccca13/html5/thumbnails/2.jpg)
Scenario
• You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult problem. You address this problem by developing a new locomotion technique for virtual worlds.
• What now?– Prove that your design is better than alternatives– What’s “better” and how do you “prove” it?
![Page 3: Evaluating VR Systems. Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult](https://reader036.vdocuments.mx/reader036/viewer/2022082821/5697c0131a28abf838ccca13/html5/thumbnails/3.jpg)
Better how?
• Usability– Intuitiveness, flexibility, functionality
• Presence/Copresence– Pi, Psi
• Performance– Accuracy, precision, efficiency
• Effectiveness– Training, therapy, distraction
![Page 4: Evaluating VR Systems. Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult](https://reader036.vdocuments.mx/reader036/viewer/2022082821/5697c0131a28abf838ccca13/html5/thumbnails/4.jpg)
Proof
• Compare system with alternative(s) by conducting human subjects experiments (user studies)
• How do we go about this “scientifically”?– Design an experiment to identify and potentially
magnify differences you expect to exist between systems?
– Is the experiment a good (valid one)?
![Page 5: Evaluating VR Systems. Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult](https://reader036.vdocuments.mx/reader036/viewer/2022082821/5697c0131a28abf838ccca13/html5/thumbnails/5.jpg)
Classic VR Experiment
• Recruit participants from the local population (population sample)
• Randomly divide that sample into two groups, control and intervention.
• Control group gets normal VR, and intervention group gets new VR
• Compare observations of the two groups to determine if they are significantly (non-randomly) different
![Page 6: Evaluating VR Systems. Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult](https://reader036.vdocuments.mx/reader036/viewer/2022082821/5697c0131a28abf838ccca13/html5/thumbnails/6.jpg)
Experimental Design Validity
• Internal Validity– Does your design properly address possible bias
factors that may lead to incorrect interpretation of observations?• E.g. Selection bias
– Can you definitively establish a cause and effect relationship?• Correlation is not causation
• External Validity– Generalization of results to other settings
![Page 7: Evaluating VR Systems. Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult](https://reader036.vdocuments.mx/reader036/viewer/2022082821/5697c0131a28abf838ccca13/html5/thumbnails/7.jpg)
Observations (Measures)
• Constructs must be operationalized as measures (metrics).
• Data produced by the metric proportional to hypothetical value of the construct
• Construct Validity– “Are you measuring what you think you’re measuring?”– Very difficult to establish (Think Pi, Psi) – Requires community to believe you– Comes after having reliability, predictive validity
![Page 8: Evaluating VR Systems. Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult](https://reader036.vdocuments.mx/reader036/viewer/2022082821/5697c0131a28abf838ccca13/html5/thumbnails/8.jpg)
How do you compare the data?
• Assume (Hypothesize) that all of the data comes from the same exact population.– Null Hypothesis– Alternative Hypothesis (your real belief)
• Find the likelihood of that seeing your particular distribution with random samples of that data
• If the probability of seeing your particular distribution is less than some pre-determined value, REJECT your hypothesis.
![Page 9: Evaluating VR Systems. Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult](https://reader036.vdocuments.mx/reader036/viewer/2022082821/5697c0131a28abf838ccca13/html5/thumbnails/9.jpg)
Statistical Tests
• Most common in VR, by far, is the Student’s T-Test (built into most spreadsheet software)– Can be used to determine the probability that a sample
population has a specific mean– Can be used to determine if two samples have the same
mean
• Compute T-Value for your case
• Determine probability of seeing this or greater T value in the T distribution
![Page 10: Evaluating VR Systems. Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult](https://reader036.vdocuments.mx/reader036/viewer/2022082821/5697c0131a28abf838ccca13/html5/thumbnails/10.jpg)
Example
• Suppose you have Metric M, which yields the following values for group 1 and group 2– [1,7,5,6,4,1,8,5,6,3,7,5,6,4,6]– [2,1,6,1,3,2,1,7,3,1,5,3,8,4,1]
• Is there a significant difference between the two groups?