The Thematic Baseline The Thematic Baseline Technique as a Means of Technique as a Means of

Improving the Sensitivity of Improving the Sensitivity of Self-report ScalesSelf-report Scales

David NunezDavid Nunez & Edwin Blake & Edwin Blakehttp://www.cs.uct.ac.za/~dnunezhttp://www.cs.uct.ac.za/~dnunez

Collaborative Visual Computing Laboratory

Department of Computer ScienceUniversity Of Cape Town

2

The CVC Lab, UCTThe CVC Lab, UCT

•Computer Graphics

•HCI & Usability

•Virtual Presence

•VR Applications

•VR Authoring for novices & non-programmers

We are looking for European research partners!

http://www.cs.uct.ac.za/Research/CVChttp://www.cs.uct.ac.za/Research/CVC

3

Self-report scales in presenceSelf-report scales in presence

Not wildly popular but widely usedNot wildly popular but widely used• Many problems – objectivity, sensitivity, Many problems – objectivity, sensitivity,

validity, etc.validity, etc.

We have found them to have low We have found them to have low validityvalidity• Especially in relation to realism/display Especially in relation to realism/display

quality criterion, although this is a well quality criterion, although this is a well known effectknown effect

4

Scale failure: Johns Scale failure: Johns et alet al, (2000), (2000)

Tried to manipulate presence as an Tried to manipulate presence as an independent variableindependent variable• Two VEs – ‘high presence’ and ‘low Two VEs – ‘high presence’ and ‘low

presence’presence’

Manipulation: 9 out of 17 factors Manipulation: 9 out of 17 factors listed by Witmer & Singer (1998)listed by Witmer & Singer (1998)• On the W&S scale, still no difference On the W&S scale, still no difference

with 30 subjects per groupwith 30 subjects per group

5

Other scales, similar resultsOther scales, similar results

Several failures to detect difference Several failures to detect difference between VE and realitybetween VE and reality• Mania (2001) [included five conditions]Mania (2001) [included five conditions]• Usoh Usoh et alet al (2000) (2000)

Construct validity of scales seems to Construct validity of scales seems to be weakbe weak• Either not capturing the construct Either not capturing the construct

(unlikely) or extremely insensitive(unlikely) or extremely insensitive

6What is it like to have your What is it like to have your

presence measured?presence measured?

Many items require an Many items require an implicit comparison implicit comparison of VE experiences to some familiar of VE experiences to some familiar experiences (eg. SUS scale)experiences (eg. SUS scale)

• To what extent were there times when the To what extent were there times when the VE was the reality for you?VE was the reality for you?

• Please rate your sense of being in the VE Please rate your sense of being in the VE on a scale of 1 to 7, where 7 represents on a scale of 1 to 7, where 7 represents your normal sense of being in a placeyour normal sense of being in a place

7

How do you compare to How do you compare to thatthat??

Which experience does one compare to?Which experience does one compare to?• Not defined by the scale!Not defined by the scale!• Extremely common, below the level of Extremely common, below the level of

awarenessawareness

The demand characteristics will lead The demand characteristics will lead subjects to provide an answersubjects to provide an answer• Each will make the comparison differentlyEach will make the comparison differently• This will increase the error variance of the This will increase the error variance of the

scale, and reduce its sensitivityscale, and reduce its sensitivity

8

Providing a baselineProviding a baseline Problem could be alleviated by Problem could be alleviated by

providing a providing a baselinebaseline• Something for the subjects to compare Something for the subjects to compare

toto

Extremely problematicExtremely problematic• How do we avoid experimenter effects?How do we avoid experimenter effects?• How do we capture individual How do we capture individual

differences?differences?• How do we avoid confusing presence How do we avoid confusing presence

and immersion?and immersion?

9

Natural ExpectationNatural Expectation

Have users compare to what they Have users compare to what they expect to experience expect to experience in the VEin the VE• Use Use schemata schemata and and scriptsscripts

Schemata encode relationships Schemata encode relationships between concepts, and link concepts between concepts, and link concepts together with behaviourstogether with behaviours• Great deal of communality, but allow for Great deal of communality, but allow for

individual differenceindividual difference• Abstraction of top-down processingAbstraction of top-down processing

10

Watch me prime youWatch me prime you

11

Which is it?Which is it?

12

Study: 2x2 Factorial ANOVAStudy: 2x2 Factorial ANOVA(n=103)(n=103)

Variables:Variables:• Independent: Priming, Stimulus QualityIndependent: Priming, Stimulus Quality• Dependent: SUS, Witmer & Singer’s PQDependent: SUS, Witmer & Singer’s PQ

Priming manipulationPriming manipulation• VE Relevant – VE Relevant – Related in theme to the VERelated in theme to the VE• VE IrrelevantVE Irrelevant – Unrelated in theme to the VE – Unrelated in theme to the VE

StimQual manipulationStimQual manipulation• High quality – Radiosity, sound, texture-mappingHigh quality – Radiosity, sound, texture-mapping• Low quality – Flat shading, no soundLow quality – Flat shading, no sound

13

Priming manipulation - Priming manipulation - RelevantRelevant

14

Priming manipulation - Priming manipulation - IrrelevantIrrelevant

15

FindingsFindings

StimQual manipulation effectiveStimQual manipulation effective• SUS:SUS: F(1, 99) = 9.68 F(1, 99) = 9.68 pp<0.003<0.003• PQ:PQ: F(1, 99) = 5.99 F(1, 99) = 5.99 pp<0.02 <0.02

InteractionInteraction on priming x StimQual on priming x StimQual• SUS:SUS: F(1, 99) = 10.18 F(1, 99) = 10.18 p<p<0.0020.002• PQ:PQ: F(1, 99) = 4.23 F(1, 99) = 4.23 p<p<0.040.04

16

Means plot: SUSMeans plot: SUS

QUAL:l

QUAL:h

Plot of Means

2-way interaction

F(1,99)=10.18; p<.0019

PRIME

Varia

ble:

SU

S

18

20

22

24

26

28

30

32

p n

17

Means plot: PQMeans plot: PQ

QUAL:l

QUAL:h

Plot of Means

2-way interaction

F(1,99)=4.23; p<.0423

PRIME

Varia

ble:

PQ

135

140

145

150

155

160

165

170

p n

18

DiscussionDiscussion

Priming magnifies differencesPriming magnifies differences• In In relevantrelevant priming condition, H-L priming condition, H-L

difference significant; in difference significant; in IrrelevantIrrelevant condition, not significantcondition, not significant

Priming allowed subjects to express Priming allowed subjects to express presence more easily in presence more easily in questionnairesquestionnaires• Or perhaps magnified responsesOr perhaps magnified responses

19

DiscussionDiscussion

Providing a thematic baseline could Providing a thematic baseline could improve the sensitivity of self-report improve the sensitivity of self-report scalesscales• Highlight differences; not sure of effects Highlight differences; not sure of effects

on absolute scoreson absolute scores

Suggests that higher-level semantic Suggests that higher-level semantic processes involved in presenceprocesses involved in presence