A Case-Study of a Web-Based Method for Repeated-Measures and

Multi-Source Research

Michael J. Walk, M.S.University of Baltimore

michael.walk@ubalt.edu SCiP—Chicago, IL, Nov. 2008

Web-Based Research Valuable research tool for Psychology Dominated by cross-sectional, between-

subject designs

Speaking Abstractly WebRTS:

Web-Based Research Task System A web-based system for poly-task online

research designs. Adhered to methodological and ethical

recommendations for online research. Tested WebRTS in a multi-source research

design (N = 28).

The Task Page – Purpose To control the ordering of stimulus

presentation To prevent repeating or premature

completion of tasks (Reips, 2000)

To separate the research design into short, distinct tasks (Reips, 2000)

To allow participants to return to complete at a later time

The Task Page - Function Find out who the user is Find out what tasks are done so far

Put a check mark by those tasks

Find out what the next task is Make this task an active hyperlink

Find out what tasks are in queue Make these tasks inactive text

WebRTS – Basic Page Structure Header

Same on every page = coherence (Nosek, Banaji, & Greenwald, 2002)

Logout Button (if logged in) University Logo = trustworthiness (Reips, 2000)

Content Footer

Dynamic (page links to important pages) = navigability Email the researcher link = experimenter presence Quit the study link = debriefing (Nosek, Banaji, & Greenwald, 2002)

Website FlowHomepage

Informed Consent

Login Page

Processing script

Task update script

Task 1 Task 2 Task k

Task Page

Debriefing

Yes

Last Task?

New Participant

No

Current Participant

Quit?

Internet

Give co

nsent

Pre-Consent Exit Interview

Refuse consent

WebRTS – Auxiliary Pages Forgotten password retrieval Administrator’s page

Send reminder emails to participants Add fields to database

WebRTS – Technology Specs MySQL database (hosting and database

provided FREE by www.agilityhoster.com) PHP server-side scripts JavaScript for form validation Pilot version

www.ubpsychportal.org/ssa Updated version (WebRTS 2.0)

www.ubcareerlab.org/cip

Case Study Used in a multi-source, poly-task research design

testing relationships between self-monitoring (Gangestad & Snyder, 2000) and accuracy of predicting personality ratings (e.g., Walk, Mitchell, & Yun, 2008)

Anecdotal usability evidence all positive Brief follow-up survey administered online in Aug.

2008 (3 months after Case Study) N = 6 (21%) Green & Pearson’s (2006) Web-Site Usability Instrument

16 items, (1) strongly disagree to (7) strongly agree 4 Open-ended questions

Notable Usability Results I completed the task on the Web site

without much effort. (M = 6.33, SD = .82) After learning to use part of the Web site, I

easily learned to use another part. (M = 6.17, SD = 1.17)

The Web site interface was consistent throughout the site. (M = 6.17, SD = 1.17)

All items M > 5

Item means

Open-Ended Questions What part or page of the website did you like

the best? I liked it all. It was user friendly. I liked the fact that the survey was multiple choice.

This meant that it took less time to complete the tasks.

I liked the fact that the website was easy to navigate when completing the tasks.

Easy to readfollow The set-up of the whole thing was easy to use. i dont know

Open-Ended Responses (cont.) What part or page of the website did you like

the least? No complaints. Unfortunately, I did not like the part of the survey that

had someone else fill out a survey… The part of the website that I did not like was the

portion in which you had to have a supervisor or co-worker complete…

lacks color, kind of boring NA I dont know

WebRTS 2.0 System pages are “easily” configurable

Titles, pictures, colors, researcher names, host institution, etc.

Can set open / close dates Can set max participants Can randomly assign to ordering conditions Prevents repeat submissions by the same

user.

WebRTS – Uses and Applications Repeated measures designs Longitudinal designs Poly-task cross-sectional designs Multi-source (e.g., participant & rater)

designs Diary designs

ReferencesGangestad, S. W., & Snyder, M. (2000). Self-monitoring: Appraisal and

reappraisal. Psychological Bulletin, 126(4), 530-555.Green, D., & Pearson, J. M. (2006). Development of a web site usability

instrument based on ISO 9241-11. Journal of Computer Information Systems, 66-72.

Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002). E-research: Ethics, security, design, and control in psychological research on the internet. Journal of Social Issues, 58(1), 161-176.

Reips, U. (2000). The web experiment method: Advantages, disadvantages, and solutions. In Birnbaum, M. H., ed. Psychological Experiments on the Internet. Academic Press: San Diego, CA.

Walk, M. J., Mitchell, T., & Yun, G. (2008). Know thy social self? Self-monitoring predicts accuracy in rating one’s reputation. Poster session presented at the 20th Annual Meeting of the Association for Psychological Science, Chicago, IL, May 2008.

Questions?