edmedia2011 online.surveys
DESCRIPTION
Presentation by Dr. Valerie Irvine at EdMedida 11 in Lisbon, Portugal for the Graduate Training StreamTRANSCRIPT
Online Surveys: Best Practices and Ethics
Dr. Valerie IrvineCo-Director, TIE Research LabUniversity of Victoria, Canadahttp://[email protected]: _valeriei
Acknowledgement to Dr. John Williams, University of Northern Iowa
Some Advantages…
• Large Sample Sizes• High Statistical Power• Diverse and Distant Geographical Areas• External Validity and Generalizability• Relatively Easy (relatively!!!)
Use of Internet for Research• From 1998 to 2003, 425% increase in listed studies on
one website (Birnbaum, 2004)
• Google “Online Experiment” • 42,100 pages in 2007• 107,000,000 in 2011• Explore some, but note not to be a participant if you
are clicking through it
Online vs. Traditional
• Are online studies fundamentally different than traditional studies?• Direct contact between researcher and participant?• Use of computers?• Age verification?• Electronic data transmission potentially less
confidential?
Ethical Considerations• Ethical considerations are generally the same for online
studies as for traditional studies.
• Various ethical guidelines still apply• For example, US Code of Federal Regulations:
Protection of Human Subjects or Canada SSHRC Tri-Council Guidelines for Research Ethics
• Guild Guidelines• American Psychological Association’s Ethical
Principles of Psychologist’s and Code of Conduct• Institutional/Ethical Review Boards (IRB or ERB)
Lack of Direct Contact Beneficial?
• May increase response rate because of increased perception of anonymity
• May increase ability to discontinue a study because of less social pressure
Lack of Direct Contact Detrimental?• Harder to judge understanding of consent
form or debriefing• Harder to provide immediate clarification
of participant questions• Harder to verify age
• These problems however are not unique to online studies and also occur in mailed surveys, telephone interviews, and group testing
Seven Ethical Issues
• Informed Consent• Deception• Debriefing• Right to Withdraw• Security of test materials• Confidentiality and Anonymity• Avoiding Harm
Informed Consent
• There is a difference between asking for consent and documenting consent• Asking for consent is easy, an informed
consent statement on a webpage will do.• Documenting consent is more involved.
Options to Documentation of Consent• ERBs typically require some indication of consent.
• A webpage with “I consent” and a “I do not consent” options (buttons) may be sufficient.
• Emailed consent may be adequate
Deception
• The issue with deception in online studies is related to debriefing• Participants may not read or understand debriefing
information• Researchers may have difficulty addressing emotional
turmoil related to deception.
• Studies with minimal deception • Weigh risks vs. possible harm• If justified for other methods with limited contact than
probably justified for online studies
Right to Withdraw
• In-person studies may exert social pressure for participants to stay.
• Online studies may exert little pressure to remain and easy ways to exit a study should be provided• “Quit the Study” links on each page
Withdrawing Data
• Withdrawing data is more problematic• Some studies may collect data throughout the
study. Withdrawing data from these studies will require manual deletion
Withdrawing Data
• Studies which collect data upon completion of the study may have links to exit and not submit data if the participant so choses• E.g., surveymonkey • Watch out for inadvertently losing participants
should they miss clicking the submit button***
Withdrawing Data
• The consent form could tell participants that if they withdraw by clicking a “Quit the study” button, they will be asked whether they would like to withdraw their data, but if they withdraw by closing the browser window, the data they have already submitted will be analyzed.
Confidentiality and Anonymity• Keeping Data Secure
• The steps for data security in online studies is the same as with traditional studies• Restrict access to data• Separate identifying information
• Issue with U.S. Patriot Act• See surveymonkey’s response• See Mt. Alison’s University’s response
Confidentiality and Anonymity• Keeping Data Secure
• Restricting Access• Requires physical security of the server• Requires electronic security of the server collecting
data and the computer storing the data
• Identifying Information• Names, email address, addresses, or even usernames or
locations• Identifying information should be stored in separate files,
databases, or computers
Confidentiality and Anonymity• The level of security needed with directly
depend upon the sensitive of the information collected and the amount of identifying information.• Data can be encrypted on the server and during
transmission.• Data values and labels can be assigned in a way that is
meaningless to others without the key• <input type=”radio” name=”Item3” value=”b”>• rather than• <input type= “radio” name= “How often have you stolen a
car?”• value= “More than once”>
Confidentiality and Anonymity• Public use computers may reduce confidentiality
and anonymity.• Pages may be cached on the computer.• Others using the computer may be able to view study
information in URLs.• Corporations and businesses have legal rights to
search and view information on the computers owned by them. Participants using these computers may be revealing information about themselves.
• IP addresses are sometimes considered identifying information.
Confidentiality and Anonymity• Establishing Identity
• Asking for information that only the participant knows (i.e. student number)
• Sending individualized participation invitations• Recipient of invitation may NOT be the individual expected
Avoiding Harm
• Because much online research is widely advertised, consider additional risks to participants from other countries and cultures. • Everyday topics, words, and pictures may be
distressing or illegal in the participant’s country. • Forewarn potential participants of such content
through the use of appropriate study titles and descriptions.
Avoiding Harm• Researchers should provide detailed information
regarding threats to confidentiality, if interception of the material could result in severe social or legal consequences for participants in some countries.
Measurement• T = O + E
(true score equals observed score plus error)
• Double-barrel items: Agree or Disagree to “I like cats and dogs”
• Numerical ranges: Your Age: 19-25, 25-35, 35-50, 50-65
Measurement• Presentation of Likert Scales
• Horizontal and keep ‘don’t know’ out of it for spatial response formation
• Provide definitions to avoid confusion• Put the definition BEFORE the question
Getting Read• Avoid high-risk words that may send it to
SPAM• Send HTML and plain text email invitations• Insert a large START HERE link• Have a familiar person or person of status
send the email• Do NOT harvest email addresses from the web• Be sure to send reminder prompt
Common Mistakes in Conducting Surveys (online)
• Increasing risk of losing participants with submit button • (e.g., saying “thank you for your participation”
ABOVE the submit button)• In some delivery modes, naming two fields the
same• Ability to check more than one option (radio vs.
checkboxes)
Common Mistakes in Conducting Surveys (online)• Survey webpages being long and overwhelming
• Participants will skip questions or enter false data• Use “skip logic” to shorten # of questions
• Avoid stupid questions in online versions• “I completed this survey:
• By paper and pencil• Online”
• Use of open-ended when structured answers could prevent long data coding time
Pilot it first
• Complete a few online surveys first to see pitfalls and good practices
• Get friends and colleagues to pilot the survey BEFORE you deploy it
Examples of Online Research• http://psych.hanover.edu/research/exponnet.html
• See attitudes toward online counselling (surveymonkey)
• http://psychexps.olemiss.edu (needs plug-ins….)
References
Birnbaum, M. H., & Reips, U. -D. (2005). Behavioral research and data collection via the Internet. In R. W. Proctor and K.-P. L. Vu (Eds.), The handbook of human factors in Web design (pp. 471-492). Mahwah, New Jersey: Erlbaum.
Buchanan, T., Johnson, J. A., & Goldberg, L. E. (2005). Implementing a five-factor personality inventory for use on the Internet. European Journal of Psychological Assessment, 21, 115-127.
Ess, C. (2007). Internet Research Ethics. In A. Joinson, K. McKenna, T. Postmes, and U.-D. Reips (Eds.), The Oxford Handbook of Internet Psychology, pp 487-502. New York: Oxford University Press.
Krantz, J. H. & Dalal, R. (2000). Validity of web-based psychological research. In M.H. Birnbaum (Ed.), Psychological Experiments on the Internet (pp. 35 – 60). San Diego, CA: Academic Press.