©  2015  Brian  N.  Larson   Topic:  Think-­‐aloud  protocols   Page  1  

Topic:  Think-­‐aloud  protocols  This  topic  page  includes  introductory  information,  a  list  of  readings,  and  questions  to  guide  your  reading  and  prepare  you  for  class  discussion;  it  may  also  include  an  individual  or  group  assignment,  which  may  or  may  not  be  graded.  

Introduction  Think-­‐aloud  protocols  have  been  around  for  more  than  40  years.  They  were  intended  as  a  way  to  gather  information  about  the  cognition  of  human  agents  dealing  with  ill-­‐structured  problems.  Ill-­‐structured  problems  contrast  with  well-­‐structured  problems;  the  latter  type  of  problem  is  one  where  the  beginning  state,  end  states,  and  available  moves  are  all  well  known  or  articulated  up-­‐front.  Think  of  playing  tic-­‐tac-­‐toe.  Ill-­‐structured  problems  lack  some  of  that  certainty;  in  fact,  while  doing  an  ill-­‐structured  problem,  one  might  be  uncertain  about  exactly  what  the  problem  is.    In  its  most  straight-­‐forward  form,  the  interviewer  or  facilitator  asks  the  participant  (sometimes  called  the  “subject”)  to  work  through  a  task  and  to  talk  all  the  while  about  what  she  is  thinking  while  she  is  doing  it.  The  participant  is  told  to  ignore  the  facilitator  and  not  to  ask  the  facilitator  for  assistance.  The  facilitator  is  present  only  to  remind  the  participant  to  keep  thinking  aloud  (e.g.,  by  saying  “Keep  thinking  aloud”).  In  theory,  this  should  result  in  the  participant  sharing  what  is  happening  in  her  cognition  from  moment  to  moment  without  her  “editing”  it  or  attempting  to  shape  it  to  make  herself  look  good  (addressing  two  concerns  with  qualitative  interviewing).    The  major  work  on  protocols  is  Ericsson  and  Simon  (1993);  that  book  is  on  reserve  at  the  library  and  is  also  available  electronically  through  the  library.  An  earlier  article  by  them  (1980)  discusses  some  of  the  main  theoretical  concerns  with  collecting  “recollections,”  which  is  what  qualitative  interviews  (about  which  we  have  already  read)  do.  That  article  is  available  on  T-­‐Square.  

Readings  for  this  topic  • van  den  Haak,  M.,  De  Jong,  M.,  &  Jan  Schellens,  P.  (2003).  Retrospective  vs.  concurrent  

think-­‐aloud  protocols:  Testing  the  usability  of  an  online  library  catalogue.  Behaviour  &  Information  Technology,  22(5),  339–351.  http://doi.org/10.1080/0044929031000  

• Optional:  Ericsson,  K.  A.,  &  Simon,  H.  A.  (1980).  Verbal  reports  as  data.  Psychological  Review,  87(3),  215–251.  

Reading  questions  Read  the  following  questions  and  comments  and  think  about  them  as  you  read  van  den  Haak  et  al.:  

• The  authors  refer  to  “face  validity.”  Look  this  up  at  Wikipedia  or  elsewhere.  Why  do  you  think  that  protocol  analysis  has  high  face  validity?  

• They  also  refer  to  other  kinds  of  validity,  including  predictive  validity,  congruent  validity,  and  reliability.  Think  about  what  each  of  these  terms  means.  Why  do  the  authors  and  other  usability  folks  care  about  these  characteristics  of  their  research  methods?  

• In  what  ways  do  the  authors  suggest  that  think-­‐aloud  protocols  may  be  “reactive”?  • Make  sure  you  understand  the  difference  between  concurrent  think-­‐aloud  (CTA)  and  

retrospective  think-­‐aloud  (RTA).  See  p.  340  for  definitions.    

©  2015  Brian  N.  Larson   Topic:  Think-­‐aloud  protocols   Page  2  

• At  p.  341,  the  authors  discuss  reasons  that  RTA  might  be  better  than  CTA,  and  reasons  why  it  might  not  be.  Make  sure  you  understand  the  widely  accepted  advantages  and  disadvantages  of  RTA.  Note  that  the  authors  propose  to  test  some  of  these  views  with  their  study.  

• Note  the  organization  of  the  authors’  presentation  of  their  own  study.  After  their  lengthy  introduction,  they  carefully  describe  their  methods,  present  the  results  they  obtained,  and  then  discuss  the  implications  of  those  results.  This  is  what  is  sometimes  called  the  IMRaD  pattern:  Introduction,  Method,  Results,  and  Discussion.  You  may  want  to  consider  it  for  your  own  projects  in  this  class.  

• On  page  344,  the  instructions  for  the  participant  for  the  study  in  this  article  are  quoted.  Locate  and  identify  them.  You  may  want  to  use  a  similar  instruction  if  you  choose  to  use  CTA  for  your  projects  in  this  class.  

• Note  how  the  authors  organized  the  problems  their  participants  had/found  into  categories:  Layout  problems,  data  entry  problems,  comprehensiveness  problems,  feedback  problems.  Think  about  your  own  projects  in  this  class:  What  categories  of  problems  might  you  anticipate  users  of  the  communications  you  are  examining  to  have?  

• When  the  authors  (p.  349)  say  that  CTA  resulted  in  identifying  significantly  more  problems  “detected  by  means  of  observation  only,”  what  do  they  mean?  Contrast  the  fashion  in  which  RTA  proved  more  useful.  

• The  authors’  overall  conclusion  is  a  bit  perplexing  (p.  350):  “the  results  of  this  study  indicate  that  concurrent  and  retrospective  think-­‐aloud  protocols  can  be  regarded  as  equivalent,  but  clearly  different  evaluation  methods.  A  strong,  and  new  argument  in  favour  of  RTA  protocols  is  that  they  may  be  less  susceptible  to  the  influence  of  task  difficulty,  both  in  terms  of  reactivity  and  in  terms  of  completeness  of  the  verbalisations.”  What  to  they  mean  by  “regarded  as  equivalent,  but  clearly  different  evaluation  methods”?  Can  you  rephrase  that  statement  to  make  it  sound  less  oxymoronic?  

Classroom  activities  for  this  topic  • Observe  a  think-­‐aloud  protocol  in  action.  

Assignments  for  this  topic  • Post  notes  regarding  the  in-­‐class  think-­‐aloud  protocol  to  your  group’s  forum  before  the  

next  class  after  the  TAP.  

Works  cited  Ericsson,  K.  A.,  &  Simon,  H.  A.  (1993).  Protocol  analysis :  verbal  reports  as  data  (Rev.  ed.).  MIT  Press.  Ericsson,  K.  A.,  &  Simon,  H.  A.  (1980).  Verbal  reports  as  data.  Psychological  Review,  87(3),  215–251.