growing pains: launching our first longitudinal study
DESCRIPTION
In our ongoing quest to discover how our users think, our UX group conducted a four-month-long study. As we would soon discover, this was no mean feat, as there was very little precedent from the UX community on the nuts and bolts of longitudinal studies. We learned the hard way how to actually set up a longitudinal study, how to maintain it, how to collect and analyze data, and perhaps most importantly, how to get real insights that can go into design solutions. In this session, I'll present a case study of our methodology, our insights, what mistakes we made, and what you can do to make qualitative research a success in your organization.TRANSCRIPT
Growing painsLeanna Gingras - @leegoesplacesInternet User Experience 2011
Hi.
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
Grab ‘n Go Gus
Savvy Sally
“that’s not it…that’s not it either…”
“this looks good enough.”
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
1. Our research need
http://www.flickr.com/photos/lwr/2446626352/sizes/l/in/photostream/
Grab ‘n Go Gus
“this looks good enough.”
IUE October 2011 @leegoesplaces
Selecting our tools
IUE October 2011 @leegoesplaces
“A longitudinal study is a correlational research study that involves repeated observations of the same variables over long periods of time.”
--Wikipedia
http://dux.typepad.com/dux/2009/05/usability-over-time-longitudinal-research-studies.html“I participated in the "Best Practices in Longitudinal Research" workshop at CHI 2009 with a small group of academics and practitioners from five countries. … Here's a brief sampling of a few topics from the day's discussions.”
User experience over time: An initial framework“This paper presents an in-depth, five-week ethnographic study that followed 6 individuals during an actual purchase of the Apple iPhone". We found prolonged use to be motivated by different qualities than the ones that provided positive initial experiences.”
(and many abstracts and workshops)
IUE October 2011 @leegoesplaces
UX resources are sparse
IUE October 2011 @leegoesplaces
To learn how to retain participants: Minimizing respondent attrition in longitudinal research: Practical implications from a cohort study of adolescent drinking
More on retaining participants: Retaning Participants in Longitudinal Community Research: A Comprehensive Protocol
Examples of how much to pay: Psychology Department at the University of Vermont
IUE October 2011 @leegoesplaces
Psychology resources galore
IUE October 2011 @leegoesplaces
2. Planning
http://www.flickr.com/photos/dacotahsgirl/299198494/sizes/o/in/photostream/ IUE October 2011 @leegoesplaces
“The goal of this study is to learn how a novice researcher’s research patterns and strategies change over time.”
IUE October 2011 @leegoesplaces
Decide the research question
“The goal of this study is to learn how a novice researcher’s research patterns and strategies change over time.”
IUE October 2011 @leegoesplaces
Decide who to study
“All of it.”
IUE October 2011 @leegoesplaces
Decide what data to collect
“We will be collecting data from usability tests, interviews, Twitter entries, diary entries, and focus group activities.”
IUE October 2011 @leegoesplaces
Decide what data to collect
1 - Initial interview & usability test1 - Contact the recruits for longitudinal study1 - Orientation, 1 – Follow-up email 3 - check-in emails1 - Mid-semester interview based on tweets1 - End-semester interview1 - Focus group
IUE October 2011 @leegoesplaces
Plan touchpoints and activities
IUE October 2011 @leegoesplaces
Plan compensation strategy
IUE October 2011 @leegoesplaces
Plan retention strategy
http://www.flickr.com/photos/goincase/2720494170/sizes/o/in/photostream/
3. Execution
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
The actual study
1 usability test10-30 tweetsAt least 3 diary entries3 in-person interviewsFocus group
IUE October 2011 @leegoesplaces
Screener usability tests
IUE October 2011 @leegoesplaces
Screener usability tests
Agenda:
• Explain study expectations and compensation
• Plan dates for touchpoints throughout semester
IUE October 2011 @leegoesplaces
Kickoff meeting
IUE October 2011 @leegoesplaces
Twitter guidelines
IUE October 2011 @leegoesplaces
Diary due dates
IUE October 2011 @leegoesplaces
Diary entries
Standard set of questions:“Tell me about your toughest moment.”
Observations from Twitter/diary to build interview from:“Does she understand where to find scholarly stuff?”
Exercise:“Let’s revisit one of your earlier Tweets. Walk me through finding resources on Faust. Is there anything you’d do differently?”
IUE October 2011 @leegoesplaces
Interviews
“so how did your research change over the semester?”
IUE October 2011 @leegoesplaces
End of semester focus group
“so how did your research change over the semester?”
IUE October 2011 @leegoesplaces
End of semester focus group
“so how did your research change over the semester?”
IUE October 2011 @leegoesplaces
End of semester workshop
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
Building activities
IUE October 2011 @leegoesplaces
Articulating activities
YAY! We’re done, right?
http://www.flickr.com/photos/markfive/4121088992/ IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
4. Analysis
• Does she understand where to get academic content?
• What keywords did she try?
http://www.island94.org/2009/09/boston-subway-in-vector-format-svg/
Analyze as we go
IUE October 2011 @leegoesplaces
Asked specific questions of the data
“The goal of this study is to learn how a novice researcher’s research patterns and strategies change over time.”
What do they learn? What tools and hacks are useful to them and why? What do they hate and why?
IUE October 2011 @leegoesplaces
“What do they hate and why?”
IUE October 2011 @leegoesplaces
Archived everything
IUE October 2011 @leegoesplaces
What’d we find out?
IUE October 2011 @leegoesplaces
Finding: Emotions run high
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
Finding: Wikipedia is a broad resource
IUE October 2011 @leegoesplaces
Finding: Resources drive research
IUE October 2011 @leegoesplaces
Yeah, but what about the deliverables?
IUE October 2011 @leegoesplaces
Cross-unit brainstorming
IUE October 2011 @leegoesplaces
Persona-building
Project steering
IUE October 2011 @leegoesplaces
Recycle, recycle, recycle
IUE October 2011 @leegoesplaces
IUE October 2011 @leegoesplaces
5. Lessons learned
IUE October 2011 @leegoesplaces
Study management takes time
Retain ‘em
IUE October 2011 @leegoesplaces
Nag, nag, nag
IUE October 2011 @leegoesplaces
Make it easy for them
IUE October 2011 @leegoesplaces
Adapt.
IUE October 2011 @leegoesplaces
1. Define research need2. Planning 3. Execution4. Analysis and findings5. Lessons learned
IUE October 2011 @leegoesplaces
Questions?
http://www.flickr.com/photos/picken/5694078937/sizes/o/in/photostream/ IUE October 2011 @leegoesplaces
Leanna [email protected]
Thanks