When Too Many is Just Enough - Jeff Pass at UXCamp DC 2013

Download When Too Many is Just Enough - Jeff Pass at UXCamp DC 2013

Post on 12-May-2015




2 download


Discussion of why it is sometimes desirable to have more than the required (either in terms of validity or statistical significance) number of participants in card sorts and similar usability exercises. Specifically, how large-scale usability exercises allow usability professionals and site owners to reach out to a broad audience and include them in the usability lifecycle.


  • 1.It starts with a guy introducing himself Hey, Im Jeff Pass. Im the one that mentioned male impregnation during my presentation at last years UX Camp DC remember me?

2. Then he makes his pitch Im working on a presentation for the IA Summit. Its called, When Too Many Is Just Enough: Citizen Engagement and Federal Government Websites. Id like to workshop part of it with you. Heres the (boring) summary: Statistical significance is an important consideration in usability studies. You need a certain minimum sampling to ensure valid results, but at the same time, too large a sampling creates more work without increasing significance or validity. Nevertheless, there is a case to be made for casting a wide net and engaging a larger audience in order to actively engage users. 3. Then he gets to the point... So today I want to talk about online card sorts. Open, closed, reverse, it really doesnt matter any kind of card sort you like as long as it is online. More to the point, I want to talk about how many participants you want/need. 4. So, youve got yourself a card sort 5. How many participants do you need? (1) According to Optimal Workshop, for an open card sort you need: 6. How many participants do you need? (2) According to Optimal Workshop, for a closed card sort you need: 7. How many participants do you need? (3) And there are lots of other opinions 8. So how many did we do? ??? 9. Over 1109 (1110 to be precise) 10. But why? What were you thinking? The 2012 Digital Government Strategy calls for large-scale citizen engagement. Increased participation doesnt increase validity (and can complicate analysis and reporting), but it does allow us to directly engage the wider public in usability studies and give them a voice in organizing the (federal) websites that are meant to serve them. 11. And how did you engage them?Directly Social media was our recruiterA blog post was our screener OptimalSort and TreeJack were our test vehicles**And yes, we did include a free text comment option and yes, analyzing the results wasonerous, but the additional effort was well worth it because while quantitative data (the sortresults) had pretty well stabilized by 25-30 participants, every comment provided valuablequalitative data. 12. So what do you think?Share your thoughts and experiences about large-scale usability studies and direct user engagement But enough about me; what do you think about me? Err I mean what do you think about large-scale usability studies, direct user engagement, and too many being just enough? 13. Thanks for your time & participation!Jeffrey Ryan PassLead User Experience ConsultantAquilent (www.aquilent.com)jeff.pass@aquilent.com@jeffpassUX Camp DC 2013