GUERILLA HCI And Customer Based Design Quentin Christensen Quentin.Christensen@live.com

Agenda Customer Activities Research Methods Shipping Activities Special thanks to Jakob Nielsen, his 1994 Guerilla HCI article is the basis for this presentation http://en.wikipedia.org/wiki/Comparison_of_usability_evaluation_methods http://www.useit.com/papers/guerrilla_hci.html http://www.useit.com/alertbox/discount-usability.html

Cost savings in a medium scale software project Original usability cost estimate by [Mantei and Teorey 1988] $128,330 Scenario developed as paper mockup instead of on videotape - $2,160 Prototyping done with free hypertext package - $16,000 All user testing done with 3 subjects instead of 5 - $11,520 Thinking aloud studies analyzed by taking notes instead of by video taping - $5,520 Special video laboratory not needed - $17,600 Only 2 focus groups instead of 3 for market research - $2,000 Only 1 focus group instead of 3 for accept analysis - $4,000 Questionnaires only used in feedback phase, not after prototype testing - $7,200 Usability expert brought in for heuristic evaluation + $3,000 Cost for "discount usability engineering" project $65,330 Source: Jakob Nielsen 1994 http://www.useit.com/papers/guerrilla_hci.html

CustomerActivities Research Methods Focus Groups Customer Focused Design Customer Visits Councils Customer Partner Marketing/Sales Customer Buddies Internal Partners Partner Relationships Concept Box Testing Usability Lab Testing Discount Usability Testing Scenario Customer design review Simplified think aloud Pluralistic walkthroughs Heuristic evaluation UX/UI/String Reviews Polish SWAT Real world projects Scenario testing & health White papers Early customer deployment Dogfood Shipping Activities

Customer Activities Focus Groups Customer Focused Design Customer Visits Councils Customer Partner Marketing/Sales Customer Buddies Internal Partners Partner Relationships What should I build?

Focus Groups / Customer Focused Design Focus Groups Create a topic Get several people in a room Ask them questions Record their responses Customer Focused Design Same as focus group List out the design ideas Have the room rate the design ideas Confirm the priorities with the group

Customer Visits Travel to the customers location Spend a day or two meeting various users Use one on one interviews and focus groups Try to collect artifacts if possible Take detailed notes

Councils 3-20 people that are experts in the area of the problem you are solving 3 Types: Customers people that will use your product Partners developers that will extend your product Marketing/Sales the people that will sell your product Create the council when you begin planning the product Meet with the council throughout development Quarterly meetings Collect feedback Present early designs Product demos Discuss product plans Show UI

Customer Buddies 1-3 users that you have a deep ongoing relationships with Frequent meetings and emails Conduct UI walkthroughs during iteration, early & often Discuss major design problems and design alternatives In depth design discussion Clarification of requirements

Internal Partners People at your organization that will be users of your product Easy to setup a meeting and discuss issues! Help find usability study participants In depth discussions UI walkthroughs Dogfood preparation

Partner Relationships Relationships with developers that will build on or extend your product Get their feedback as experts that understand the market Design a platform that supports their needs Help them go to market earlier and increase adoption Two types: Independent Software Vendor (ISV) sell and markets an app that uses your platform Systems Integrator (SI) sells your app and helps customers deploy and integrate with other software

Research Methods Concept Box Testing Usability Lab Testing Discount Usability Testing Scenario Customer design review Simplified think aloud Pluralistic walkthroughs Heuristic evaluation How should I build it?

Testing With 3-5 Users http://www.useit.com/alertbox/20000319.htm

Scenario Explain in a paragraph Reduce to the bare minimum Small and easy to change Use low cost prototypes to describe and test on users Try this with: Your team members UX Individual users When: You first start designing Example: Jim is authoring a document on hurricane patterns in the Gulf of Mexico. He shares the document with Armando, who works at a different university, for review. Armando adds several comments and Jim is delightfully surprised that he can see the comments right in the document and easily make the updates.

Customer Design Review Meet with a small group of users Present a design / prototype Explain the design and functionality Lead the discussion Ask probing questions Focus on uncovering requirements Try this with: Individual users Groups of customers When: You have a decent prototype Early and late in the design

Simplified Think Aloud Test one user at a time Have a list of tasks / use cases Show a prototype Users perform the tasks Detailed design, but low fidelity Take notes and measure performance Try testing multiple designs Try this with: Individual users UX Your team members When: Nearly final design After coding + some polish

Cognitive / Pluralistic Walkthroughs Walk through the design with people from various backgrounds together Ask everyone to write on paper how they would accomplish each task Discuss amongst the group Present the correct answer if the group goes too far off course Dont lead or try and explain everything to users Have a list of tasks and a prototype Try this with: Customers UX Your team members When: Nearly final design After coding + some polish

Heuristic Evaluation 3-5 evaluators Have a prototype or actual implementation Walk through the design Rate the design based on heuristic standards Try this with: Your team members UX When: Any time

Heuristic Evaluation1. Visibility of system status The system should always keep users informed about what is going on, through appropriate feedback within reasonable time. 2. Match between system and the real world The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order. 3. User control and freedom Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo. 4. Consistency and standards Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions. 5. Error prevention Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action. 6. Recognition rather than recall Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate. 7. Flexibility and efficiency of use Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions. 8. Aesthetic and minimalist design Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility. 9. Help users recognize, diagnose, and recover from errors Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution. 10. Help and documentation Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large. http://www.useit.com/papers/heuristic/heuristic_evaluation.html

Microsoft eDiscovery Examples User tasks Score sheet Scenario Health Design for eDiscovery Preservation

User Tasks 1. A legal event has occurred that requires you to hold content 1. Identify the mailbox for Dan Jump and Quentin Christensen, document center SharePoint site, and file share 2. Get an estimate of how many items there are 3. Reduce the amount of content using some keywords 4. Hold the content to make sure it cannot be deleted 5. Gather content to ensure it is held 2. You know more about the case and need to limit the content so you can export it, but you dont want to affect how it is preserved 1. Use keywords to limit the data set 2. Get the content to the correct set of data you need 3. Close your work so you can access it another time 3. You now need to gather the data set you have specified 1. Select the content you want to gather 2. Send it to a location where you can easily access it 3. Verify you have the correct content and are complete 4. You need to print a report to prove you adequately held content 1. Create a report that shows the dates of all the content you preserved

Heuristic Score Sheet http://office/15/teams/exchange/SEC/FindMyVault/FederatedFind/Shared Documents/eDiscovery Scenario Status.xlsx

Scenario Health Status Scenario Task Bugs/Notes Create a case Create a discovery set Add a mailbox Add a SharePoint site Get the number of items Enter a filter and view the stats Preserve and save the discovery set Check the preservation in Exchange Delete an email - notice it is kept in the dumpster Check the SharePoint site see it is preserved Edit a doc - it is copied to the preservation library Delete a doc - it is copied to the preservation library View auditing report of when content was preserved Create a new query Specify a query with a wildcard and proximity Narrow down the query scope View results from Exchange, SharePoint, and file shares Show the hovercard with a preview of the message Open the message Select multiple refinements for SharePoint View the hovercard for SharePoint items Save the query Export the query

Design for eDiscovery Preservation

3:00PM 1/23/2008 Search SaveEstimate Hits eDiscovery Document Name Windows Internet Explorer Favorites | Website 1 Website 2 Website 3 http://mysite/ediscovery Search Home Custodians Locations Searches Exports Reports All of these words: The exact phrase: Any of these words: None of these words: Scope Date Range: to Email Property Restrictions File Property Restrictions Language Restrictions All Mailboxes Mailboxes All Locations Locations Result Type Keywords Keywords: Add Mailboxes Add Locations Add Mailbox CancelSave Custodian: Lionel Penuchot Add To Case Name: Luka Abrus Preserve Mailbox Notify Place Associated Locations on Hold Associated Locations > < Import Locations Lukas Desktop Fileshare B Team Site Project Team Site

3:00PM 1/23/2008 eDiscovery Document Name Windows Internet Explorer Favorites | Website 1 Website 2 Website 3 http://mysite/ediscovery Search Case Manager Home Cases Holds Custodians Content Sources Content Sources Add Content Source Name Type Items Size Data Wiz Site SharePoint Site 3,460 1.384GB Data Wiz Files Fileshare 3,403 1.323GB

3:00PM 1/23/2008 eDiscovery Datum v. Contoso Windows Internet Explorer Favorites | Website 1 Website 2 Website 3 http://portals/sites/ediscovery/contoso Search Home Identify & Preserve Custodians Locations Search and Process Searches Exports Reports Case Settings Case Owners Status Preservation Settings Tom Perham; Active Preserve all content Select preservation type per mailbox Filter preserved content Date Range: to Query: Add Additional Filters Mailboxes Preserve all content Select preservation type per location Filter preserved content Locations Preview

3:00PM 1/23/2008 eDiscovery Document Name Windows Internet Explorer Favorites | Website 1 Website 2 Website 3 http://mysite/ediscovery Search Case Manager Home Cases Collections Create a New Case Datum v. ContosoName: Tom PerhamCase Owner: CancelSave Collections Add Collection None Collections CancelSave Name Date Created Managed By On Hold Type Widget F Files 9/26/2010 Tom Perham Yes Mailbox; SharePoint Steve Den 3/7/2010 Tom Perham Yes Mailbox Widget D 2/4/2010 Tom Perham No SharePoint Creative Files 6/2/2009 Luka Abrus Yes File Share; SharePoint Controller Files 3/6/2009 Tom Perham Yes Search Results Active Collections | Closed Collections | Create New Collection

Add Collection Filter: Preview Date Range: to Preserve Add Sources

Sources Preservation Settings Preserve this content Save and Apply Changes Cancel Update Statistics Filter (optional) Add a filter to reduce the content (e.g. date>10/10/2010, images) Collection Statistics Source Item Count Size Cull Rate Total 0 0 0% Name: Name your collection Preservation Type Statistics Content Type Statistics Add a collection Queries

CASE HOME COLLECTIONS SEARCHES EXPORTS SOURCES DOCUMENTS REPORTS Filter (optional) Add a filter to reduce the content (e.g. date>10/10/2010, images) Name: Name your collection Date: to: Sources Name Source Type Quentin Christensen Exchange Mailbox http://contoso/teams/sales SharePoint Site //back/scratch/quentin File Share Add sources Searches Name Modified Items Size Add new item Preservation Type Statistics Content Type Statistics Statistics Source Items Size Total Preservation Settings Preserve this content Collect content Save CancelUpdate Statistics Preview Results

HOME COLLECTIONS QUERIES SOURCES EXPORTS DOCUMENTS Apply Filter Save CancelPreview Results Name* Name your collection Sources Source Source Type Preservation Status Items Size Total Manage sources Filter Add a keyword to filter content Start Date: End Date:Keywords: Preservation Certain repositories such as SharePoint and Exchange support in place preservation. Selecting preserve this content will protect it in its original location. Preserve Author: There are no sources in this collection. To add sources, click Manage Sources. Initial Design

Initial Build

Redesign

Beta1

Shipping Activities UX/UI/String Reviews Polish SWAT Real world projects Scenario testing & health White papers Early customer deployment Dogfood

UX/UI/String Reviews Get team members, UX, design in a room Walkthrough the experience Testers write down bugs and file later UX/UI focus on the overall experience, flow, interactions, look and feel. Get into the details on spacing, color, font, etc. String Focus on the text that is displayed to users, be sure to cover all error cases Get a list of all text in areas you own Try to view it in context, but be sure to review everything Get recommendations on how to change the string

No, this Polish SWAT 1. Identify page/area/experience to polish 2. File one bug to track the SWAT 3. Identify existing bugs and track them 4. Do a UI review, identify all issues 5. Track issues in a shared OneNote 6. Review with your team 7. Create a design with all changes 8. Fix all issues 9. Dev sends link to changes 10. Review in person, identify remaining issues, fix, repeat 11. Your done: 1. Everyone agrees its ready 2. Polished no remaining UI bugs Members: Dev / Test UX/Design SMEs

Polish SWAT Tips Double check CSS make it consistent (IE F12 Developer Tools) Get it right - but dont stress UI and CSS you dont control, go for consistency not the way you want it to look Meet with the developer, make real time changes Focus pick one page or task Schedule a week the SWAT is a top priority, meet ad hoc not on a schedule Agility move to each step as fast as possible, if the developer completes the update meet immediately Agreement its done when you all agree its done Its Done the page looks ready for customers (aside from any issues you dont control)

Example: Microsoft eDiscovery Before SWAT After SWAT23 initial issues identified 10 new issues identified during iteration Looks way better after 1 week Spec

Real World Projects Build real world applications using the latest build Build a real thing, use real data: Team up get 3-4 people Dedicate a week it should be the majority of your time for that week File as many bugs as possible! 50-100 for a team or even one person is doable Can you complete the scenario?

Example: Microsoft Sales Portal Rebuild the Microsoft Sales Portal site on the new version of the product (Alpha Build) 1 week, 4 people full time,112 bugs filed

Scenario Testing & Health Identify core tasks for your scenario Conduct regular test passes Rate each task Green no issues or very minor issues Yellow minor visual issues, not great experience Red major visual or functional issues Black cannot complete task Mark bad bugs scenario blocking Track usability & experience issues

Example ID Assigned To Sub Status Title 290742 ramansom Paging doesnt work for exchange sources 319499 boxinli Bug Understood When an update panel refreshes search no longer works 325512 carloa No stats on the saved searches list view; need Items and Size 328714 srivr Auth: The realm in the token is hardcoded to 'microsoft.com' 333406 shiyao Sum of # of documents displayed in refiners is greater than the total number of documents returned in the query results (for very simple result sets) 350069 boxinli Need a way to cancel a query which has been issued to the SDP 360106 carloa Case Manager: When failing to talk to search, the seaching window in the query edit form keeps searching forever 361198 carloa Creating a new Query should search all sources in the case with a blank filter 373183 ramansom Search fail to get token when searching using OAuth 376625 grahamc Local Fix Exchange SourceFilter includes Path property; Exchange exports don't work 229559 1 sveinar Local Fix Thread.CurrentPrincipal in the flow is not correct 231439 7 carloa The hover panel is gone; Object required in searchui.js line 1 col 9342 234268 8 carloa Refiners fall back to non Discovery refiners if you specify a specific collection as the query scope 234552 5 carloa Loading advanced query options dialog gives a 404 235438 0 carloa Automation Failure: Search returned results from the SharePoint Source [ACTUAL: 0, EXPECTED: Not 0] (Method: 235881 0 jessals Investigating We need a way to resolve a web title from search results 237634 3 carloa Query for the top-level of the query fragment web part doesn't include hiddenconstraints 237639 1 carloa Expanding a query fragment in the query fragment web part first queries with HiddenConstraints and then without 238619 8 grahamc Local Fix Creating an export fails; The given key was not present in the dictionary. at Microsoft.Office.Server.Discovery.Export.GetExportContent() 238717 0 grahamc Local Fix SourceFilter in export.xml for an export includes properties that shouldn't have values; export fails Scenario Health Scenario Blocking Bugs Status Scenario Task Create a case Create a discovery set Add a mailbox Add a SharePoint site Get the number of items Enter a filter and view the stats Preserve and save the discovery set Preservation works in Exchange Preservation works in SharePoint View auditing report of when content was preserved Create a query Specify a query with a wildcard and proximity Narrow down the query scope View results from Exchange, SharePoint, and file shares Show the hovercard with a preview of the message Open the message Select multiple refinements for SharePoint View the hovercard for SharePoint items Save the query Export the query

White Papers A big doc of how to do it Guidance, config, setup, best practices Write a draft early (before Beta1) Give to early customers Get feedback hard to setup? Are the requirements reasonable? Add to it when you make a decision that customers should know about

Early Customer Deployments Beta users Adoption Program High touch: Weekly synch ups Prototype testing Beta 1 Go live Beta2 RTM Ongoing feedback Bug finding Lots of support from engineering Prove features and fix gaps

Dogfood Internal use of your features before official release Important source of validation and bug finding Every product feature should have a dogfood plan

Secret Guerilla Tactics & Tips If you dont have at least one user to talk to get one At least do one test with 3 people before release! Dont lead users - give tasks, observe, and take notes Ask users to think aloud Know your user, you are not the user When giving tasks dont use the phrases in the UI Dont take everything you hear at face value Dont be afraid to show early designs to real customers Do projects to validate your features Use real world data and examples Measure your health and readiness to ship