implementing crowdsourced testing

MH AM Tutorial

4/29/13 8:30AM

Implementing Crowdsourced Testing

Presented by:

Rajini Padmanaban

QA InfoTech

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Rajini Padmanaban

As director of engagement at QA InfoTech, Rajini Padmanaban leads the engagement and relationship management for some of QA InfoTech's largest and most strategic accounts. Rajini has more than eleven years of professional experience, primarily in the software quality assurance area. She actively advocates software quality assurance through evangelistic activities; writes on test trends, technologies, and best practices; speaks at conferences including STAREAST 2012 and STARWEST 2012; and provides insights on software testing to analyst firms such as Gartner and IDC. Read Rajini’s official blog and reach her at [email protected].

Corp04 1

Your Software Testing PartnerWe help you build better software

Implementing Crowd Sourced Testing

Rajini PadmanabanDirector of Testing Engagements

Agenda (1 of 2)

Slide 2

Topic Time (in

minutes)

A Peek into Software Quality 10

Crowd Sourced Testing - Defined 10

Understanding Varied forms for Crowd Sourcing 15

Let’s be the Crowd – Exercise Time 20

Crowd Sourced Testing Relevance in Current

Scenario

10

Limitations of Crowd Sourced Testing 15

Corp04 2

Agenda (2 of 2)

Slide 3

Topic Time (in

minutes)

Practices for a Successful Crowd Sourced Test

Effort

30

Know What Not to Crowd Source 15

Getting Stake Holder Buy-in 15

Case Study & Examples 30

Myths and Facts 15

Conclusion and Q&A 15

Requirements

A device to connect to the internet for one of our exercises (laptop, smart phones, tablets )

Slide 4

Corp04 3

Test is responsible for

QualityFocused on creating a quality deliverable

Ensure we don’t sacrifice quality for the sake of schedule

Empower the rest of the team to partake in improving quality

A Peek into Software Quality

Slide 5

Defining Quality

• ISO 9000: Degree to which a set of inherent characteristics fulfill requirements

• Six Sigma: Number of defects per million opportunities

• Philip B. Crosby: Conformance to requirements

• Joseph M. Juran: Fitness for use by the customer

• Gerald M. Weinberg: Value to some person

• Robert Pirsig: The result of care

• American Society for Quality: A subjective term for which each person has his or her own definition. In technical usage, quality can have two meanings:

a. The characteristics of a product or service that bear on its ability to satisfy stated or implied needs

b. A product or service free of deficiencies

Slide 6

Corp04 4

Software Development

Slide 7

Attacking the Inverse Exponential

TestabilityFault InjectionStress TestingEnd-to-EndExploratoryCode InspectionLeverage the Beta Test Crowd

Slide 8

Corp04 5

Portfolio Selection Theory

Diversification in investing tells us that risk lessens as the number of investments in the portfolio increases

10%10% 10%

10%

10%

10%

10%

10%10%

10%

Slide 9

Audience Interaction Time=

What Does Quality Mean to you ?

Slide 10

Corp04 6

What is Crowd Sourced Testing?

Dailycrowdsource.com

Crowd sourcing is the process of getting work, usually online, from a crowd of people. The word is a

combination of the words 'crowd' and 'outsourcing'. The idea is to take work and outsource it to a crowd

of workers

CrowdSourcing.org (similar definition on Wikipedia)

Welcome to the new world of crowd sourced testing, an emerging trend in software engineering that

exploits the benefits, effectiveness, and efficiency of crowd sourcing and the cloud platform towards

software quality assurance and control. With this new form of software testing, the product is put to test

under diverse platforms, which makes it more representative, reliable, cost-effective, fast, and above all,

bug-free

Crowd Source Testing.com

Crowd sourcing your software testing consists of delegating onto a number of internet users the task of

testing your web or software project while in development to ensure that it contains no defects, referred

to as bugsSlide 11

I prefer the

word engaging

rather than

delegating


.so what really is Crowd Sourced Testing?

• Is it all about a pool of testers leveraged and paid per valid bug? No, not just that

• Think:• Sourcing relevant people from within your company

• Across disciplines and levels• Sourcing end users from various disciplines – e.g.

teachers, students, nurses, bankers• Partnerships with universities, organizations leveraging

domain knowledge• In essence, think of the community at large to test your

productSlide 12

Corp04 7


Slide 13

Services

CompaniesProduct

Companies

External

Crowd

Sourcing

Internal

Crowd

Sourcing

Technologies DomainsCompany Scale

Understand Factors that Motivate the Crowd

Portfolio Selection Theory

Crowd Sourced Testing: Portfolio Selection Theory applied in QA

10%10% 10%

10%

10%

10%

10%

10%10%

10%

Slide 14

Corp04 8

A Practical Solution to Leverage

Slide 15

Set # 1

• Crowd Sourcing:• Is not restricted to any single company, technology,

domain• Has a much larger yet simpler meaning than what it is

often portrayed to be

Slide 16

Corp04 9

Types of Crowd Sourcing - Explained

Slide 17

Crowd Creation

Crowd Voting

Crowd Wisdom

Crowd Funding


• Crowd Creation

• Invite crowd to create subject content• Common usage areas: software

development, translations, photos repository• Content usage by crowd or by organizations• Typical crowd motivators – Money, fun,

community involvement, brand loyalty• Well known examples – Linux, iStockPhoto,

99 designs

Slide 18

Corp04 10


Slide 19


• Crowd Voting

• Leverage crowd’s judgement to organize, filter, stack rank content

• Common usage areas: retail, media, simple yet powerful decisions across domains

• Results used by an organization• Typical crowd motivators – Fun, community

involvement, brand loyalty• Well known examples – American Idol,

Threadless.com

Most popular of crowd

sourced versions

(1:10:89 rule)

1% create

10% vote and rate

89% consume-Jeff Howe, Author of CrowdSourcing

Slide 20

Corp04 11


Slide 21


• Crowd Wisdom

• Harnesses crowd’s knowledge to solve problems, predict future outcomes

• Common usage areas: reality shows, quality assurance, exchanges markets

• Results used by an organization, crowd• Typical crowd motivators – Money, fun,

product transparency, brand loyalty• Well known examples – Who wants to be a

Millionaire

Slide 22

Corp04 12


Did You Know?

The crowd’s answer in

“Who Wants to be a

Millionaire” was right 91%

of the time compared to

“Ask an Expert”, which was

right 65% of the time

Source: The Wisdom of

Crowds, James Surowiecki

Slide 23


• Crowd Funding

• Leverages crowd to finance individuals or groups that might otherwise be denied credit or opportunity

• Common usage areas: ideas in developing nations, educational domain

• Results have far reaching impact• Typical crowd motivators – Money, social

causes / community involvement• Several interesting examples at:

http://www.alumnifutures.com/2012/07/crowdsourced.html

Slide 24

Corp04 13

Set # 2

• Crowd Sourcing Options are:• plentiful• available for a diverse set of community needs – both

commercial and not-for-profit• to be customized based on results needed and the

crowd motivating factors

Slide 25

Exercise Time – Let’s Crowd Source

Slide 26

Who has won the most oscars?

1.Walt Disney

2.Elizabeth Taylor

3.Meryl Streep

4.Jack Nicholson

Walt Disney – he won 26

oscars including 4 in a

same year

Corp04 14


Slide 27

On which national flag is there an eagle and

a snake?

1.China

2.Mexico

3.Greece

4.Spain

Mexico derived from an Aztec

legend that a city would be built

where they spot an eagle eating a

serpent, which is now Mexico City


Slide 28

Which American state produces most

potatoes?

1. Oregon

2. California

3.Idaho

4.Washington

Idaho, followed by Washington

and Oregon (in 2011)Source:

http://www.potatopro.com/Newsletters/20111110.htm

Corp04 15

Did You Know?

The crowd’s answer in

“Who Wants to be a

Millionaire” was right 91%

of the time compared to

“Ask an Expert”, which was

right 65% of the time

Source: The Wisdom of

Crowds, James Surowiecki

Set # 3

Collective wisdom of the crowd often surpasses that of an expert

• Please take about 10 minutes to test www.amazon.com from usability and accessibility standpoints

• Need tools - screen readers, magnifiers?• http://en.wikipedia.org/wiki/List_of_screen_readers• http://mediaaccess.org.au/digital-technology/assistive-

tech/screen-magnifiers/ - e.g. Magnifier on Windows, Zoom

for Mac

Slide 30


Corp04 16

• Discuss test results / feedback –• Goal of our study: demonstrating richness

and diversity of crowd’s feedback as end users

• Feedback from our visually challenged crowd

Slide 31


Amazon

Accessibility Issues

• Shrinking release cycles• Close scrutiny on overall spend• Collective ownership of quality• Need to:

• Focus on product domain knowledge; not just disciplinary knowledge

• Understand competing products• Creatively emulate end user scenarios• Mimic user environments via lab and simulations• Consider global product distribution effects

Slide 32

Product Quality Scenario – as it stands today

Corp04 17

A Practical Solution to Leverage

• Distribute quality effort • Enhance productivity through global solutions• Pool in end users into quality implementation• Use live environments to test• Analyze potential partnerships for SME• Flexible, selective and cost effective testing

.........Bring in the Crowd

Slide 33

When does Crowd Sourced Testing

Succeed?

• Diversity of knowledge, background, experience

• Independence in testing process

• Wide spread domain background for product requiring SMEs

• End user scenarios and environments difficult to simulate in-house

• Work aligns with factors that motivate crowd

Slide 34

Corp04 18

Inherent Challenges

Slide 35

• Random test efforts do not fit into quality strategy• Choosing, sustaining a crowd sourced team

• Ongoing motivation, floating crowd• Keeping everyone in sync on product dynamics• Communication challenges• Management overhead including logistics• Stakeholder buy in• Securing product IP before release

Any Challenges from

your experience that

you would like to share?

Perceived Limitation

Slide 36

Quality of product adversely impacted by an amateur crowd

- Solidify your implementation plans- Use instrumentation wherever possible- Refer case studies – understand successes and failures

Corp04 19

Set # 4

• Current product scenario demands creative solutions to test within existing constraints

• Crowd Sourcing is not a no-brainer solution to all problems• Not even a stand-alone solution in most cases• Understand its strengths and challenges in customizing

it to your needs• Understand “What, When and How to Crowd Source in

Testing”

Slide 37

Practices for Successful Crowd Testing

Customize your

practices mindful of

your constraints

Slide 38

Corp04 20

Slide 39

• When to Crowd Test:• Product works reasonably well E2E • Ready to incorporate crowd’s feedback• No time or resources for formal testing .try to

avoid this situation

• Source content files are ready• Ongoing feedback from a chosen SME team

at specific stages

“What, When and How” of Crowd Testing

Slide 40

Examples

Started off with informal testing due to

lack of timeOngoing MVP programs at Microsoft

Product ready to be tested E2E and

feedback incorporated

Corp04 21

Slide 41

• What to Crowd Test:• User facing features• Areas where external team feedback is

important – e.g. design, feature set, performance

• Specialized areas of test:• localization (context based verification)• performance• compatibility, devices testing

• Content testing – valuable SME knowledge

• Align crowd’s focus areas into test strategy• Know what not to crowd source


Slide 42

Examples

Exam Grading

Content Testing using SME /

Localization

Corp04 22

Slide 43

• How to Crowd Test:• Pick the right areas, team and time

• Minimize duplication, overhead of sifting throughknown issues

• Clear internal ownership:• Communication, technical query resolution• Prompt follow up and responses• Team up-to-date on product changes

• Use of collaborative tools• Think about interactions amongst testers


Slide 44

• How to Crowd Test:• Leverage cloud, VPC for ease, secured access• Identify crowd motivators

• Use management theories – Maslow’s hierarchy of needs?

• Identify tasks that align with crowd motivators• Work on stakeholders buy-in


Self Actualization – Pursue Inner Talent,

Creativity, Fulfillment

Self Esteem – Achievement, Mastery,

Recognition, Respect

Belonging – Friends, Family, Spouse, Lover

Safety – Security, Stability, Freedom from Fear

Psychological – Food, Water, Shelter, Warmth

Corp04 23

Set # 5

• Understand when a crowd test effort succeeds:• Diversity of knowledge, background, experience• Independence in the testing process• Wide spread domain background for product requiring SMEs• End user scenarios and environments difficult to simulate in-house• Work aligns with factors that motivate the crowd

Any Examples, Best

Practices from your

experience that you

would like to share?

Slide 45

Slide 46

• Features with moving pieces, demanding close collaboration

• Sensitive IP• Environment specific complex testing• Tasks requiring immediate and regular turnaround

- BVTs• Core testing activities

• Test automation, TDD scripts• Regression testing • First round of performance, security, integration testing• Mundane testing tasks that don’t need diversity

Know What Not to Crowd Test

Corp04 24

Slide 47

• Simple performance tests such as PLTs – use tools• Understand pros and cons specific to your scenario –

study by University of Texas, Austin* on Crowd Sourcing for Usability Testing vs. Lab Usability Testing on a college website

Some Examples

Lab Usability Test Crowd Sourced Usability Test

Participants 5 55 (14 spammers)

Participant Demographics Students Crowdworkers

Age 24 to 33 19 to 51

Education level Bachelor’s degree and Master’s degree All levels

Experience with similar

websites

Yes: 100% Yes: 77%

No: 23%

Speed Approximately 30 min. per session. Less than 4 hours total.

Participant

Costs

None $2.92 for pilot test

$23.41 for final test

(Avg: $0.48/tester)

* - http://arxiv.org/ftp/arxiv/papers/1203/1203.1468.pdf

Slide 48

Some Examples

Major Problems Identified Lab Usability Test Crowd Sourcing Usability Test

Font size too small

Out-of-date information

Menu overlap

Irrelevant picture

Invisible tools

Information not cross-linked

Lack of sort function

Navigation unclear

Search box difficult to locate

Corp04 25

Slide 49

Some Examples

Advantages Disadvantages

More participants Lower Quality Feedback

High Speed Less Interaction

Low Cost Spammers

Various Backgrounds Less Focused User Groups

In Conclusion:

1. In this scenario, usability testing would be better off done by students of the college website;

choosing target users is very important

2. If crowd sourced testing is relevant based on your user profile, design the effort with care; using

same tests and questions as that of lab testers for crowd testers may not yield great results (see

section 4.1.2 on Test Redesign)

Slide 50

Stake holder Buy In

• Stakeholder resistance to Crowd Sourcing largely inline with model’s challenges:• Product IP, privacy issues• Internal team motivation• Quality of test effort and product• Additional overhead in effort management• Randomized test effort – tactical in nature

Corp04 26

Slide 51

Stake holder Buy In

Identify stakeholders; prioritize the team;

engage early

Walk through the test strategy including crowd

sourcing plans; understand their

concerns

Explain Solutions Undertake Pilot if

needed

Ongoing Communication

1. Supplemental test technique

2. Acknowledge problems

3. Explain the technique as

applicable to your product; don’t

assume their know-how

1. Educated decision

2. Explain implementation plans –

what & what not, when, how,

3. Define checks and balances for

internal team and CS team roles

4. Map solutions to address each

concern identified earlier

1. Practical demonstration of the

model; adds to your confidence too!

2. Additional overhead at start but

pays off in longer run

3. Run a pilot like a regular CS test

program but of smaller scale

1. Show steady progress

2. Stick to pre-defined

communication protocols

3. Communicate the good and the

bad

4. Be on top of new stakeholders

or new concerns

Set # 6

• Acknowledging what not to crowd source will:• Fasten stakeholder approval process• Help not impact internal team motivation

Slide 52

Corp04 27

Core

Work Skills

Unique

Work Skills

Future

Work Skills

In-Job

Behaviors

Organizational

Citizenship

Behaviors

Successful Work Patterns

Slide 53

Core

Work

Skills

Unique

Work

Skills

Future

Work

Skills

In-Job

Behaviors

Organizational

Citizenship

Behaviors

Successful Work Patterns

Slide 54

Corp04 28

Localization Testing

• Hard, Large-Scale Problem

• Windows 7 ships in 100

languages

• Thousands of strings and screens per release

• Traditional model of localization testers expensive and difficult to find

Slide 55

link

Let’s Play

Slide 56

Corp04 29

The Language Quality Game

Slide 57

Results

Significant Quality Improvements for Windows 7

Positive Impact on Ship Schedule

Team Morale and Subsidiary Engagement

Internal sourcing alleviates security and access issues

Total Screens Reviewed: Over 500,000

Total Number of Reviewers:

Over 4,500

Screens per Reviewer: Average 119

Slide 58

Corp04 30

Amplify Skill with Volume

Individual dialects,

nuances, hard to

detect with a single

vendor – crowd does

a better job

Slide 59

Reduce Cost with Discovery,

Instrumentation

No need to install

Telemetry to direct

effort

Slide 60

Corp04 31

Reduce Risk with Diversity

Portfolio theory in

Quality Assurance

Slide 61

Trust and Transparency Increase

Effectiveness

Inclusion = Trust, Trust = Enthusiasm

Inclusion = Trust, Trust = Enthusiasm

Subsidiary

engagement with

Language Quality

Game

Momentum for Win 7

Slide 62

Corp04 32

Players earn points for disaster relief agencies

Microsoft donates $ based on leader board

Individual players can sponsor tasks or scenarios

Slide 63

Results

Significant Quality Improvements for Communicator “14”

Positive Impact on Ship Schedule

Team Morale and Dog-food User Engagement

Players Over 1,000

Feedback increase: > 16x

Feedback received: 10,000

Slide 64Source : Ross Smith, Director of Test, Microsoft Corporation

Corp04 33

Other Examples

Slide 65

• Working with Blind Relief Association to bring in real users into accessibility testing

• Working with universities for content grading• Mobile application testing across devices within

the company

Other Examples

Slide 66

Listen to Ross Smith, Director of Test at Microsoft,

on his thoughts on Crowd Sourced Testing

Corp04 34

Crowd Sourced Testing – Myths and Facts

Slide 67

Crowd Sourced Testing impacts core testing team adversely; threatens their positioning

in the product team

Really?

1. Crowd Sourced Testing - supplemental technique, not stand

alone

2. Does not work in all situations; areas of niche to be reserved for

internal testing

3. Internal team to build a sense of empowerment that the crowd

adds to product quality

# 1


Slide 68

Crowd Sourcing is only for software testing. Development is a very technical and

specialized area to leverage the crowd for

Really?

1. Think of Open Source Software….Linux - popular example of

open software collaboration

2. Crowd Sourcing easier for testing than development as crowd

often represents user base

3. Security, IP easier to manage in testing – no access to source

code

4. Practices discussed extendible to development as well

5. Top Coder.com another good example

# 2

Corp04 35


Slide 69

Management overhead is significantly higher in Crowd Sourced Testing. Given the short

project deadlines, we do not have time or resources to manage a crowd sourced test

team

Partly True…

1. Management overhead slightly more...but this is inevitable in

current day global development models

2. Crowd is a smart and self – sufficient group; YOU DON’T WANT

TO MICRO-MANAGE

3. Best practices of what, when, how to crowd source will make

the effort streamlined and not chaotic

# 3


Slide 70

Crowd Testers can be ramped up or down at very short notice, giving great head count

flexibility

Partly True…

1. Scale potential of resources is huge

2. However, getting right resources at right time is challenging

3. Maintain a pool / common database and engage with the crowd

on ongoing basis even in lean periods

# 4

Corp04 36

In Conclusion

Slide 71

Let’s:• Pictorially walkthrough Crowd Sourcing

• Revisit Take-Aways• Look at Call to Action

Crowd Sourcing – Pictorial Walkthrough

Slide 72

Corp04 37


Slide 73


Slide 74

Corp04 38

Take-Aways revisited

• CS used across companies, domains, technologies

• Understand varied manifestations, crowd motivators, to customize your implementation

• Educated decision of what, when, how to crowd source

• Acknowledge model’s challenges; be transparent in seeking stakeholder approval

Slide 75

Call to Action

• Evaluate programs in your group; gradually build on them

• Start small• Try a Pilot / Proof of Concept• Register to be a crowd

sourced tester – internally, externally -http://www.qainfotech.com/CS_Reg_Step1.php

Slide 76

Corp04 39

About QA InfoTech

• An independent software quality assurance and testing company, founded in 2003, currently employing 600 people

• Five testing “Centers of Excellence” across the USA and India

• World-class testing labs

• Experience working with clients across various domains

• Bagged the “Top 100 places to work for in India*” award, three years in a row

• Focus on the right balance of people, processes, technology • CMMi III, ISO 9001:2008, 20000-1:2005 certified

QA InfoTech facilities in IndiaSlide 77

* Study conducted by Great Places to

Work Institute, India

Q&A – Let’s find answers together!

Slide 78

Corp04 40

References

• http://outsideinmarketing.wordpress.com/2010/05/08/four-types-of-crowdsourcing/

• http://explore2win.blogspot.in/2012/10/what-is-crowdsourcing.html

• http://arxiv.org/ftp/arxiv/papers/1203/1203.1468.pdf

Slide 79

Thank You

For more information, please:

• Contact us at [email protected]

• [email protected]

• Visit us at www.qainfotech.com

• Read our blog at www.qainfotech.com/blog

• Follow us on Twitter at www.twitter.com/qainfotech

USA

Office

International

Headquarters

NoidaUttar Pradesh, India Phone: +91-120-4292222(Three additional testing facilities in India)

Farmington Hills Michigan, U.S.A. Phone: +1-248-719-3409

Slide 80

implementing crowdsourced testing

Technology