2010 gtac crowd source testing mozilla style

Crowd Source Testing Mozilla Community Style

Matt EvansMozilla QA Director

Wednesday, November 10, 2010

about:talk

• Background and Scale of Mozilla

• Crowdsourcing: Crowd vs. Community

• History of Mozilla Community testing

• How Mozilla succeeds with Community testing

• I have been in the testing biz for most my career

• 9 mos as QA Director @ Mozilla

• Manage a team of ~30 Global Test Engineers

about:me

about:mozilla

Mozilla’s Mission:

is to promote openness, innovation, and opportunity on the web

I’m still getting my head around scale at Mozilla

• The scale of information flow

• The scale of the user base

Mozilla Firehose

Mozilla Firehose• Many sources of information

• bugzilla (200+bugs/day)

• forums (50+)

• blogs (50+)

• tweets (100s/day)

• wikis (1000s pages)

• irc (50+ channels)

Mozilla Scale

400+Developers

10,000+Nightly Testers

2,500,000Beta Testers

400,000,000Users

400,000,000

350,000,000

300,000,000

250,000,000

200,000,000

150,000,000

100,000,000

50,000,0000Wednesday, November 10, 2010

Story Time

Big decision: Ship or No-Ship to Beta

A Recent Firefox product team meeting

Protector of the tester community

Get feedback from tester community

We were both wrong right!

It was a big change for my perspective

Respecter of the Tester Community as QA

Team Partner

We now work much better together

Crowd Sourcing: It involves a crowdworking towards a common goal

What is new, the virtual crowd is enabled by internet technologies

• Building a global community of testers ~30k+

• Stewardship of the test community is their secret sauce

Community approach to software testing

Mozilla Labs Crowdsource Crowdsourcing Project

Key Findings• there are two types users: heavy

users and casual users. • 90-9-1 Principle

• crowd motivation is separated into two forms: intrinsic and extrinsic

• crowdsourcing relies on people, and thus works when participants are treated as people.

Mozilla Labs Crowdsource Crowdsourcing Project

Community Crowd

Mozilla uses a hybrid of crowd based feedback mechanisms and community

management

“Crowd sourced testing Mozilla Style”

The History of Mozilla Testing Community

Mozilla.org is launched

by Netscape

1998 2002

Mozilla 1.0 is released

Phoenix 0.1released

Firefox 1.0released

2005-2009 2010

Firefox 1.5-3.5released

Firefox 4.0expected release

Pioneer Community Testers

Community Testers, The Next Generation

1998 2002 2004 2005-2009 2010

1998 Netscape open sourced the Netscape browser source code.

History of Mozilla’s Tester Community

•A lot of passionate people without a development background wanted to get involved.

Asa Dotzler First Mozilla Community Tester

The Rise of the Nightly Build

•A great success

•The community of testers grew to the 1000s

Too much of good thing

•As tester community members increased so did the noise

1998 2002 2004 2005-2009 2010

• 2002+ • The rise of Firefox code base• Split from AOL/Netscape• 2003 Mozilla Foundation is formed

1998 2002 2004 2005-2009 2010

• 2007+ • The rise of Continuous Integration

and unit tests

Continuous Integration

~3.3 million automated tests executed for every committed code change

•The largest CI system built on Buildbot and Tinderbox technologies

• For every code change

• 11 full builds of the browser across supported platforms

•~300k unit/perf tests per build

Minefield is the result of Continuous Integration

Continuous Integration and Unit Testing is the Foundation of Today’s Mozilla

Community Testing Success

Community Feedback

Mozilla Runs On Bugzilla

Crash Data Analysis

Crash Data Analysis5 years of crash data

FF1.5 FF2.0 FF3.0 FF3.5

FiledFixedDups

Firefox 4 Beta Feedback

LCD Screens of Feedback and Twitter mentions are displayed throughout the Mozilla Office

Engaging the Mozilla Tester Community

Manual TestingLitmus - Test Case Management

Test and Bug Days

Mozmill Test Automation Framework

Community Focused Automated Testing

• Javascript framework for testing gecko based applications• Includes an IDE• Installs as an Firefox

extension• Better suited for

community contribution

Mozilla Community Based Automated Testing

Overview of How we succeed with Community Testing

• Continuous Integration is the foundation of today’s community testing

• Utilizing Community Feedback

• Engage in Manual Testing efforts

• Engage Community focused automated testing in the near future

Community Stewardship

Community Stewardshipquality.mozilla.org

Reaching Out to the Communityquality.mozilla.org

quality.mozilla.org

A call out to you!

Thank you

Questions?

2010 gtac crowd source testing mozilla style

mozilla community tester

test community

community approach

community testingwednesday

community feedbackwednesday

global community of

tester community members

feedback fromtester

Documents

gtac 2014: what lurks in test suites?

gtac-10225-brief1 january 28, 2021 us forest ... - usda

of 2 antibiotics in action - gtac

apresentação gtac solutions - agenda e hangar

lessons from mozilla - open innovation & crowd sourcing

mozilla jordan in the arabic mozilla regional meeting

apresentação gtac solutions desenvolvimento de...

gtac boosting your testing productivity with groovy

mozilla @ mozilla meetup cluj

mozilla egypt , how to contribute in mozilla...

siemens plm software gtac services – asia...

siemens plm software – wayne mahan, solid edge gtac

gtac 2015 and innovative development 30 min

mozilla webmaker - mozilla india

global thresholds & allocations council (gtac) kickoff...

gtac meeting 08 september 2004. gtac agenda review of last...

gtac emea telefonmenü & benutzerhandbuch

54 gta/gtac genset

francisco j. garcía rodríguez dpto. de economía y ... ·...

mozilla firefox by duane adekoya. what is mozilla firefox?...