inspiring, enabling and driving the evolution of quality

®

Copyright 2008 Adobe Systems Incorporated. All rights reserved. Adobe confidential. 1

Inspiring, enabling and driving the Evolution of Quality at Adobe leveraging the TSP

Jim Sartain

[email protected]

Senior Director, Quality

September 22nd, 2009

mailto:[email protected]�

®


Agenda

Background

Future Vision

How we inspire, enable and drive progress

Team Software Process at Adobe

Summary

®


About Adobe

More than $3 billion in revenue in FY2008, 7100+ employees

More than half of Adobe’s revenue is generated outside the United States

Industry leading technologies with global wide-spread adoption

Adobe® Reader® is distributed in 26 languages on 10 major platforms worldwide

Adobe® Flash® Player has 98% reach on connected PC’s and 8 million installs per day

More than 90% of creative professionals have Adobe Photoshop® software on their desktops

Major software development located centers in San Jose, San Francisco, Seattle, India, China, Germany and Romania

Presenter

Presentation Notes

About Adobe Adobe is headquartered in San Jose, California, and has more than 6,700 employees worldwide. It is the fifth largest software company in the world. Adobe Fiscal 2007 Revenues Greater than US$3 billion Across the world, in virtually every conceivable medium, Adobe is creating leading-edge solutions that enable media-rich communications to engage customers and address the needs of the global marketplace. Our major sales regions include: Europe, the Middle East, and Africa (EMEA) Asia Pacific Japan Americas

®


The Evolution of Quality at Adobe: 2012

Customers Employees Shareholders

70% of our customers are product promoters and detractors are rare

Shareholders see an increase in stock price tied to quality improvement savings

Love the fact they get twice as much and more important work done with less effort and drama

Aspects of this vision are already happening in some Adobe teams

Presenter

Presentation Notes

Customers 70% of our customers are product promoters and detractors are rare: Our intimate understanding of customer needs is used to deliver and continuously improve products that truly delight customers Customers count on our applications being always-on, reliable and secure: Millions of customers trust us to use and store their critical information over the internet. Employees Love the fact they get twice as much done with less effort: Highly effective development processes and tools and the use of best practices provides a work environment that delights employees and is the envy of the industry. Make significant progress on their professional development: Adobe fills 90% of its leadership positions using job rotations and internal promotions. Adobe has a strong leadership bench for key engineering disciplines (Architects , Engineering Mgrs, User Experience and Quality Leaders) Shareholders Adobe quickly responds to important customer needs: Adobe releases new software as a service quarterly and provides updates when customers want them. Software is always release ready. Shareholders see an increase in stock price tied to quality improvement savings: Cost of Quality as a % of sales is reduced increasing profits by $60M/year

®


The measureable piece of quality costs is the tip of the iceberg

Engineering Rework

Higher Support Costs

Refunds, Returns, Scrap, Patch downloads

For many software companies :

40% to 50% of Engineering Effort

50% of Customer Support Budget

Less time for innovation

Fewer promoters

Reduced product/service value

Delayed time to market

Increased employee attrition

Reduced employee engagement

Litigation costs

Indirect costs are typically two to three times direct

quality costs

Improving quality is a huge opportunity to free up resources for value-added work and make Adobe an even better place to work

Source: Principles of Quality Costs, Campanella

®


Increased Agility

Improved Customer Experience

Goals:

How we assess improvement progress

Improved Engineering Productivity

Progress on these goals is reviewed regularly by Senior Leadership

®


Increased Agility


Goals:



Are we delivering Customer Delight? Are key customer issues

being eliminated?

Questions:

How many days between code complete and release dates? Is effective Automated

Testing in place?

Are major sources of rework identified and reduced? Are teams using engineering

best practices to find and remove defects early


®


Increased Agility


Goals:



Are we delivering Customer Delight? Are key customer issues

being eliminated?

Questions:

How many days between code complete and release dates? Is effective Automated

Testing in place?

Are major sources of rework identified and reduced? Are teams using engineering

best practices to find and remove defects early


Metrics:

• Net Promoter Score• # support cases / customer

• % of dev effort spent on system testing and defect driven rework

• % defects found before System Testing

• # days effort per test cycle• Automated code coverage

®


Adobe(TSP teams)

50%

10%

>90%

Days

90%

Industry(Typical)

20%

50%

10%

Varies

Varies

What’s Important and How are We Doing?

Question

Improved Customer

Experience

ImprovedProductivity

IncreasedAgility

Goal

How satisfied are your customers? (Net Promoter Score)

% of dev effort spent testing/bug fixing?

% of defects found before system test?

Effort required to do a full test cycle?

Automated test code coverage level?

Sharing what’s possible and rewarding/recognizing improvement drives progress

WorldClass

70%

10%

>90%

90%

Hours

Sources: [Caper Jones , Applied Software Measurement,, 1996.; Software Assessments, Benchmarks, and Best Practices., 2000.

®


5

22

2

25 32

1405

10

0

100

200

300

400

500

600

700

Personal Review (Code)

Peer Review (Code)

Unit Test Integration Test System Test

Source: Six Adobe Projects using the Team Software Process

Minutes to Find and Resolve a Defect by Discovery Activity

TSP Projects are showing what is possible to the larger organization

Presenter

Presentation Notes

From Xerox

®


Best Practice Adoption at Adobe

TSP

100%

75%

50%

25%

0%

% A

dopt

ion

Time From Introduction

Innovators Early Adopters Early Majority Late Majority Late Adopters

Source: Crossing the Chasm: Geoffrey Moore, 1991

Adoption of best practices follow a traditional technology adoption curve

Peer Reviews

Scrum

®


Peer Review Adoption at Adobe

60%30%

10%

How Likely are you to Recommend Peer Reviews to Others?

Promotors (9 or 10)

Neutral (7 or 8)

Detractors (0 - 6)

Source: 284 Survey Respondents

Enthusiasm for doing reviews is high

Jan Feb Mar Apr May June July Aug

Months of Effort Saved

Number of staff doing reviews is growing

More Peer Reviews are being done

Great progress… Adoption increasing while we improve effectiveness

0

500

1000

1500

2000

2500

3000

3500

Jan Feb Mar Apr May June July Aug

# Peer Reviews

Peer Reviews Managed Using Shared Code Review Server

Peer Review effectiveness is increasing

®


Scrum & TSP can work togetherSimilarities:

Incremental, Iterative, Empirical

Dedicated, self managing teams

Proven methods to deliver higher quality

Scrum:Light framework

Assumes teams will choose best eng practices

Low “startup costs”

Requires moderate changes for engineering

Good ramp up to high quality over time

TSP:More detailed methodology

Assumes teams will use PSP eng practices

Higher “startup costs”

Requires significant changes for engineering

Almost immediate leap to high quality

Combinations:

Scrum teams adopt TSP practices & metrics

TSP teams adopt scrum framework best practices

Presenter

Presentation Notes

These represent the “tip of the iceberg”

®


TSP Rollout Strategy

• Ensure organizational leadership expects and requires delivery of software products withquality

• Use TSP to establish examples of excellence• Only choose teams that are generally excited• Ensure each project has a strong coach• Use pilot teams as internal references and

sources of champions/future coaches• Build internal coaching capacity within each

adopting organization

A viral adoption approach works best

®


Team Software Process Adoption at Adobe

Since January 2008

Featured in keynotes at Adobe Tech Summits in San Jose, California and Noida, India

325 executives, managers, developers, and team members trained

Classes conducted in San Jose, San Francisco, Noida (India), Bangalore (India), Beijing (China), and Hamburg (Germany)

9 TSP projects underway, approximately 100 developers and quality engineers using TSP

Many teams are broadly adopting engineering practices learned from their TSP training

The rate of TSP adoption is accelerating

®

Copyright 2009 Adobe Systems Incorporated. All rights reserved. Adobe confidential. 16162008 Adobe Systems Incorporated. All Rights Reserved.

Early Quality Results for our TSP Projects Are Impressive

Source: Quality Data for Six Adobe TSP Pilots

Adoption of TSP is a great way for teams to improve both quality and productivity

Total Cost of Quality = Quality Activities (e.g. Peer Reviews, Unit Testing) + Effort for Defect Rework (System Testing + Bug Fixing) TSP teams average of 9% is four times less rework than typical Teams found 93% of all defects before integration and system

test

0%20%40%60%80%

100%

Total Cost of Quality Post-Dev Yield (2)

TSP

Non-TSP

®


Other Benefits realized by teams adopting TSP

Teams are making plans they can fully commit to

Teams are taking ownership of quality

Decision making has been pushed to lowest possible level

Teams plan, track, measure, and improve

Measurement has encouraged other improvements

Improved design practices

Defect analysis to determine common causes and implement improvement methods

Better estimation methods

Automated unit tests (most teams are getting >80% statement coverage)

100% of designs and code inspected

The Benefits have been more than improved quality

®


What you must do to Inspire, Enable and Drive Improvement

Build an inspiring vision for the future

Ask senior leadership to require measurably better quality with each release

Use metrics to identify what is important and help the teams assess how they are doing

Be an enabler of teams… be a coach, not a cop

Broadly communicate the vision, plans and progress

Share successes and recognize improvement

®


Acknowledgements

Watts Humphrey for inventing the Team Software Process and his support in introducing it to Adobe

Noopur and Darryl Davis for the excellent training, consulting and coaching support that is helping make the initial projects successful

Johnny Loiacono and Barry Hills (Adobe) for their strong support of the Adobe Quality initiative

®


inspiring, enabling and driving the evolution of quality

Documents