bspin agile practice benchmarking report 2012 highlights bspin agile practices benchmarking report...

BSPIN Agile Practice Benchmarking Report 2012

Highlights

BSPIN Agile Practices Benchmarking Report October 2012


Introduction• Agile Practice Benchmarking:

BSPIN conducted a study during 2011-2012 to benchmark the Agile practices to provide insights on the effective ways of adopting Agile in small to large organizations. 12 Organizations presented their Agile experience across 3 Panel discussions Agile Practice benchmarking survey was conducted in July 2012

• Designing of Questionnaire, Consolidation, Analysis of feedback was done in coordination with Agile coach and Senior Consultant from Improvalue Consulting, Bangalore (www.improvalue.com).

• Objectives :– Explore new areas like:

• Domains where Agile has been adopted. Agile Practices in use.• Benefits and Barriers encountered by organizations on their Agile journey• Metrics Reported (Project and Product) • Coexistence/Compatibility with other standards (CMMi) and practices (Lean)• Best Practices

– Focus is on IMPLEMENTATION aspects of AGILE

• Outcome:– Learn from industry experience– Enable informed decisions

2


Agenda• Demographics - Agile Spread• Usage of Agile Practices• Estimation Techniques• Automation, Tools• Agile Metrics• Agile – Coexistence with other Methodologies/Models• Benefits and Barriers• Challenges Faced, Best Practices• Conclusions/Recommendations

3


4

• 94 responses from 75 Organizations • More participation from Small and Medium Organizations

(compared to other surveys)• Product and Service Organizations covered equally• 98% of Large Organizations and 77% of Small an d Medium

Organizations have external quality certifications• More than 50% of the respondents have been using Agile for

more than 2 years• Panel discussions – 12 Organizations shared their Agile

Experience ( 6 Large and 6 Small-Medium)

Demographics


How Agile adoption is supported?

Training Methods used:

Support to Agile

5

• This section was answered by 72% of the organizations

• As the organization gains more experience on Agile, Agile becomes a part of QMS

• Only 6% of the organization indicated “Self Management by teams” with no other support.

• This was answered by 90% of respondents

• About 40% of the organization had more than one method of training

• About 20% of the organization indicated only “In-house” with no other support.


Agile Spread

6

• SCRUM is the mostly adopted methodology

• Increase in usage of KANBAN - – Many Organizations evolved from

manufacturing background where LEAN is adopted

• Agile experience of the organization has no major effect on the methodology chosen

• Nearly 70% of the organizations are using 1 or 2 methods

• Of the 7% using more than 4 methodologies, 3% are small organizations


Agile Spread

ABREVIATIONS(1) BFS – Banking and Financial Services (2) IA – Industrial Automation (3) A,A,A – Avionics, Aviation, Aerospace (4) SI – Systems Integration

7

• Agile has been used in almost all domains including safety critical domains like Pharma, Avionics and other Embedded Systems.

• Telecom domain which is mostly hardware dependent has reported highest percentage of Agile implementation.

• More than 75% of the respondents have reported the implementation of ‘Distributed Agile’

• Agile has been used in most of the project types – including Testing and Systems integration

Others Domains Reported:•OS, Middleware, Kernel•Platform development •Control Systems

• Product Integration• Media and Publication, Education• Travel, Energy and Storage


Usage of Agile Practices

8

• Core Agile (Scrum) Practices have been well adopted.• Engineering Practices usage (Refactoring, Test Driven Development) needs to be strengthened.

82% have release cycles <= 4 weeks

• 48% of the respondents use 2 to 3 weeks release cycles.


Agile Practices Vs Agile Experience

9

• Core Agile practices are embraced right from the beginning

• There is a significant increase in the usage of Open Work Areas (compared to other surveys)

• Some of Scrum practices take more time to stabilize

• Organizations can have increased rigor early in the Agile adoption for better benefits.

• The Engineering practices strengthen over time

• Refactoring and TDD need more focus for effective Agile adoption


Agile Estimation Techniques

10

• About 53% of the respondents use "User Stories" estimation at proposal/bid stage

• About 80% of the respondents are using one or more of the following estimation techniques: i. User Stories estimationii. WBSiii. Guesstimation

• Organizations are taking more than 2(two) years to move away from Guesstimation to WBS which needs attention.

Usage of traditional estimation techniques at proposal/bid stage and User Story Estimation/ WBS at iteration stage seems to be in vogue


Extent of Automation

11

• Only 76% of the respondents answered this section

• Automation of unit testing is low, inline with low adoption of TDD

• Automation of code reviews is low, as evident from low adoption rate of Refactoring

• Automation done in later stages of testing and quality compromised- missing out the essence of Agile

• The automation level increases with Agile experience of the organization

• The size of the organization had no impact on the level of automation

Organizations have shared multiple Tools : For Agile Project Management and Engineering.


Tools Used

12

Requirements

• Caliber RM

• DOORS

• Greenhopper

• JIRA

• RALLY

• Rational Requisite Pro

• Red Mine

• ScrumWorks

• Sharepoint

• TFS

• VersionOne

• Caliber RM

• DOORS

• Greenhopper

• JIRA

• RALLY

• Rational Requisite Pro

• Red Mine

• ScrumWorks

• Sharepoint

• TFS

• VersionOne

Code Reviews

• CheckStyle

• Eclipse plugins

• FxCop

• IBM AppScan

• Rational Software Analyzer

• SONAR

• StyleCop

• TFS

• Toad

• xTest

• xUnit

• CheckStyle

• Eclipse plugins

• FxCop

• IBM AppScan

• Rational Software Analyzer

• SONAR

• StyleCop

• TFS

• Toad

• xTest

• xUnit

Testing

• HP Quality Center

• HP Quick Test Professional

• JIRA

• LoadRunner

• MS VS

• Selenium

• Silk

• VSTS

• Winrunner

• xTest

• xUnit


• HP Quick Test Professional

• JIRA

• LoadRunner

• MS VS

• Selenium

• Silk

• VSTS

• Winrunner

• xTest

• xUnit

Defect Tracking

• Bugzilla

• TeamForge

• Clearquest


• JIRA

• Rational ClearQuest

• TFS

• Bugzilla

• TeamForge

• Clearquest


• JIRA

• Rational ClearQuest

• TFS

Project Management

• CollabNet

• Green Hopper

• HP-Project and Portfolio Management

• JIRA

• Mingle

• Rally

• Scrum Works

• TFS

• Version One

• XPlanner

• CollabNet

• Green Hopper

• HP-Project and Portfolio Management

• JIRA

• Mingle

• Rally

• Scrum Works

• TFS

• Version One

• XPlanner

Not a complete list. Widely used tools are reported here.


Traditional Practices with Agile

13

• 56% of the respondents reported:– Upfront collection and finalization of

requirements – Upfront architecture

• This practice was predominantly observed in Product companies when compared to Service Providers


Management Metrics

14

Other Metrics Reported:

Schedule: Schedule Variance, Sprint Stretch Factor

Effort: Effort Variance,

Scope: Commitment Accuracy, (De)Scoping Index,Requirements Stability, Scope burn down

Quality: Product Metrics, Review Efficiency, Defect Turnaround Time, Defect Resolution Productivity, Defect Removal effectiveness

Testing: Test Yield, Test case pass percentage

• 90% of the respondents have answered this question

• The usage of metrics increases with Agile experience to measure effectiveness

Recommend BSPIN to conduct ‘Agile data Benchmarking’ to obtain industry benchmark


Engineering Metrics

15

• Monitoring Product Metrics is fairly good except for the Code related activities

• Measuring Code complexity is key for refactoring – needs more focus

• Measuring Code coverage is key for quality – needs more attention

• There seems to be a steady improvement as the organization gains more experience on Agile

• However, Code Complexity and Code Coverage metrics needs more focus


Agile - CMMi Coexitence

16

• This question was answered by 72% of the respondents

• 68% of the respondents think that Agile and CMMi can coexist. Of these 60% are from organizations that have ISO or CMMi certification.

• 72% of the respondents reported that their Agile projects were also being audited (as per the corresponding certification process)

What does the Survey Report?

Reasons cited for “No” and “not Sure” :i.Different focus (People centric Vs Process centric)ii.Agile is project management, engineering framework and CMMi is a process improvement frameworkiii.It is compatible till L3 except IPM, OPD

Reasons cited for “Yes”::i.Agile does not throw away the processes. It is a mature way of looking how efficient the processii.CMMi prescriptions can be applied to any methodology.


CMMi on Agile

17

What does CMMi v1.3 say?

To help those who use Agile methods to interpret CMMI practices in their environments, notes have been added to selected process areas. These notes are added, usually in the introductory notes, to the following process areas in CMMI-DEV: CM, PI, PMC, PP, PPQA, RD, REQM, RSKM, TS, and VER.. . .Even if you are not using an Agile approach, you might still find value in these notes.. . .Some general background and motivation for the guidance given on Agile development approaches are found in the SEI technical note CMMI or Agile: Why Not Embrace Both! [Glazer 2008].

To help those who use Agile methods to interpret CMMI practices in their environments, notes have been added to selected process areas. These notes are added, usually in the introductory notes, to the following process areas in CMMI-DEV: CM, PI, PMC, PP, PPQA, RD, REQM, RSKM, TS, and VER.. . .Even if you are not using an Agile approach, you might still find value in these notes.. . .Some general background and motivation for the guidance given on Agile development approaches are found in the SEI technical note CMMI or Agile: Why Not Embrace Both! [Glazer 2008].

Possible reason for higher maturity organizations to agree that Agile and CMMi are

compatiblePractitioner Groups of CMMi and Agile can integrate the practices

to adopt the best of both

Reference: Section-5: Interpreting CMMI When Using Agile Approaches – CMMI-DEV, V1.3 by SEI


Coexistence with other models Integrating LEAN with Agile

• 33% of the respondents reported using LEAN techniques with Agile

• Most organizations (26%) have adopted KANBAN

Integrating Agile with other Life Cycle Models

• 46% of the respondents reported using Agile best practices in other LCMs

• Agile practices predominantly used:

18


Benefits of Agile adoption

19

Key Benefits:

•Better management of Requirement and Priority changes, Risks

•Improved time-to-market and customer satisfaction


Benefits Perceived – Role wise

20

• Benefits perceived by organizations increase with Agile experience, inline with practices adopted and metrics collected/analyzed

• Ability to handle risks and improved customer satisfaction were rated high by Agile team and Agile Coach

• Managing changing priorities and better project visibility were rated high by senior management and process professionals


Barriers to Agile adoption

21

Agile Coaches/Consultants and Senior Management to focus on methodical transition to AGILE

These observations are in line with the world wide survey reports by other organizations


Barriers Perceived – Role wise

22

• Barriers reported by Senior Management and Process Professional are much higher than Agile Coach/Consultant and Agile Team

• The size and Agile experience of the organization had no significant impact on the barriers

0%

10%

20%

30%

40%

50%

60%Project technical complexity / size

Lack of focus among PM / PMO

Lack of onsite - offshorecollaboration

Agile cannot handle Maintenance

Lack of Skilled Resources

Resistance to change

Agile produces Insufficientdocumentation

Lack of tools to support shortiterations

Management not interested

Budget constraints

Senior Management Process Professional Agile Coach Agile Team


Benefits and Barriers - Impact

23

• Time-to-market• IT and Business Collaboration • Handle risks earlier/ better

• Time-to-market• IT and Business Collaboration • Handle risks earlier/ better

• Manage changing requirements• Better Team-morale• Better collaboration

• Manage changing requirements• Better Team-morale• Better collaboration

• Manage changing priorities• Better project visibility• Customer/End-user satisfaction

• Manage changing priorities• Better project visibility• Customer/End-user satisfaction

• Project technical complexity / size• Lack of onsite - offshore collaboration• Lack of tools/ Automation for short iterations

• Project technical complexity / size• Lack of onsite - offshore collaboration• Lack of tools/ Automation for short iterations

• Lack of coordinated transition to agile• Lack of focus among PM/PMO Groups• Resistance to change

• Lack of coordinated transition to agile• Lack of focus among PM/PMO Groups• Resistance to change

• Agile cannot handle Maintenance• Lack of skilled resources• Agile produces Insufficient documentation

• Agile cannot handle Maintenance• Lack of skilled resources• Agile produces Insufficient documentation

Team

ManagementCustomer


• Can think of Hybrid model based on the scope clarity.

• Redefine/Customize the traditional metrics.

• Handholding by Agile coach. Mandatory Agile skill training.

• Stakeholders to come together and decide.

• Policies need to be flexible to have Team benefits

• Address the dependency issues in Daily stand-up meetings. Effective Scrum of Scrums

• Senior management to focus on infrastructure needs.

• Teams to look into non-functional aspects during planning meetings. Detail planning of testing effort for manual testing

• Benchmark within the organization . Participate in external Forums.

• Breakdown stories for ease of development. Help customers articulate User Stories and Product Backlog.

• Usage of Tools

• Enhance People Agile skills; especially on Engineering side.

• Fixed Price Model is a challenge, T&M have worked well.

• Collection of Traditional Metrics in Agile Projects.

• Moving away from Command & Control; Self Organizing Teams

• Scope creep and sign-offs.

• Compensation / Appraisal related issues due to no individual Heroes

• Communication/ Collaboration challenges between the distributed sub-teams.

• Infrastructure needs for Continuous integration. Teams Not exposed to usage of tools and automation.

• Frequent release cycles in sprints thus not enough time for testing. Very less focus on Non functional Test. Some legacy application testing cannot be fully automated.

• No predefined performance benchmarks.

• User story articulation by different stakeholders.

• Focus on Maintainability and code quality.

• Not all teams will be empowered or trained for ‘Test Driven Development’. Need External Support.

Challenges ExpressedChallenges Faced Possible Solution

24


• Record the retrospection and use some Agile tools for tracking them.

• Stake holder involvement in defining test scenarios.

• More encouragement from management to adopt Test Driven and Behaviour Driven Development.

• Invest on tools and infrastructure to run automated regression suite on every check in

• Representative at onsite for each offshore team: Overlap of the key people from offshore with onsite team.

• Slightly heavy onsite team for better coordination.

• Good build plan. Report invocation from Build.

• Teams based on functionality rather than skills

• Buddy Testing from developer machine to save the deployment time.

• All the team members who are part of particular sprint should be part of the scope and estimation. • Measurement of code complexity for quantification.

• Use of Agile training from time to time, participation in Agile conferences and Forums.

Best Practices Reported

25


Conclusions and Recommendations

26

• Agile as a methodology is well established - more than 50% of respondents have used it for past 2 years and included it in QMS. Scrum is the most widely used methodology followed by Kanban

• Agile usage is spread across multiple domains and project types; Over 75% of the respondents have confirmed using Distributed Agile

• While Management practices (predominantly from SCRUM) have been well adopted, the adoption of engineering practices (predominantly from XP) needs more rigor

• Usage of metrics and automation level increased with Agile experience• Coexistence of Agile with other industry standards (like ISO and CMMi) has been agreed upon

NEXT STEPS• Data Benchmarking of Agile projects• Practitioner Groups of CMMi and Agile can integrate the practices to adopt the best of both

Bibliography•CMMi for Development, Version 1.3 by SEI•“2011 State of Agile Development Survey Results” by VersionOne•“Agile adoption in India Survey Report – 2011” by ThoughtWorks•“Agile India 2012 Survey Results” by Valtech


Discussions

THANK YOU

27

bspin agile practice benchmarking report 2012 highlights bspin agile practices benchmarking report...

Documents