Pedro Serrador, PMP, P.Eng., MBA, PhD

Serrador Project Management

Does Agile Work? -A Quantitative Analysis of Project Success

Dr. Pedro Serrador Serrador Project Management

Dr. Jeffrey K. PintoBlack School of Business

Penn State – Erie

� Review trends in the use of agile in industry� Understand whether Agile is associated with

more successful projects� Discuss what situations and moderators impact

the relationship between agile use and success

� And as early as 1958, Koontz noted that “no effective manager makes a plan and then proceeds to put it into effect regardless of what events occur”

(Koontz, 1958, p. 54).

“It is a bad plan that admits of no modification.”

- Publilius Syrus (~100 BC)

� Evolved in the mid-1990s as part of a reaction against "heavyweight" methods,

� An adaptive software development process was introduced in a paper by Edmonds (1974). Notable early Agile methods include Scrum (1995), Crystal Clear, Extreme Programming (1996), Adaptive Software Development, Feature Driven Development, and Dynamic Systems Development Method (DSDM) (1995).

� These are now typically referred to as Agile Methodologies, after the Agile Manifesto published in 2001.

� Later, some of these authors formed The Agile Alliance, a non-profit organization that promotes agile development.

� Started in 1994, studied over 35,000 application development projects

� In 2000:

Source: Standish “Chaos” Report, Jim Johnson lecture at XP2002 conference, http://www.xp2003.org/xp2002/talksinfo/johnson.pdf

� It has been known since the 70s that defects in software development are cheaper to fix the earlier they are found.

� At each stage in development the cost of a defect or missed requirement gets higher.

� The most expensive time to find a defect? After a project is delivered.

Relative Cost to Fix a Defect, after Poston (1985)

� Individuals and interactions over processes and tools

� Working software over comprehensive documentation

� Customer collaboration over contract negotiation� Responding to change over following a plan

� Customer satisfaction by rapid, continuous delivery of useful software

� Working software is delivered frequently (weeks rather than months)

� Working software is the principal measure of progress� Even late changes in requirements are welcomed� Close, daily cooperation between business people and

developers� Face-to-face conversation is the best form of communication

(co-location)� Projects are built around motivated individuals, who should be

trusted� Continuous attention to technical excellence and good design� Simplicity� Self-organizing teams� Regular adaptation to changing circumstances

� Software development� Small teams� New Product development � Low criticality� Senior developers� Requirements change often� Cultures that thrive on chaos

� Large teams� Inexperienced team members� Geographically dispersed teams� High criticality� Requirements do not change often� Culture that demands order

� Small releases� Iterative and incremental development� Collocation � Release plan/feature backlog � Iteration plan/feature backlog � Self organizing teams� Pairing (in XP)� Test-driven development� Tracking� Simple, lean and adaptable

Traditional development Agile development

Fundamental assumption

Systems are fully specifiable, predictable, and are built through meticulous and extensive planning

High-quality adaptive software is developed by small teams using the principles of continuous design improvement and testing based on rapid feedback and change

Management style Command and control Leadership and collaboration

Knowledge management

Explicit Tacit

Communication Formal Informal

Development model Life-cycle model The evolutionary-delivery modelDesired organizational form/structure

Mechanistic (bureaucratic with high formalization), aimedat large organizations

Organic (flexible and participative encouraging cooperative social action), aimed at small and medium sized organizations

Quality control Heavy planning and strict control. Late, heavy testing

Continuous control of requirements, design and solutions.Continuous testing

Information Technology Project Management, Seventh Edition (Schwalbe, 2014)

� Although Agile has been used for project planning for a number of years now, to date, the majority of research examining its usefulness has been:

� anecdotal, � single-case studies� based on small sample sizes � single-organization � single-industry settings

Method- Global on-line questionnaire

Methodology Approach- Post-positivism and quantitative

Target Audience- Project managers

Unit of Measure- Completed projects

� 859 participants provided information on at least one project.

� 1,386 projects.

� 1002 projects provided data on agile

� People from over 60 countries

� Each participant provided data on two projects, one more successful and another less successful.

� One question on agile planning during execution was misunderstood by some respondents.

� A transformation was undertaken on one subset of responses to correct the error

� Homogeneity was confirmed between the two groups (group 1 – no transformation needed and group 2 – transformed)

� Project efficiency – meeting cost, time and scope goals

� Stakeholder success – satisfying the expectations of project stakeholders who are the best judges of overall success

Efficiency Factor = mean of the following three responses:1.How did the project do in meeting project budget goals?2.How did the project do in meeting project time goals?3.How did the project do in meeting project scope and

requirements goals?

Stakeholder Success Factor = mean of the following four responses:1.How did the project sponsors and stakeholders rate the success

of the project? 2.How do you rate the project team’s satisfaction with the project?3.How do you rate the client’s satisfaction with the project’s

results?4.How do you rate the end users’ satisfaction with the project’s

results?

Percentage Agile/

Iterative

Count Percent Cumulative -

Percent

80-100% 80 6 6

60-79% 152 11 17

40-59% 347 25 42

20-39% 162 12 54

1-19% 194 14 68

0% 451 32 100

Missing 0 0 100

The following moderators were examined:◦ quality of the vision/goals of the project◦ project complexity◦ project team experience level◦ stakeholder engagement level

Agile Planning Effort index

= total effort expended on planning after the planning phase

the total project effort (in person days)

Percentage Agile/

Iterative

Upfront planning effort

index

Agile planning

effort index

Efficiency

Factor

Stakeholder Success

Factor

Valid N

80-100% 0.161 0.149 4.82 3.64 80

60-79% 0.147 0.138 4.66 3.57 152

40-59% 0.164 0.132 4.79 3.54 347

20-39% 0.135 0.101 4.63 3.41 162

1-19% 0.150 0.091 4.46 3.18 194

0% 0.154 0.048 4.58 3.21 451

All Groups 0.153 0.105 4.64 3.38 1386

p(F) 0.173 0.000 0.087 0.000

Regression Project success rating

R² p-level

Methodology type 0.030 0.000

Combined Agile measure = mean of the following two responses as a summated scale of normalized values:

� Methodology type � Agile planning index

Where Agile planning index is defined as follows:Agile Planning Index

= total effort expended on planning after the planning phase

total project effort (in person days)

1

1.5 2

2.5 3

3.5 4

4.5 5

0.01

0.03

0.05

0.07

0.09

0.11

0.13

0.15

0.17

0.19

0.21

0.23

0.25

0.27

0.29

0.31

0.33

0.35

0.37

0.39

0.41

0.43

0.45

0.47

0.49

0.51

0.53

0.55

0.57

0.59

Success Factor

Effo

rt Ind

ex

Ma

xim

um

Imp

act

Ma

xim

um

Imp

act

Variables entered Step 1 Step 2 Step 3

Main EffectsCombined Agile

measure .875***

Moderators

Quality of vision/goals

Interaction TermsQuality of vision/goals*Combined

Agile measure

R2 0.019***

+ p < .10, * p < .05, ** p < .01, *** p < .001

Variables entered Step 1 Step 2 Step 3

Main EffectsCombined Agile

measure .875*** .572**

Moderators

Quality of vision/goals -.513***

Interaction TermsQuality of vision/goals*Combined

Agile measure

R2 0.019*** 0.151***

+ p < .10, * p < .05, ** p < .01, *** p < .001

Variables entered Step 1 Step 2 Step 3

Main EffectsCombined Agile

measure .875*** .572** -.253

Moderators

Quality of vision/goals -.513*** -643***

Interaction TermsQuality of vision/goals*Combined

Agile measure.401+

R2 0.019*** 0.151*** 0.152+

+ p < .10, * p < .05, ** p < .01, *** p < .001

Highly Successful

Mediocre

Successful

Unsuccessful

Failure

Agile Factor Impact

Methodology type

Efficiency Factor

Stakeholder Success Factor

Valid N R2 Regression p value vs. Success

Manufacturing 4.81 4.44 3.3 87 0.008 0.404Financial services 4.78 4.62 3.27 167 0.015 0.112Construction 4.74 4.54 3.65 36 0.035 0.274High technology 4.53 4.69 3.43 166 0.042 0.008*Health care 4.5 5.04 3.52 80 0.057 0.032*Telecommunications 4.34 5.11 3.71 95 0.005 0.477Retail 4.31 4.56 3.23 31 0.000 0.889Professional services 4.27 4.53 3.32 56 0.056 0.078Utilities 4.26 4.44 3.41 35 0.001 0.826Other 4.25 4.48 3.19 109 0.121 0.0002*Education 3.8 4.87 3.15 25 0.063 0.224Government 3.65 4.3 3.17 115 0.010 0.277All Groups 4.43 4.62 3.36 1002 0.019 0.007*

*- indicates statistical significance

Methodology

type

Efficiency

Factor

Stakeholder

Success

Factor

Success

FactorValid N R2

Regression

p value vs.

Success

Pacific 3.67 4.70 3.28 3.30 22 0.072 0.227

Russia and FSU 3.67 4.44 3.33 3.27 9 0.214 0.209

Europe 4.00 4.69 3.41 3.39 142 0.002 0.592

Indian subcontinent 4.17 4.30 3.44 3.29 83 0.037 0.081

Middle East 4.33 4.33 3.31 3.22 73 0.000 0.911

North America 4.55 4.74 3.43 3.41 527 0.029 0.000*

Latin America 4.61 4.48 3.06 3.11 58 0.118 0.008*

Far East 4.90 4.73 3.50 3.46 24 0.001 0.862

Africa sub-Sahara 5.00 4.67 3.07 3.18 32 0.016 0.485

Australasia 5.08 4.67 3.00 3.16 32 0.003 0.766

All Groups 4.43 4.62 3.36 3.33 1002 0.019 0.007*

*- indicates statistical significance

� How do different types of projects fare under Agile?

39

Project team sizeNumber of

ProjectsR2 p-value

1-5 224 0.036 0.005

6-15 375 0.019 0.007

16-30 186 0.042 0.005

31-50 91 0.014 0.261

51-100 78 0.000 0.864

101-500 41 0.015 0.444

500+ 7 0.052 0.622

40

Complexity of the

project

Number of

ProjectsR2 p-value

Low 73 0.016 0.278

Medium 512 0.016 0.004

High 417 0.035 0.000

41

Local vs. remote

team

Number of

ProjectsR2 p-value

One city or

region424 0.015 0.009

National 263 0.042 0.001

International 315 0.024 0.005

42

43

level of use of

technology

Number of

ProjectsR2 p-value

Lowtech 101 0.039 0.049

Mediumtech 437 0.010 0.037

Hightech 408 0.023 0.002

Super hightech 56 0.078 0.037

Degree of stakeholder

engagement

Number of

ProjectsR2 p-value

Very Engaged 400 0.001 0.430

Somewhat

Engaged452 0.021 0.002

Mostly not

Engaged132 0.085 0.001

Not at all Engaged18 0.139 0.128

44

Experience level

of team

Number of

ProjectsR2 p-value

High 349 0.010 0.057

Medium 538 0.030 0.000

Low 115 0.023 0.100

45

Quality of goals/vision

statement

Number of

ProjectsR2 p-value

Excellent 191 0.000 0.772

Good 569 0.014 0.005

Poor 198 0.033 0.010

Very Poor/ Not

used44 0.085 0.055

46

N=1002 Stakeholder

Success Factor

Efficiency

factor

Stakeholder Success Factor

(group 2 only)

R2 0.152 0.096 0.171

p=0.089 p=0.083 p=0.010

� Agile been adopted in multiple industries and across national borders

� Agile projects plan more than traditional projects

� Agile has achieved it’s best success to date within some industries; notably, high technology, healthcare, and professional service

� The quality of goals and vision of a project are very important to the success of Agile projects

� The amount of agile adoption is positively correlated with project success

� Agile has a greater impact on stakeholder success measures than efficiency measures

� After removing moderator effects, agile use was found to be have an R2 of .15 with project success

� Agile appears to work well with experienced and less experienced teams, if stakeholder engagement is low and in complex environments

� The success of different agile methodologies should be studied in greater detail.

� The role of project complexity in the relationship between Agile and success is an area that warrants further study.

� Analysis of projects where more time is spent replanning during execution is warranted to understand if they are more or less successful.

� Does the relationship between how planning is structured impact Agile success?

� It would be useful to revisit this field in a few years to determine the ongoing diffusion rate of Agile given its record of success

� Test as you build� Review as early as possible with stakeholders� Close team work� Good communication with business

customers

� Get team support� Get management support� Explain it to the business� Finalize your plan!

� Find champions with in the team� Get them involved in the planning� Get them enthusiastic and they will help sell

the idea!

� See what the team is ready for� Look at what fits the environment� Don’t try to do it all right away!

� Resistance from some team members� Business or management may not be

receptive� Drift into old processes� When time crunch comes, processes can fall

to the wayside� Stay the course through problems and delays

� Review trends in the use of agile in industry� Understand whether Agile is associated with more

successful projects� Discuss what situations and moderators impact the

relationship between agile use and success

Pedro Serrador, PMP, P.Eng., MBA, PhD

Serrador Project Management

pedro@serrador.net

www.serrador.net

1-866-927-0506

Find me on LinkedIn.com

Contact Information

Ahern, T., Leavy, B., Byrne, P. J., 2014. Complex project management as complex problem solving: A distributed knowledge management perspective. International Journal of Project Management, in press.

Atkinson, R., 1999. Project management: cost, time and quality, two best guesses and a phenomenon, its time to accept other success criteria. International Journal of Project Management, 17(6), 337-342.

Baccarini, D., 1996. The concept of project complexity — a review. International Journal of Project Management, 14(4), 201–204.

Boehm, B., 1996. Anchoring the software process. IEEE Software, 13(4), 73-82.

Boehm, B., 2002. Get ready for agile methods, with care. Computer, 35(1), 64-69.

Budzier, A., Flyvbjerg, B., 2013. Making sense of the impact and importance of outliers in project management through the use of power laws. Paper presented at IRNOP. Oslo.

Collyer, S., Warren, C., Hemsley, B., Stevens, C., 2010. Aim, fire, aim - Project planning styles in dynamic environments. Project Management Journal, 41(4), 108-121.

Conway, J., Lance, C., 2010. What reviewers should expect from authors regarding common method bias in organizational research. Journal of Business and Psychology, 25(3), 325-334.

Cooke-Davies, T., 2002. The "real" success factors on projects, International Journal of Project

Management, 20, 185-190.

Cooper, D., Schindler, P., 2001. Business Research Methods. New York: Irwin/McGraw-Hill.

Coram, M., Bohner, S., 2005. The impact of agile methods on software project management. Proceedings of the 12th IEEE International Conference and Workshops on Engineering of Computer-Based Systems (pp. 363-370). Washington, DC, USA: IEEE Computer Society.

Crawford, L. H., Hobbs, J. B., Turner, J. R., 2004. Project categorization systems and their use in organisations: an empirical study. Proceedings of PMI Research Conference. London.

Dvir, D., Raz, T., Shenhar, A., 2003. An empirical analysis of the relationship between project planning and project success. International Journal of Project Management, 21(2), 89-95.

Dybå, T., Dingsøyr, T., 2008. Empirical studies of agile software development: A systematic review. Information and Software Technology, 50(9), 833-859.

Field, A., 2009. Discovering Statistics Using SPSS. London: Sage Publishing.

Fitzgerald, B., 1996. Formalized systems development methodologies: a critical perspective. Information Systems Journal, 6(1), 3-23.

Hällgren, M. a.-O., 2005. Deviations, ambiguity and uncertainty in a project-intensive organization. Project Management Journal, 36(3), 17-26.

Jones, R., Deckro, R., 1993. The social psychology of project management conflict. European Journal of Operational Research, 64, 216-228.

Jugdev, K., Müller, R., 2005. A retrospective look at our evolving understanding of project success. Project Management Journal, 36(4), 19-31.

Kerzner, H., 2003. Project Management: A Systems Approach to Planning, Scheduling, and Controlling (8th ed). New York: Wiley.

Kloppenborg, T. J., Manolis, C., Tesch, D., 2009. Successful project sponsor behaviors during project initiation: an empirical investigation. Journal of Managerial Issues, 21(1), 140-159.

Koontz, H., 1958. A preliminary statement of principles of planning and control. The Journal of the Academy of Management, Vol. 1, No, 1(1), 45-61.

Koskela, J., Abrahamsson, P., 2004. On-site customer in an xp project: empirical results from a case study. In T. Dingsøyr (Ed.), Software Process Improvement (Vol. 3281, pp. 1-11). Springer Berlin / Heidelberg.

Lindvall, M., Basili, V., Boehm, B., Costa, P., Dangle, K., Shull, F., . . . Zelkowitz, M., 2002. Empirical Findings in Agile Methods. (D. Wells, & L. Williams, Eds.) Extreme Programming and Agile Methods—XP/Agile Universe 2002, 2418, 81-92.

Magazinius, A., Feldt, R., 2011. Confirming distortional behaviors in software cost estimation practice. Proceedings of the 37th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA) (pp. 411-418). IEEE.

Mann, C., Maurer, F., 2005. A case study on the impact of scrum on overtime and customer satisfaction. Agile Conference, July, 2005. Proceedings, (pp. 70-79).

Martinsuo, M., 2013. Project portfolio management in practice and context. International Journal of Project Management, 31, 794-803.

Miller, R., Hobbs, B., 2005. Governance regimes for large complex projects. Project Management Journal, 36(3), 42–50.

Müller, R., 2009. Project Governance. Gower Publishing: Surrey, UK.

Müller, R., Turner, R., 2007. The influence of project managers on project success criteria and project success by type of project. European Management Journal, 25(4), 298-309.

Munns, A., Bjeirmi, B., 1996. The role of project management in achieving project success. International Journal of Project Management, 14(2), 81-87.

Norman, G., 2010. Likert scales, levels of measurement and the “laws” of statistics. Advances in Health Sciences Education, 15(5), 625-632.

Nunnally, J., 1978. Psychometric theory. New York: McGraw-Hill.

Pinto, J. K., Slevin, D. P., 1988. Project success: definitions and measurement techniques. Project Management Journal, 19(1), 67-72.

Pinto, M. B., Pinto, J. K., Prescott, J. E., 1993. Antecedents and consequences of project team cross-functional cooperation. Management Science, 39, 1281 – 1297.

Podsakoff, P., MacKenzie, S., Lee, J., Podsakoff, N., 2003. Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879-903.

Prabhakar, G., 2008. What is project success: a literature review. International Journal of Business and Management, 3(8), 3-10.

Ritter, T., 2007. Public sector IT projects have only 30% success rate. CIO for Department for Work and Pensions. Retrieved fromhttp://www.computerweekly.com/blogs/public-sector/2007/05/public-sector-it-projects-have.html

Serrador, P., 2013. The impact of planning on project success: a literature review. Journal of Modern Project Management, 1(2), 28-39.

Serrador, P., Turner, J. R., 2013. The Impact of the planning phase on project success. Paper presented at IRNOP. Oslo, Norway.

Serrador, P., Turner, J. R., 2014. The Relationship between project success and project efficiency. Unpublished working paper.

Sharma, S., Durand, R., Gur-Arie, O., 1981. Identification and analysis of moderator variables. Journal of Marketing Research, 291-300.

Shenhar, A. J., Dvir, B., 1997. Mapping the dimensions of project success. Project Management Journal, 28(2), 5-9.

Shenhar, A. J., Dvir, D., Levy, O., Maltz, A. C., 2001. Project success: a multidimensional strategic concept. Long Range Planning, 34(6), 699-725.

Smits, H., 2006. 5 levels of agile planning: from enterprise product vision to team stand-up. Rally Software Development Corporation. Rally Software Development Corporation. Retrieved from http://www.rallydev.com/downloads/document/2-five-levels-of-agile-planning-from-enterprise-product-vision-to-team-stand-up.html

The Standish Group. (2011). CHAOS Manifesto 2011. Accessed June, 2011, The Standish Group. Retrieved from http://standishgroup.com/newsroom/chaos\_manifesto\_2011.php

Thomas, M., Jacques, P. H., Adams, J. R., Kihneman-Woote, J., 2008. Developing an effective project: planning and team building combined. Project Management Journal,, 39(4), 105-113.

Turner, J. R., Cochrane, R. A., 1993. Goals-and-methods matrix: coping with projects with ill defined goals and/or methods of achieving them. International Journal of Project Management, 11(2), 93-102.

Williams, T. M., 1999. The need for new paradigms for complex projects. International Journal of Project Management, 17(5), 269–273.