Agile Workshop
Leveraging Agile Concepts & Techniques To Enable Enterprise Modernization
7/9/13 © The Mercator Group 1
Agenda
• Workshop I: Guide to Leveraging Agile (3 to 4 hours) • Workshop II: Hands-on Exercise (1.5 days)
– Develop a Product Backlog – Develop a Sprint Backlog – Conduct a Sprint – Lessons Learned
7/9/13 © The Mercator Group 2
Objective of Workshop I
A Practical Guide to leveraging Agile concepts & techniques
• Brief primer on Agile • Introduction to Agile Methodologies
– Scrum – Agile BPM and Model Driven Requirements
• Primer on how to manage Agile efforts • Guidelines on how to stand up an Agile capability and
adopt Agile to meet your organizational needs
7/9/13 © The Mercator Group 3
What Is Agile?
Agile is set of principles that proponents believe to be a better way of developing software
• Agile grew out of a “Manifesto” issued in 2001 by a group of 17 influential programmers, that included the following:
– Individuals and interactions over processes and tools – Customer collaboration over contract negotiation – Working software over comprehensive documentation – Responding to change over following a plan.
• Agile gains its substance through the methodologies that implement its principles, such as Scrum, which is not only a methodology, but a framework most other methods are based on.
• All Agile methodologies have in common a set of core principles: – Focus on collaboration – Shared-ownership and responsibility – Adaptation and continuous learning – Iteration – Open-ended requirements and time-boxed implementation
7/9/13 © The Mercator Group 4
How Agile Fundamentally Changes the Development Process
• Under Agile, cross-functional and self-organizing teams work together to collaborate in all phases of the development process
• Unlike other approaches to software development, changing requirements are not only accepted, but thought of as fundamental to the process
• Agile completely transforms the development process, fundamentally altering roles, responsibilities, required skills, process and culture:
– End users are no longer “customers,” but part of a team with joint responsibility for the outcomes
– Documentation is more of an output of the process, than a control mechanism to the process
– The quality of delivered functionality takes precedence over completing and end-to-end solution, at least as initially perceived
• Strong governance, software architecture, and software change management processes are even more important to manage the higher level of change that occurs as part of the process
7/9/13 © The Mercator Group 5
How Do You Implement Agile?
Figuring out what works for your organization – not one single orthodoxy
Start With Agile Development As A Philosophy
Understand Agile Methodologies
Adopt One To Fit Your Organizational Needs
Agile fundamentally changes roles, relationships, culture, and power structures, and needs to be viewed as an organization change effort
7/9/13 © The Mercator Group 6
Agile Is Supported by Multiple Methodologies
• All Agile methodologies stress collaboration, joint ownership, flexibility, and iteration.
• While Scrum is the most used framework for Agile methodologies, there are many other influences (eXtreme Programming, Model Driven Architecture, Lean Manufacturing).
• Many vendors have their own proprietary Agile approach, e.g. IBM has created Disciplined Agile Delivery (DAD) methodology.
• TMG has developed Agile BPM for process oriented application development.
• It is up to the individual organization to decide what best fits its need.
• For the purpose of this workshop we focus on Scrum, with some level of discussion on Agile BPM, which we have successfully implemented at GSA.
7/9/13 © The Mercator Group 7
Agile Enables the Leveraging of Modern Software Capabilities
• Software has typically been treated as an expensive and rigid asset requiring a lot of upfront planning and design to ensure a return on investment, and control risk
• While still expensive, software environments have become very flexible and adaptive to change:
– Modern environments allow for easy configuration of Lego-block like capability, versus laborious and error prone coding
– Service Oriented Architecture, combined with powerful object oriented abstraction, have further improved the flexibility of software, and the availability of reuse
– Most packages today have easy to understand modeling environments that can be mastered by non-programmers
– APIs allow for easy integration of diverse environments, including email, portals, office automation tools, and other applications
• Adopting Agile is required for an organization to leverage these capabilities, and provide quicker and iterative turnaround of capabilities, cheaper and more flexible environments, and higher quality and better aligned software.
7/9/13 © The Mercator Group 8
Agile Using The Scrum Method
• Scrum is an industry standard framework for implementing Agile: – Integrated project teams (IPTs) share joint responsibility for achieving the
goals of the effort – Business stakeholders, end users, analyst, trainers, testers and developers
work as a unit to develop a solution; usually with interim releases of functionality
– Iterations, referred to as Sprints (Scrum methodology); typically organized into two to six week timeframes for defining and developing a functional subset of the required capabilities
– The team is guided by a Product Owner who is responsible for prioritizing work activities, and manage the addition of new or changing requirements into the effort.
– A Scrum Master helps lead the team by providing expertise in the methodology. The Scrum Master is not the Project Manager.
– The outcome of the effort is working software, while minimum documentation tends to be a byproduct of Development, not an input to Development
7/9/13 © The Mercator Group 9
Key Scrum Roles and Responsibilities
Scrum adds some additional role descriptions to the software development process
7/9/13 © The Mercator Group 10
Scrum Role Responsibilities Product Owner Sets objectives and priorities, ensure outcome reflect the
needs of the business, and their priorities Team Self-managed teams sharing the responsibilities for
getting the work done Scrum Master Facilitates Scrum activities, coaches and focuses on
removing impediments Product Assurance Team
Works with Product Owner to ensure Scrum Objectives and prioritization reflect a wide organizational view
Business System Architect (TMG Opinion)
Provides expertise in facilitating tradeoffs among business objectives, technology, and project resource or timeline limitations
Notional Scrum Project Organization
Executive Sponsors Steering Committee
Product Owner
Project Lead
Project Management
Team
End Users Analysts Software Professionals
Scrum Master
Bus. Systems Architect
Product Assurance
Team
7/9/13 © The Mercator Group 11
Roles in blue boxes represent roles particular to the Scrum methodology
Agile Teams
• Should be ten or less people • Geographically co-located, with shared team space, or
mitigation strategy for dispersed teams • Fulltime • Receive Agile training • Open-minded
7/9/13 © The Mercator Group 12
The Product Backlog
• The Product Backlog is an ordered list of requirements/features controlled by the Product Manager
• The list is ordered according to the order in which the requirements should be implemented
• Many Agile methodologies define the list as the ordered set of User Stories to be implemented
• TMG advises the Product List be made up of a more definitive and detailed set of requirements that represent capabilities to be implemented (business requirements), and that it be supported by some level of process design
• The Product Backlog will be broken into a set of time-boxed Sprints for implementation
7/9/13 © The Mercator Group 13
The Sprint Backlog
• The Sprint Backlog further decomposes the Product Backlog into a set of activities required to implement the the Product Backlog included in that Sprint
• The activities represented in the Sprint are what is managed
7/9/13 © The Mercator Group 14
The Scrum Lifecycle
Prioritized Requirements – Divided into a time-boxed set of activities - To deliver an increment of functionality – Which will become part of an iterative release of capabilities supporting a business process
7/9/13 © The Mercator Group 15
User Stories - Process Modeling - Product Backlog
As a Requestor, I need to enter CLINs in my procurement request.
USER STORY <Role> <Activity> <Context>
Subset of sub-process within Procurement Request process
PROCESS MODEL
PRODUCT BACKLOG All of the components of the process models (role, activities, inputs, outputs, deliverables, rules, etc.) are then documented as part of the product backlog to be developed iteratively.
<Role> <Activity>
Many User Stories
Few Process Models
One Backlog
7/9/13 © The Mercator Group 16
The Agile Business Process Management (BPM) methodology
• TMG’s Agile BPM methodology brings together Agile with Business Process Management Systems (BPMS)
• BPMS allows for the quick configuration and building of process-based software by a non-programmer
• Business Process Management System capabilities available in standalone tools, and as part of many software packages (SAP, Force.com, etc.) provide an excellent way to quickly model and develop system capabilities in an interactive and collaborative environment.
• It allows for easy collaboration between end-user subject matter experts and business analyst to build functionally operational capabilities
7/9/13 © The Mercator Group 17
The Agile Business Process Management (BPM) methodology
• Using BPMS, substantial applications can go from conception to implementation in anywhere from 3 to 16 weeks.
• Taking a business process approach to defining software services has proven to be the most optimal approach to developing effective system capabilities.
• This approach also has the important benefit of eliminating the stove-pipes and data issues so common to approached using a more traditional functional decomposition method of defining software needs.
7/9/13 © The Mercator Group 18
TMG Agile BPM Methodology
* See TMG “practical Guide to Agile BPM”
7/9/13 © The Mercator Group 19
TMG Agile BPM Methodology
Stage Phase Tools/Methods
Inception Scope • User stories • Value chain • High-level process modeling
Iteration Model • Business workflow modeling • BPMS workflow modeling • Process Improvement • Scrum • Facilitated stakeholder sessions • Process simulation
Assemble • BPMS coding • System integration • Scrum • Stakeholder/user testing • Data management
Execute • System deployment • User acceptance testing • Change management
Maturation/Evolution Measure • Process performance metrics • Stakeholder feedback
Analyze • Performance expectation analysis • Continuous process improvement • ROI analysis
Refine • BPMS model refinement (major and minor) • Scrum • Facilitated stakeholder sessions
7/9/13 © The Mercator Group 20
Why Agile Makes Sense?
• Agile recognizes that software development is an empirical discipline: – The effort is time and resourced boxed, what varies is the functionality that is delivered. – Time and resources are fixed. – Requirements are subject to change as further insight is achieved as part of the process.
• Agile takes advantage of “lessons learned” from traditional approaches. Allows for:
– Changes in requirements – Flexibility in both time and resources constraints – Stakeholder acceptance and buy-in – Better focus and responsiveness to “must have” requirements
• The majority of time is spent building what the end user needs, but may not get through the complete list of what is desired
• Agile focuses on optimizing the quality of what is delivered over completeness, cost and schedule:
• Achieving 80% of initial scope is considered a success, as long as the 80% of delivered capabilities is of excellent quality
7/9/13 © The Mercator Group 21
Agile Doesn’t Lessen the Importance of Program Management
• Agile requires strong program management just like any software development effort
• It differs in: – Plans are less static – There is a realization that upfront estimates are subject to change as
discovery takes place as a byproduct of the development effort – Work is always time-boxed and prioritized based on business need – Focus on quality of what is delivered, versus completing the upfront plan
• Agile does require: – Strong governance processes to ensure the added flexibility provided by Agile
does not get abused – Independent architectural oversight to ensure it doesn’t become an excuse for
non-delivery
7/9/13 © The Mercator Group 22
Project Planning
• The necessity of planning and the amount of planning necessary. – Underplanning results in the inability to answer questions like, “When
will you be done?” – Overplanning results in creating a plan that may not be accurate or
useful in execution.
• Planning in Agile – Knowledge and insight are being constantly updated, as such, Agile
planning is an ongoing process and Agile plans are subject to change. – Focus is on features rather than activities. – Take uncertainty into account in the planning phase.
7/9/13 © The Mercator Group 23
Estimation
• Estimation of a project refers to the question, “how long does it take to complete the project?”
• The entire project’s amount of work are broken down in a Work Breakdown Structure (WBS) into smaller units of work.
• The units of work can be based relatively to the amount of work required such as story points and ideal days, or time based such as days or hours.
• Relative estimates based off of work required rather than time required are generally more accurate, because it is difficult to estimate how long a project will take the people actually implementing it to do.
7/9/13 © The Mercator Group 24
• The rate of progress within software development pertaining to Agile methods is called “Velocity.”
• The equation of the velocity of a software development team is similar to the velocity of an object (Velocity of an object = Distance / Time).
• Velocity would be determined by taking the amount of work done (distance), divided by the length of time taken to do the work (time). – Ex: if 20 story points are achieved in the initial week, the velocity
of the team is 20 story points per week. • Velocity is then used to determine overall time the project
will take. – If the said project is a 200 story point project, it is assumed the
project would take 10 weeks. 7/9/13 © The Mercator Group 25
Measuring Progress
Tracking Progress: Burn Rate and Velocity
7/9/13 © The Mercator Group 26
A Program Management View of Agile
Managing Agile Projects Description
Carefully plan iterations Start with a business process design, and move from start to finish through the process. This will result in a logical sequencing of capabilities and resolution of data disconnects across the process
Get team comfortable with shared-ownership
The agile process blurs lines of responsibilities and puts the end user front and center in the process. Establishing team chemistry and implementing an integrated project team approach is essential to success.
Manage change and churn The Agile process, supported by the right tools, can make changing requirements too easy, resulting in excessive churn (“should we make this blue or red?”)
Manage operational risks Agile projects can over-concentrate on end-user functionality, at the cost of administrative and back end integration vital to the operational efficacy of the system.
Ensure operational support capabilities are in place
The level of software change activity goes way up in an agile environment, requiring added attention to the software configuration management function.
Leverage modeling tools The efficacy of the end system increases dramatically with upfront modeling of functionality (Model Driven Requirements – MDR). The team should leverage modeling software to elicit requirements.
Establish an organizational change management process
Agile is a major transformation of the software development process, and needs to be treated as an organizational transformation effort
Develop and maintain project structure
An agile approach, in a subtle, but substantial way, effects the roles of everyone involved in the software lifecycle process. If not managed correctly, it could become a bit chaotic. It is important to maintain project management discipline and leverage the tools that provide for participation and visibility into the effort.
Agile leverages all we have learned in 40 years of software development
7/9/13 © The Mercator Group 27
Refactoring
• Refactoring is improving the quality of the underlying software without affecting behavior.
• In the rush to meet deadlines, it is almost certain that shortcuts will be taken in developing the software.
• These short cuts may allow for 200 to 1600 users to more quickly take advantage of productivity improvements, and provide a huge ROI
• However, over time, they can make the software more brittle and costly to maintain
• To prevent this, all projects should: – Architect the software so refactoring can take place (good object
oriented techniques) – Factor in resources to ensure refactoring can occur
7/9/13 © The Mercator Group 28
Misconceptions of Agile
• Agile means no plan. – Agile doesn’t support development without planning. Rather,
planning and estimation is a continuous process.
• Agile doesn’t allow for documentation. – Documentation should be written at the right time, and only just
the right amount.
• Agile only works for trivial projects. – Agile has been used for:
• End to end hospital healthcare processes • Complex financial applications • Adopted by DoD • Etc.
7/9/13 © The Mercator Group 29
Benefits of an Agile Mindset
• Typically results in software that aligns most closely to business needs.
• Working valuable software is delivered on a frequent basis, allowing for immediate feedback, and use.
• Agile provides more flexibility than Waterfall in requirement changes, even late in development.
• The end product will have fewer defects than if Waterfall methodology was applied.
• Early feedback (non-scientific) is there is a lower rate of failure, even given the level of change required.
7/9/13 © The Mercator Group 30
Agile in the Government
• Key components of Agile from the Agile Manifesto are collaboration takes precedence over contracting, and change should be welcomed
• Of course, most development efforts in the government take place in a structured budgeting and contractual environment, with change being difficult
• Despite these potential limitations, the Government can greatly benefit from adopting some form of Agile:
– Collaboration is beneficial to all parties, and increases buy-in by all participant – Software development environments are much more amenable to a more flexible
and agile development process, and it would be a mistake not to take advantage of this
– Iteration provides for better oversight and program management and reduces risk across the board
– Accepting some level of change is only acknowledging the realities of software development.
7/9/13 © The Mercator Group 31
Adopting Agile Techniques to the Enterprise
Technique Method Difficulty Value Collaborative cross-functional teams
Implement Integrated Project Teams (IPTs)
Low to Medium High
Iterative joint-development Model Driven Requirements (MDR)
Medium High
Self organization and management
Culture change High High
Abstract requirements definition
More flexible contracting
High to Very High Very High
Time-box based management (Open-ended Requirements)
Shared risk contracting
Very High High
Ope
n-en
ded
Req
uire
men
ts
Pre
scrib
ed
Req
uire
men
ts
Government Comfort Zone
Functional Specifications Model Driven Requirements
Middle Ground
User Stories
Agile
7/9/13 © The Mercator Group 32
Modified Agile for Government or a First Step
• A pure Agile approach is extremely difficult to implement in a Government environment due to the general contractual requirements imposed by regulations, and organizational structures and culture that promote clear delineations of responsibilities
• Something has to give: – Agile needs to be broken down into three stages that allow for better definition of the work to be performed
– SOWs need to provide more leeway in how requirements are defined (somewhere between User Stories and detailed functional requirements)
• The TMG three-stage Agile approach is as follows: 1. Conduct business modeling and define Product Backlog: Identify and define Business scope, Roles,
Processes, System touch points and capabilities, Data Requirements
2. Conduct Working Model Sprints: Work with end users to build Working Models and identify functional system integration requirements
3. Assemble and Deploy Working Models: Integrate working models into the operational environment, and provide any additional required customization.
• The three-stage approach: – Maintains much of the collaboration benefits of Agile – Allows for improved control and standard contractual relationships – Provides for overlapping of stages to leverage cross-team collaboration
7/9/13 © The Mercator Group 33
A Stage-Based Approach to Agile for the Government
Conduct business modeling and define Product Backlog
User Stories – Business Processes – Capability Descriptions – Product Backlog
……
……
Conduct Working Model Sprints
Standalone functioning modules
……
Assemble and Deploy Working Models
Integrated operational Modules
Detailed Prioritized Business Capability Requirements
Col
labo
rativ
e C
ross
-Fun
ctio
nal T
eam
s
7/9/13 © The Mercator Group 34
Changing the Process
• Agile is a drastic transformation of the development process, and must be implemented in the context of an organizational change effort.
• A center of excellence should be established, and the required support mechanisms need to be put in place.
• Infrastructure support processes, such as software change management and software integration need to be iron clad.
• The effort should start with a manageable, but significant application.
• The Agile life cycle should be well documented, and all participants properly trained.
• Performance measures should be put in place that not only measure ROI, but measure the quality of the software released.
• A broader program promoting cross-function collaboration should be implemented.
7/9/13 © The Mercator Group 35
Recap and Closing
7/9/13 © The Mercator Group 36
Why Agile?
• Agile Scrum provides a mechanism for significantly reducing risk associated with software development, and significantly improving the quality of the delivered product: – The iterative nature of Agile, with interim deliverables of functionality,
provides natural check points for assessing progress, and reducing the risk so obvious in traditional waterfall methods
– The intense collaborative nature of Agile, along with the focus on the quality of what is built, versus a focus on timelines and resources, results in much improved software that is better aligned to the business needs.
• Large Waterfall efforts have a very high mortality rate – Too often, large enterprises trust the delivery of major software asset
investments to system integrators, only to find out, after most of the money is spent, they will not get what was agreed to, and will be faced with a savvy contractor making a case for them to spend a lot more money.
– This acquisition approach has inherent conflict of interest built into it. Adopting an acquisition strategy that enables Agile, and implementing Agile, significantly reduces the chances of the above scenario occurring.
7/9/13 © The Mercator Group 37
Why Not Agile?
• Moving to Agile is a significant cross-organizational effort: – A true implementation of Agile requires a significant change in
budgeting, contracting, and, most importantly, organizational software development process and culture.
– Agile changes governance, processes, roles, power structures, and skill requirements simultaneous across organizational and functional areas. It is hard to do, and requires a prescriptive organizational change effort.
• An organization may have in place long-term existing contractual relationships that preclude implementing Agile.
7/9/13 © The Mercator Group 38
Is there a hybrid approach?
In the long-term, Agile is clearly displacing traditional waterfall as the primary approach to software development. Modern software development environments provide incredible productivity increases, and improvements in the quality of what can be made possible with software. These environments can only be leveraged by implementing some form of Agile. In the short term, organizations have implemented four strategies for moving to Agile:
7/9/13 © The Mercator Group 39
Ten steps to standing up Agile
1. Allow and encourage vendors to implement Agile on their own, and incorporate the changes in how you manage and interact with their projects
2. Educate senior management across the organization on Agile
3. Have the CIO appoint a senior manager in the role of what is referred to as a Scrum Master (appoint a senior manager with responsibility to implement the program)
4. Carefully select a set of projects to pilot a first implementation of the complete Agile Scrum process
5. Just as carefully, select the participants for those projects
6. Stand-up a center of excellence to support the effort
7. Put in place the appropriate governance and software change processes required
8. Provide just in time training for the team
9. Implement the pilots and measure results
10. Put in place continuous process improvement
7/9/13 © The Mercator Group 40