Chaos, Consistency, Creativity:

A Journey Through Agile Auditability

Steve Nunziata, PMP, PMI-ACP, CSM, SAFe SPCAgile Austin Monthly Meeting, October 14th, 2014

About Steve…

http://www.linkedin.com/pub/steve-nunziata/5/519/484/

PMP, ACP, CSM, SAFe SPC

EDS, Nike, Adidas, USAA

Agile Trainer & Coach

New Jersey / Oregon

Bassist Extraordinaire

Alamo Agilistas / PMI

Background: My Story

Zero to Sixty (Days): Chaos to Consistency

So… Why Are We Here? Opportunity:

Educate internal auditors to evolve away from formal artifacts and accept Agile tenets of visibility and transparency to demonstrate adherence to defined Quality standards.

We will collaborate on an approach to define an Agile Risk & Control framework that can start you on your journey.

How Would You Like: A 50% - or more – reduction in project ‘paperwork’ to demonstrate adherence to compliance processes?

Waterfall Agile

59

30

Project Compliance

Arti facts

A framework for consistent application of Agile practices and ceremonies across a large – and growing – organization?

SAMPLE: YMMVReally?

Remember…Use the Force

Remove, you must, Stories from the Backlog,

That, within an Iteration, completed, will not be…

AgendaCh

aos Failings of

Today’s Risk Management Processes

Cons

iste

ncy Why Audit

Execution Models Need to Evolve Cr

eativ

ity Creating an Agile Auditable Framework

Managing Risk – How Important is it?

The primary goal of a business is to… stay in business.

It is therefore necessary to continually evaluate, monitor, and address threats to retain market share. Otherwise, what would happen?

Managing Risk – The Risk Management Process

Risk Identificatio

n

Risk Assessment

Risk Response Risk Review

Managing Risk – ISO 9001 Summary

Part 4 – The Company must establish, document, and maintain a Quality Management System (QMS)

Part 5 – Management commitment in evidence for the QMS

Part 6 – Necessary resources must be determined & provisioned

Part 7 – Plan & Develop processes for product realization. The processes must produce documents that can be (1) reviewed for acceptance; and (2) used as proof of conformance

Part 8 – All reports of non-conformances, both of the product or the process, shall be reported upon, analyzed and lead to corrective action

Managing Risk – Risk & Control Compliance Framework

Risk Controls

Control Tests

Reporting & Review

Operational Risks

Incomplete Requirements Ineffective or Incomplete

Software Solution Poor User Experience Poor Project Execution

Plan

Formal Requirements Baseline Process

Project Execution Schedule Review

Code Peer Reviews

Evidence of Formal Signoffs

Published Meeting Minutes

Documented Decisions / Logs

Formal results of Audit published for review; opportunities for improvements noted

Auditors

Are Risk Management Processes Inherently anti-Agile?

Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif

SDLC & Process Audit Execution Models: Challenges

While Agile adoption and evolution has continued unabated over the past several years, traditional process audits have largely been unable to keep pace. Why might this be?

SDLC & Process Audit Execution Models

Req’s Analysis Design Build Test Deploy

Systems Development Life Cycle – Linear View

SDLC & Process Audit Execution Models

Source: http://julianeverett.wordpress.com/

Red Dotted Line: Waterfall

Blue Dotted Line: Agile

RISK

TIMEProject Risk Profile – Agile & Waterfall

SDLC & Process Audit Execution Models

Daily

24 H

ours

Iteration

2-4 Weeks

Release

~3 Months

Closure

~9-12 Months

SDLC Execution – Waterfall, Incremental, & Agile

SDLC & Process Audit Execution Models

Process Audit vs. SDLC Execution Gap Analysis

Closure

~9-12 Months

Release

~3 Months

Iteration

2-4 Weeks

Daily

24 H

ours

SDLC & Process Audit Execution Models

DailyIteration

2-4 Weeks

Release

~3 Months

Closure

SDLC and Process Audit Execution: Optimal Quality State

5 Steps to Establishing an Agile Auditable Framework

Risk Validation

Inventory Agile Practices

Create Acceptable Parameters

Determine Method of Control

Establish Operational Parameters

1

2

3

4

5

5 Steps to Evolving an Agile Auditable Framework

Risk Validation

Review and Validate the current Risk & Control Framework, ensuring traceability from Risks to Controls to Control Tests.

Operational Risk: Risk Control: Control Test:

Failure to Manage Project Risks

Risk Management Process

Evidence of a Periodic Risk Review (Risk Log)

Issue Management Process

Formal, Complete Issues Log

1

5 Steps to Evolving an Agile Auditable Framework

Inventory Agile Practices

Inventory the Agile Practices supported by the organization. Scrum practices and ceremonies provide a good start.

Match the Agile ceremonies to the list of Risks in the current Risk & Control Framework. Can a Ceremony or Practice provide an acceptable substitute? How / Why?

2

5 Steps to Evolving an Agile Auditable Framework

Inventory Agile Practices

Introduce the Agile Practice as a Control. Could it work? Could it be effective? What would be the value of the current control set – should anything remain, or can they be dismissed?

Operational Risk: Risk Control: Control Test:

Failure to Manage Project Risks

Risk Management Process

Evidence of a Periodic Risk Review

Agile Daily Standup

2

5 Steps to Evolving an Agile Auditable Framework

Create Acceptable Parameters

Research Industry standard ‘best practices’ for the ceremonies or practices you plan on using as a Control (mitigation strategy) for the Risk. A great example is Version One’s The Agile Checklist

Create a matrix defining minimally acceptable behaviors, along with anti-patterns, and radiate the desired outcomes in a common area

3

5 Steps to Evolving an Agile Auditable Framework

Create Acceptable Parameters

Agile Ceremony: Daily Standup

Best Practice Acceptable Partial Unacceptable

Occurs 5 Days per Week

Occurs 4 Days per Week

Occurs 3 Days per Week

Occurs <3 Days per Week

3 Core Questions Addressed

3 Core Questions Addressed

<3 Core Questions Addressed

<3 Core Questions Addressed

…Your Organization?

…Your Organization?

…Your Organization?

….Your Organization?

3

5 Steps to Evolving an Agile Auditable Framework

Determine Method of Control

Does the new Control Test require someone observe an Agile Ceremony, or is there a consistent formal artifact from an Agile practice that can be viewed?

4

5 Steps to Evolving an Agile Auditable Framework

Establish Operational Parameters

Review the total number of Control Tests. How many require observation from an Auditor?

Establish the Audit cycle & reporting time (Weekly? Sprint Level? Release Level? Other..?)

Train and deploy Audit resources

Execute an Audit cycle… and report to Risk Owners

Learn… and continue to evolve!

5

5 Steps to Evolving… Creativity

Host a Retrospective Ceremony with some of the Agile teams to uncover: What may be challenging teams in conforming to minimal standards? What opportunities can they recommend to evolve to controls? Are the audits providing value in holding roles accountable for their deliverables? Finally – when minimal standards are easily achieved – it’s time to take the next steps in maturity, and shift the pattern.

5 Steps to Evolving - Going Beyond... Challenge: can you evolve traditional, formal artifacts into a

more Agile framework? How can you continuously improve?

Picture Source: http://agile101.wordpress.com/2009/07/27/agile-risk-management-assessing-risks-step-2-of-4/

Positive Outcomes Better alignment of Controls and Tests to the project execution model

Real time, actionable feedback & reporting to teams and Risk owners

Scalable for future methodologies & practices

Continual quality assessments; a project can have multiple reviews

Sets a benchmark for Agile maturity across an Organization

‘Humanizes’ the Audit (not ‘check the box’) – gives teams a voice

Experience – 50% reduction in Controls… while doubling Quality

Leading – NOT lagging – metric; address problems before they manifest

Opportunity for two-way communication and learnings

Challenges Optimal model is labor intensive Inherent subjectivity in assessments (‘Auditor Bias’) Potential for teams to feel ‘over controlled’ Oversight and administration of the process Communication and support for changes Determining boundaries of adherence vs. non-adherence, and appropriate remedies Ever-evolving process; can feel like an ‘arms race’

Common Questions Does this model Scale? How much time per week would this require? Isn’t this just the Scrum Master’s… or (insert role here) – job? Could we use Pair Programming as a Control? What is the future of Agile Quality Assurance?

Objectives Met?

Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif

Remember: Auditors are the Board of Health!

Questions?

Thank You!

Information Sources Malik Imran Ullah & Waqar Ali Zaidi, “Quality Assurance Activities in Agile – Philosophy to Practice”. Sep. 2009.

Larry Whittington, “ISO9001:2008 Requirements Summary in Plain English”. http://www.whittingtonassociates.com/

Tor Stalhane, Geir Kjetil Hanssen, “The application of ISO 9001 to Agile Software Development”. 2008.

Buck Kulkami, “Agile Projects: An Emerging Challenge for IT Auditors”.

Information Sources R. Gopinath, “Guideline: How to Audit and Agile Project?”

George Schlitz, “Is your Agile Audit and Compliance Process really Agile?”

Christelle Scharff, “Guiding Global Software Development Projects using Scrum and Agile with Quality Assurance”