satisfying auditors: plans and evidence in a regulated environment

MM PM Tutorial

10/13/2014 1:00:00 PM

"Satisfying Auditors: Plans and

Evidence in a Regulated Environment"

Presented by:

James Christie

Claro Testing

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

James Christie

Claro Testing James Christie is a testing consultant with thirty-one years of IT experience. Before moving into testing, James spent six years as an IT auditor, so he has experience on both sides of the fence. With experience in information security management, project management, business analysis, and development, he is particularly interested in links between testing, auditing, governance, and compliance. James spent fourteen years working for a large UK insurance company, then nine years with IBM working with large clients in the UK and Finland. A member of the Information Systems Audit and Control Association, James has been self-employed for the past eight years.

9/11/2014

1

Satisfying Auditors:Plans and Evidence in a Regulated Environment

James Christie

How I ended up in software testing via auditing.

Why Alice in Wonderland was relevant to my attempts to understand what goes on in big companies.

An introduction – to me and the tutorial 1a

9/11/2014

2

“The chief difficulty Alice found at first was in managing her flamingo”

An introduction – to me and the tutorial 1a

“When I use a word,”

Humpty Dumpty said in rather a scornful tone,

“it means just what I choose it to mean —neither more nor less”.

1bNothing seemed to make sense

9/11/2014

3

1cSomething like sanity

Something like sanity 1c

9/11/2014

4

Y2K – a testing time 1d

Image courtesy Stuart Miles & FreeDigitalPhotos.net

Information security management – the IBM way 1d

9/11/2014

5

Some internal audit departments have an image problem

Ambiguous? I’m not sure exactly what this means,

but it’s not good.

1d

Back to testing again 1d

9/11/2014

6

And out… 1d

External & internal auditors 2a

9/11/2014

7

“External auditors are watchdogs not bloodhounds”2b

Providing an opinion to the shareholders about whether the accounts are true and fair.

Images courtesy Artur84/FreeDigitalPhotos.net

Providing an opinion to the shareholders about whether the accounts are true and fair.

External auditor independence

Such a big problem it’s more than just a problem.

2b

9/11/2014

8

“Commercial suicide”, alleged quote from current chair of UK Financial Conduct Authority.

John Griffith-Jones

External auditor independence

Challenging client management?

2b

Images courtesy Artur84/FreeDigitalPhotos.net

Problem #1 - up or out

Images courtesy Stuart Miles, Renjith Krishnanur84/FreeDigitalPhotos.net

”It would seem that about 50% of newly qualified ACAs ”It would seem that about 50% of newly qualified ACAs ”It would seem that about 50% of newly qualified ACAs ”It would seem that about 50% of newly qualified ACAs do not have enough practical experience to continue do not have enough practical experience to continue do not have enough practical experience to continue do not have enough practical experience to continue their careers as their careers as their careers as their careers as auditors”auditors”auditors”auditors”

Michael Michael Michael Michael IzzaIzzaIzzaIzza, ICAEW Chief Executive, 2009, ICAEW Chief Executive, 2009, ICAEW Chief Executive, 2009, ICAEW Chief Executive, 2009

2b

9/11/2014

9

””””Most internal auditors would join me in assessing the Most internal auditors would join me in assessing the Most internal auditors would join me in assessing the Most internal auditors would join me in assessing the external audit partners and senior managers as arrogant external audit partners and senior managers as arrogant external audit partners and senior managers as arrogant external audit partners and senior managers as arrogant beyond their competencebeyond their competencebeyond their competencebeyond their competence””””

Norman Norman Norman Norman Marks, 2010 (Marks, 2010 (Marks, 2010 (Marks, 2010 (chief audit exec at major global corporations for 20+ years))))

Problem #2 – quality of people2b

Problem #3 - sampling

Auditors can’t check Auditors can’t check Auditors can’t check Auditors can’t check all the all the all the all the figures. That figures. That figures. That figures. That would make audits would make audits would make audits would make audits far too expensive.far too expensive.far too expensive.far too expensive.

But they can’t just But they can’t just But they can’t just But they can’t just take figures on trust.take figures on trust.take figures on trust.take figures on trust.

So they So they So they So they sample.sample.sample.sample.

How much do they How much do they How much do they How much do they samplesamplesamplesample????

How do they choose How do they choose How do they choose How do they choose the sample?the sample?the sample?the sample?

Cartoons courtesy Scott Adams

2b

9/11/2014

10

Add up everything that moves through the books; Add up everything that moves through the books; Add up everything that moves through the books; Add up everything that moves through the books; all revenue plus all costs, to get turnover=t. Let’s all revenue plus all costs, to get turnover=t. Let’s all revenue plus all costs, to get turnover=t. Let’s all revenue plus all costs, to get turnover=t. Let’s say t=£25 million.say t=£25 million.say t=£25 million.say t=£25 million.

Problem #3 – sampling on the Internal Controls Basis

Apply the accounts total & control score to the Apply the accounts total & control score to the Apply the accounts total & control score to the Apply the accounts total & control score to the sampling formula to get the sampling interval, sampling formula to get the sampling interval, sampling formula to get the sampling interval, sampling formula to get the sampling interval, e.g. interval = t/(100*s). So interval = £50,000.e.g. interval = t/(100*s). So interval = £50,000.e.g. interval = t/(100*s). So interval = £50,000.e.g. interval = t/(100*s). So interval = £50,000.

Assess the internal controls and assign a score, s, Assess the internal controls and assign a score, s, Assess the internal controls and assign a score, s, Assess the internal controls and assign a score, s, egegegeg 1= atrocious & 5=excellent. Assume s=5.1= atrocious & 5=excellent. Assume s=5.1= atrocious & 5=excellent. Assume s=5.1= atrocious & 5=excellent. Assume s=5.

2b

Problem #3 – sampling on the Internal Controls Basis

Pull out a bank note. Take the last three digits. Pull out a bank note. Take the last three digits. Pull out a bank note. Take the last three digits. Pull out a bank note. Take the last three digits. Express that as a decimal fraction of 1,000.Express that as a decimal fraction of 1,000.Express that as a decimal fraction of 1,000.Express that as a decimal fraction of 1,000.Let’s say 0.851.Let’s say 0.851.Let’s say 0.851.Let’s say 0.851.

Sample your way through the accounts, examining Sample your way through the accounts, examining Sample your way through the accounts, examining Sample your way through the accounts, examining every transaction you hit at the sample interval, every transaction you hit at the sample interval, every transaction you hit at the sample interval, every transaction you hit at the sample interval, hitting every single transaction of £50,000+.hitting every single transaction of £50,000+.hitting every single transaction of £50,000+.hitting every single transaction of £50,000+.

Apply the fraction to the interval to get the Apply the fraction to the interval to get the Apply the fraction to the interval to get the Apply the fraction to the interval to get the starting point, i.e. £42,550.starting point, i.e. £42,550.starting point, i.e. £42,550.starting point, i.e. £42,550.

2b

9/11/2014

11

Problem #3 – sampling on the Internal Controls Basis; gaming the system

(aka cheating)

Why might you want to manipulate the method, Why might you want to manipulate the method, Why might you want to manipulate the method, Why might you want to manipulate the method, and how would you do it?and how would you do it?and how would you do it?and how would you do it?

You can rig the internal controls score to get the You can rig the internal controls score to get the You can rig the internal controls score to get the You can rig the internal controls score to get the result you want. The higher the score, the higher result you want. The higher the score, the higher result you want. The higher the score, the higher result you want. The higher the score, the higher the sampling interval, and the less work the the sampling interval, and the less work the the sampling interval, and the less work the the sampling interval, and the less work the auditors have to do.auditors have to do.auditors have to do.auditors have to do.

You can rig the formula, but that’s obvious and You can rig the formula, but that’s obvious and You can rig the formula, but that’s obvious and You can rig the formula, but that’s obvious and you’d have to justify it.you’d have to justify it.you’d have to justify it.you’d have to justify it.

REMEMBER THE FEE IS SET BEFORE THE INTERNAL REMEMBER THE FEE IS SET BEFORE THE INTERNAL REMEMBER THE FEE IS SET BEFORE THE INTERNAL REMEMBER THE FEE IS SET BEFORE THE INTERNAL CONTROL CONTROL CONTROL CONTROL SCORE IS CALCULATED.SCORE IS CALCULATED.SCORE IS CALCULATED.SCORE IS CALCULATED.

2b

Internal Audita totally different perspective

“Internal auditing is an “Internal auditing is an “Internal auditing is an “Internal auditing is an independentindependentindependentindependent, objective , objective , objective , objective assurance and consulting activity designed to assurance and consulting activity designed to assurance and consulting activity designed to assurance and consulting activity designed to add value and improve add value and improve add value and improve add value and improve an organization's an organization's an organization's an organization's operations. operations. operations. operations.

It helps an organization accomplish its It helps an organization accomplish its It helps an organization accomplish its It helps an organization accomplish its objectives by bringing a systematic, disciplined objectives by bringing a systematic, disciplined objectives by bringing a systematic, disciplined objectives by bringing a systematic, disciplined approach approach approach approach to evaluate and improve the to evaluate and improve the to evaluate and improve the to evaluate and improve the effectiveness of risk management, control, and effectiveness of risk management, control, and effectiveness of risk management, control, and effectiveness of risk management, control, and governance processesgovernance processesgovernance processesgovernance processes.”.”.”.”

Global Institute Global Institute Global Institute Global Institute of Internal of Internal of Internal of Internal AuditorsAuditorsAuditorsAuditors

2c

9/11/2014

12

Internal Audit

The people are different

Image courtesy cooldesign & FreeDigitalPhotos.net

2c

Top six qualities internal auditors need

IIA’s 2013 Global Pulse of the Profession survey

Critical thinking

Communication skills

Risk-management

IT knowledge

Data mining/analytics

Accounting

2c

9/11/2014

13

1 - Critical thinking

2 - Communication skills

3 - Risk-management

4 - IT knowledge

5 - Data mining/analytics (frauds! ☺)

6 – Accountancy knowledge

Communications

Data mining/analytics

Risk-management assurance

IT knowledge

Accountancy knowledge

Top six qualities internal auditors need

IIA’s 2013 Global Pulse of the Profession survey

2c

Internal auditors know more

Deeper business knowledge

Greater tacit knowledge

Greater nous (streetwise)

More mature & stronger characters?

Image courtesy Krormrathog & FreeDigitalPhotos.net

2c

9/11/2014

14

Are internal auditors stronger?

You can’t bully good internal auditors.

(If you can bully them then they don’t last long).

2c

The internal audit hothouse

Internal audit is used as a training ground for high quality staff.

There is a potential downside to staff rotation. Where do they go next?

Image courtesy Chanpipat & FreeDigitalPhotos.net

2c

9/11/2014

15

Risk and the financial crash

Risk is a tricky concept and auditors didn’t handle it well.

Image courtesy cooldesign & FreeDigitalPhotos.net

3

“...the chance, high or low, of somebody being harmed by the hazard, and how serious the harm could be”

(UK Health & Safety Executive)

Image courtesy jscreationzs & FreeDigitalPhotos.net

What is risk anyway?

“the effect of uncertainty on objectives”

(ISO 31000)

“a set of circumstances that hinder the achievement of objectives”

(David Griffiths)

3a

9/11/2014

16

UK HSE risk matrix

What is risk anyway?3a

Enrico Fermi – the brilliant nuclear physicist who worked on the project to develop the atomic bomb.


1939. The probability that nuclear fission could be controlled for power or weapons?

10%

1945. The probability that the atomic bomb would set the atmosphere on fire and wipe out life on earth?

10%

1950. The probability that humans would develop the technology to travel faster than the speed of light by 1960? 10%

3a

9/11/2014

17

Tim O’Riordan & Patrick Cox, 2001.Science, Risk, Uncertainty & Precaution. University of Cambridge.

3a


Simple, understandable and totally misleading?

Complex, accurate(?) and totally uninformative?

Risk – the big dilemma?

or

Images courtesy Luigi Diamanti, Mr Brightman & FreeDigitalPhotos.net

3a

9/11/2014

18

Rick Buy – Chief Risk Officer.His stated aim was to ”condense ”condense ”condense ”condense all the risks of all the risks of all the risks of all the risks of

the corporation into a single metricthe corporation into a single metricthe corporation into a single metricthe corporation into a single metric”.”.”.”.

Risk – the big dilemma?3b

Risk – and how we lost sight of it

Image courtesy of Just2shutter / FreeDigitalPhotos.net

“With half a decade’s hindsight, it is clear the crisis had multiple causes. The most obvious is the financiers themselves –especially the irrationally exuberant Anglo-Saxon sort, who claimed to have found a way to banish risk when in fact they had simply lost track of it.”

The Economist

Image courtesy pakorn / FreeDigitalPhotos.net

3c

9/11/2014

19

Risk – and how we lost sight of it

Image courtesy of Just2shutter / FreeDigitalPhotos.net

“The weaknesses of group risk in HBOS were a matter of design, not accident.”

Parliamentary Commission on Banking Standards; “An Accident Waiting To Happen: The Failure of HBOS”

3c


* Fixed probability* Time period* Amount at risk

Eg, 95% probability that the maximum loss in a week will not exceed £1m.

Definitely not 5% probability of losing just £1m in a week.

Value at Risk - or losing sight of risk


3c

9/11/2014

20

Value at Risk – ignoring Black Swans

Decision makers and auditors lost sight of what VaR actually means.

Above the “VaR break” all bets are off – we’re into Black Swan territory.

And that’s pretty much what happened.

3c

Big 4 audit fees for 2007

““““…fees are now coming before independence, objectivity (and sometimes, even competence) in important parts of the accounting profession.””””

Paul Moore (ex partner KPMG, ex Head of Group Regulatory Risk, HBOS - 2013)

3d

9/11/2014

21

Big 6 foul ups in US

US PCAOB Audit Failures 2012 (2011)

Grant Thornton 65% (43%)

BDO 55% (39%)

Ernst & Young 48% (36%)

PWC 39% (41%)

KPMG 34% (23%)

Deloitte 25% (42%)

(% of audits inspected deemed to be “audit failures” by regulator)

3d


Has external audit had its day?

“External audit is now largely out-dated. The binary nature of the opinion renders it useless.”Richard Anderson chairman of the Institute of Risk Management, 2011

“With or without new rules, the main worry for auditors may be that people wonder whether their reports are worth a bean.”

The Economist, April 2014

3e

9/11/2014

22

Has external audit had its day?

“The fact that the audit process failed to highlight developing problems in the banking sector does cause us to question exactly how useful audit currently is.”

House of Commons Treasury Committee “Banking Crisis”, 2009

“The problem is that there's not a lot of evidence that (external) auditors are very good at assessing risk.”

Charles Cullinan, Bryant College, USA

3e

Is internal audit better placed?

PeoplePeoplePeoplePeople

TimeTimeTimeTime

Business knowledgeBusiness knowledgeBusiness knowledgeBusiness knowledge

IndependenceIndependenceIndependenceIndependence

Business modelBusiness modelBusiness modelBusiness model

None of these are guaranteed advantagesNone of these are guaranteed advantagesNone of these are guaranteed advantagesNone of these are guaranteed advantages

Greater scrutiny of corporate culture?Greater scrutiny of corporate culture?Greater scrutiny of corporate culture?Greater scrutiny of corporate culture?

3f

9/11/2014

23

Evidence and Opinion

How do we know anything?What matters? Who cares?

“To know that we know what we know, and to know that we do not know what we do not know, that is true knowledge.” Copernicus

4a4a

Ontology

What is the nature of reality?

What isisisis a windmill?

What is real?

4a

9/11/2014

24

How can Don Quijote know anything?

Epistemology

What can he know about windmills?

From Sancho Panza?

From his senses?

From books!

How can he know about windmills?

Trigger’s Broom 4a

9/11/2014

25

A positivist worldview?

Have we treated testing, and auditing, as if they are like scientific experiments where we know and control all the variables?

Have we been too keen to assume the world we are investigating is a neater and more ordered place than it really is?

4a

Is an interpretivistworldview more helpful?

A dangerous extreme for testers?Certainly for auditors!

There is no single, fixed reality. Everything is a social construct so we have to understand what we are looking at rather than criticising or condemning.

4a

9/11/2014

26

A balanced approach?(just doing the best we can)

We might not know things with certainty, but we can make statements based on evidence & keep refining our opinion.

Positivists might think that certainty is out there and we can know it.

Interpretivists might not say anything useful; they’re all features, not bugs!

4a

They are They are They are They are overoveroverover----simplifiers. They simplifiers. They simplifiers. They simplifiers. They take a complicated issue and take a complicated issue and take a complicated issue and take a complicated issue and deliver a simplistic and deliver a simplistic and deliver a simplistic and deliver a simplistic and superficially plausible answer. superficially plausible answer. superficially plausible answer. superficially plausible answer. They offer clear, actionable advice They offer clear, actionable advice They offer clear, actionable advice They offer clear, actionable advice but…but…but…but…

Thanks to Thanks to Thanks to Thanks to StianStianStianStian Westlake for this.Westlake for this.Westlake for this.Westlake for this.

Berks

4b

Don’t be a berk or a wanker

9/11/2014

27

A wanker (that’s me)

They want to be robust They want to be robust They want to be robust They want to be robust and comprehensive, and and comprehensive, and and comprehensive, and and comprehensive, and forget about clarity & forget about clarity & forget about clarity & forget about clarity & brevity.brevity.brevity.brevity.

4b

Don’t be a berk or a wanker

Rikard Edgren

“Reality isn’t binary… we “Reality isn’t binary… we “Reality isn’t binary… we “Reality isn’t binary… we don’t know everything in don’t know everything in don’t know everything in don’t know everything in advance.advance.advance.advance.

We should observe the We should observe the We should observe the We should observe the software without a software without a software without a software without a hypothesis to nullify.”hypothesis to nullify.”hypothesis to nullify.”hypothesis to nullify.”

Image courtesy digitalart binary / FreeDigitalPhotos.net

4c

The binary trap

9/11/2014

28

The questions we can answer yes/no with most certainty are probably those that don't matter.

The danger is that we focus on them because the light is better there.

The binary trap4c

It’s not meant to be easy, it’s meant to be valuable.

Test scripts are not testing.

Checklists are not auditing.

4c

The binary trap

9/11/2014

29

Relying on scripts and checklists assumes that the information we want is under the streetlight.

It assumes that we can know in advance what matters, what we need to look for.

It assumes that the important questions can be answered with a “yes” or “no”.

The relevance to testers4c

If we focus only on what was specified we will not see what was needed but neither specified nor built (5).

And we won’t see what was not specified or needed, but was built (6).

Thanks to James Lyndsay, Iain McCowatt, James Bach & Michael Bolton.

and auditors want to know too

Either could be damaging.

The relevance to testers4c

9/11/2014

30

Good auditors learn by listening.

Bad auditors don’t listen. Their checklist tells them the “right answers”.

UK & US regulators are pushing auditors away from binary opinions. EU???

An auditor – “one who hears, a listener4c

Risk Based AuditingWhat is it? How do they do it?

Image courtesy David Castillo Dominici/FreeDigitalPhotos.net

We don’t understand risk well.

We don’t understand auditing.

So do we really know what Risk Based Auditing means?

5

9/11/2014

31

Risk Based Auditing – what is it?

2- RBA provides assurance that risks are being managed effectively.

3- RBA focuses effort on the areas most likely to suffer problems.

1- RBA identifies risks so that management can eliminate them.

4- RBA focuses on the risks that pose the greatest threat to company objectives.

5a

1- RBA identifies risks so that management can eliminate them.

Risk Based Auditing – what is it?5a

3- RBA focuses effort on the areas most likely to suffer problems.

2- RBA provides assurance that risks are being managed effectively.

4- RBA focuses on the risks that pose the greatest threat to company objectives.

9/11/2014

32

Controls based auditor; “how can I be sure no-one will steal bricks while the house is being built?”

Script driven tester; “what tests should I write for using these bricks to build a nice house?”

Risk based auditor; “could someone hit the cashier over the head with a brick and steal the payroll? Is that significant?”

The exploratory tester?

Image courtesy Piyachok Thawornmat/FreeDigitalPhotos.net

Risk Based Auditing – what is it?5a

There’s compliance, and then there’s compliance

Big difference between the cops and mere processes!

5a

9/11/2014

33

Reasonable assurance about risksnot absolute

5a

Appropriate…

sufficient…

reasonable…

material

Auditors are looking for reasonable assurance, not absolute assurance.

Risks that matter

“Audit priorities (should) align with those of the board and executive management. Risks that keep our stakeholders up at night also should be of concern to us.”Richard Chambers, CEO & President of Institute of Internal Auditors

“The problem is that there's not a lot of evidence that (external) auditors are very good at assessing risk.” (reminder!)Charles Cullinan, Bryant College

Image courtesy digitalart /FreeDigitalPhotos.net

5a

9/11/2014

34

Attitude of the Institute of Internal Auditors

Compliance auditing; “tipping out the pieces of a jigsaw puzzle on to the Audit Committee table rather than turning those pieces into a picture.”

Sarah Blackburn, ex President of IIA UK

Image courtesy Stuart Miles/FreeDigitalPhotos.net

5a


“In a risk-based approach to security, compliance is provided by security –security is not necessarily provided by compliance.”John Wheeler, Gartner Inc

Moving this way?

5a

Attitude of the Institute of Internal Auditors

““““Many organizations look at compliance as a set of check boxes… but compliance is not the goal, it’s a result.”Mike Rothman, Security Incite

9/11/2014

35

Risk Based Auditing - doing it


There are no right answers (probably).

The checklist is not the audit. It’s just a tool.

Auditors who rely on checklists are unprofessional compliance monkeys. It demeans and deskills the profession.

5b

Risk Based Auditing - planning it

A development that is under way

RequirementsRequirementsRequirementsRequirements

Test strategyTest strategyTest strategyTest strategy Test resultsTest resultsTest resultsTest resultsDetailed test Detailed test Detailed test Detailed test plan/scriptsplan/scriptsplan/scriptsplan/scripts

Design Design Design Design documentsdocumentsdocumentsdocuments

Development Development Development Development standardsstandardsstandardsstandards

Problem recordsProblem recordsProblem recordsProblem records

Other?Other?Other?Other?

Handover packHandover packHandover packHandover packProject planProject planProject planProject plan

Change recordsChange recordsChange recordsChange records

A development that went live two months ago

A live system that's been running for four years

5b

9/11/2014

36

A development that is under way









5b


A development that went live two months ago









5b


9/11/2014

37

A live system that's been running for four years









5b


Conway’s Law – a personal hobby horse.

“Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations”

Melvin Conway

Image courtesy jscreationzs/FreeDigitalPhotos.net

5b


9/11/2014

38

The communications and organisational structure are a useful guide to where the worst flaws will be in the project and system.

My auditor’s corollary (or heuristic) to Conway’s Law.

Risk Based Auditing - planning it5b

Conway’s Law – a personal hobby horse.

Image courtesy jscreationzs/FreeDigitalPhotos.net

Risk Based Auditing

IDEF0 & decomposing an application

5b

9/11/2014

39

Risk Based Auditing

Exploratory testing?

Breaking the application


5b

Don’t tell me, show me (auditor’s mantra)

“Don’t tell me the moon is shining, show me the glint of light on broken

glass”

Anton Chekhov

5b

9/11/2014

40

And why does it matter?

Different parts of the world have different models –with different outcomes.

6

What is Governance?

κυβερνάω [kubernáo] – to steer?

6a

What is Governance?

9/11/2014

41

Corporate governance is the board’s job

� Should not involve day to day operational management by full-time executives

� Supervising management & reporting to shareholders

� Setting the strategic aims & values

� Leadership to put them into effect

� Values based on principles of transparency, accountability, probity and long term sustainability

Paraphrased from the UK Financial Reporting Council’s “UK Corporate Governance Code”

6a

IT governance is the responsibility of corporate management

� Evaluates stakeholders’ needs and sets objectives to satisfy them

� Directs and sets priorities

� Monitors performance

Paraphrased from ISACA’s definition

6a

9/11/2014

42

IT management

� Plans

� Builds

� Runs

� Monitors

� All in alignment with the strategic direction set by the governance body

Paraphrased from ISACA’s definition

6a

Why governance is a good thing

If we get governance wrong then we suffer

Images courtesy Scott Adams & Stuart Miles/FreeDigitalPhotos.net

6b

9/11/2014

43

Governance - Risk Management

Three Lines of Defence

Functions that own and manage risks; operational management (the front line)

Functions that oversee risks; risk management and compliance function

Functions that provide independent assurance; internal audit

IIA strongly recommended guidance

6c

Governance – comply or explain

“Comply or explain” is the UK approach.

Also Germany and Netherlands.

UK Corporate Governance Code, Deutscher Corporate Governance Codex & Code Tabaksblat

US style

Comply or else!

(my experience)

Images courtesy Stuart Miles & FreeDigitalPhotos.net

6d

9/11/2014

44

Governance – different countries, different models, different outcomes

etc

6e

ISACAInformation Systems Audit & Control

Association

ISACA and COBIT 5Why they matter

77

9/11/2014

45

ISO/IEC 38500:2008 Model for Corporate Governance of IT

7a

COBIT 5 interpretation of IT governance

Control Objectives for Information and Related Technology

7b

9/11/2014

46

COBIT 5 interpretation of IT governance7c

A Quality Management System with quality standards.

AP011 Manage QualityAP011 Manage QualityAP011 Manage QualityAP011 Manage Quality

“Best practices” to be used as a “reference when improving and tailoring”.

Based on industry “goodgoodgoodgood practices”.

No mention of specific standards (or even the need to go looking for standards to adapt).

ISACA expect the following

COBIT 5 interpretation of IT governance7c

9/11/2014

47

“Validate all requirements through approaches such as peer review, model validation or operational prototyping”.

BA102 Manage BA102 Manage BA102 Manage BA102 Manage Requirements DefinitionRequirements DefinitionRequirements DefinitionRequirements Definition

“If appropriate, implement the selected option as a pilot to determine possible improvements”.

“Review the alternative solutions… and select the most appropriate one based on feasibility… risk and cost.”


7c


“… using agreed-on and appropriate phased or rapid agile development techniques”.

BA103 Manage Solutions BA103 Manage Solutions BA103 Manage Solutions BA103 Manage Solutions Identification & BuildIdentification & BuildIdentification & BuildIdentification & Build


“Proactively evaluate for design weaknesses (e.g., inconsistencies, lack of clarity, potential flaws) throughout the life cycle”.

BA103.02 Design detailed BA103.02 Design detailed BA103.02 Design detailed BA103.02 Design detailed solution componentssolution componentssolution componentssolution components

7c


9/11/2014

48

“Develop, resource and execute a QA plan aligned with the QMS to obtain the quality specified in the requirements definition and the enterprise’s quality policies and procedures.”

BA103.06 Perform quality BA103.06 Perform quality BA103.06 Perform quality BA103.06 Perform quality assuranceassuranceassuranceassurance



7c


“1. Define a QA plan & practices including, e.g., specification of quality criteria, validation and verification processes, definition of how quality will be reviewed, necessary qualifications of quality reviewers, and roles and responsibilities for the achievement of quality.”




7c


9/11/2014

49

“2. Frequently monitor the solution quality based on project requirements, enterprise policies, adherence to development methodologies, quality management procedures and acceptance criteria.”




7c


“3. Employ code inspection, test-drivendevelopment practices, automated testing, continuous integration, walk-throughs and testing of applications as appropriate.”




7c


9/11/2014

50

“Establish a test plan and required environments to test the individual and integrated solution components, including the business processes and supporting services, applications and infrastructure.”



BAI03.07 Prepare for BAI03.07 Prepare for BAI03.07 Prepare for BAI03.07 Prepare for solution testingsolution testingsolution testingsolution testing

7c


“Execute testing continually during development.”



BAI03.08 Execute BAI03.08 Execute BAI03.08 Execute BAI03.08 Execute solution testingsolution testingsolution testingsolution testing

Not “keep busy writing scripts till the testing phase”.

7c


9/11/2014

51

“1. Undertake testing of solutions and their components in accordance with the testing plan. Include testers independent from the solution team…”




7c


“2. Use clearly defined test instructions, as defined in the test plan, and consider the appropriate balance between automated scripted tests and interactive user testing.”




7c


9/11/2014

52

“3. Undertake all tests in accordance with the test plan and practices including the integration of business processes & IT solution components and of non-functional requirements (e.g., security, interoperability, usability).”




7c


“4. Identify, log and classify (e.g., minor, significant and mission-critical) errors during testing... Ensure that an audit trail of test results is maintained.”




7c


9/11/2014

53

“5. Record testing outcomes and communicate results of testing to stakeholders in accordance with the test plan.”




7c


“2. Ensure that the test plan reflects an assessment of risk from the project.”

BA107.03 Plan acceptance BA107.03 Plan acceptance BA107.03 Plan acceptance BA107.03 Plan acceptance teststeststeststests

BA107 Manage Change BA107 Manage Change BA107 Manage Change BA107 Manage Change Acceptance and TransitioningAcceptance and TransitioningAcceptance and TransitioningAcceptance and Transitioning

Not in BA103 surprisingly.


7c


9/11/2014

54

“3. Ensure that the test plan addresses the potential need for internal or external accreditation of outcomes of the test process (e.g., financial regulatory requirements).”




7c


“5. Ensure that the test plan identifies testing phases appropriate to the operational requirements and environment.”




7c


9/11/2014

55

“6. Confirm that the test plan considers test preparation … training requirements, … test environment, planning/performing/documenting/retaining test cases, error and problem handling, correction and escalation, and formal approval.”




7c


“6. Consider using clearly defined test instructions (scripts) to implement the tests.”

BA107.05 Perform BA107.05 Perform BA107.05 Perform BA107.05 Perform acceptance testsacceptance testsacceptance testsacceptance tests

That’s the end of testing in COBIT 5



7c


9/11/2014

56

COBIT 5 – big lessons for testers

No insistence on “best practice”

Countless references to ISO standards for;- Risk management- Security- Release management- Configuration management- Service level management- Incident management- Problem management- Business continuity - etc No mention of testing standards

No insistence on detailed scripts or test cases.

None at all!

7d

Institute of Internal Auditors

IIA standards - good news(seriously!)

8

9/11/2014

57

The Snowflake Theory of IT Audit

“Every IT environment is unique and represents a unique set of risks. The differences make it increasingly difficult to take a generic or checklist approach to auditing.”

Institute of Internal AuditorsGlobal Technology Audit Guide, Management of IT Audit, 1st edition, 2006

8a

IIA IT Audit Management Standard

“Frameworks and StandardsFrameworks and StandardsFrameworks and StandardsFrameworks and StandardsOne challenge auditors face when executing IT audit work is knowing what to audit against. Most organizations have not fully developed IT control baselines for all applications and technologies. The rapid evolution of technology could likely render any baselines useless after a short period of time.”

Institute of Internal AuditorsGlobal Technology Audit Guide, Management of IT Audit, 2nd edition, 2013

Image courtesy Salvatore Vuono & FreeDigitalPhotos.net

8b

9/11/2014

58

ISO standards are not mentioned except in an appendix “… for consideration”.

COBIT 5 is a recommended source of “control objectives” against which auditors can work. It offers “robust and generally accepted IT-specific control objectives… that helps management to conceptualize an approach for measuring and managing IT risk”.

Institute of Internal AuditorsGlobal Technology Audit Guide, Management of IT Audit, 2nd edition, 2013

IIA IT Audit Management Standard8b

IIA Auditing IT Projects Standard

A basic primer in software development (not a criticism – humility is not a bad thing).

Every organisation uses a different mix of methods, standards & tools. Auditors must understand these. They’re the ones that matter.

Institute of Internal AuditorsGlobal Technology Audit Guide, Auditing IT Projects, 2009

8c

9/11/2014

59

Mentions ISO project management standards, but not testing standards.

Favourably disposed towards Agile (one of the top ten factors for project success).


Importance of COBIT 5 is stressed – though the IIA does think it’s mainly about project management.

8c

IIA Auditing IT Projects Standard

“Internal auditors should not expect organizations to fully implement PMBOK, PRINCE2, COBIT, or any other large set of best practices. Rather, they should expect to see that these practices have been customized and integrated into the organization’s project management methodology.”


IIA Auditing IT Projects Standard8c

9/11/2014

60

Sarbanes Oxley

Does Sarbox deserve its scary reputation?

Yes, but…No, but…

9

Is Sarbanes Oxley scary?

Yes, especially section 404. That’s the requirement that management and the external auditors must report on internal control over financial reporting.

It’s a lot of work and it scares people who can make life difficult.

Image courtesy Simon Howden & FreeDigitalPhotos.net

But, it’s only for US companies, but… but…

9a

9/11/2014

61

No, so long as you don’t have Wally in charge of compliance.

Comply with COBIT 5 and Sarbox need not be a problem for testers.

That’s one of the reasons COBIT 5 is so important.

Cartoon courtesy Scott Adams

Is Sarbanes Oxley scary?9b

“Documentation is never required ‘for the auditors’.

If it is required it is because it is needed to manage the project, or it is a requirement of the project that has to be justified like any other requirement.”

James Christie

“Do standards keep testers in the kindergarten?”

Testing Experience, Dec 2009http://clarotesting.com/page20.htmhttp://clarotesting.com/page20.htmhttp://clarotesting.com/page20.htmhttp://clarotesting.com/page20.htm

Image courtesy Simon Howden & FreeDigitalPhotos.net

Is Sarbanes Oxley scary?9b

9/11/2014

62

US Food & Drugs Administration

What does the FDA expect?

10


Strong Credible

Images courtesy Simon Howden, Stuart Miles & FreeDigitalPhotos.net

10a


9/11/2014

63


“Test procedures, test data, and test results should be documented in a manner permitting objective pass/fail decisions to be reached.”


General Principles of Software Validation, FDA 2002

General Principles of Software Validation, FDA 2002

10b



10c


“The FDA is open to agile processes and realizes that the current approach to software validation is not working”

Griffin Jones CAST 2011

9/11/2014

64

AAMI TIR45:2012

“Guidance on the use of AGILE practices in the development of medical device software”


10c


“Agile can be adapted to the unique needs of medical device software… … and (can satisfy) regulatory requirements.”

AAMI TIR45:2012

Shows how Agile maps onto IEC 62304 (the standard specifying lifecycle requirements for developing medical software).


10c


9/11/2014

65

“The exploratory stage ofclinical device development is intended to allow for any iterative improvement of the design of the device, advance the understanding of how the device works and its safety, and to set the stage for the pivotal study.”

Image courtesy digitalart & FreeDigitalPhotos.net

FDA draft guidance 2011

http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm265553.htm

10d



Image courtesy Master & FreeDigitalPhotos.net


10e


9/11/2014

66


� Clear

� Objective (not requiring interpretation)

� Authentic

� Demonstrable integrity

� Readable & available

Evidence that will stand up in court


10e


Check out Griffin Jones’ work. See his talk on YouTube. http://www.youtube.com/watch?v=i8he7Rejn5s


� Attributable and not repudiable

� Full record & audit trail for changes

� Contemporary




10e


Seriously consider filming testing.

9/11/2014

67





10e


The evidence has to be sufficient (quality and quantity) so that 3rd

parties will have to come to the same conclusion if they review it, without interpretation by the testers.


10f


“the more energy put in to preparation, the less likely direct observations are captured”

Griffin Jones on Twitter

9/11/2014

68


10f


Evidence of planning is emphatically not evidence of what was done.

Detailed test script documentation is not evidence of test execution.

Is a beautifully constructed project plan evidence that the project finished on time?


10f


Image courtesy digitalart & FreeDigitalPhotos.net

Get help

9/11/2014

69

Test Strategy & PlanningWhat does a good auditor expect?

Images courtesy Stuart Miles & Master/FreeDigitalPhotos.net

StrategyStrategyStrategyStrategy not form filling

Relevance, not boiler-plate

11a

Test Strategy & Planning

Images courtesy Stuart Miles & Master/FreeDigitalPhotos.net

11a

Thoughtfulness, not massive documentation

Honesty, not spurious confidence

What does a good auditor expect?

9/11/2014

70

The strategy is not the process.

The strategy isn’t part of the plan, it shapes the plan.

We’re hopeless at strategy

Cartoon courtesy Scott Adams

Test Strategy & Planning 11a

My experience - we randomly mix up processes, strategy & planning.

James Bach talking like an auditor sensation!Test Strategy & Planning 11a

9/11/2014

71


Brainless optimism. �

Problems are not removed with a stroke of the pen.

Problems do not disappear if they are ignored.

Budding auditoror tester?


9/11/2014

72

Kopimism - “the act of copying is sacred”.

Copy/pasting is not cool. It’s evidence of a lack of thought.

Writing a strategy is not a matter of fleshing out a template, or recycling an old strategy.

The Kopimism HeresyTest Strategy & Planning 11a

“Strategies” running to 50+ pages.

“Assumptions” & “risks” that are just wishes that bad things won’t happen (ifififif they’re even stated).

Failure to learn from experience.

Go live dates announced before work is sized or staff secured.

Successive draft versions of project plans that get more optimistic without obvious plausible reasons.

Images courtesy digitalart/FreeDigitalPhotos.net

11bTest Strategy & PlanningMore warning signs (a personal list)

9/11/2014

73

Requirements can’t be traced through to testing.

Images courtesy digitalart/FreeDigitalPhotos.net

11bTest Strategy & Planning

“Testing must be traceable to requirements”. �

Vague defect management process.

Environments?

Conflicting demands on resources.

Conway’s Law.

More warning signs (a personal list)

A better way?

Test Strategy & PlanningRST Heuristic Test Strategy Model

11c

9/11/2014

74

Really good, but…

it’s not a template,

it won’t think for you,

it won’t stop you making blunders I’ve seen with traditional approaches,

and you have to follow the spirit, not the letter, and THINK.

Test Strategy & PlanningRST Heuristic Test Strategy (Plan? Model

11c

The strategy has to show how you’ve thought your way through from the problem to a plausible answer.

Image courtesy David Castillo Dominici & FreeDigitalPhotos.net

The plan should show how you’ll implement the strategy.

Test Strategy & Planning“Plan =strategy + logistics”

11d

9/11/2014

75

Images courtesy Simon Howden, Stuart Miles, Artur84 & stockimages/FreeDigitalPhotos.net

What does a good auditor

expect?

12

Test Execution

Test ExecutionCOBIT 5

Remember COBIT 5. That says it all.

Record & communicate everything you said you’d do.

Exploratory testing?Rapid Software Testing?

What does a good auditor

expect?

Images courtesy Simon Howden, Stuart Miles, Artur84 & stockimages/FreeDigitalPhotos.net

12a

9/11/2014

76

Test execution deviating from the plan. Hmm!

Changes to defect management & reporting and test priorities during the test execution.

Lack of an audit trail for defects/fixes & a lack of reliable, contemporary evidence.

Image courtesy digitalart/FreeDigitalPhotos.net

Test ExecutionWarning signs (an official list from COBIT 5)

12b


Test ExecutionWarning signs (an official list from COBIT 5)

12b

In summary, auditors expect the plan to be relevant.

There are good reasons to change plans and schedules during testing, but auditors will be very suspicious of anything that looks like winging the testing because the plan was rubbish, or rigging the testing schedule to hit the implementation date.

9/11/2014

77


Test ExecutionWarning signs (a personal list)

12c

Reporting that implies a link between test case passes & progress.

Confusion between defect fix priority & defect severity.

Massaging defect severity down and up.

Treating usability issues as cosmetic.


Test ExecutionWarning signs (a personal list)

12c

Large numbers of defects being rejected.

Defects rejected because there’s no matching test case or requirement.

Defects rejected because the requirements are assumed to be correct.

Failure to write reusable automated tests.

9/11/2014

78

Test Reports

Images courtesy Stuart Miles & David Castillo Dominici/FreeDigitalPhotos.net

13a

What does a goodauditor expect?

The same as a good test manager.

What does a goodauditor expect?


13b

Test Reports(with thanks to Rapid Software Testing)

Learning about the product

Learning about how the product

was tested.

Learning about how good the testing was.

9/11/2014

79

Images courtesy Stuart MilesFreeDigitalPhotos.net

Putting the jigsaw together.

Don’t empty the box onto the table.

Put the pieces together to assemble a clear picture, to tell a compelling story.

Test Reports13c


Auditors live and die by evidence.

Opinions are not casual observations. They must be backed by evidence.

Test Reports13d

9/11/2014

80


Finally, say what you mean and mean what you say.

Auditors will take your statements at face value.

Test Reports13d

Testing Standards14

9/11/2014

81

Wrap Up


Never follow the letter of the law and ignore the spirit.

Never do something just because “that’s what the auditors will expect”.

Do the right thing and be ready to justify it.

Go and speak to the auditors.

Say what you mean and mean what you say. And never lie!

15a


Email: [email protected]

Twitter: @james_christie

www.clarotesting.wordpress.com

www.clarotesting.com

15b

Wrap Up

satisfying auditors: plans and evidence in a regulated environment

Technology

practical experience

testing time

testing consultant

software testing

qualified acas

icaew chief executive

satisfying auditors

project management