1© Thomas E. Festing – 2013
Project Risk Assessment© Thomas E. Festing – 2013
© Thomas E. Festing – 2013 2
Education/Certifications:
CISA/CRISC
BSBA In Accounting (Back When Dirt Was New)
Been Working On A Masters Since 1981
35 Years Married:
Wife & Two Children
Moved 17 times in 35 years
Two Dogs
2 year old Grandson
11 Years In The US Army - Captain:Communications & Automation
Banking & Finance
Have driven an Abrams M1 Tank (mighty fine)
Things, That If I Told You –
Someone “May Come And Take Us Both Away”!
10 Years Public Accounting:
7 years with Arthur Andersen
IT audit and consulting support
IT Risk Assessment & Governance
10 Years Internal Audit/Risk Management :
JP Morgan Chase & Prudential Home Mortgage
Risk assessments, Privacy, ITGC reviews
Infrastructure / Data Center Technology
1 Year CIO:Non-traditional Credit Card Industry
“Owned” IT functions
State of Ohio:
Office of Budget & Management
Internal Audit / IT General Controls
Risk Assessments /Consulting support
A Little Background – Tom Festing
© Thomas E. Festing – 2013
3© Thomas E. Festing – 2013
Today’s objectiveis not to makeyou a “guru” …
This is not a “technology” presentation
… but to make youaware of a risk-basedapproach!
…. It is from a“business risk”perspective!
© Thomas E. Festing – 2013
4© Thomas E. Festing – 2013
Project Risk Assessment
• Why are we here?• Definition – Risk Assessment• The Business Problem
– Problem– Solution– Objective/Approach
• The Process– Frame It– Collect It– Analyze It– Tell All
• Life Cycle
• Questions/Comments
5© Thomas E. Festing – 2013
Risk professionals are confronted with developing processes that integrate withholistic risk management strategy complementing Enterprise Risk management, ITrisk, privacy, and NIST 800-30 Risk Assessments.
This session provides an example of a repeatable survey-based process that aids inassessing the business risks associated with managing large/complex projects.
These risks transcend traditional functionality and code testing - expanding topotential business weaknesses within the areas of project governance,management, business requirements, design /architecture, implementation, andsecurity.
Review an approach that provides quantifiable results supporting stratification ofpotential issues by organization demographics - including business functions,position, experience, and participation at both a business and IT levels.
Review deliverable examples that provide benchmark data that can be used bysenior management and project sponsors to support accountability and follow-onassessments to evaluate changes in project communications and execution.
Why Are We Here Today?
6© Thomas E. Festing – 2013
The Business Solution
What do you really want!Proactively identify areas where large project managementmay be at risk.Stratify possible risk areas based on demographics – who is “insync” and who is “out”.Identify areas that can be adjustedto help ensure success.Understand how to make it easy!
What will you get!Explanation/walkthrough of process.Copy of sample survey questions.Description of a “tool”.CPE credits … and ME!
What do you really want!Proactively identify areas where large project managementmay be at risk.Stratify possible risk areas based on demographics – who is “insync” and who is “out”.Identify areas that can be adjustedto help ensure success.Understand how to make it easy!
7© Thomas E. Festing – 2013
Extract - Holistic Risk Assessment Strategy
Focus• Large/complex projects• Provides quantifiable
analysis stratifying keygovernance areas
Project Risk Assessments
Pol i ci es andP ro ced ures P erson nel V en dor M anage me nt Con trol Cap abi li t ies E ase of U se E xposure Pop ul at ion E xte rn al Faci ng User Se curi ty A dm. Sec urity Moni tori ng Performan ce
Moni torin gTran sact io nMon itori ng Da ta P ri va cy Da ta A vai l abi l it y/
I mp ortance C han ge Ac ti vi ty Age /R el ease L evel In vent oryR edu ndan tArchi t ectu re
BC /D R & Re cove rySp eed
Suf fi ci ent Con fi den t No/L ow Ex posure S tron g Exp ert Techn ical Supp ort No E xterna l Faci ng Cen tral i zed I n Pl ace Est abl ish ed T racks D et ail Act i vit y Publ i c D at a C urrent Lo w Lo w I mp act Lo w C urren t I mport antE xist s Limit ed C onfi denc e Moderat e E xposu re Ma rgi nal Some Ex peri ence D epar tmen tal Int ernal E xposu re Mi xed E nvironment A d-Hoc Ad -H oc Pow er ful U sers Int ernal I mpor tan t Mediu m Medi um Med ium Par t ial ly Cus tomer Cr it i cal
D ef ici ent - No ne No Co nf id ence High E xposu re Weak Gen eral User "Everyo ne" Ex terna l Exp osure Decen tral i zed Do es Not E xist D oes No t Exi st Do es Not Ex ist Pri vat e Hi ghl y Impo rt an t High Hi gh Lo w Hi gh Missi on Cri ti cal
Po li ci es an d
Procedu resPersonn el V en dor M anage me nt Con trol Cap abi li t ies E ase of U se E xposure Pop ul at ion E xte rn al Faci ng Use r Secu rit y Ad m. Se curit y Moni t orin g
Pe rforman ce
Mon it ori ng
T ra nsact ion
M oni to ri ngDat a P ri vacy
Dat a A vai l abi li t y/
Import anceC han ge Act i vit y Age /R el ease L evel In vent ory
R edu ndan t
Archi t ectu re
BC /DR & R ecove ry
Sp eed
3 2 2 2 9 3 2 2 2 2 2 9 3 5 2 3 2 6
5 9 5 4 5 9 9 9 9 5 4 9 3 5 5 5 9 10
2 5 2 2 9 2 2 5 5 2 2 9 5 2 2 3 9 10
5 2 2 3 9 9 2 2 5 2 2 9 3 5 2 3 2 10
5 1 0 2 9 9 9 9 2 6 3 9 9 5 8 1 0 3 5 10
5 9 6 5 5 5 2 5 5 2 2 9 3 8 3 3 2 10
5 9 9 2 6 9 9 9 2 8 2 9 3 5 2 3 2 6
5 6 2 2 7 6 2 6 5 3 2 9 3 5 2 3 2 10
6 1 0 10 3 9 9 2 6 9 6 9 9 3 3 1 0 3 5 10
3 2 5 5 7 6 3 2 5 2 8 9 3 5 2 3 2 10
5 1 0 10 1 5 4 5 2 2 2 10 10 7 5 2 3 5 10
5 5 2 2 9 9 5 5 2 2 5 9 3 5 2 3 2 10
5 9 2 2 7 5 5 5 2 2 5 9 3 5 5 3 2 10
3 5 2 5 9 9 9 5 5 9 9 9 5 2 5 3 2 10
5 5 2 2 3 2 9 5 2 2 2 9 3 5 2 3 2 10
5 9 5 2 2 10 10 5 7 5 2 9 3 5 2 3 2 10
5 9 5 2 2 2 1 5 2 5 2 9 3 5 2 3 2 10
5 5 5 3 5 10 10 5 5 5 2 9 3 5 2 3 2 6
6 1 0 10 6 2 9 9 9 9 9 6 9 3 5 3 3 5 10
5 5 9 5 9 2 2 5 6 2 2 2 3 2 2 5 2 5
5 5 9 5 9 2 2 5 2 2 2 2 3 2 2 5 2 5
5 1 0 10 5 5 5 9 9 8 3 10 10 9 5 2 3 5 10
Func tiona lity Re cove rability/A va ila bilityG over nance Se cur ity Ma na ge me nt Monitoring D ata Ma na ge me nt
Execute
Identify Threat
Identify Vulnerabilities/ Predispos ingConditions
Determine Likelihood of Occurrence
Determine Impact
Determine Risk
Com
mun
icat
ions
&In
form
atio
nSh
arin
g(D
eliv
erab
les)
Mai
ntai
ning
Risk
Asse
ssm
ent(
Life
Cycl
e)
Preparing For Risk Assessment (Understand)
H igh
Like
lihoo
d
Low Impact
High
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
IT Risk Assessment
IT Risk Assessment
Focus• Assess IT technology risk drives• Creating A Multi-Year Audit Plan• Technology device/process focus
NIST 800-30
Focus• Regulatory requirement• Identifying security threats/ likelihood/ impact• Business focus
APPLICATION
STORAGE/REMOVABLE
MEDIA
REPORTS
TRANSIT
GOVERNANCE
Focus• Regulatory requirements• Data Privacy
Privacy
Enterprise Risk Management
IT Governance
Cont
rolE
ffect
iven
ess
Regu
lato
ry
Effic
ienc
y
Avai
labi
lity
Risk Assessment
AuditPlan/Scope
OptimizedRisk Environment
Governance
Strategy
Enterprise Risk Mgmt
Common LinkageLinkage to ERM
Managing Change
8© Thomas E. Festing – 2013
Risk Assessment
A Risk Assessment is a logical first step in a methodical riskmanagement process ...
that provides a framework for creating a quantifiable orqualitative value of the risk ...
linking to threat sources and vulnerabilities …
supporting determining the inherent likelihood and impact ….
that could hinder an organization from attaining its businessgoals and objectives in an efficient, effective, and controlledmanner – be it process, technology, people, or vendor generated.
9Thomas E. Festing – 2013
Project Risk Assessment Process
10Thomas E. Festing – 2013
Ready?Project Risk Assessment Process
11© Thomas E. Festing – 2013
I used to think that 90% was all aboutthe journey – and thought everyonewas excited about the trek as I was.
The end was just the conclusion of the“fun stuff”.
Project Risk Assessment – New Agenda
Nope…
Well – its not.
You need to see what’s at the end soyou can see if the 90% is worth the10%.
It also allows you to see why the pathis not “easy”.
So here’s the modified agenda.
12© Thomas E. Festing – 2013
• A “Peek” To End Deliverables• Definition – Risk Assessment• The Business Problem
– Problem– Solution– Objective/Approach
• The Process– Tell All– Frame It– Collect It– Analyze It– Tell All
• Life Cycle
• Questions/Comments
Why do I even wantto take this trip!
What will I take the trip in … is it sound!
10% - End Deliverable
90% - Fun Stuff
10% - End Deliverable .. recap
Sustain … don’t make the same trip twice
Project Risk Assessment – New Agenda
What do I get!
How many CPE did I get for this?
13© Thomas E. Festing – 2013
THE PEEK
14© Thomas E. Festing – 2013
Wouldn’t it make more sense to be able toget a peek at what we will “get … like:
Define critical project risk and demographic areas!
Strategy for collecting and analyzing data!
Understanding what/how to communicate results!
How you can track improvements.
The Peek
May be handy – especially if we run out of time ……
Understand how this links to ERM and other riskassessments!
15© Thomas E. Festing – 2013
The Business Problem
Large projects tend to fail.
Need to find a way to identify potential risk areas.
Need to find a way to track improvements.
16© Thomas E. Festing – 2013
Management Governance
Program Management
Business Requirements
Design & Development
Implementation/Operations
Information Security
Critical Risk Areas
Use a limited number of broad-based“business relevant” control areas.
Need a common language.
They do need to link to standardcontrol areas so they are “defendable”and tie to audit & ERM.
Build standard survey questions foreach “Control Area”.
End results - 6 Categories / 22 Sub-areas / 47 Questions.
17© Thomas E. Festing – 2013
Collect / Analyze Data By Demographics
Stratifying & consolidating data bydemographics provides a way to gaugeresponses and provide differentperspectives.
Gaining input across different“levels” and organizational groupshelps identify who is or is not ………
18© Thomas E. Festing – 2013
Need to:Identify demographicsCollect input with anonymityKeeps it relevant & limitedExample demographics ….
FunctionalArea/Responsibility
Executive Leadership
Line Of Business
Information Technology
Vendor/Consultant
Position
Executive Management
Senior Managers/Directors
Supervisors
Staff
Years Experience
1-3 Years
3-6 Years
6-9 Years
> 9 Years
Project Involvement
Core Team Member
Subject Matter Expert
Tester
None
Collect / Analyze Data By Demographics
Use On-Line Survey
19© Thomas E. Festing – 2013
Collect & Analyze Data
Success is not driven byslogans, mandates, and
t-shirts – but by thesupport of the
diversified team
20© Thomas E. Festing – 2013
Collect & Analyze Data
Use On-Line Survey(Survey Monkey)
Data Collection
Management Governance
Business Requirements
Design & Development
Implementation/Operations
Information Security
Program Management
By Critical Risk Areas(6 Categories / 22 Sub-areas)
By Demographics &47 specific Question
“Crunch” Data
CORE RISK ENGINE
Import To CoreRisk Engine
21© Thomas E. Festing – 2013
A way to communicateso management can“size” and “track”.
Future Point In Time
Confidence Level
Impa
ctPr
ojec
tSuc
cess
H i g h
L o w
Not ConfidentExtremelyConfident
910
54
78 1
221
22 1
6
15
14
17
18
19
1
2
311
13
6
20
Overall Top /Bottom 5 areas
Detail By QuestionArea/Demographics
Confidence Level
53
12
64
97
81 0
5 3 126 49 7810
1
2
6
78
910
11
16
15
14
13
21
20
412
22
19
5
3
18
17Im
pact
Proj
ectS
ucce
ssH i g h
L o w
Not ConfidentExtremelyConfident
Baseline
Risk AresHeat Map
Communicate To Management
22© Thomas E. Festing – 2013
GOAL: CONCLUSION - RECOMMENDATION - PRIORITIZATION
Significant areas indicated a protracted lack ofconfidence. Area average – exceptInformation Security – were lower at thismilestone than the previous two assessments.
Communicate To Management
23© Thomas E. Festing – 2013
Privacy RiskAssessment
DR/BC BusinessImpact Assessment
Business ProcessAssessment
Legal/RegulatoryRisk Assessment
BUSINESS AREAS6
Enterprise RiskManagement (ERM)
1
Detective/Monitoring
Preventative/Logical
Availability/Data Management
Change Management
Governance/Process
COMMON RISKDRIVERS
2
CORE RISK ENGINE
Business Process
Risk Driver
Technology Device
Audit Plan
IT Business Plan
4
“WhatIf”
PRIORITIZATION
RESIDUAL/INHERENT RISK
TECHNOLOGY CHANGE
RISK TOLERANCE
5
TECHNOLOGY AREAS
IT Risk Assessment3
Coordinated ERM and Risk Assessments
Project RiskAssessment
8
LIFECYCLE
Threat AssessmentNIST 800-30
6
Various Risk ReportsVarious Risk Reports
Various Risk Reports
OTHER RISKASSESSMENTS
24© Thomas E. Festing – 2013
PEEK END
Now the detail fun journeyfor the “why” and “how”!
25© Thomas E. Festing – 2013The Business Problem
26© Thomas E. Festing – 2013
For years we have convinced ourselves that all we needed to do was carry forwardour audit approach and strategy from year to year – or just focus on NIST.
Today – we have traditionally built structured risk-basedapproaches based on a combination of financial riskmanagement, technology risk assessment framework, andrisk-based audit scoping that works in concert with theoverall enterprise risk management model to guide audit’sassurance of “reasonable” levels of residual risk.
We acknowledged that there was always “change”, and somelevel of “business risk” that management would accept.
No one disputes that project risk increases based onthe project size and duration.
After all, what you didn’t know couldn’t hurt you! …or isit “If it doesn’t kill you – it makes you stronger”?
Risk Assessments
27© Thomas E. Festing – 2013
Organizations are confronted with having to develop efficient, effective,and repeatable assessment tools to aid in assessing the business riskswith managing large enterprise projects.
Pressure is placed on more than just functionality and code testing, butnow expands to where to focus limited resources to target potentialweaknesses within the areas of project:
Governance,Project Management,Business Requirements,Design/Architecture,Implementation, andSecurity.
Pressures are being applied by Audit Committees and Boards tounderstand cost / benefits and what proactive steps are being taken toreduce industry “failure” rates!
The Business Problem
28© Thomas E. Festing – 2013
“.. While larger projects are more likely to fail than smaller projects,around half of all project failures, irrespective of project size, wereput down to functionality issues and substantial delays.”
2012 Gartner – Survey
The Business Problem
Specifically identified:Cost:
Not identifying budget variances/overruns early.Changes in scope – with related impactto cost vs. budget.
Functionality:Not capturing business functionalityexpectations.Quality.Infrequent project status meetings.Misalignment with business strategy.
Late!
“.. Failure rate of IT projects with budgets exceeding $1 million wasfound to be almost 50% higher than for projects … below $350,000.”
“.. Smaller projects experienced a one-third lower failure rate thanlarge projects . keep small … not exceeding six months in duration …”
Sounds like projectmanagement &Governance to me!
29© Thomas E. Festing – 2013
The Real Business Solution
Provides quantifiable analysis techniques gained by evaluatinginput across demographic cross sections by:• Functions,• Position,• Experience, and• Participation at both a business and IT levels.
Provides quantifiable results support stratification of potentialissues both by project area and organization layer.
This session provides an example approach of a repeatablesurvey-based process that:
Delivers benchmark data to support:• Audit focus during the project.• Follow-on assessments to evaluate if changes in project
communications and execution are achieving the desired results.
30© Thomas E. Festing – 2013
I. Identify and prioritize risks at key points during the project:Pre project as part of normal risk assessments (baseline)During project/comparisonPost project
Business Objective/Approach
II. Develop Risk Assessment Survey:Address demographic cross sections by functions, position, experience,and participation on the “targeted” project.Leveraged industry governance models – NIST, COBIT, ITIL, etc.
III. Initial Survey:Focus on level of confidence (likelihood)Survey analysis including stratifying responses / developing Risk Matrix
31© Thomas E. Festing – 2013
ITIL COBIT
Execute
Identify Threat
Identify Vulnerabilities/ Predispos ing Conditions
Determine Likelihood of Occurrence
Determine Impact
Determine Risk
Com
mun
icat
ions
&In
form
atio
nSh
arin
g(D
eliv
erab
les)
Mai
ntai
ning
Risk
Asse
ssm
ent(
Life
Cycl
e)
Preparing For Risk Assessment (Understand)
NIST
IT Governance
Cont
rolE
ffect
iven
ess
Regu
lato
ry
Effic
ienc
y
Avai
labi
lity
Risk Assessment
Audit Plan/Scope
OptimizedRisk Environment
Governance
Strategy
Enterprise Risk Mgmt
Approaches must link toIndustry Guidelines &Enterprise Risk Management.
This way we know weare not wastinglimited resources(time, people, money).
IT RiskAssessments
Financial RiskAssessments
Risk-BasedAudit Plan
So we are covered andcan now all go home!
Business Objective/Approach
32© Thomas E. Festing – 2013
IT Risk Assessments
Financial RiskAssessments
Risk-Based Audit Plan
ITIL COBIT
Execute
Identify Threat
IdentifyVulnerabilities/
Predispos ingConditions
DetermineLikelihood ofOccurrence
DetermineImpact
Determine Risk
Com
mun
icat
ions
&In
form
atio
nSh
arin
g(D
eliv
erab
les)
Mai
ntai
ning
Risk
Asse
ssm
ent(
Life
Cycl
e)
Preparing For Risk Assessment (Understand)
NIST
IT GovernanceCo
ntro
lEffe
ctiv
enes
s
Regu
lato
ry
Effic
ienc
y
Avai
labi
lity
Risk Assessment
Audit Plan/Scope
OptimizedRisk Environment
Governance
Strategy
Enterprise Risk Mgmt
Project RiskAssessment
What about ProjectManagement?
Too often we are great at pointing outpost implementation observations:
Absence of adequateprocess/IT controls
Financial overages
Failed efficiencies
$
Business Objective/Approach
33© Thomas E. Festing – 2013
IT Governance
Cont
rolE
ffect
iven
ess
Regu
lato
ry
Effic
ienc
y
Avai
labi
lity
Risk Assessment
Audit Plan/Scope
OptimizedRisk Environment
Governance
Strategy
Enterprise Risk Mgmt
Initial assessment - sizeproject early – establish abaseline and identify areasthat may require more focus.
Periodic snapshots to assesseffectiveness of projectgovernance, communications.Track against baseline.
Post implementationevaluation.
Business Objective/Approach
The approach can’t just be “rear view mirror” based to be effective.
34© Thomas E. Festing – 2013
The Process
35© Thomas E. Festing – 2013TELL ALL
36© Thomas E. Festing – 2013
Communications To Management
With the ability to collect &mine the survey data …..….. You have unlimited waysto assess current andtrending project data …..
Business Unit
F unctiona l Are a T ota l
Leg al/Inte rnal Audit
F unctiona l Ar eaTo tal
Exe cutive M ana gem ent
Functio nal Area T o tal Sta keho lde rLea der ship
Fun ctio nal Are a T otal
Vend or/ Co nsultant
F unctiona l Are a T ota l
Sta ff Super visor Adm in is tr atorExecutive
Ma nag eme ntSta ff Supe rvisor Adm in is tr ator Exe cutive M anag em ent Exe cutive Ma nag em ent Sta ff Ad ministrato r Sta ff Super visor
3.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 1.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.002.00 10.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 2.00 2.00 1.00 4.00 5.00 1.00 1.00 2.002.00 10.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 4.00 5.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 4.00 5.00 1.00 1.00 2.00
3.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
Cost:Not identifying budgetvariances/overruns early.Changes in scope – with relatedimpact to cost vs. budget.
Functionality:Not capturing business functionalityexpectations.QualityInfrequent project status meetings.Misalignment with business strategy
Late!Remember Gartner’s areas ofinterest …
37© Thomas E. Festing – 2013
Response By PositionsExecutive Management
Senior Manager/Director
Supervisors
Staff
Track Response By Functional Areas
Executive Leadership
Business Unit
Information Technology
Legal/Internal Audit
Vendor/Consultant
Sample
60-100 Target Participants:
Historically 80-95% responded
Typically 40-45% Provided additional comments
Understand The Environment /Approach
Review Project Documentation
Conduct Risk Assessment Survey
Communications To Senior Management
Report By Project Areas
I. Management Governance
II. Program Management
III. Business Requirements
IV. Design & Development
V. Implementation / Operations
VI. Information Security
Rating Options
• EXTREMELY CONFIDENT
• CONFIDENT
• SOMEWHAT CONFIDENT
• NOT CONFIDENT
• DO NOT KNOW
38© Thomas E. Festing – 2013
Responses were initially assessed from three (3)perspectives:1. Individual responses2. Average responses by individual questions3. Average by Survey Area
Significant areas indicated a protracted lack ofconfidence. Area average – exceptInformation Security – were lower at thismilestone than the previous two assessments.
Communications To Management - Overall
© Thomas E. Festing – 2013 39
Compare Projected(Inherent Baseline) to
Survey Result
Heat Map - Inherent vs. Residual RiskAssessment
Confidence Level
53
12
64
97
81 0
5 3 126 49 7810
1
2
6
7
8
9
10
11
16
15
14
13
21
20
4
12
22
19
5
3
18
17
Impa
ctPr
ojec
tSuc
cess
Hig
hLo
w
Not ConfidentExtremely Confident
Baseline Risk Assessment Matrix Why The Variance In
“Security”?Baseline Residual Risk Matrix – From Survey
Confidence Level
Impa
ctPr
ojec
tSuc
cess
Hig
hLo
w
Not ConfidentExtremely Confident
9
10
5
4
78
12
21
22
16
15
14
17
18 19
1
2
3
11
13
6
20
40© Thomas E. Festing – 2013
Low ConfidenceCommunicationsVendor Management/TransitionOversight accountabilityRequirement Definition identified, benefit quantifiedToo many workarounds
Communications To Management –By Residual Risk
Residual Risk Assessment:Participants felt more confident whencompared to the Inherent Baseline RiskMatrixHigh Risk Stripe:
Business RequirementsDesign & DevelopmentManagement GovernanceImplementation
Information Security moved out of “highrisk” Strip
• Highest and Lowest Rated Areas:High Confidence
Strong “can do”System design – already hardenedSecurity administration - centralizedRole-based user profiles establishedExperienced PM team
Confidence Level
Impa
ctPr
ojec
tSuc
cess
Hig
hLo
w
Not ConfidentExtremely Confident
9
10
5
4
78
12
21
22
16
15
14
17
18 19
1
2
3
11
13
6
20
41© Thomas E. Festing – 2013
1 Organizational Oversight2 Project Risk Management3 Application Technology Capability
Management Governance
4 Project Team5 Resources6 Organizational Readiness7 Vendor Management8 Communication Strategy
Program Management
9 Requirement Definitions10 Business/Process Change
Business Requirements
11 System Development12 Data Management13 Legacy Process/System Management
Design & Development
14 Implementation Strategy15 Vendor Transition16 Training (Business/IT/Security/Third Party17 Service (Help) Desk18 Disaster Recovery & Business Continuity19 Go Live20 Post Evaluation
Implementation/Operations
21 Role Based Profiles22 Security Administration
Information Security
Communications To Management –Prioritization/Observation
• Critical Initiative Areas include:– Management Governance– Program Management– Business Requirements
• Other areas can be builtout/refined as the progressmatures:
– Design & Develop– Implementation / Operations– Information Security
‹#›© Thomas E. Festing – 2013
Communications To Management – OverallDetail By Functions/Position
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Overall AdministrativeLeadership
InformationTechnology
Business Other
Overall By Functional Area
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Overall Executive Manager/Director Supervisor Staff
Overall By Organization Hierarchy
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Overall ManagementGovernance
ProgramManagement
BusinessRequirements
Design &Development
Implementation/ Operations
InformationSecurity
Overall ByInitiative Area
‹#›© Thomas E. Festing – 2013
012345
What is the observation here?“Supervisors” may know something others don’t!
Overall ByPosition
Example Deliverables- Detail By Position/Area
44© Thomas E. Festing – 2013
Participants are provided space to provide additional comments.
00.5
11.5
22.5
33.5
44.5
5
Communication Strategy
Example Deliverables –Free Form Text
Areas that need focus:Revalidaterequirements as partof on-going meetingsCloser vendormanagementTrainingUSE Communications
Key words were “searched” including:CommunicationsRequirementsVendorTrainingResources
45© Thomas E. Festing – 2013
Executive and Managementfunction have an overall favorableconfidence level.
Project vendors indicate a solid“confident” level that they will beable to deliver the $25 Millionproject.
What doesthis tell you?
Communications To Management –Detail By Functional Area
However – Business, IT, and“compliance functions”gravitate to a much lowerconfidence.
There appears to be a“significant” gap betweenthose with “skin in the game”and those who will have touse/ support the project! Thirsty!
46© Thomas E. Festing – 2013
HOW
47© Thomas E. Festing – 2013
FRAME IT
Back To Step 1: Process - Survey Framework
48© Thomas E. Festing – 2013
Process - Survey Framework
Input guidance form industryframeworks:
– ITIL (Efficiency Focus)– COBIT (Control Focus)– NIST (Regulatory/Security Focus)
ITIL COBIT
Execute
Identify Threat
Identify Vulnerabilities/ Predispos ing Conditions
Determine Likelihood of Occurrence
Determine Impact
Determine Risk
Com
mun
icat
ions
&In
form
atio
nSh
arin
g(D
eliv
erab
les)
Mai
ntai
ning
Risk
Asse
ssm
ent(
Life
Cycl
e)
Preparing For Risk Assessment (Understand)
NIST
Output matrix identified:– 50 + sub-category areas– 156 + specific “control/governance” data point
IT Governance
Cont
rolE
ffec
tiven
ess
Regu
lato
ry
Effic
ienc
y
Avai
labi
lity
Risk Assessment
Audit Plan/Scope
OptimizedRisk Environment
Governance
Strategy
Enterprise Risk Mgmt
Addresses key areas including:- Control Effectiveness- Efficiency- Availability- Regulatory
49© Thomas E. Festing – 2013
156 data points is too unwieldy -so additional business focusedcompression was performed.
Input matrix:– 50 + sub-category areas– 156 + specific “control/governance” data point
1 Organizational Oversight
2 Project Risk Management
3 Application Technology Capability
Management Governance
4 Project Team5 Resources6 Organizational Readiness7 Vendor Management8 Communication Strategy
Program Management
9 Requirement Definitions
10 Business/Process Change
Business Requirements
11 System Development12 Data Management13 Legacy Process/System Management
Design & Development
14 Implementation Strategy15 Vendor Transition16 Training (Business/IT/Security/Third Party
17 Service (Help) Desk18 Disaster Recovery & Business Continuity19 Go Live20 Post Evaluation
Implementation/Operations
21 Role Based Profiles22 Security Administration
Information Security
Output:– Compressed to 6 categories– Reduced to 22 risk areas– Consolidated to ~ 50
streamlined questions
Process - Survey Framework
50© Thomas E. Festing – 2013
1 Organizational Oversight2 Project Risk Management3 Application Technology Capability
Management Governance
10
4 Project Team5 Resources6 Organizational Readiness7 Vendor Management8 Communication Strategy
Program Management
11
9 Requirement Definitions10 Business/Process Change
Business Requirements
4
11 System Development12 Data Management13 Legacy Process/System Management
Design & Development
10
14 Implementation Strategy15 Vendor Transition16 Training -Business/IT/Security/Third Party17 Service (Help) Desk)18 Disaster Recovery & Business Continuity19 Go Live20 Post Evaluation
Implementation/Operations
8
21 Role Based Profiles22 Security Administration
Information Security
4
Process - Survey Framework
Example - Questions 14 & 15
Develop survey questionfor each of the 22 areas.
47
51© Thomas E. Festing – 2013
Semi-Qualitative
Create assessmentcriteria / rating
1 Not Confident Extremely unlikely that this will be achieved/implemented/ performed.
- View Weekly = 10
- View Monthly = 7
- View Quarterly = 4
- Never = 1
- Unaware of Portal = 2
Confident Will be achieved/ implemented/ performed –with only minor adjustments.7
Somewhat Confident May be achieved/ implemented/ performed.– but may require significant effort.4
Extremely Confident There is no doubt that this will be achieved/implemented/ performed.10
Don’t Know I am unaware if this has been established oraddressed.2
Rate Definition
Process - Survey Framework
52© Thomas E. Festing – 2013
Section Overview:• Helps frame the participant’s understanding.• e.g. “BUSINESS REQUIREMENTS: These are the ‘what do I
need and what will I change’ aspects of the initiative. Theseinclude areas associated with defining/ prioritizing businessrequirements, identifying process changes, projectingefficiencies, and security/access needs.”
Questions:• Specific.• May be repeated in more than 1 section to help with
validation of responses.
Rating Criteria:• Repeat rating criteria to keep participant focused.• .
Everything depends on the survey questions!
Process - Survey Framework
Pg. 7 / Questions #14 & 15
53© Thomas E. Festing – 2013
Semi-Qualitative
Functional Area Department (e.g. Exec. Mgmt., Line of Business, Legal/Compliance, Vendor, etc.)
Project Involvement Participation in project (e.g. Member of project team, SME,Tester, none, etc.)
Position Position/Level (e.g. Executive, Director, Manager, Staff, etc.)
Years Experience Familiarity with systems and organization (e.g. 1 – 15 years)
Definition
Process - Survey Framework
Identify “demographics” stratification:
Pages 1 & 2Survey Questions 1-4
54© Thomas E. Festing – 2013
Executive
Director
Manager
Supervisor
Staff
Audit
Process - Survey Framework
What can be gainedby assessing resultsbased ondemographics?
Knowledge/Understanding Paradigm
55© Thomas E. Festing – 2013
Confidence Level
53
12
64
97
810
5 3 126 49 7810
Impa
ctPr
ojec
tSuc
cess
Hig
hLo
w
Not ConfidentExtremely Confident
1
2
3
1 Organizational Oversight2 Project Risk Management3 Application Technology Capability
Management Governance
9
10
9 Requirement Definitions10 Business/Process Change
Business Requirements11
13
12
11 System Development12 Data Management13 Legacy Process/System Management
Design & Development
16
15
14
20
19
18
17
14 Implementation Strategy15 Vendor Transition16 Training (Business/IT/Security/Third Party17 Service (Help) Desk18 Disaster Recovery & Business Continuity19 Go Live20 Post Evaluation
Implementation/Operations
21
22
21 Role Based Profiles22 Security Administration
Information Security
6
78
45
4 Project Team5 Resources6 Organizational Readiness7 Vendor Management8 Communication Strategy
Program Management
Process – Example Baseline Heat Map
9
10
12
1619
21
22
7
45
1
3
1 Organizational Oversight2 Project Risk Management3 Application Technology Capability
Management Governance
56© Thomas E. Festing – 2013
…. and links tocontrol strategies!
IT Governance
Cont
rolE
ffect
iven
ess
Regu
lato
ry
Effic
ienc
y
Avai
labi
lity
Risk Assessment
Audit Plan/Scope
OptimizedRisk Environment
Governance
Strategy
Enterprise Risk Mgmt
Best of all … the process is“defendable” ….
Process - Survey Framework
57© Thomas E. Festing – 2013COLLECT IT
Back To Step 2: Process - Survey Framework
58© Thomas E. Festing – 2013
Process – Communication Strategy
“Monday” - Executive sponsor:ImportanceAnonymityMore detail to follow
“Tuesday” - Survey Administrator:Survey overviewTimeline 1 ½ weeksAnonymityQ&A dial in
“Wednesday” - Send Survey
Week 2 – Notification:
“Friday” - Dial In participant Q&A
Week 3 – Survey:“Wednesday” - Send “Reminder”“Friday” - Send “Reminder”
Week 4 – Survey:“Tuesday” - Send “Reminder”“Thursday” - Close Survey“Friday” - “Thank You” fromExecutive Sponsor
6 Elapsed Weeks
Week 1 – Scope/Planning
Week 6:Communicate Deliverables
Week 5:Data analysisQC
“Monday” - Executive sponsor:ImportanceAnonymityMore detail to follow
“Tuesday” - Survey Administrator:Survey overviewTimeline 1 ½ weeksAnonymityQ&A dial in
“Wednesday” - Send Survey
Week 2 – Notification:
“Friday” - Dial In participant Q&A
Week 3 – Data Collection:“Wednesday” - Send “Reminder”“Friday” - Send “Reminder”
Week 4 – Data Collection:“Tuesday” - Send “Reminder”“Thursday” - Close Survey“Friday” - “Thank You” fromExecutive Sponsor
6 Elapsed Weeks
Week 1 – Scope/Planning
Week 6:Communicate Deliverables
Week 5:Data analysisQC
59© Thomas E. Festing – 2013
Data capture/entry:
Risk Assessment – Data Input/collection
Via electronic survey (e.g. Web based Survey Monkey)
Supports large volume, anonymous , autoimport into Core Risk Engine, & providesdetail analysis by demographics.
User directly into the template
Task 2- 2E-2 / E5
1
H2
Type Of H ar m Impac tSem i-Quan tati ve
V alu eR ange Of
Effec tSemi -Quanta tiv e
Va lu eL evel O f
Impac tSemi-Q uant at ive Value
(Calculation Average H- 3 / H-4)
Attac ks :
Exploitation of s ys tem weakness es (1)Op eratio nalAssetIn di vid ualOrga ni zatio n
Moder at e 5 High 8 Moder at e 6. 5 M oder at e 6. 45
Injec tion of v irus/ malware (2)Moder at e 5 M oder at e 5 Moder at e 5 M oder at e 6. 4
Soc ial Engineering/ Phis hing (3) Moder at e 5 M oder at e 5 Moder at e 5 M oder at e 6
Data Management
Inappropriate acc es s (read/W rite/Delete/Modify) (4)High 8 High 8 Hi gh 8 High 7. 4
Exploit mobile media to ac ces s information (Tapes/ FlashDrives /CD ) ( 5)
High 8 V ery Hi gh 1 0 Ver y Hi gh 9 M oder at e 6. 9
Equipment Dispos al expos es un-cleared information (6)Moder at e 5 Low 2 Low 3. 5 M oder at e 3. 95
Availability of Data - U nint entional los s of dat a due to systemcorr upt ion (7)
Moder at e 5 High 8 Moder at e 6. 5 M oder at e 5. 65
Phys ical Acc ess - Loss of int egrity/t heft of information due toinappropriate ac ces s to sys tems (8)
High 8 M oder at e 5 Moder at e 6. 5 M oder at e 5. 45
Tr ansmiss ions - exploit ac ces s to networkconnect ions/ transmis sions
Internal Network - c ompromise int er nal networ k (9) Moder at e 5 V ery Hi gh 1 0 Hi gh 7. 5 M oder at e 5. 75
External - exploit ac cess to external net workconnections /trans mis sions (10)
High 8 High 8 Hi gh 8 High 7. 1
Monitor ing - Inability to detec t/identif y attacks :
Sec urity - Inability to detec t/identify s ecurity-based att ac ks orsyst em failure (11)
Moder at e 5 V ery Hi gh 1 0 Hi gh 7. 5 High 7. 25
Inability t o detect or identify perfor manc e/capacity-basedat tacks or s ys tem availabilit y/ failure (12)Note: Average ( a) & ( b) below
Low 3 .5 V ery Hi gh 1 0 Moder at e 6. 75 M oder at e 5 .9 75
Per formance - (12a) Moder at e 5 V ery Hi gh 1 0 Hi gh 7. 5 M oder at e 6. 65
Capac it y (12b)Low 2 V ery Hi gh 1 0 Moder at e 6 M oder at e 5. 3
Change Act ivity - Expos ure inc reas ed due to age ofhar dware/s oftwar e, patch levels , c ompat ibility, number/t ype ofchanges (13)
Low 2 V ery Hi gh 1 0 Moder at e 6 M oder at e 5. 8
Los s O f Facilit ies
Natural (eart hquake/flood/etc.) (14) High 8 V ery Hi gh 1 0 Ver y Hi gh 9 M oder at e 5. 9
Non-Natural/Man Made (f ir e/ elec trical/etc .) ( 15) High 8 V ery Hi gh 1 0 Ver y Hi gh 9 M oder at e 6
Semi-Q uantative Value(Calculation Average 10
& 11)
1211
O ver all Ris kThr eat Event
Overall Impac t
Task 2-5
H-3
Table H-5 (Average H -3 or H4)
H -4
T ask 2-2 T ask 2 -3E-2 / E 5
Ca lcula tio nAve rag e
1
C ap ab ility Value Inte nt Valu e Ta rge ting Val ue Rat i ng Val ue R ati ng Val ue R ati ng V alue
Atta ck s:
Exp loi tatio n of sys tem we ak ne sses (1 )Adver sar ialAcciden talSt ruct ur al
Moder at e 5 Hig h 8 Hi gh 8 Hi g h 7 Pr edic a ted 4 Hig h 8 Moder at e 5 Hi g h 8 Modera t e 6.4
In jectio n of vi rus /ma lwa re (2)Adver sar ialAcciden tal Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Conf ir med 10 Hig h 8 Moder at e 5 Hi g h 8 Hig h 7.8
Soc ia l E ng in ee rin g/ Ph is hing (3 ) Adver sar ial Moder at e 5 Hig h 8 M oder a te 5 Modera t e 6 Ex pec t ed 8 Hig h 8 Moder at e 5 Hi g h 8 Hig h 7
Da ta M an ag em en tIn ap pr op ria te ac cess (re ad /Write /Delete /Mo dify) (4) Adver sar ial
Acciden tal Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ex pec t ed 8 Moder at e 5 Moder at e 5 Hi g h 8 Modera t e 6.8
Exp loi t mo bi le med ia to ac ces s in form ation (T ap es /FlashDrive s/C D) (5 )
Adver sar ial L ow 2 L ow 2 Hi gh 8 Modera t e 4 Ex pec t ed 8 L ow 2 Moder at e 5 M oder a te 5 Modera t e 4.8
Eq uipm en t Disp os al e xp os es un -clea red in fo rm ation (6 )Adver sar ialAcciden tal Hig h 8 L ow 2 M oder a te 5 Modera t e 5 Ex pec t ed 8 L ow 2 Moder at e 5 L ow 2 Modera t e 4.4
Av ai la b il i ty of Data - Uninte ntio na l loss o f d ata du e to sy stemcorru p tio n (7)
Adver sar ialAcciden talSt ruct ur al
Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Pr edic a ted 4 L ow 2 Moder at e 5 M oder a te 5 Modera t e 4.8
Ph ysica l Acc ess - Lo ss of in teg rity/th eft of in form atio n du e toin ap pro p ria te acc ess to s yste ms (8)
Adver sar ial Moder at e 5 L ow 2 Hi gh 8 Modera t e 5 Ex pec t ed 8 L ow 2 Moder at e 5 L ow 2 Modera t e 4.4
Tra nsm issio ns - e xp lo it acce ss to n e twor kcon n ectio ns /tra nsm ission s
In tern al Netwo rk - co mp rom is e in tern al ne two rk (9 )Adver sar ialAcciden tal Moder at e 5 Hig h 8 Hi gh 8 Hi g h 7 Pr edic a ted 4 L ow 2 Moder at e 5 L ow 2 Modera t e 4
Exte rna l - ex ploi t acc ess to e xte rna l ne two rkcon ne ction s/tra ns mission s (10 )
Adver sar ial Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ex pec t ed 8 Moder at e 5 Moder at e 5 M oder a te 5 Modera t e 6.2
Mo nito ring - Ina bil ity to d ete ct/id en tify a ttac ks:
Sec urity - Ina bi l ity to de tect/id en tify se cu rity -ba sed a ttack s orsyste m fai lure (11 )
Adver sar ialAcciden talSt ruct ur al
Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ant ic ipa ted 6 Hig h 8 Moder at e 5 Hi g h 8 Hig h 7
In ab i li ty to d ete ct or id en ti fy pe rfo rma nc e/c ap aci ty-b ase dattac ks o r sy stem ava ila bi li ty/ fa i lu re (12 )No te: Av era ge (a ) & (b) be lo w
Adver sar ialAcciden talSt ruct ur al
Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ant ic ipa ted 6 L ow 3.5 Moder at e 5 L ow 3. 5 Modera t e 5.2
Pe rfo rm an ce - (1 2a ) Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ant ic ipa ted 6 Moder at e 5 Moder at e 5 M oder a te 5 Modera t e 5.8
Ca pa ci ty (1 2 b) Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ant ic ipa ted 6 L ow 2 Moder at e 5 L ow 2 Modera t e 4.6
Ch an ge Activ ity - E xpo su re incre ase d d ue to ag e ofha rdwa re/s oftwa re, p atc h leve ls , co mp atib il i ty, n u mb er/typ e o fcha ng es (1 3)
Adver sar ialAcciden talSt ruct ur al
Moder at e 5 Hig h 8 Hi gh 8 Hi g h 7 Ant ic ipa ted 6 Moder at e 5 Moder at e 5 M oder a te 5 Modera t e 5.6
Los s Of F aci l itie s
Na tura l (e arth qu ak e/floo d/e tc.) (1 4) Envir onm ent al Moder at e 5 L ow 2 L ow 2 Low 3 Pos si ble 2 L ow 2 Moder at e 5 L ow 2 Low 2.8
No n -Na tura l/Ma n Ma de (fi re/ electrica l/e tc.) (1 5)
Adver sar ialAcciden talSt ruct ur alEnvir onm ent al
Moder at e 5 L ow 2 M oder a te 5 Modera t e 4 Pos si ble 2 L ow 2 Moder at e 5 L ow 2 Low 3
G-4
L ik eliho o dS ucc ee ds
9
T ask 2-4
OV
ERA
LLLI
KEL
IHO
OD
Com
bin
atio
nof
Ave
rag
eC
har
act
eris
tics
(3/4
/5-c
olu
mn
k),
Re
lev
ance
(6-
colu
mn
M),
Like
liho
od
of
Init
iati
on
(7-
colu
mn
O),
ITR
isk
Ass
ess
me
nt/
Con
tro
lDri
ver
s(8
-co
lum
nQ
),&
Like
liho
odo
fSu
cce
ss(9
-col
um
nS)
Ov era ll C ha racte ristics
T ask 2 -1
Rele van ce
C alcu latio nAve ra ge 3 ,4,& 5
Tas k 2-4
3 4D-4
Like lih o od OfInitiatio n
G2 /G3
Value Val ueTh rea t E ve ntTh rea t
S o urce
Vu ln e rab il ityP re disp os itio n(Ris k Driv er s)
( L ink To Ris kDr iver s ) Va lu e
Th rea t So urc e Ch ara cte ristics
D-5
Ta sk 2 -3Se e IT Risk As ses sme nt
D-22
D-3T as k 2-1
6E -4
7Risk T em pla te - F
85
Ove rallLike lih oo d
1 0G-5
Consolidate and “cut & paste” averageresponse into Core Risk Engine “input pages”– lose anonymity and detail by demographics,requires increased meetings.
One-On-One Meeting – manual entry
Best for interviews with senior executives – provides morebusiness insight. Consolidate and “cut & paste” into Core RiskEngine “input pages”.
Data capture/entry:Data capture/entry:
One-On-One Meeting – manual entry
Best for interviews with senior executives – provides morebusiness insight. Consolidate and “cut & paste” into Core RiskEngine “input pages”.
Via electronic survey (e.g. Web based Survey Monkey)
Supports large volume, anonymous , autoimport into Core Risk Engine, & providesdetail analysis by demographics.
User directly into the template
Task 2- 2E-2 / E5
1
H2
Type Of H ar m Impac tSem i-Quan tati ve
V alu eR ange Of
Effec tSemi -Quanta tiv e
Va lu eL evel O f
Impac tSemi-Q uant at ive Value
(Calculation Average H- 3 / H-4)
Attac ks :
Exploitation of s ys tem weakness es (1)Op eratio nalAssetIn di vid ualOrga ni zatio n
Moder at e 5 High 8 Moder at e 6. 5 M oder at e 6. 45
Injec tion of v irus/ malware (2)Moder at e 5 M oder at e 5 Moder at e 5 M oder at e 6. 4
Soc ial Engineering/ Phis hing (3) Moder at e 5 M oder at e 5 Moder at e 5 M oder at e 6
Data Management
Inappropriate acc es s (read/W rite/Delete/Modify) (4)High 8 High 8 Hi gh 8 High 7. 4
Exploit mobile media to ac ces s information (Tapes/ FlashDrives /CD ) ( 5)
High 8 V ery Hi gh 1 0 Ver y Hi gh 9 M oder at e 6. 9
Equipment Dispos al expos es un-cleared information (6)Moder at e 5 Low 2 Low 3. 5 M oder at e 3. 95
Availability of Data - U nint entional los s of dat a due to systemcorr upt ion (7)
Moder at e 5 High 8 Moder at e 6. 5 M oder at e 5. 65
Phys ical Acc ess - Loss of int egrity/t heft of information due toinappropriate ac ces s to sys tems (8)
High 8 M oder at e 5 Moder at e 6. 5 M oder at e 5. 45
Tr ansmiss ions - exploit ac ces s to networkconnect ions/ transmis sions
Internal Network - c ompromise int er nal networ k (9) Moder at e 5 V ery Hi gh 1 0 Hi gh 7. 5 M oder at e 5. 75
External - exploit ac cess to external net workconnections /trans mis sions (10)
High 8 High 8 Hi gh 8 High 7. 1
Monitor ing - Inability to detec t/identif y attacks :
Sec urity - Inability to detec t/identify s ecurity-based att ac ks orsyst em failure (11)
Moder at e 5 V ery Hi gh 1 0 Hi gh 7. 5 High 7. 25
Inability t o detect or identify perfor manc e/capacity-basedat tacks or s ys tem availabilit y/ failure (12)Note: Average ( a) & ( b) below
Low 3 .5 V ery Hi gh 1 0 Moder at e 6. 75 M oder at e 5 .9 75
Per formance - (12a) Moder at e 5 V ery Hi gh 1 0 Hi gh 7. 5 M oder at e 6. 65
Capac it y (12b)Low 2 V ery Hi gh 1 0 Moder at e 6 M oder at e 5. 3
Change Act ivity - Expos ure inc reas ed due to age ofhar dware/s oftwar e, patch levels , c ompat ibility, number/t ype ofchanges (13)
Low 2 V ery Hi gh 1 0 Moder at e 6 M oder at e 5. 8
Los s O f Facilit ies
Natural (eart hquake/flood/etc.) (14) High 8 V ery Hi gh 1 0 Ver y Hi gh 9 M oder at e 5. 9
Non-Natural/Man Made (f ir e/ elec trical/etc .) ( 15) High 8 V ery Hi gh 1 0 Ver y Hi gh 9 M oder at e 6
Semi-Q uantative Value(Calculation Average 10
& 11)
1211
O ver all Ris kThr eat Event
Overall Impac t
Task 2-5
H-3
Table H-5 (Average H -3 or H4)
H -4
T ask 2-2 T ask 2 -3E-2 / E 5
Ca lcula tio nAve rag e
1
C ap ab ility Value Inte nt Valu e Ta rge ting Val ue Rat i ng Val ue R ati ng Val ue R ati ng V alue
Atta ck s:
Exp loi tatio n of sys tem we ak ne sses (1 )Adver sar ialAcciden talSt ruct ur al
Moder at e 5 Hig h 8 Hi gh 8 Hi g h 7 Pr edic a ted 4 Hig h 8 Moder at e 5 Hi g h 8 Modera t e 6.4
In jectio n of vi rus /ma lwa re (2)Adver sar ialAcciden tal Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Conf ir med 10 Hig h 8 Moder at e 5 Hi g h 8 Hig h 7.8
Soc ia l E ng in ee rin g/ Ph is hing (3 ) Adver sar ial Moder at e 5 Hig h 8 M oder a te 5 Modera t e 6 Ex pec t ed 8 Hig h 8 Moder at e 5 Hi g h 8 Hig h 7
Da ta M an ag em en tIn ap pr op ria te ac cess (re ad /Write /Delete /Mo dify) (4) Adver sar ial
Acciden tal Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ex pec t ed 8 Moder at e 5 Moder at e 5 Hi g h 8 Modera t e 6.8
Exp loi t mo bi le med ia to ac ces s in form ation (T ap es /FlashDrive s/C D) (5 )
Adver sar ial L ow 2 L ow 2 Hi gh 8 Modera t e 4 Ex pec t ed 8 L ow 2 Moder at e 5 M oder a te 5 Modera t e 4.8
Eq uipm en t Disp os al e xp os es un -clea red in fo rm ation (6 )Adver sar ialAcciden tal Hig h 8 L ow 2 M oder a te 5 Modera t e 5 Ex pec t ed 8 L ow 2 Moder at e 5 L ow 2 Modera t e 4.4
Av ai la b il i ty of Data - Uninte ntio na l loss o f d ata du e to sy stemcorru p tio n (7)
Adver sar ialAcciden talSt ruct ur al
Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Pr edic a ted 4 L ow 2 Moder at e 5 M oder a te 5 Modera t e 4.8
Ph ysica l Acc ess - Lo ss of in teg rity/th eft of in form atio n du e toin ap pro p ria te acc ess to s yste ms (8)
Adver sar ial Moder at e 5 L ow 2 Hi gh 8 Modera t e 5 Ex pec t ed 8 L ow 2 Moder at e 5 L ow 2 Modera t e 4.4
Tra nsm issio ns - e xp lo it acce ss to n e twor kcon n ectio ns /tra nsm ission s
In tern al Netwo rk - co mp rom is e in tern al ne two rk (9 )Adver sar ialAcciden tal Moder at e 5 Hig h 8 Hi gh 8 Hi g h 7 Pr edic a ted 4 L ow 2 Moder at e 5 L ow 2 Modera t e 4
Exte rna l - ex ploi t acc ess to e xte rna l ne two rkcon ne ction s/tra ns mission s (10 )
Adver sar ial Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ex pec t ed 8 Moder at e 5 Moder at e 5 M oder a te 5 Modera t e 6.2
Mo nito ring - Ina bil ity to d ete ct/id en tify a ttac ks:
Sec urity - Ina bi l ity to de tect/id en tify se cu rity -ba sed a ttack s orsyste m fai lure (11 )
Adver sar ialAcciden talSt ruct ur al
Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ant ic ipa ted 6 Hig h 8 Moder at e 5 Hi g h 8 Hig h 7
In ab i li ty to d ete ct or id en ti fy pe rfo rma nc e/c ap aci ty-b ase dattac ks o r sy stem ava ila bi li ty/ fa i lu re (12 )No te: Av era ge (a ) & (b) be lo w
Adver sar ialAcciden talSt ruct ur al
Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ant ic ipa ted 6 L ow 3.5 Moder at e 5 L ow 3. 5 Modera t e 5.2
Pe rfo rm an ce - (1 2a ) Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ant ic ipa ted 6 Moder at e 5 Moder at e 5 M oder a te 5 Modera t e 5.8
Ca pa ci ty (1 2 b) Hig h 8 Hig h 8 Hi gh 8 Hi g h 8 Ant ic ipa ted 6 L ow 2 Moder at e 5 L ow 2 Modera t e 4.6
Ch an ge Activ ity - E xpo su re incre ase d d ue to ag e ofha rdwa re/s oftwa re, p atc h leve ls , co mp atib il i ty, n u mb er/typ e o fcha ng es (1 3)
Adver sar ialAcciden talSt ruct ur al
Moder at e 5 Hig h 8 Hi gh 8 Hi g h 7 Ant ic ipa ted 6 Moder at e 5 Moder at e 5 M oder a te 5 Modera t e 5.6
Los s Of F aci l itie s
Na tura l (e arth qu ak e/floo d/e tc.) (1 4) Envir onm ent al Moder at e 5 L ow 2 L ow 2 Low 3 Pos si ble 2 L ow 2 Moder at e 5 L ow 2 Low 2.8
No n -Na tura l/Ma n Ma de (fi re/ electrica l/e tc.) (1 5)
Adver sar ialAcciden talSt ruct ur alEnvir onm ent al
Moder at e 5 L ow 2 M oder a te 5 Modera t e 4 Pos si ble 2 L ow 2 Moder at e 5 L ow 2 Low 3
G-4
L ik eliho o dS ucc ee ds
9
T ask 2-4
OV
ERA
LLLI
KEL
IHO
OD
Com
bin
atio
nof
Ave
rag
eC
har
act
eris
tics
(3/4
/5-c
olu
mn
k),
Re
lev
ance
(6-
colu
mn
M),
Like
liho
od
of
Init
iati
on
(7-
colu
mn
O),
ITR
isk
Ass
ess
me
nt/
Con
tro
lDri
ver
s(8
-co
lum
nQ
),&
Like
liho
odo
fSu
cce
ss(9
-col
um
nS)
Ov era ll C ha racte ristics
T ask 2 -1
Rele van ce
C alcu latio nAve ra ge 3 ,4,& 5
Tas k 2-4
3 4D-4
Like lih o od OfInitiatio n
G2 /G3
Value Val ueTh rea t E ve ntTh rea t
S o urce
Vu ln e rab il ityP re disp os itio n(Ris k Driv er s)
( L ink To Ris kDr iver s ) Va lu e
Th rea t So urc e Ch ara cte ristics
D-5
Ta sk 2 -3Se e IT Risk As ses sme nt
D-22
D-3T as k 2-1
6E -4
7Risk T em pla te - F
85
Ove rallLike lih oo d
1 0G-5
Consolidate and “cut & paste” averageresponse into Core Risk Engine “input pages”– lose anonymity and detail by demographics,requires increased meetings.
Data capture/entry:
60© Thomas E. Festing – 2013
On-line/Internet based Tool supports sendingsurvey to participants (e.g. Survey Monkey).
Tool supports collection of data.
Position - Please identifyyour position within theorganization / functional
area.
Project - Please identifywhat type of direct
participation you havehad with the project.
Years of Service withand/or using Agency
systems.
OrganizationalOversight
Executive Management None 11-15 years Somewhat Confident Confident Somewhat ConfidentStaff None 1-5 years Don't Know Somewhat Confident Somewhat ConfidentSupervisor None Greater than 15 years Somewhat Confident Confident ConfidentAdministrator SME - representing Business/IT/Vendor6-10 yearsSupervisor SME - representing Business/IT/Vendor6-10 years Not Confident Not Confident Somewhat ConfidentStaff Member of the "Project Team"Greater than 15 years Somewhat Confident Confident Somewhat ConfidentSupervisor None 1-5 years Somewhat Confident Somewhat Confident Not ConfidentExecutive Management Member of the "Project Team"Greater than 15 years Confident Confident Somewhat ConfidentAdministrator SME - representing Business/IT/VendorGreater than 15 years Somewhat Confident Not Confident Not Confident
Tool supports downloading rawdata into excel for data mining.
Process – Data Collection
Tool tracks who hasn’t respondedand will send e-mail “reminders”.
61© Thomas E. Festing – 2013
Position - Please identifyyour position within theorganization / functional
area.
Project - Please identifywhat type of direct
participation you havehad with the project.
Years of Service withand/or using Agency
systems.
OrganizationalOversight
Executive Management None 11-15 years Somewhat Confident Confident Somewhat ConfidentStaff None 1-5 years Don't Know Somewhat Confident Somewhat ConfidentSupervisor None Greater than 15 years Somewhat Confident Confident ConfidentAdministrator SME - representing Business/IT/Vendor6-10 yearsSupervisor SME - representing Business/IT/Vendor6-10 years Not Confident Not Confident Somewhat ConfidentStaff Member of the "Project Team"Greater than 15 years Somewhat Confident Confident Somewhat ConfidentSupervisor None 1-5 years Somewhat Confident Somewhat Confident Not ConfidentExecutive Management Member of the "Project Team"Greater than 15 years Confident Confident Somewhat ConfidentAdministrator SME - representing Business/IT/VendorGreater than 15 years Somewhat Confident Not Confident Not Confident
ANALYZE IT
Back To Step 3: Process - Survey Framework
62© Thomas E. Festing – 2013
Process – Analysis
Position - Please identifyyour position within theorganization / functional
area.
Project - Please identifywhat type of direct
participation you havehad with the project.
Years of Service withand/or using Agency
systems.
OrganizationalOversight
Executive Management None 11-15 years Somewhat Confident Confident Somewhat ConfidentStaff None 1-5 years Don't Know Somewhat Confident Somewhat ConfidentSupervisor None Greater than 15 years Somewhat Confident Confident ConfidentAdministrator SME - representing Business/IT/Vendor6-10 yearsSupervisor SME - representing Business/IT/Vendor6-10 years Not Confident Not Confident Somewhat ConfidentStaff Member of the "Project Team"Greater than 15 years Somewhat Confident Confident Somewhat ConfidentSupervisor None 1-5 years Somewhat Confident Somewhat Confident Not ConfidentExecutive Management Member of the "Project Team"Greater than 15 years Confident Confident Somewhat ConfidentAdministrator SME - representing Business/IT/VendorGreater than 15 years Somewhat Confident Not Confident Not Confident
Downloaded Raw Data andparticipants free form textcomments into the “Tool’s” PivotTables.
Busine ss Uni t
Functio nal Are a To ta l
Le ga l/Inte rna l Aud it
Functio nal AreaTota l
Exe cutive Mana ge me nt
Functiona l Are a To tal Stake holderLea de rship
Functiona l Area Total
Ve ndo r/Consul tan t
Functio nal Are a To ta l
Staff Supervi sor Ad ministra tor ExecutiveMa nag ement
Staff Supe rvisor Ad ministrator Executi ve Ma nag ement Exe cutive Mana ge me nt Staff Ad ministra tor Staff Sup ervisor
3.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 1.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.002.00 10.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 2.00 2.00 1.00 4.00 5.00 1.00 1.00 2.002.00 10.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 4.00 5.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 10.00 2.00 1.00 16.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 4.00 5.00 1.00 1.00 2.00
3.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
3.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.003.00 9.00 2.00 1.00 15.00 2.00 2.00 1.00 2.00 7.00 3.00 3.00 1.00 5.00 6.00 1.00 1.00 2.00
63© Thomas E. Festing – 2013
• Objective of Scatter Diagrams:
Example Deliverables - “Scatter Diagram”
– Results posted in the order received:• Green = “exceeding confident” and “confident”• Red = “somewhat confident” & “not confident”• Blue = “Don’t Know ”• Yellow/White = No Response/Blank
– QA of “raw” data collection for reasonableness andanomalies:
• By Function (e.g. Business/IT/etc.)• By Position (e.g. Manager/Staff/etc.)
• Overall Observations:– Determine distribution of responses– Identify/minimal “same response” to questions (column)– Identify concentrations /trending of “confident”, “not
confident”, or “don’t know” combinations– Limited number of unanswered questions
• Objective of Scatter Diagrams:
64© Thomas E. Festing – 2013
Assists with assessing overall dataquality.
Example Deliverables - “Scatter Diagram”
Overall Observations Highlights:• Good distribution of responses• Minimal “same response” or
concentrations of “4” & “5” or“1” & “2” combinations
• Limited number of unansweredquestions – as an example:– 95%+ questions answered– 75% of unanswered question
linked to 3 respondents
© Thomas E. Festing – 2013 65
Scatter By "Don't Know" Question # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
Functional Area - Pleaseidentify what functional areayou belong to.
Position - Please identify yourposition within the organization/ functional area.
OrganizationalOversight -CORE
Organizational
Organizational
Organizational
Organizational
Organizational
Organizational
ProjectRiskMan
ApplicationTec
ApplicationTec
ProjectTeam -
ProjectT eam -
Resources -Peo
Resources -Man
Organizational
Organizational
Organizational
VendorManage
VendorManage
Communication
RequirementDefi
Business/Process
Business/Process
Business/Process
SystemDevelop
SystemDevelop
SystemDevelop
SystemDevelop
SystemDevelop
SystemDevelop
SystemDevelop
SystemDevelop
DataManagement
DataManagement
DataManagement
DataManagement
LegacyProcess/
Implementation
VendorT ransitio
T raining(Busines
Service(Help)
BusinessContinuit
PostEvaluation -
RoleBasedProfi
SecurityAdministr
SecurityAdministrAdministration Leadership Executive Management 1 2 2 2.5 2 2 2 2 2.5 2.5
Administration Leadership Executive Management 4 1 2 1 2 4 4 2 4 4 1 2 2 1 2 4 5 4 5 4 2 5 2 2 4 2 4 4 4 4 4 4 4 4 4 4 1 2 4 4 2 4 4 4 4 4Business Executive Management 5 5 4 4 1 5 5 4 2.5 2.5 2 4 5 5 4 4 4 2.5 2 2.5 2 4 2 4 2.5 5 2 2.5 4 2.5 4 2.5 2.5 2.5 2 2 2.5 2.5 2 2.5 2 2.5 4 2.5 2.5Legal/Finance/Compliance/AuditExecutive Management 4 4 1 4 4 4 2 4 2.5 2.5 2.5 2.5 2 4 4 2.5 4 5 5 2.5 2 2.5 2.5 2.5 1 2.5 5 2.5 4 2.5 2.5 4 2.5 2.5 2.5 2.5 4Business Executive Management 4 2 2 2 4 4 4 2 4 2 2 4 2 4 2.5 2 4 2 4 2 2 4 2 2 2.5 2 4 2 2 2.5 2.5 2 2.5 2.5 2 2 2 2 2.5 2 2 2 2 4 4 4Administration Leadership Executive Management 2 2 2 2 4 5 5 1 2.5 2.5 4 4 2 1 5 2.5 4 2.5 4 2.5 4 4 2 2 2.5 2.5 2.5 2.5 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Administration Leadership Executive Management 4 4 2.5 4 5 2 5 4 2.5 2.5 4 4 2.5 5 5 2.5 5 2.5 4 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 4 2.5 2.5 2.5 2.5 2.5 2.5Legal/Finance/Compliance/AuditExecutive Management 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1Administration Leadership Executive Management 5 4 5 2 2.5 4 4 4 5 4 4 4 4 5 5 4 5 4 5 4 2 2 4 4 4 4 5 4 4 4 5 2 5 2 4 4 5 5 2 4 2 2 2 5 5 5Administration Leadership Executive Management 5 4 4 4 2 5 5 4 5 5 4 5 5 4 5 5 5 5 5 4 4 4 5 5 4 5 5 4 5 5 4 5 5 5 5 5 4 4 4 4 4 5 5 5 5 5Administration Leadership Executive Management 4 4 4 4 4 4 4 4 2 4 4 4 4 4 5 5 5 5 4 4 4 4 4 4 4 5 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 4Information Technology Manager/Director 2 5 4 5 2 4 5 5 4 5 5 5 4 5 5 4 4 4 4 5 4 4 4 4 5 5 5 5 5 5 4 5 4 2 5 4 4 4 4 4 4 4 4 4 4 4Information Technology Manager/Director 2 2 4 2 1 4 4 5 4 5 5 2 4 5 2 4 5 5 4 2 4 2 5 2 5 5 5 4 5 2.5 4 5 1 1 5 1 4 2.5 5 1 5 5 5Legal/Finance/Compliance/AuditManager/Director 4 4 5 5 5 4 2 4 4 4 4 4 4 5 4 4 4 4 4 5 4 4 4 4 4 5 5 4 4 5 4 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4Legal/Finance/Compliance/AuditManager/Director 2 2 2 2 2 4 4 2 4 4 2 4 2 1 4 4 4 2 4 2.5 1 2 2 2 4 1 4 2.5 4 2 2 4 4 2.5 4 4 1 4 2.5 4 2 2 4 4 2Legal/Finance/Compliance/AuditManager/Director 2.5 2 2.5 2.5 2.5 2.5 2.5 5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2 2 2.5 4 4 2.5 2.5 4 4 4 2.5 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Business Manager/Director 4 4 2 4 2 5 5 2 5 5 4 4 5 5 5 4 4 5 5 4 5 4 4 4 5 5 5 5 5 5 5 5 5 4 4 2.5 4 4 4 4 4 4 4 5 4 4Legal/Finance/Compliance/AuditManager/Director 1 4 1 4 2.5 4 4 2 2.5 2.5 2.5 2.5 2.5 2.5 4 2.5 4 2.5 2.5 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Legal/Finance/Compliance/AuditManager/Director 4 2 1 2 2 2 4 2 2 2 2 4 2 4 4 4 4 1 2 4 2 2 2 2 2 4 4 4 4 2 2 4 4 2 4 2 4 1 2 4 2 2 2 2 4 2Business Manager/Director 2.5 4 4 4 2.5 5 4 4 4 4 5 5 5 4 4 2.5 2.5 4 4 4 4 4 2 4 4 4 4 2 4 5 4 4 4 4 4 4 4 4 2.5 2.5 4 4 4 4Business Manager/Director 5 4 5 5 2 5 5 4 4 4 5 5 5 5 4 5 5 5 5 5 5 5 4 4 4 5 5 5 5 4 4 5 5 4 5 5 5 5 4 4 4 4 5 5 5 5Information Technology Manager/Director 2 5 2 4 2 2 5 2 4 4 5 2 4 4 5 4 4 5 5 4 2 4 2 4 4 2 5 4 4 2 4 5 4 5 4 4 2 5 2 2 2 4 2 4 5 5Administration Leadership Manager/Director 5 4 5 5 2 2 5 4 4 5 5 5 5 2 5 5 2 4 5 5 4 5 4 4 2.5 5 5 2.5 5 2.5 4 4 2.5 2.5 2.5 2.5 5 4 4 2.5 2.5 4 2.5 5 2.5 2.5Business Manager/Director 4 5 2 2 4 4 4 4 4 2 4 2 1 1 5 4 4 2 5 4 2 2 2 2 4 5 4 2.5 4 4 2 2 4 1 2 2 1 4 4 2.5 4 2 2 2 2 2Business Manager/Director 2 4 2.5 4 2 2 4 4 4 2.5 4 2 2 2 4 2.5 2.5 2.5 4 4 4 4 2.5 2.5 2 2 2.5 2.5 2.5 2 2.5 4 2.5 2.5 2.5 2.5 2 2.5 2.5 2 2.5 2.5 2 4 4 4Business Manager/Director 4 4 2 4 5 5 5 4Legal/Finance/Compliance/AuditManager/Director 4 2 2 4 5 2 2 4 2 4 4 2 2 1 4 4 2 4 2 2 2 4 1 2 2 1 4 4 2 2 1 4 4 2 2 2 2 2 2 4 2 2 4 2 2 2Information Technology Manager/Director 2 4 4 4 2 4 5 4 5 4 5 4 4 2 2 2 2 4 5 4 2 4 2.5 4 5 5 5 5 5 5 4 5 5 5 5 5 2 5 5 5 4 5 5 5 5 5Information Technology Manager/Director 2 4 1 2 4 4 4 5 2.5 2.5 4 2 2.5 2 2 5 5 2 4 4 4 4 2.5 4 4 5 5 4 4 4 2.5 5 4 1 4 1 1 4 4 4 2 4 2 4 5 5Business Manager/Director 1 4 1 1 1 1 1 2 1 1 1 4 1 2 1 1 1 1 1 1 4 5 5 2.5 5 5 5 4 5 4 5 1 1 1 1 1 2 1 1 1Business Manager/Director 5 5 4 5 2.5 5 4 4 2.5 2.5 5 4 4 5 4 5 5 2.5 2.5 2.5 2.5 4 4 4 2.5 5 2.5 2.5 2.5 5 2.5 5 2.5 5 2.5 2.5 4 2.5 2.5 5 2.5 5 2.5 5 2.5Information Technology Manager/Director 2 2 2 2 4 5 4 2 2 2 4 4 2 4 4 4 5 2 2 2 2 2 2 2 2 2 5 2 2 2 1 2 4 2 2 2 2 2 2 5 1 2 2 4 4 4Information Technology Manager/Director 4 4 2 4 2 4 5 4 4 4 5 4 4 4 4 4 4 4 4 4 2 4 4 4 4 4 4 4 4 4 2 2 4 4 5 2 2 2 2 4 4 4 4 5 5 5Business Manager/Director 4 4 4 4 2 5 5 4 2.5 2.5 4 5 4 2 2 2 5 2.5 4 5 4 5 4 4 2.5 4 4 2.5 4 4 2.5 4 2.5 2.5 2.5 2.5 2 2.5 2.5 2.5 2.5 2.5 2.5 4 4 4Business Manager/Director 2 1 2.5 2.5 4 4 2 2.5 2.5 4 4 2.5 4 2 2.5 4 2 2.5 2.5 2.5 2.5 2 2.5 2.5 2.5 2 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2 2 2.5 2.5 2.5 2.5 2.5 2.5Legal/Finance/Compliance/AuditManager/Director 4 2 4 4 4 5 4 4 5 5 2 2 4 4 4 4 4 5 5 4 2 4 4 4 4 4 5 4 4 4 4 4 4 4 4 4 2 4 4 4 4 4 4 4 4 4Business Manager/Director 4 2 2.5 2.5 4 4 4 4 4 4 4 4 4 2 2 2 4 2.5 2.5 4 4 5 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Legal/Finance/Compliance/AuditSupervisor 2 4 2 4 5 4 4 5 5 4 2 4 4 4 4 4 2.5 2.5 4 4 2.5 2 4 2.5 5 5 2.5 2.5 4 2.5 2.5 2.5 2.5 2.5 2.5 4 2.5 2.5 4 4 4 4 4 5 4Information Technology Supervisor 4 4 2 4 2 4 4 4 4 2 2 2 1 4 4 2 4 4 4 4 2.5 2.5 2.5 2.5 2 4 4 2 2 2.5 2.5 4 4 1 2 2.5 2 2 2.5 1 2 4 4 4 4Legal/Finance/Compliance/AuditSupervisor 4 4 2 2 4 2 4 2 5 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 4 4 4 2 2 4 2 4 5 5Business Supervisor 2 1 1 1 4 1 1 1 1 1 4 4 2 1 5 1 1 2.5 2.5 2 2 2 1 4 4 5 5 5 5 5 4 5 5 5 5 5 2 2 2 1 4 2 2 4 4 4Administration Leadership Supervisor 4 2 4 2.5 4 4 4 2 2 2 4 2 4 2 2.5 2.5 2 2.5 2.5 2 2.5 4 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Information Technology Supervisor 4 5 4 2 2 2 4 4 2 4 4 4 4 4 2 2 1 4 2 2 2 1 2 5 4 2 2 4 4 4 2 5 4 2 2 2 4 2 2 2 2 2 4 4 4Business Supervisor 4 4 2.5 4 5 5 4 4 2.5 2.5 2.5 5 2.5 4 4 2.5 4 4 4 4 4 5 5 5 2.5 5 5 2.5 5 2.5 4 2.5 4 2.5 2.5 2.5 5 5 2.5 5 5 2.5 5 5 5 5Administration Leadership Supervisor 2 4 4 2 2 5 5 4 4 5 2 2 5 2 5 5 4 5 2.5 2 4 5 5 2 2.5 5 5 4 4 2.5 4 2 2.5 2.5 2.5 2.5 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Business Supervisor 4 4 4 4 4 4 2.5 4 5 2 4 4 4 2 5 4 4 2.5 2.5 2 4 4 2 2.5 2.5 4 2.5 2.5 2.5 4 2.5 2.5 2.5 2.5 2.5 4 4 4 4 4 2 4 4 4 4 4Business Supervisor 2 2 2 2 1 2 2 2 2 2 1 2 2 1 2 2 1 2 2 4 2 2 1 1 1 5 5 2 2 5 2.5 4 5 5 5 5 1 1 5 4 4 5 1 5 5 5Business Supervisor 4 4 4 2 2 5 4 4 5 5 4 4 4 2 5 4 5 4 4 4 5 4 2 2.5 2.5 5 2.5 2.5 5 5 4 5 5 4 5 4 2 4 4 5 4 5 4 5 5 5Administration Leadership Supervisor 2 2 4 2 2 1 2 1 1 1 2 4 2 1 2 2 1 1 4 2 4 4 2 4 2 1 2 2 4 2 4 4 4 4 4 4 1 4 4 4 2 4 4 4Business Supervisor 4 4 4 4 2.5 4 5 4 4 4 4 4 2.5 2.5 4 4 4 4 4 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 4 4 4Information Technology Supervisor 4 4 4 2 4 2 2 4 4 4 2 2 4 4 4 2 4 4 5 4 2 4 4 2 2 4 5 4 4 4 2 4 4 2 2 2 4 4 2.5 4 4 2.5 4 4 4 4Information Technology Supervisor 4 5 5 4 4 4 4 4 4 5 4 4 2 4 4 4 2 2 2 2 2 4 4 2 2 2 4 4 2 4 2 2 4 4 4 4 4 4 2 2 4 4 4 4 5 5Legal/Finance/Compliance/AuditSupervisor 4 2 1 4 4 2 4 2 2 2 2 2 1 4 2 4 1 4 4 2 1 2 2 2 4 2 1 2 2 1 4 1 1 2 2 2 4 1 2 2 2 1 1 2 1Business Supervisor 4 4 4 2 4 4 2 4 4 2.5 5 5 4 2 2.5 4 5 4 4 2 4 4 4 5 2.5 4 5 2 5 5 4 5 5 4 5 4 4 2 4 4 2 5 2 5 5 5Business Supervisor 4 2 4 2 2 2 4 2 2 2 2 4 4 1 2.5 2 2 2.5 2.5 2 4 2.5 2.5 2.5 2.5 4 4 4 4 2.5 2.5 4 4 2.5 2.5 2 1 2 2.5 2 2.5 2.5 2.5 4 4 4Legal/Finance/Compliance/AuditSupervisor 1 2 1 1 1 1 1 1 2 1 2 1 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1Business Supervisor 4 2 4 2 4 4 2 2 2 2 4 4 4 5 4 2 4 4 4 4 4 5 5 5 4 5 5 4 5 5 4 5 4 4 4 5 4 5 5 2 4 4 5 5 5 5Business Supervisor 5 5 5 5 5 5 5 5 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 4 5 5 5 4 5 4 4 4Business Supervisor 4 4 4 4 2 4 2 4 4 4 5 5 4 2 5 4 4 4 5 4 2 4 2 4 4 5 5 2.5 5 5 5 5 2.5 2.5 2.5 2.5 4 5 4 5 5 4 4 5 5 5Business Supervisor 4 2 4 4 4 2 4 4 4 5 4 4 4 4 4 4 5 4 4 4 4 2 4 4 4 4 5 4 4 4 4 4 4 4 4 2 4 4 4 4 4 5 4 5 4 4Business Supervisor 5 4 4 4 5 5 1 4 1 1 2 4 2 1 4 4 1 5 5 4 1 1 1 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 1 2.5 2.5 2.5 2.5 2.5 2.5Information Technology Staff 2 2 2 4 2 5 4 2 2 2 2 4 2 1 4 4 2 2 2 2 2 2 1 1 2 2 2 2 4 4 2 5 2 1 4 1 1 2 1 2 1 1 2 4 4 4Administration Leadership Staff 5 5 5 5 5 5 5 2.5 2.5 2.5 5 5 5 5 2.5 2.5 5 2.5 2.5 2.5 5 2.5 2.5 2.5 2.5 5 5 2.5 5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 5 5 5 5 2.5 2.5 2.5 5 4Information Technology Staff 4 4 4 2 4 4 4 2 2 4 4 5 2 2 5 4 4 2 4 2 4 5 4 5 2 4 2 4 2 2 2 4 1 1 2 1 4 5 1 4 4 4 4 4 5 5Legal/Finance/Compliance/AuditStaff 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 1 1 1 1 1 1 1 2 2 1 1 1 1 1 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 1 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Legal/Finance/Compliance/AuditStaff 4 1 4 2 1 1 2 2 2 2 1 1 2 1 5 2 1 2 1 1 4 5 4 2 2 4 4 4 4 4 4 4 4 4 4 4 2 2 2 1 2 1 2 4 2 2Legal/Finance/Compliance/AuditStaff 4 4 2 4 5 2 5 5 5 5 4 4 2 4 5 5 5 2.5 4 5 4 5 4 5 5 5 5 5 5 5 5 5 5 5 5 4 5 5 5 4 5 4 4 5 5Information Technology Staff 4 4 2.5 4 4 4 5 4 4 4 4 4 5 5 4 4 5 4 5 5 2.5 2.5 2.5 2.5 5 5 5 5 4 4 5 5 4 4 5 4 4 4 4 4 4 5 4 4 5 5Information Technology Staff 2 2 1 1 1 1 2 2 1 1 2 2 1 1 2 1 1 2 1 2 1 1 1 1 2 2 2 1 1 1 1 2 1 1 1 2 2 2 1 1 1 1 1 2 2 1Administration Leadership Staff 4 4 2 2 4 4 2 2 2.5 2.5 4 4 2.5 1 4 2.5 2.5 2.5 2.5 4 1 4 2 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 1 2.5 1 2.5 2.5 2.5 1 2.5 2.5 2.5Legal/Finance/Compliance/AuditStaff 2 4 2 4 2 2 4 4 2.5 2.5 2 2 4 2 2 4 4 4 4 4 4 2 2 4 4 4 4 4 4 4 4 4 2.5 4 2 2 4 2 2 4 2 2.5 4 2 4 4Business Staff 4 4 4 2 2 4 4 2 4 4 2.5 2.5 2.5 2.5 2 4 4 4 4 4 4 4 4 4 4 4 5 5 5 4 4 4 4 4 4 4 4 4 2.5 5 5 2.5 2.5 4 5 5Legal/Finance/Compliance/AuditStaff 5 5 2 4 4 5 4 4 5 5 5 4 4 4 2 2 5 4 5 2.5 5 4 4 2.5 5 5 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2 2.5 1 2.5 2.5 2.5 2.5 2.5 2.5Legal/Finance/Compliance/AuditStaff 2 4 2 2 2 2 2 4 4 4 4 4 2 2 2.5 2 4 2 2 2.5 2 2 2 4 4 4 2 4 4 4 1 4 2 2 4 4 2 2 2 4 2 4 4 4 2 2Business Staff 4 2 2.5 2.5 4 4 4 2 2.5 2.5 2.5 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Legal/Finance/Compliance/AuditStaff 2 2 2 1 2 1 1 2 2 2 2 1 1 1 1 1 1 2.5 2.5 1 2.5 1 1 2 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Information Technology Staff 2 2 2 2 2 2 2 2 4 2 1 4 2.5 1 2.5 1 1 2.5 2.5 1 2.5 2.5 2.5 2.5 1 2 2.5 2.5 2 4 1 2 4 1 4 2 1 1 1 1 1 1 1 4 4 4Information Technology Staff 2 4 2 2 2 2 2 2 2 2 4 2 1 1 4 2 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 2 2 2 2Legal/Finance/Compliance/AuditStaff 4 4 4 2 5 4 4 4 5 4 4 4 4 2.5 2.5 2 2.5 2.5 4 2.5 4 4 2.5 2.5 2 5 2.5 5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Business Staff 4 4 4 5 5 5 4 5 4 2.5 4 4 4 5 5 4 5 2 2.5 4 2.5 4 4 4 2.5 4 5 5 5 2.5 2.5 2.5 5 2.5 2.5 2.5 4 4 4 4 5 4 4 4 4 4Business Staff 2 2 4 4 4 2 2 2 4 2.5 2 2 2 2 2 2 2 2.5 2.5 2 2 2 2 2 2.5 2 2 2.5 2 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2 2 2 2 2 2 2 2 2 2Information Technology Staff 2.5 5 2.5 2.5 2.5 2.5 1 5 2.5 2.5 4 2 2 2 1 1 1 2.5 2.5 2 2 2.5 2.5 2.5 4 4 4 4 2 4 2 4 4 4 4 4 4 4 1 4 4 2 2 4 4 4Business Staff 4 2 2 2 4 2 4 2 4 4 5 4 4 5 2.5 2 5 4 4 2 4 4 5 4 4 5 4 2 5 5 2 5 4 4 5 4 4 2 2 4 4 2 2 4 4 4Business Staff 5 4 4 2 4 4 5 2.5 2.5 2.5 4 5 4 4 5 4 5 2.5 2.5 2 4 4 4 4 2.5 4 4 2 4 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2 2.5 2.5 4 2.5 2.5 4 4 4Business Staff 2 4 2 4 4 2 1 5 2.5 2.5 4 2 4 4 2 2 2 2 2 2 2 4 4 4 2 4 4 4 4 2 2 2 4 4 5 5 4 4 2.5 4 4 4 4 5 2 2Business Staff 4 2 2 2 2 4 2 4 4 4 4 4 2 2 2 2 2 4 4 4 4 4 4 4 2 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4Information Technology Staff 4 4 2 2 2 2 5 2 4 1 4 4 4 2 5 4 2 2.5 4 4 2 2 2 2 4 5 5 4 4 4 2 2 2 2 2 2 4 4 2 2 2 2 2 2 2 2Legal/Finance/Compliance/AuditStaff 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4Business Staff 4 2 1 5 4 1 2 4 5 4 4 2 1 1 2 4 1 2.5 2.5 4 1 1 1 1 2 2 5 2.5 2.5 2.5 2 2.5 2.5 2.5 2.5 2.5 2.5 4 4 4 4 2.5 2 2 5 5Business Staff 4 2 2 2 1 4 1 4 2.5 2.5 4 4 2 1 2 2.5 1 2.5 2.5 1 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 4 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5Business Staff 2 2 2 2 1 2 2 2 2.5 2.5 4 4 4 2 4 2 2 2 4 4 2 2 4 4 4 4 2.5 2.5 2.5 4 4 4 4 4 2.5 2.5 4 4 2 4 2.5 2.5 4 4 4 4Information Technology Staff 4 4 1 4 5 4 4 4 2.5 2.5Information Technology Staff 2 4 2 4 2 4 4 4 4 4 4 4 2 2 2 4 4 5 5 4 2 4 2.5 2.5 4 2 5 5 4 2 2.5 4 2 2 5 2 2 2 2 2 2.5 5 4 2.5 5 5Legal/Finance/Compliance/AuditStaff 2 2 1 2 4 4 4 2 2.5 2.5 2 2 2 1 2 2 1 2 2.5 1 2.5 1 2.5 2 2.5 2.5 2.5 2.5 2 2.5 2.5 1 2.5 2.5 2.5 2.5 1 1 2 2 2.5 2.5 2.5 2 2.5 2.5Information Technology Staff 4 4 4 4 4 2 4 2 4 1 4 4 4 4 4 2.5 2.5 4 4 4 2.5 2.5 2.5 2.5 4 5 4 2 4 4 2 2 2.5 2.5 2.5 4 2.5 2.5 4 2.5 1 2 4 2.5 2.5Information Technology Staff 4 4 2 4 2 4 4 4 4 4 2 4 2 4 4 5 4 5 4 2 4 4 4 4 2 5 4 4 4 4 4 4 2 2 4 2 2 4 4 4 4 4 4 4 4
– “Not aware of the generalgovernance area (suggestingcommunication, training &awareness).
Scatter Diagram - By “Don’t Know”
• Without detailed analysis – provides“Visual Indications” of highestConcentration:
– Area - System Development &Data Management
– Position - More common amongStaff & Executives
• “Don’t Know” is an acceptableresponse and may be an indicator of:
– Area may not be consistentlyperformed (suggesting strongerproject management oversight).
© Thomas E. Festing – 2013 66
Concentration Of “Don’t Know” – “Example”• Identifies the questions that have the highest “Don’t Know”
responses.
Data Management (#12)(Question 15)
1. Data Privacy Security reviews assess data confidentiality & read/write/delete controls over data in application, storage, transit, and removable media.2. Test/Production Test data strategy has been developed and approved to address testing and privacy requirements (data obfuscation/masking).3. Back-up / Historical Data
RetentionBack-up /retention strategies are reviewed by Business, Legal/Compliance, and Audit – including new/legacy/conversion data (Disk & Tape).
4. Conversion Legacy system data is inventoried and conversion rules developed to help ensure accuracy and completeness for the new systems.
• In this example – the greatest concentration were in the DataManagement areas.
• You can then mine further to see if there is focus in certainquestions:
– System Development - “scope creep” & “inventory tooperations / DR”
– Implementation/Go Live – “service (Help) desk” & “DR”• Can identify concentration by positions … e.g. in the
example - “Senior Management/Executive”:
– Expected more “green”
– Identify how may respondents have a an excessiveamount of “Don’t Know”
67© Thomas E. Festing – 2013
Survey Responses – Example Stratification OfSurvey Questions By Position (Staff)
Link strong and weakresponses to specificquestions and findout “why”!
Overall ByQuestionPosition
68© Thomas E. Festing – 2013
Example Deliverables - “Scatter Diagram”
The ability to “mine” responses at aquestion level can often be invaluable!
69© Thomas E. Festing – 2013
Survey Responses – By Area
The ability to “mine”responses at an“area/question” levelprovides the ability to focuson specific areas of strengthor weakness!
Areas like ….Vendor Transition,Training, Service Desk …
Areas like ….Vendor Management,Communication Strategy …
70© Thomas E. Festing – 2013
TELL ALL - RECAP
71© Thomas E. Festing – 2013
Tell All – Recap Of Previous Slides
72© Thomas E. Festing – 2013
Yes …
Life Cycle
73© Thomas E. Festing – 2013
Cost:Not identifying budgetvariances/overruns early.Changes in scope – with relatedimpact to cost vs. budget.
Functionality:Not capturing businessfunctionality expectations.QualityInfrequent project status meetings.Misalignment with businessstrategy
Late!
Remember Gartner’s areas of interest …
Project management governance areasmay be adjusted (e.g. communication/training /metrics /etc.).
Effectiveness can be assessed as part ofperiodic “soundings”/ “drive-by”.
Sometimes a “course adjustment” isneeded to ensure the project continues tomeet the organizations objectives/strategy.
A course adjustment can be 1800
Project Risk Assessment – Life Cycle
74© Thomas E. Festing – 2013
Project Risk Assessment – Life Cycle
Confidence Level
Impa
ctPr
ojec
tSuc
cess
H i g h
L o w
NotConfident
ExtremelyConfident
910 5
4
78 1
221
22 1
6
151
4
17
18
19
1
2
311
13
6
20
012345
3.64.7
3.8
2.5 2.6
Initial
Implement0
5
Initial
Implement0
5
Initial
Implement0
5
75© Thomas E. Festing – 2013
Results from Project Risk Assessments link toother “review/assessment” areas:
IT Governance
Cont
rolE
ffect
iven
ess
Regu
lato
ry
Effic
ienc
y
Avai
labi
lity
Risk Assessment
Audit Plan/Scope
OptimizedRisk Environment
Governance
Strategy
Enterprise Risk Mgmt
Financial RiskAssessments
Business Objective/Approach
IT RiskAssessments
Change ActivitySecurity Management (Preventative Controls)
Monitoring (Detective Controls)
Data Management Recoverability/Availability
Governance (Directive Controls)
Business Integrity
ITIL COBIT
Execute
Identify Threat
IdentifyVulnerabilities/ Predispos ing
Conditions
DetermineLikelihood ofOccurrence
DetermineImpact
DetermineRisk
Com
mun
icat
ions
&In
form
atio
nSh
arin
g(D
eliv
erab
les)
Mai
ntai
ning
Risk
Asse
ssm
ent(
Life
Cycl
e)
Preparing For Risk Assessment (Understand)
NIST
Risk-BasedAudit Plan
76© Thomas E. Festing – 2013
The key benefit is …..
Information is “power” –harness it ….
The risk assessmenthelps frame“option”, “belief”,& “perception”….
“Perception” can become “reality” - somanage it ….
Get ahead of the waveand have a mechanismto start the“conversation” & don’tbe part of the 50%failed projects ….
If you don’t measure “it” – how willyou know if “it” improves ….© Thomas E. Festing – 2013
77© Thomas E. Festing – 2013
Post slogans,mandates, and buyt-shirts … & coffee
cups!
The “B” Option!
78© Thomas E. Festing – 2013
Questions/Comments
79© Thomas E. Festing – 2013
Tom FestingState Of Ohio
Office of Internal Audit
Columbus, Ohio 43215
THANK YOU