demystifying risk based audit methodology - … · demystifying risk‐based audit methodology...

Demystifying Risk‐Based Audit Methodology

Instructor Jay Ranade

CISA, CISM, CISSP, ISSAP, CBCP, CGEITPh. 1‐917‐971‐9786

[email protected]@aol.com

mailto:[email protected]


2www.technodyneuniversity.com 2www.technodyneuniversity.com

Instructor IntroductionJay, a certified CISA, CISM, CISSP, ISSAP, and CBCP, is an internationally renowned expert on computers, communications, disaster recovery, IT Security, and IT controls. He has written and published more than 35 IT‐related books on various subjects ranging from networks, security, operating systems, languages, and systems. He also has an imprint with McGraw‐Hill with more than 300 books called “Jay Ranade Series”. He has written and published articles for various computer magazines such as Byte, LAN Magazine, and Enterprise Systems Journal. The New York Times critically acclaimed his book called the “Best of Byte”. He is currently working on a number of books on various subjects such as IT Audit, IT Security, Business Continuity, and IT Risk Management.

Jay has consulted and worked for Global and Fortune 500 companies in the US and abroad including American International Group, Time Life, Merrill Lynch, Dreyfus/Mellon Bank, Johnson and Johnson, Unisys, McGraw‐Hill, Mobiltel Bulgaria, and Credit Suisse. He was a member of the ISACA International's Publications Committee (2005‐2007).

He teaches graduate‐level class on Information Security Management and Enterprise Risk Management at New York University. He also teaches accounting information systems, IT auditing, and internal auditing at St. John’s University.

Currently, he is the Director of Education for TechnoDyne University, a NJ, USA‐based professional educational services organization specializing in CARGOS (Controls, Auditing, Risk, Governance, Operations, and Security)

He is four times world champion in Arm Wrestling and two times world champion (2002 and 2003) in martial arts breaking. He has appeared on ESPN and ESPN2 numerous times.

• Contact information– [email protected]– [email protected]

– USA +1‐917‐971‐9786

• TechnoDyne University502 Valley Road, Suite 103Wayne, NJ 07470USA

Instructor Information



Why Risk Based?


Two aspects

• Risk‐based IT auditing has 2 aspects• Risk‐based audit planning

– Annual basis– CAE accountable for that– That results in individual audit engagements

• Risk‐based individual engagement planning– This is for each engagement– Engagement auditor does that


What is flow of Risk Management?

• Threat exploits a vulnerability• That’s damages an asset• That damages a business process

– From AIC perspective• That’s the risk• You put controls in place to mitigate the risk• Till (executive) management says it is an acceptable risk

• Note: auditors give assurance on that to the board


Limited Audit Resources

• Limited audit resources in any organization• Audit’s value proportional to cost element• Cost of controls must be less than protection to the asset– CAPEX and OPEX perspective

• Controls cost money and controls impede business

• Controls and value must be balanced


Who Develops Audit Plan?

• Accountability lies with CAE• Lack of IT knowledge with CAE• Those who have knowledge of IT lack knowledge of business processes

• Knowledge of business processes and IT is required for an audit plan

• It is revised on an annual basis• Remember that according to QARs, developing risk‐based plan is the weakest link in IAA


Organizational Factors

• Organizational factors affect audit plans• What is industry sector

– Financial, pharmaceutical, energy, health care

• Size of the organization• Business processes

– Unique for each organization

• Geographical locations

Four Steps in Risk‐based Audit Planning


1. Know the business

• What are organizational strategies• What are business objectives• What is the risk profile• How is operations structured• How does IT support business

– Support role– Is IT business enabler?


2. Know the Audit Universe

• Organization is collection of business processes• Applications support business processes• Infrastructure supports applications• IT Service Management supports infrastructure• Technologies constitute infrastructures• New projects are created continuously• All of them constitute the universe of audit• But you can not audit all……….


3. Perform risk assessment

• Why? Because you audit subset of audit universe

• Risk assessment should be a process• Risk factors help prioritize audit subjects• Remember‐ Risk is to business processes, not IT


4. Now, create Audit Plan

• Time to create audit engagements • Audit frequency determined by significance of business process

• Management can also add to engagement subjects– Assurance and consulting

• Business will validate the plan but audit committee has final say

1. Know the business


Each Organization is Different

• Different mission• Different goals• Different objectives• Different business models• Different market base• Different supply channels• Different product generation or service generation processes• Different delivery mechanisms

• So there is no cookie cutter approach to audit planning


What is the operating Environment?

• How business processes are structured to meet business objectives?

• Documents needed to understand– Mission statement– Vision statement– Strategic plans (4‐5 years horizon)– Annual business plans (one year)– Annual reports and supplements– Regulatory filings


Operating Environment cntd.

• Key processes contributing to success of the entity

• Remember that business processes differ– For each operating unit (BU)– For each support functions (IT)– For each entity‐level project (corporate)



• Operating units include core processes to meet objectives– Manufacturing, sales, distribution, services

• Support functions support core operational functions– Governance, compliance activities, HR, finance, cash management, treasury, procurement

– Oh yes, IT as well ;)



• Now you know the business processes• A business process has three components

– Manual– IT– Third party dependency



• Our focus here is IT – Business processes need IT application systems– Business application systems need infrastructure

• DB, OS, networks, facilities

– Infrastructure needs supporting IT processes• SDLC, operations, security, change management, problem management and many

more

– And lets throw in compliance activities• Regulatory, financial reporting

– They all have risks elements and contribute to risk‐based audit planning

22www.technodyneuniversity.com

2. Know the Audit Universe


Audit Universe

• What is audit universe– Finite and all encompassing collection of audit areas– Organizational entities– Locations related to business functions

• Most comprehensive list of audits if CAE had UNLIMITED RESOURCES and TIME ;)

• It is independent of risk assessment• There are 2 parallel universe ;)

– IT Audit universe and Business Audit universe


Audit Universe

• You have to know what is possible before you know what is feasible

• To know the audit universe you should know– Organization’s objectives– Business model– IT support model


Audit Universe‐ Business Model

• Business Model– Organization has business objectives– Operations units and support functions support those objectives

– And each of them has business processes• Business processes of sales units, marketing units, • Support functions have their own processes

– IT applications support these processes– Infrastructure supports applications


Audit Universe‐ Centralized vs. Decentralized• Centralized functions good for individual audits– Network audit, Security admn. audit, DBM audit, server admn. audit, help desk audit etc

– These functions, if centralized, are ideal for individual audits

– Audit team can cover a lot with a single audit– Single GC audit paves way for application audits on a platform


Audit Universe‐ Centralized vs. Decentralized• Centralized functions good for individual audits– Centralized audit functions reviewed at least annually

– In decentralized, each location is a different audit at GC level

– In a decentralized environment, with diverse technologies, multiple number of reviews needed


Audit Universe‐ IT Support Processes

• IT Support Processes– Infrastructure supported by support processes– ITIL is the leader in support processes– Change management, asset management, configuration management, release management, incident management, problem management

– Their effectiveness determines effectiveness of infrastructure to support applications

– Site audit is about “how they are followed and not effectiveness” because standard processes are always affective


Audit Universe‐ Audit Subject Areas?

• What are audit subject areas– Goal is to create most affective audits and coverage

– Business risk is NOT evaluated at this stage– Defining too small audit subject areas hinder audit effort

• Because there is admn. overhead for each audit– Large (long) audits can hinder client productivity


Audit Universe‐ Audit Subject Areas?

• What are audit subject areas– There is no right or wrong way, depends upon organization culture

– 2‐3 IT auditors for 3‐4 weeks is appropriate audit size for a subject area

– Need highly technical people for GC audit and general auditors for AC

– Management accountability consideration for grouping audit subjects

• Else resolution of audit issues becomes an issue– Scope of each audit must be defined properly


Audit Universe‐ Business Applications

• Business audit universe• IT audit universe• Business applications usually audited with business audit universe

• And GCs audited as a separate entity• ERP applications span many business processes

– So, they are given special consideration


Audit Universe‐ Now the RA

• Now you do risk assessment• So that you can create a subset of audit universe

• And that is the basis of annual audit plan

3. Perform Risk Assessment


Why Risk Assessment?

• Objectives are related to (this is what business wants from IT)– Confidentiality– Integrity– Availability– Reliability– Efficiency– Effectiveness– Compliance


What is RA Process?

• RA based on IT risk– Likelihood– Impact on the organization

• Audit based on if adequate controls in place to bring risk down to acceptable level

• Audit plan will be based on selecting a subset of universe based on RA


Perform Risk Ranking (RR)…..• Risk Ranking

– Impact and likelihood of occurrence

– Each risk may not be significant in the audit universe

– Weight differentiates relative importance over others

– E.g. for SOX compliance, an area directly related to accuracy of financial statements carries a higher weight vis‐à‐vis an area not directly related to financial statements


Three RR Techniques ‐ 1

• Direct probability estimates and expected loss functions. Or application of probability to asset value– Insurance industry uses this method, IT auditors do not

• Based on ALE = SLE x ARO – where SLE = AV x EF



• Observable or measurable factors to measure risk or class of risk

– Good for macro risk assessment not micro risk assessment

– This approach is OK if all auditable units are homogenous in the audit universe



• Weighted or sorted matrices. • Use of threats vs. component metrics

– Good for micro risk assessment– Weight of component taken into consideration

• All components are not equal– E.g. web‐facing applications carry more weight than non web‐facing application

– Used for application level risk assessment


Likelihood Scale

Likelihood ScaleH 3 High probability that the risk will occur.

M 2 Medium probability that the risk will occur

L 1 Low probability that the risk will occur


Impact Model

• Impact is to business process and not IT

• Different impact models for different organizations

• Impact can be financial, reputational, asset‐specific, client‐retention specific


Impact Model Scale

Impact Scale (Financial)H 3 The potential for material impact on the organization’s

earnings, assets, reputation, or stakeholders is high.M 2 The potential for material impact on the organization’s

earnings, assets, reputation, or stakeholders may be significant to the audit unit, but moderate in terms of thetotal organization.

L 1 The potential impact on the organization is minor in size or limited in scope.


RR Score Model‐ an example

• Refer to the spreadsheet –Financial Impact–Quality of internal controls–Changes in audit unit–Confidentiality, integrity, availability


Recommended Annual Cycle

Level Composite RiskScore Range

RecommendedAnnual Cycle

H 35–54 Every 1 to 2 years

M 20–34 Every 2 to 3 years

L 6–19 Every 3 to 5 years


ITG Frameworks

• COBIT– 4 domains and 34 processes– 218 Control objectives– CMM scale maturity level for each IT process– Good for large organizations

• ITIL v.3– Service strategy, design, transition, operations, continuous improvement

• ISO 27001/27002


Prioritizing Applications

• Business processes are supported by applications

• So, computer applications form the hub of risk‐based audit plan

• So, how do you prioritize applications?

3A. Prioritizing Applications


Examples of IT‐AC

• 3‐way match for AP– PO, vendor invoice, recipient of goods/services

• Depreciation of CAPEX is recorded in the correct period

• Received goods are accrued upon receipt only• SoD based on job function

– Governed by the principle of CARRE

• Goods procured with approved PO


IT‐AC Transaction Audits

• No one person should...– Initiate the transaction– Approve the transaction– Record the transaction– Reconcile balances– Handle assets– Review reports

• ∙ At least two sets of eyes needed


Facts about GCs and ACs

• IT‐ACs depend upon reliability of IT‐GCs• If GCs are malfunctioning, ACs don’t have any value– E.g. if change controls are weak, auditing internal processing of application has no value


Complex vs. Non‐Complex IT Environment

• Application controls differ in both• Complex IT Environment’s characteristics

– Source code is developed in‐house – Customized prepackaged software is adapted to organization’s needs

– Changes made to systems, databases, and applications– Production deployment of pre‐packaged applications, changes, and code


Complex vs. Less‐Complex IT Environment cntd.

• Less‐Complex IT Environment’s characteristics– Existing IT environment not changed much– Pre‐packaged software implementation with no major modifications in current year

– User‐configurable options that do not change application functioning

– Not many IT development projects


Complex vs. Non‐Complex IT Environment cntd.

• Less complex environment = more complex auditing – Because less complex environment does not have many inherent or configurable application controls for risk management

• So, degree of transactional or support application will drive scoping, implementation, effort level, and knowledge to perform application control review

• Auditing is about DE and OE of controls– Less complex environment does not have many controls


Manual Controls vs. IT‐ACs cntd.

• Risk Factor– Pre‐packaged application does not allow for code changes

– However, application controls within complex ERP (SAP, PeopleSoft) can be disabled w/o code change

– And Packaged applications are ALL parameter driven for control changes


AC and Risk Assessment


Financial Reporting Risks

• Summary (Very important)– Revenue is from Business Units– Payables, payroll, treasury is corporate– But risks are in business processes– Controls are in processes– Processes can span business units– IT‐applications support business processes– IT‐AC are in IT applications – Controls are also in underlying technology which is IT‐GC

– Control weakness in any of them can affect financial statement, so we do end‐to‐end audit


So, How do You do IT‐AC Risk Assessment?

• Define the universe of – Applications supporting processes– Databases supporting those applications (GC)– Technology supporting those applications (GC)

• Remember that 3 associated GCs directly affect applications– Change management– Logical security– Operational controls

• Remember that a table change in an application can eliminate controls thus bypassing change management controls for code changes



• Two methods to do risk assessment– Qualitative– Quantitative

• Qualitative is subjective– Risk (1= low impact, 5= high impact)– Controls (1= strong control, 5= weak control)– Determine risk and control weights for each of the 10 factors

• Quantitative is objective– Annual < $100,000 is risk level 1– Annual > $2,000,000 is risk level 5



• Qualitative is subjective (cntd.)– Calculate (risk factor rating x current risk weight) or (risk factor rating x current control weight) for all 10 risk factors for an application

– Add score for 10 risk factors– Calculate for all applications that need assessed– Sort results in descending order of composite score– Create audit plan based on higher composite risk score



• 10 factors and their weight for each application RA– Application contains primary controls (30)– DE of AC (20)– Complex or Less‐complex application (15)– Application deals with privacy issues (20)

• Depends if affected by EuroSOX, GLBA, HIPAA, Turnbull etc.

– Application supports more than one critical business processes (20)



• 10 factors and their weight for each application RA (cntd.)– Frequency of application change (15)– Complexity of application change (20)– Financial impact of change (25)– Overall effectiveness of IT‐GCs (25)– Audit history of controls (10)

• Previous audits discovered serious DE and OE deficiencies


Risk Factor Rating Rating Risk/Control Rank (1 to 5)

Risk Score Remarks

App. Has Primary controls 30 4 120

DE of AC 20 3 60

Complex or non‐complex application

15 3 45

Privacy Issues or confidentiality issues

20 1 20

Support > one critical application 20 4 80

Frequency of application change 15 1 15

Complexity of application change 20 4 80

Financial impact of changes 25 5 125

IT‐GC Effectiveness 25 1 25

Audit History of controls 10 5 50

Cumulative Score 600

Example‐ Application = A/P


Next step……..

• Note: Total possible cumulative score is 1000. You may change risk factor rating or risk/control factor based on your subjective judgment

• Sort in the descending order of cumulative score• Select higher score applications based on audit resource availability

• Important: Irrespective of cumulative score, audit will include evaluation of input, processing, and output controls


AC and GC SoD Principles

• GC SoD follow DOPESS principles • AC SoD follow CARRE principles

• Note: Risk Assessment model depends upon may factors. It depends on your environment. We discussed only 10 in our case. There are some of those factors as well (given in the next foil).


17 factors for Application Assessment

• This one has 17 factors for Application Assessment:

• Quality of internal controls• Economic conditions‐ fraud increases in bad economy• Recent accounting system changes• Time elapsed since last audit• Operational complexity• Operational environment change• Recent changes in key positions• Time in existence• Competitive environment


17 factors for Application Assessment

• Prior audit results• Assets at risk• Transaction volume• Regulatory agency impact• Staff turnover• Impact of application failure• Sensitivity of transactions• Monetary volume


AC Audit Methods

• Business Process Method– ACs present in all the systems that support particular business process

– BP may span many BUs– ERP transactional applications arising out of BPR – In non‐ERP applications, review all applications spanning a BP

– Consider downstream and upstream interfaces (aka inbound and outbound interfaces)


AC Audit Methods

• Single Application Method– Suitable for non‐ERP and non‐integrated environment

– Not a recommended method for ERP applications

• There could be many data feeds going in and coming out of a module

• Difficult to assess ERP with single application


Business Process Method Auditing An Example


Four Types of BP Audits

• Mega Process– End‐to‐end audit or integrated audit– E.g. in AP, it is procure‐to‐pay process– Level 1

• Major Process– One component of mega process

• E.g. one of the AP components• procurement or receiving or payment of goods

– Level 2


Four Types of BP Audits cntd.

• Minor Process– Component of major process

• E.g. PR and PO sub‐process of procurement process

– Level 3

• Activity– System transactions that create, modify or delete data in a sub‐process

– Level 4– IT auditor’s traditional domain– But levels 1,2,3 are very important


Example of Mega Process Procure‐to‐pay

Level 2 Level 3 Level 4Procurement PR and requisition processing

PO ProcessingA, C, DA, C, D

Receiving Goods (services) receipt processingGoods return Processing

A, C, DA, C, D

Accounts Payable

Vendor managementInvoice processingCredit memo processingProcess paymentsVoid payments

A, C, DA, C, DA, C, DA, C, DA, C, D


Example of Mega Process Procure‐to‐pay cntd.

• Highlighted items in previous slide are called “triple control”

• Level 4 is where IT auditor concentrates• But, if you don’t know level 1,2, and 3, risk is not mitigated because– Controls at the lower level (level 4) do not compensate for controls at the higher levels

4. Create the Audit Plan


The End Result

• Audit plan a subset of the audit universe• It is an outcome of risk assessment• Additions to audit plan from senior management and audit committee

• Everything must be risk based


The Real Audit Plan

• In risk assessment driver is risk, influencer is resources

• In creating audit plan, driver is resources and influencer is risk


Audit Plan

Driver = ResourcesInfluencer = Risks

Risk Assessment

Driver = RisksInfluencer = Resources

UnderstandRisks

Allocate Resources

Key Activities

• Obtain explicit input from stakeholders.• Identify relevant risks.• Assess risks.• Prioritize risks.

Key Activities

• Understand universe of potentialaudits subjects.• Allocate and rationalize resources.• Reconcile and finalize the audit plan.

Objectives For Risk Assessments And Audit Plans


Requests from Stakeholders

• Stakeholder requests from board, audit committee, senior management, operating managers

• Special audit assurance from stakeholders• Consulting services requests from stakeholders

• Fraud investigations requests come throughout the year

• Consulting engagements to be included in the audit plan


Audit Frequency

• Multiyear plans presented to audit committee and management

• 3 to 5 years is normal for planning• May need external resources • Annual plan is a subset of multiyear plan• Audit frequency established at RA time


Frequency vs. Resource Allocation

Priority Frequency Resource AllocationH Immediate action,

usually within the first year

Annual reviews or multiple actions within the cycle

High allocation

M Mid‐term action within the audit cycle

One or several audit engagements within the cycle; could be postponed

Base allocation

L Audit engagements usually not planned within the cycle

At most one audit engagement planned within the cycle

Limited allocation

Frequency and resource allocation of audit activities


Audit Plan Contents

• Different types of IT audits• Integrated business process audits

– IT processes (as in COBIT, ISO, and ITIL)– SDLC reviews– Application controls– Technical infrastructure audits– Network audits

• Financial reviews, operational reviews, compliance reviews

• SoD• New threats and innovations


Integration of IT Auditing

• Low integration IT audit– Isolated from non‐IT activities

• Partially Integrated– Associated with business process reviews– Application reviews

• Highly integrated– IT audit part of business process engagement– Multidisciplinary team


Audit Universe Low‐integrated

Audit Plan

Partially IntegratedAudit Plan

Highly IntegratedAudit Plan

Business Processes• Operational• Financial• Compliance

Non‐IT audit Non‐IT audit Integrated approach

Applications Systems• Application controls• IT general controls

IT audit Integrated approach

Integrated approach

IT Infrastructure Controls• Databases• Operating systems• Network

IT audit IT audit Integrated approach

IT auditing and integrated auditing


AUDIT RESOURCES

Consider alternative audit approach (CSA)

TOTAL

AUDIT

UNIVERSE LOW

HIGH

TARGETED RESULT

Chart of targeted audit results


Audit Plan –A Living Document

• New threats and new vulnerabilities evolve• IT has higher rate of change than non‐IT activities

• New technologies‐ e‐commerce, web applications

• Therefore, audit plan is a living document


Executive buy‐in and Plan Approval

• Audit plan presented to audit committee and senior management

• Also discussed with CIO, CTO, IT managers, business application owners

• Client interaction during RA is important • Buy‐in brings cooperation, hence value to the organization


Questions