EXCEPTIONAL EXCEPTIONS

Hussein Issa

(Spring 2015)

BACKGROUND

Background

• Audit by Exception (Vasarhelyi and Halper, 1991)

• Continuous Auditing literature is rich with studies that propose

statistical and machine learning techniques to identify

exceptions (Dull et al., 2006; Groomer & Murthy, 1989a; Alexander Kogan et al.,

1999; Vasarhelyi & Halper, 1991)

• Problem?

• Result?

3 5/31/2015 34th WCARS

Large numbers of Exceptions!

-Information overload!

-Ambiguity

-Information relevance

Motivation & Contribution

Motivation:

• Prior continuous auditing research focused on detecting

exceptions efficiently

• We need to develop a classification hierarchy or ranking of

detected exceptions to limit the need for human intervention

Contribution:

• Fill the gap in the continuous auditing literature with regards to

providing the human users with assistance in dealing with

large number of exceptions

• Propose exceptions prioritization methodologies

4 5/31/2015 34th WCARS

Relevant Literature-Exceptional Exceptions

• Analysis usually yields large amounts of exceptions, causing

information overloads due to sub-optimal business processes

or over-conservative CA/CM system (Alles et al 2006, 2008 ;

Debreceny et al. 2003)

• Human users perform complex aggregation and processing

tasks poorly (Iselin, 1988; Kleinmunitz, 1990)

• Murthy 2012 emphasized the lack of and need for studies

addressing the problem of large numbers of exceptions

5 5/31/2015 34th WCARS

IDENTIFYING AND PRIORITIZING

IRREGULARITIES USING A RULE-

BASED MODEL WITH A WEIGHTING

SYSTEM DERIVED FROM EXPERTS’

KNOWLEDGE

Motivation, Research Questions, & Findings

Motivation: • Plenty of studies for exception identification, few address processing

• Majority of expert systems assign the same weight to rules

• HOWEVER: business rules, and accordingly their violations, do not have

the same importance

Research Questions: 1. How can we integrate the judgment of the domain experts (in this case the

auditors) in a rule-based expert system?

2. How can we develop a weighting system for the various rules in that expert

system?

3. How can we use this weighting system to prioritize exceptions?

7 5/31/2015 34th WCARS

Suspicion Score

8 5/31/2015 34th WCARS

Rank

4

2

3

1

6

5

Framework

9 5/31/2015 34th WCARS

Framework

10 5/31/2015 34th WCARS

6

Rule-Based System

• Set of IF-THEN rules

• Popularity stems from simplicity, interpretability, flexibility

• Data: Simulated Order to Cash data

• Originally 33 analytics, narrowed down to 12 by experienced auditors

• 12 analytics can be categorized as tests for: – Segregation of duties

– Unauthorized transactions

– Missing documents

– Non-matching documents.

11 5/31/2015 34th WCARS

Rules Weights Inference

• Business rules and accordingly their violations, do not have

the same significance

• 28 participants with 3 and more years of experience:

– Conduct 17 pairwise comparisons

– Select the transaction they believe to present higher control risk

– Provide justification if their assessment

12 5/31/2015 34th WCARS

Customer

Transaction

ID

Custom

er ID

Invoice

Number

Invoice

Date

Created

by

Inventory

Item ID

Invoice

Selling

Price

Invoiced

Amount

Shipment

ID

Shipment

document

Created By

Shipment

document

Approved By

10034 1146 10001203 10/15/1997 1546 63

800 9600 10000877 1002 1546

4110 1000 10000262 3/1/1998 1117 628

600 1800

SOD Missing Values

(Orphaned invoices)

Rules Weight Inference-LP

• Special Case Linear Program

– 16 pairs

– Each transaction can violate one and only one

• General Case Linear Program

– 17 pairs

– One transaction violated two rules

13 5/31/2015 34th WCARS

Exceptions Identification & Prioritization

Identification:

• Apply expert system to the whole population to find all the records

that violate one or more rules

• Remaining records are assumed to be normal, thus presenting

negligible risk

Prioritization:

• Calculate the Suspicion Score for each exception such that:

14 5/31/2015 34th WCARS

Exceptions Prioritization-Example

15 5/31/2015 34th WCARS

Rank

4

2

3

1

6

5

Investigation & Feedback

Investigation:

• Auditors are provided with Prioritized exceptions

• Scope of investigation depends on their time/budget

constraints

Feedback:

• Helps adjusting the rules that make up the expert system.

• Enables us to modify the weights of the rules according to the

audit teams’ findings

– incorporated as a new set of constraints in the general case Model

• Effect of the original experiment will decrease over time with

more feedback from auditors

16 5/31/2015 34th WCARS

Weights –Special vs. General Models

17 5/31/2015 34th WCARS

• Excessive Write offs ranked highest

• SOD in general ranked low

• Rules with direct impact on financial numbers ranked high

• Operational controls ranked low Analytic Special Case Model General Case Model

Weight Rank Weight Rank

Analytic_1_SOD_Customers 1.00 11 1.00 11 Analytic_2_Unauthorized_Sales_Order 2.86 2 2.71 3 Analytic_3_Unathorized_Price 1.38 9 1.35 9 Analytic_4_SOD_Credit_Adjustment 1.21 10 1.20 10 Analytic_5_Match_Shipping_to_SO 1.79 8 1.72 8 Analytic_6_Match_Invoice_to_Ship 2.33 4 2.22 5 Analytic_7_Missing_Sales_Orders 2.31 5 3.18 1 Analytic_8_Unauthorized_Shipments 2.27 6 2.17 6 Analytic_9_SOD_Ship_Invoice 1.88 7 1.81 7 Analytic_10_Orphaned_Invoices 2.85 3 2.70 4 Analytic_11_SOD_Invoice_Receipt 1.00 12 1.00 12 Analytic_12_Excessive_Write_Offs 3.11 1 2.94 2

Analytic Special Case Model General Case Model

Weight Rank Weight Rank

Analytic_12_Excessive_Write_Offs 3.11 1 2.94 2

Analytic_2_Unauthorized_Sales_Order 2.86 2 2.71 3

Analytic_10_Orphaned_Invoices 2.85 3 2.7 4

Analytic_6_Match_Invoice_to_Ship 2.33 4 2.22 5

Analytic_7_Missing_Sales_Orders 2.31 5 3.18 1

Analytic_8_Unauthorized_Shipments 2.27 6 2.17 6

Analytic_9_SOD_Ship_Invoice 1.88 7 1.81 7

Analytic_5_Match_Shipping_to_SO 1.79 8 1.72 8

Analytic_3_Unathorized_Price 1.38 9 1.35 9

Analytic_4_SOD_Credit_Adjustment 1.21 10 1.2 10

Analytic_1_SOD_Customers 1 11 1 11

Analytic_11_SOD_Invoice_Receipt 1 12 1 12

Weights – Internal vs. External Auditors

18 5/31/2015 34th WCARS

All Responses Internal Auditors External Auditors

Analytic Weight Rank Weight Rank Weight Rank

Analytic_1_SOD_Customers 1.00 11 1.00 10 1.00 11

Analytic_2_Unauthorized_Sales_Order 2.71 3 2.65 4 2.84 3

Analytic_3_Unathorized_Price 1.35 9 1.51 8 1.24 9

Analytic_4_SOD_Credit_Adjustment 1.20 10 1.45 9 1.06 10

Analytic_5_Match_Shipping_to_SO 1.72 8 1.98 7 1.59 8

Analytic_6_Match_Invoice_to_Ship 2.22 5 2.29 6 2.22 5

Analytic_7_Missing_Sales_Orders 3.18 1 3.10 1 3.37 1

Analytic_8_Unauthorized_Shipments 2.17 6 2.36 5 2.08 6

Analytic_9_SOD_Ship_Invoice 1.81 7 1.00 11 1.79 7

Analytic_10_Orphaned_Invoices 2.70 4 2.75 3 2.76 4

Analytic_11_SOD_Invoice_Receipt 1.00 12 1.00 12 1.00 12

Analytic_12_Excessive_Write_Offs 2.94 2 2.93 2 3.05 2

Takeaway from today

I think the main take away from this presentation is:

A. Continuous auditing sucks, dude!

B. It is better to stick to traditional sample based auditing

C. Number of exceptions is always very low

D. Hussein is awesome!!! (Hint: correct answer)

19 5/31/2015 34th WCARS

Too many exceptions!

Information Overload

Research Opp. in EE