asosai performance audit guidelines

8/8/2019 ASOSAI Performance Audit Guidelines

1/82

Fifth ASOSAI Research Project

Performance AuditingGuidelines

October 2000


2/82

2 5th ASOSAI Research ProjectPerformance Auditing Guidelines


3/82

3

Contents

Foreword

Preface

1 Introduction 7

Performance auditing guidelines 7Definition and scope of performance auditing 8

Performance auditing mandate and objective 10

Performance audit process 10General principles of performance auditing 13

2 Strategic Planning 16

Introduction 16Selecting audit topics 17

3 Initiating the Performance Audit 21

Introduction 21

Planning individual audits 21Understanding the business of an auditee 23

Audit criteria 25

4 Implementing the Performance Audit 30

Introduction 30

Audit approaches 32Refining the audit program 33

Audit test programs 35

Developing findings and recommendations 36

5 Evidence and Documentation 38

Audit evidence 38Competence, relevance and sufficiency of evidence 38

Characteristics of performance audit evidence 40

The evidential process 42

Documentation 44

6 Reporting 48Introduction 48

Reporting process 49

Report contents 52

7 Follow-up Processes 54

Introduction 54Assessing and reporting on follow-up activity 55


4/82


8 Audit ManagementSome Important Issues 57

Consultation with auditees 57Quality assurance and quality control 59

Appendix A Important considerations for performance auditing in an

Information Technology environment 65Appendix B Potential impacts of performance auditing 70

Appendix C Key features of the auditee and its environment requiringunderstanding 72

Appendix D Important issues in the performance audit of government

programs/activities 75

Glossary of Terms 79


5/82

5

Foreword

In 1997, at its 6 th Internat ional Seminar , held in Jakar ta , the Asian

Organization of Sup reme Au d it Institutions (ASOSAI) Assembly ap proved

the Jaka rta Declaration of Gu idelines on Prom oting Efficient an d Effective

Pu blic Ad min istration through Performan ce Au d iting. The Declaration

resolved that broad guidelines for performance auditing be developed.

In September 1998 the 26th ASOSAI Governing Board meeting, held in

Beijing, app roved that the 5th A SOSAI Research Project w ou ld be on the

topic of performance auditing guidelines.

The general framework for the development of ASOSAI performance

au d iting gu idelines has been ded u ced from the Jakarta Declaration. Theseguid elines are not a d etailed instruction m anu al and do not replace the

need for management and staff to use their professional judgement to

ensure the d elivery of a quality au dit p rodu ct.

The team for the research project comprised: Mr. Peter Robinson (Team

lead erAustralia), Messrs. I. P. Singh and A . K. Thaku r (Ind ia), Mr. Ab.

Rahma n bin Moham med and Ms.Azizah Arshad (Malaysia) and Mr. John

Oldroyd (New Zealand) .

T he r e sea r ch t e am deve loped a d r a ft o f t he pe r f o r m ance aud i t i ngguidel ines that was c i rcula ted to ASOSAI members for review and

comment in N ovember 1999. Comm ents from ASOSAI members were

taken into account in develop ing the final d raft of the guid elines. The

team wishes to express its appreciation for the contributions made by

m ember Sup reme Aud it Institution s. In October 2000, at the 8th ASOSAI

Assembly held in Chiang Mai, Thailand, the ASOSAI Governing Board

approved the performance auditing guidelines.


6/82


Preface

Perform ance au d iting is essentially a p rocess that u ses available evidence

to form an op inion on th e extent to w hich an agen cy utilises its resources

in an econom ic, efficient and effective man ner. Althou gh p erforman ce

a u d i t i n g i n v o l v e s t h e u s e o f p r o f e s s i o n a l j u d g e m e n t , a r o b u s t

methodology can provide an objective framework for this judgement.

Performance auditing offers considerable scope for adding real value to

pu blic ad ministration. It also aims to ensu re that ind ividu als and agencies

managing public resources remain accountable for their actions.

To be effective, performance auditors must have a clear understanding

of the objectives and p riorities of any area su bject to aud it. They mu stalso have an un d erstand ing of the p olicies for the p lanning and cond uct

of audi ts , audi t s tandards , audi t methodology and processes of the

particular Supreme Audit Institution (hereinafter referred to as SAI).

These guidelines provide a m ethodology and a broad framework for the

cond uct of the p erformance aud iting process and also p rovide the basis

by wh ich the qu ality of the aud it produ ct can be jud ged. The guid elines

will assist performance auditors:

to be attuned to the needs, expectations and p riorities of clients, wh ichin the case of SAIs are the legislature, government institutions and

those entrusted with the task of managing public affairs;

to add value to agencies through performance audit ing services;

t o p r ov ide an env ir onm en t tha t enab le s pe r fo r m ance aud i to r s t o

enh ance their skills and achieve their fu ll potential;

to achieve audi t excellence; and

to manage p erformance aud it operations efficiently and effectively.

The guidelines take into account relevant INTOSAI Auditing Standardsand are based on generally accepted principles of performance auditing

d is t il led f rom th e exp er ience of ASOSAI memb ers . They therefore

comprise contemporary performance auditing methodology and reflect

a bes t pract ice consensus for the cur rent environment , among the

mem ber SAIs. The guid elines are a living d ocum ent, w hich will need

to be upd ated as the environment changes and as p erformance aud iting

methodology and practice develop.


7/82

7

1. Introduction

Performance auditing guidelines1.1 These guid elines p rovid e a framew ork for iden tifying, collecting

and analysing information an d provid e practical guidan ce to:

assist SAI performance auditors to manage and condu ct performance

audits ;

prom ote consistent, economical, efficient an d effective performance

audit practice;

e st a b li sh a b a s is fo r fu r t h e r d e v e lo p m e n t o f p e r fo r m a n ce a u d i t

methodology and professional development; and

set out a basic framework within w hich professional jud gement may

be exercised on analysing performance and reporting conclusions.

1.2 These gu ide l ines a re des igned for use by SAI audi tor s and

consultants who are involved in planning, carrying out, reporting and

man aging a performance audit . As such, the guid eline chap ters have

been structured in line with the major stages of the performance audit

process . The guidel ines wil l help audi tors to deal wi th the complex

demands of performance auditing.

1.3 The greater u se of information techn ology by ag encies, includ ing

the Internet, has meant that performance audits will be increasingly

un d ertaken in information techn ology environments. As SAIs may be

called up on to develop d etailed guidelines for p erformance aud iting in

an information technology environment, some of the more important

considerations hav e been outlined in Ap pend ix A.

1.4 Perform ance aud itors can be faced w ith consid erable variety and

ambigu ity in their work. They need skills in analysing organisationalactivities and m anag emen t practices. They can be faced w ith the need to

become fam iliar with a w ide ran ge of organ isational contexts and su bject

ma tters. They need the ability to write reports dealing with comp lex

issu es in a logical and thorou ghly sup por ted fashion. The gu idelines can

provide some assistance in these areas, but it is largely incumbent on

p erform ance aud itors to d evelop their skills in these areas by other m eans.


8/82


Definition and scope of performance auditing

1.5 Th e INTOSAI Auditing Standards1 define a performance audit

as an audit of the economy, efficiency and effectiveness with which the audited

ent ity uses its resources in carrying out its responsibilities.2

1.6 Economy is min imising the cost of resources used for an activit y, having

regard to the appropriate quality.3 Econom y issues focu s on t he cost of the

inpu ts and p rocesses. Econom y occurs w here equ al-qu ality resour ces

are acquired at lower prices.

1.7 Efficiency is the relationship between the output, in terms of goods,

services or other results, and the resources used to produce them.4 Efficiency

exis ts where the use of f inancia l , human, phys ical and informat ion

resources is such that output is maximised for any given set of resource

inpu ts, or inpu t is minimised for any given qu antity and quality of outp ut.

1.8 Effectiveness is the extent to which objectives are achieved and the

relationship between the intended impact and the actual impact of an activity.5

Effectiveness add resses the issue of whether th e program / activity has

achieved its objectives. When focusing on effectiven ess, it is imp ortant

to dis t inguish between the immediate outputs (or products) and the

u ltimate imp acts (or ou tcomes). Effectiveness is achieved , for instan ce,

w here there is imp roved achievement of a progr ams objectives. Ou tcomes

are important to the effectiveness of program s/ activities but may be more

difficult to measure and assess than the inpu ts and outp uts. Ou tcomes

will often be influenced by external factors and may require long-term

rather than short-term assessment.

1.9 Performance audit ing, also referred to as Value For Money

aud i t ing , shares s imi la r app roaches and method ologies to p rogram

evaluation bu t d oes not gen erally extend to assessing p olicy effectiveness

or policy alternatives. Ap art from examining the impa ct of outp uts, an

evaluation could includ e issues such as w hether ag ency plann ing reflected

the p rogram objectives, w heth er the ob jectives were consistent w ith p olicyand op tions for chan ging th e policy to achieve m ore effective outcomes.

1 Auditing Standards, Auditing Standards Committee, International Organization of Supreme Audit

Institutions, June 1992, p. 69.

2 Refer to Appendix B for examples of potential impacts of performance auditing.

3

Auditing Standards, Auditing Standards Committee, INTOSAI, June 1992, p. 664 Ibid, p. 67

5 Ibid, p. 67


9/82

9

1.10 Performance auditing promotes public accountability and is an

aid to good corp orate governan ce. It also encomp asses the concept of

trad itional financial, compliance and p rop riety au d its. Com pliance au d its

d eal w ith regularity and legality aspects. Prop riety refers to the concept

of the best pr actice in pu blic man agem ent of pr ogram s/ activitiespu blic

funds should not be misused by the managers for personal benefit and

expend iture on program s should not exceed w hat the occasion deman ds.

Performance auditors should utilise these requirements in the course of

their audit, wherever applicable.

1.11 The performan ce au d itor will also be exp ected to ad d ress concerns

re la t ing to equi ty and e thics whi le assess ing the ef fect iveness of a

program/ activity. Equity in the context of program m anagement relates

to fairness and imp artiality in use of pu blic fund s. Ethics in ma nagin g

publ ic expendi ture enjoins the qual i t ies of hones ty and integr i ty inpersonal conduct and devotion to the duty as manager of public funds.

The SAI performance auditor should be conscious of equity and ethics

issues.

1.12 Performance aud its may examine and report on:

the qual ity of informat ion and advice available to Government for

the formulation of policy;

the existence and effectiveness of adm inistrative machinery in place

to inform the Government whether program objectives and targets

have been determined with a view to fulfilling policy objectives;

wh ether, and to wh at extent , s ta ted program object ives have been

met;

the economy, efficiency, effectiveness and ethics of the means u sed to

implement a program / activity; and

the intended and u nintended direct and indirect impacts of programs/

ac t ivi t ies ; for example , the environmental impact of government

activity.

1.13 Auditors should not confine the audit to what has been done

but should also examine what has not been done to meet the policy

objectives. Given the size, comp lexity and d iversity of agency operations,

it is generally imp racticable to attem pt to assess the overall performan ce

of d epartm ents or agencies. Consequ ently, performa nce aud its are usually

directed towards specific functions, activities, programs or operations

of the agency.

Introduction


10/82


Performance auditing mandate and objective

1.14 The aud it mand ate is generally set out in statute and dictates the

extent to w hich an SAI can aud it pu blic sector agencies. Where the

mandate is derived out of a broad interpretation of the term audit or

has been accepted by convention, auditors should be guided by their

best ju d gemen t in the context of any su ch conven tion. The performa nce

aud i t mand ate ordinar ily specifies the m inimu m aud i t and repor t ing

requirements; indicates what is required of the auditor ; and provides

the auditor with authority to carry out the work and report the results.

Accordingly, the auditor should be guided by the audit mandate when

planning the au d it. In m ost coun tries performance aud its are excluded

from commenting on Government policy.

1.15 Perform ance aud iting h as the objective of imp roving p u blic sectoradm inistration an d accoun tability by ad ding value throu gh an effective

program of per formance audi ts and re la ted products such as bet ter

pr actice gu ides. One of the main objectives of p erform ance aud iting is

to assist the peoples representatives in exercising effective legislative

control and oversight.

1.16 Performance aud its hav e a du al role. They p rovid e clients w ith

information and assurance about the quality of management of public

resources and they also assist pu blic sector m anag ers by ident ifying an d

prom oting bet ter managem ent pract ices . Per formance audi t ing maytherefore lead to better accoun tability, imp roved econom y a nd efficiency

in th e acquisition of resources, im pr oved effectiveness in a chieving p ub lic

sector p rogram objectives, a higher qu ality in pu blic sector service delivery

and imp roved man agement planning and control. The emp hasis placed

on the assurance role of performance auditing over the role of public

sector improvement will vary between SAIs.

Performance audit process

1.17 The first stage in p erforma nce aud iting is strategic plann ing, wh ich

requires the developm ent and maintenance of information on th e agency

that w ill assist in id entifying poten tial areas for aud it. Potential top ics

can then be analysed and ranked to form annu al audit strategy docum ents

for each agency. Chap ter 2 of these guid elines deals fu rther w ith strategic

planning.


11/82

11

1.18 On ce a top ic has been selected for p erform ance aud it, the au d it is

initiated by the d evelopm ent of a plan d etailing th e cond uct of the au dit.

A preliminary study may be undertaken to gather information on the

topic and to identify s ignif icant issues for audit . Chapter 3 of these

guid elines deals with initiating the p erformance au d it.

1.19 The imp lementation stage of a perform ance aud it is dealt with in

particular in Chapters 4 and 5 of these guidelines and involves:

development and execut ion of an audi t program of procedures;

co ll ect ion and docum en ta t i on o f su f fi ci en t r e levan t and r e li abl e

evidence, including quantitative and qualitative analysis;

formulation of aud it findings, conclusions and recommend ations; and

development of discussion papers and confirmation of aud it find ings

at an exit conference.

1.20 A report on the audit may then be drafted for consideration by

the government, the legislature, the agency and th e pu blic. Throu ghou t

each s tage of the per formance audi t the emphasis should be on the

prod uction of a final report that is balanced an d has value ad d ed imp act.

The report-writ ing process should be viewed as a continuous one of

formu lating, testing an d revising conclusions, if necessary, about the au d it

topic. Therefore, issues such as the expected imp act and va lue of the

aud it, the likely imp rovements and savings resulting from the au d it andmethods of communicat ing audi t conclus ions should be cons idered

throu ghou t the aud it. Chapter6 of these guidelines deals furth er with

reporting.

1.21 Follow-up procedures should id entify and docum ent aud it imp act

and the progress of the agency in imp lementing au dit recomm end ations.

The cond u ct of follow -u p au dits is vital to the follow-up process to provid e

feedback to the SAI, the legislature and the government on performance

a u d i t e f f e c t i v e n e s s i n p r o d u c i n g i m p r o v e m e n t s i n p u b l i c s e c t o r

management. Follow-up processes are dealt with in Chapter7 of theseguidelines.

1.22 The key stages in the performance audit cycle are outl ined in

Figure 1.

Introduction


12/82


Figure 1Key Stages in the Performance Audit Cycle

Information

feedback

to the

strategic

planning

process


13/82

13

General principles of performance auditing

Auditing standards

1.23 In conduct ing a per formance audi t the audi tor should fol low

INTOSAI Audi t ing Standards as we l l a s r e l evan t S AI s t anda r ds

applicable to performance audits.

1.24 Th e INTOSAI general auditing standards common to aud itors

and SAIs are:

(a) The auditor and the SA I must be independent ;

(b) The auditor and the SA I must possess the required competence; and

(c) The auditor and the SA I must exercise due care and concern in

complying with the IN TOSA I Auditing S tandards. This embraces

due care in planning, specifying, gathering and evaluating evidence,and in reporting findings, conclusions and recommendations. 6

1.25 The performance auditor and SAI must be, and be seen to be,

indep end ent and objective in the cond uct of aud its. That is , the SAI

mu st have freedom to act in accordance with its audit mandate without external

direction or interference of any kind.7

1.26 Th e INTOSAI general auditing standards also note that SAIs

should:

(a) Recruit personnel with su itable qualifications;

(b) Develop and t rain SA I employees to enable them to perform their

tasks effectively, and to define the basis for the advancement of

auditors and other staff;

(c) Prepare manuals and other written guidance and inst ructions

concerning the conduct of audits;

(d) Su pport the skills and experience available wit hin the SA I and

identify the skills which are absent; provide a good distribution of

skills to auditing tasks and assign a sufficient nu mber of persons forthe audit; and have proper planning and supervision to achieve its

goals at the required level of due care and concern; and

(e) Review the efficiency and effectiveness of the SAIs internal standards

and procedures. 8

Introduction

6

Ibid, p. 237 Ibid, pp. 24, 68

8 Ibid, p. 23


14/82


Professional behaviour

1.27 The aud itor shou ld comp ly with ethical principles and codes of

c o n d u c t g o v e r n i n g t h e a u d i t o r s p r o f e s s i o n a l b e h a v i o u r a n d

respon sibilities, w hich includ e:

in tegrity;

object ivity and fa irness;

confidentia lity; and

t echn ica l st anda rds .

1.28 Althou gh th ese gu idelines set out a coherent basis for condu cting

a performance audit, professional judgement (albeit largely applied on

the basis of relevant ru les and procedures) remains the m ost imp ortant

ingredient in performance audit work.

1.29 The performance aud itor should adop t an attitud e of professional

scepticism th rou ghou t the au d it, recognising that circum stances may exist

that could cause the information relating to performan ce to be materially

mis-stated.

Reasonable assurance

1.30 A performance audit conducted in accordance with applicable

audi t ing s tandards provides reasonable assurance as to whether the

information relating to performa nce is free from mat erial mis-statemen t.

Reasonable assurance re la tes to the accumulat ion of audi t evidence

necessary for the auditor to conclude whether there are any mater ial

mis-statements in the information relating to performance.

1.31 What is reasonable is dependent on the facts of that situation

and is to be d etermined by w hat evidence could reasonably be expected

to be gathered and wh at conclusions could reasonably be expected to be

draw n in the p articular situation.

Terms of the engagement1.32 The SAI should formally notify the auditee 9 of the details of the

engagement, preferably before the commencement of the audit, to help

in avoid ing any misun d er s tand ings . Wi th some engagem ents , the

objec t ive and scope of the audi t and the audi tor s ob l iga t ions a re

established by law. Even in those situ ations id entifying the terms of

engagement may still be informative for the agency.

9 Auditee refers to the agency being audited.


15/82

15

Work performed by assistants

1.33 INTOSAI AuditingStandards10 state: The work of the audit staff at

each level and audit phase should be properly supervised during the audit, and

docum ented work should be reviewed by a senior member of the audit staff. When

work is delegated to members of the audi t team, the audi tor shouldcarefully direct, supervise and review the work delegated.

1.34 When mu lti-d isciplinary au d it teams are u sed, adequ ate direction,

supervis ion and review are necessary to ensure that team members

d ifferent p erspectives, experience and specialties are app rop riately u sed

in the aud it. All team m embers shou ld u nd erstand the objectives of the

aud it, the terms of reference of work assigned to them and the natu re of

obligations imposed on them by app licable aud iting stand ards. Adequ ate

d irection, sup ervision an d review w ill assist in such an un d erstand ing.

Using the work of an expert

1.35 When u sing the w ork performed by an expert, the aud itor should

obtain sufficient ap prop riate audit evidence to ensure that such w ork is

adequate for the purposes of the audit.

1.36 An exp ert is a person or firm p ossessing sp ecial skill, know ledge

and experience in a particu lar field oth er than au d iting. Becau se of the

diverse range of activities subject to performance auditing, the auditor

may need to obta in audi t evidence in the form of repor ts , opinions ,

valuations and statements of an expert. Although the aud itor may u se

the wor k o f an expe r t a s aud i t ev idence , t he aud i to r r e t a in s f u l l

responsibility for the conclusions in the audit report.

10 Auditing Standards, Auditing Standards Committee, INTOSAI, June 1992, p. 45

Introduction


16/82


2. Strategic Planning

Introduction2.1 The SAI needs to select and sched ule au d it tasks that help achieve

its role and corpora te objectives. Au d it top ics shou ld be ran ked , for

examp le u sing a risk-based ap pr oach to topic selection, in ord er to make

the best use of lim ited SAI resou rces. For this pu rp ose the SAI ma y

prepare a program model which identifies the programs, projects and

activit ies undertaken by an agency, along with other factors such as

program objec t ives , expec ted ou tputs , ou tcomes and per formance

ind icators. The selection of top ics for perform ance aud it can be based

on a combination of relevant risk and materiality factors.

2.2 A well-structured strategic planning process, based on a sound

rationale, is necessary to ensu re tha t the resources of the SAI are u sed in

the most efficient an d effective m anner. The objectives of strat egic au d it

planning are to:

provid e a firm basis for the SAI man agement to give strategic direction

for futu re au dit coverage;

identify and select aud its with the potential to improve public sector

accountability and administration;

provide a p latform for commu nication w ith agencies and the legislature

on SAI audit strategies;

produ ce a work program that can be achieved with expected available

resources;

und erstand agency r isks and take them into account in audit selection;

an d

provide a basis for SAI accountabil ity.

2.3 The key d eliverable of the strategic plan ning p rocess is a d ocum ent

w hich is prep ared for the SAI man agem ent to en able it to critically assess

the proposed planning strategy for overall consistency with the SAIs

corporate objectives. It w ill also assist ma nagem ent to make ap pr op riate

resource allocations an d an assessment of the strategic plan ning pr ocess.


17/82

17

2.4 The strategic plan should be supported by working papers that

include:

a descr ipt ion of the agency and i ts environment ;

an a s s es sm en t o f t he r is ks t o good p e r fo r m ance o f t he agencys

program s and activities;

a summ ary of the long-term strategic view of SAI performance audit

d irections in each agen cy;

recent SAI aud its , recent and prop osed inquir ies by the legislature,

and agency evaluations and internal aud its;

a l is t of potential areas for performance audit ; and

a l is t of aud its proposed for the next two or three years, the basis for

their selection and ind icative timin g wh ere possible. This list should

reflect resource availability and should also distinguish between newperformance audit topics, cross-agency audits and follow-up audits.

Selecting audit topics

2.5 Audit topics are generally selected on two grounds: firstly, to

focus on audits expected to add maximum value in terms of improved

accoun tability, econom y, efficiency and effectiveness; second ly, to en su re

appropriate coverage of program operations within the l imitations of

au d it resources available. The strategic plan ning p rocess is outlined in

Figure2.

Figure 2The Strategic Planning Process

Inputs

Potential audits

Priorities

Resourcerequirements

Audit Strategy Plan

Outputs

Budget

Previous audit strategy planning

Government views, budget papers, etc

Agency annual reports and evaluations

Media and external reports

Previous audit fieldwork

Analysis of performance indicators

Discussions with agencies, clients, etc.

Priorities of legislature

Priorities of government

Strategic Planning


18/82


2.6 The analysis of risks of poor performance or, expressed another

way, risks of inadequate economy, efficiency and effectiveness will lead

to a list of p otential aud it topics. I t can be u sefu l to rank th e topics

subjectively against the following criteria:

overa ll est imated audit impact ; financia l mate ria lity;

r isk t o good m anagem en t;

significance of the program to the activities of the agency;

visibility of the program / activity as reflected in its political sensitivity

and national imp ortance; and

lack of recent audit coverage and other internal and external review

of the program/ activity.

Overall estimated audit impact

2.7 Of major importance in the final selection of topics is the added

value expected from the au d it. A p reliminary assessment of the au d its

likely benefits should be made at this strategic planning stage.

2.8 Appendix B lists some of the potential impacts of performance

auditing and classifies the benefits by reference to: economy; efficiency;

effectiveness; quality of service; p lanning , control an d man agem ent; and

accou ntability. Qu an tification is d esirable bu t un likely to be feasible at

the strategic planning stage.

Financial materiality

2.9 This criterion is based on an a ssessm ent of the total valu e of assets,

liabilities, annual expenditure and annual revenue of the selected audit

area. The more m aterial an area is, the h igher is its priority for selection

as an aud it topic.

Risks to good management

2.10 Assessment of risks to good performance in the agency requiresthe SAI to assess whether the management of the activity to be audited

is likely to be deficient in economy, efficiency and effectiveness.

2.11 Evidence of risk to good management includes:

management inaction in response to identified w eaknesses;

adverse comment by the legis la ture or media;

non-achievement of stated objectives such as revenu e raised or clients

assis ted;

h igh st aff t u rnover ;


19/82

19

identified weaknesses in internal control;

s ignificant und erspending or overspending;

sudden program expansion or contract ion; and

overlapp ing or blurred accountabil ity relationships.

2.12 An agency p rogram or activity that is more comp lex to man age

and operates in an un certain environment is more likely to have p roblems

associated w ith p erforman ce. Some p ossible ind icators of high comp lexity

and uncertainty are:

highly decentralised operations with devolved management decision-

making responsibilities;

a mu lt ipl icity of interes ted p ar ties;

use of rapidly changing and sophisticated technology; a dynamic and compet it ive environment ; and

controversial social and p oli tical debate surrounding the issue.

2.13 The stage of the agencys program development should also be

kept in mind w hen assessing m anagement p erformance. For examp le, in

the development stages it will be particularly important for the agency

management to set measurable, operational objectives, which clear ly

identify how the program will contr ibute to the agencys objectives.

During p rogram imp lementation it will be imp ortant to ascertain w hetherapp ropriate performance measures are maintained and analysed to assess

performance, and whether there is a clear identif ication of roles and

respon sibilities for each level of the prog ram / activity. If the progr am

has been in place for some time it will be important to assess whether a

formal evaluation h as been u nd ertaken to ascertain wh ether the program

is continuing to m eet relevant n eeds and the extent to w hich th ose needs

still exist or are being met by other programs.

Significance

2.14 The s ignif icance of an audi t topic should have regard to the

magn itude of its organisational impacts. It will dep end on wh ether the

activity is comparatively minor and whether shortcomings in the area

concerned could flow on to other activities within the agency.

2.15 Significance will rate highly where the topic is considered to be

of particular importance to the agency and where improvement would

hav e a significant imp act on the operations of the agency. A low ran king

in relation t o significance w ou ld be expected w here th e activity is of a

routine natu re and the imp act of poor performan ce wou ld be restrictedto a sma ll area or be likely to hav e minim al impa ct.

Strategic Planning


20/82


2.16 Cross-agency au d its are more likely to ran k high ly on significance.

Visibility

2.17 This criterion is similar to significance bu t is m ore concerned w ith

the external imp act of the p rogram . It is related to the social, econom ic

and environmental aspects of the act ivi ty and the impor tance of i t s

opera tions to the governm ent and the p ub lic. In considering th is criterion

some w eight w ould be attached to the impact of an error or irregularity

on p ub lic accoun tabili ty. It wou ld also have regard to the d egree of

interest by the legislatu re and pu blic in the outcom e of the au d it. Sub jects

that have been identified as current themes for the SAI would generally

w arran t a h igh ran king in terms of visibility.

Coverage

2.18 Coverage refers not only to previous SAI coverage but also to

other ind epen d ent review s of the activity. Such reviews may hav e been

conduc ted by in t e r na l aud i t , ex t e r na l cons u l t an t s o r gove r nm en t

comm ittees or the activity could h ave been su bject to prog ram evaluation.

As a general rule, a low ranking would occur when there has been a

substantial review of the activity within the past tw o years. A higher

ranking would be warranted where a review has been requested by the

legislature or the previou s review ind icated th at such a follow -up sh ould

occur.

2.19 The m ateriality, risk, significance and visibility of an activity w ill

also influ ence the ran king for coverage. If an activity has ran ked h ighly

on all or most of these elements , i t would be expected that the audit

coverage wou ld be more frequen t than for a lower ranked top ic. The

frequ ency of au d it coverage w ould also d epen d on the SAI strategic au d it

plan and on the availability of resources.

Cross-agency and theme audits

2.20 SAI audi t manager s should cons ider appropr ia te themes in

prop osing aud its. Au d its that cross several agencies, ad d ressing themes

w hich are of relevance to the entire p u blic sector or a d d ressing significant

nat ional concerns , can have very high imp act . Potent ia l theme and

cross-agency au d its can be evalua ted in th e same w ay as other au d its for

inclusion in the aud it pr ogram . That is, specific topics w hich rank mor e

highly in terms of risk, impact and materiality should receive priority.


21/82

21

3. Initiating the Performance

Audit

Introduction

3.1 This chapte r ou t l ines the s teps involved wi th in i t i a t ing the

performance audit: the approach to be followed in planning individual

aud i t s , t he r equ i r em en t t o unde r s t and the aud i t e e s bus ine s s and

environment and the development of audit criteria.

Planning individual audits

3.2 INTOSAI Auditing Standards11 identify that the following are

the planning steps ordinarily includ ed in an au dit:

(a) collect in formation about the audited agency and it s organisation

in order to assess risk and t o determine materiality;

(b) define the objective and scope of the audit ;

(c) un dertake preliminary analysis to determine the approach to be

adopted and the nature and extent of enquiries to be made later;

(d) highlight special problems foreseen when plann ing the audit ;(e) prepare a budget and a schedule for the audit ;

(f) ident ify staff requirements and a team for the audit; and

(g) discuss wit h the audit ed agency t he scope, objectives and the

assessment criteria of the audit .

3.3 Planning consists of d eveloping a general strategy and a d etailed

app roach for the expected n ature, timing and extent of the aud it. The

audit plan is a key document for controlling and monitoring audits in

the SAI . The aud i tor shou ld deve lop and d ocum ent an aud i t p land escribing the expected scope and cond uct of the aud it and shou ld p lan

to have the audit work performed in an efficient, effective and timely

man ner. In add ition to the planning steps outlined above, the aud itor

should also consider:

determining the suitabili ty of aud it cr iteria;

determining an efficient and effective approach to cond ucting the audit;

11 Ibid, p. 44


22/82


p lanning to use the work of an expert ;

relevant accountabil ity relationships;

remaining mindful of the users of audi t repor ts ;

reviewing the audi tees internal audi t sys tem;

reviewing re levant p revious audi t findings/ recommendat ions; and

docum ent ing t he aud it p lan.

3.4 A d e q u a t e p l a n n i n g o f t h e a u d i t w o r k h e l p s t o e n su r e t h a t

appropriate attention is devoted to important areas of the audit , that

po t en t i a l p r ob l em s a r e i den t i f i ed and tha t t he wor k i s com ple t ed

exped itiou sly. Planning also assists in p roper assignm ent of w ork to team

mem bers and in coordination of work p erformed by other aud itors and

experts. Aud it plann ing shou ld lead to the developm ent of a detailed

aud it program that id entifies the sp ecific aud it tasks to be un dertaken.

Determining the objectives of a performance audit

3.5 Audit objectives relate to the reasons for conducting the audit

and shou ld be established early in th e aud it process to assist in identifying

the matters to be aud ited and reported. For examp le, an au dit objective

ma y be to assess w heth er an agency uses app rop riate processes to identify

the human resources it needs to achieve its objectives.

3.6 The aud it objectives should ad d ress the concerns of accoun tability

and good governance and should facilitate financial control as well as

im pr ov ing the econom y, e f f i c i ency and e f f ec t i venes s o f p r og r am

management.

3.7 T he aud i t ob j ec t i ve s s hou ld be cond i t i oned by the need to

ma ximise the net benefits and imp acts from the au d it. In setting objectives

the audit team should take into account the roles and responsibilities of

the SAI and the expected net imp act of the au d it as defined in th e strategic

aud it plan. The au dit objectives and scope are interrelated and , since

changes in one usually affect the other , they need to be consideredtogether. It is good m anag emen t practice for aud it objectives and scop e

to be discussed w ith the agency man agement.

The audit program

3.8 The auditor should develop and document an audit program. 12

An audit program outlines the requirements and procedures necessary

12 Refer to Chapter 4 for more details on developing the audit program.


23/82

23

to imp lement the au dit objectives and to m ake assessments against au dit

criteria. The main objectives of an au d it program are to:

e s ta b l is h a cl ea r r e la t i on s h i p b e t w e e n a u d i t o b je ct i v es , a u d i t

methodology, and the anticipated field work to be carried out;

ident ify and d ocument the procedures to be per formed; and

facilita te supervis ion and review.

3.9 An au dit p rogram shou ld set out and specify the following:

aud it objectives and sub-objectives concerning the efficient, economical

or effective operations of the area or agency being audited;

audit cr it er ia to be appl ied ;

the specific tasks to be performed d uring the fieldwork to determine

the extent to w hich agency operations m eet the requirements ou tlined

in audit criteria;

evidence to be collected, includ ing details of aud it sampling and the

timeframe of the activity to be covered;

procedures/ techniques for collecting the evidence;

the a lloca t ion and t iming of t a sks to be per formed by audi t t eam

members; and

any necessary specia l ins truct ions.

3.10 Progress against the planned aud it program should be monitoredand regular briefings provid ed to SAI man agement. The aud it program

should be revised as necessary du ring the course of the au d it.

Understanding the business of an auditee

3.11 It is imp ortant to develop a soun d un derstand ing of the aud itees

bus iness suf f ic ient to achieve the audi t object ives , to faci l i ta te the

identif ication of s ignif icant audit issues and to fulf i l assigned audit

responsibilities. This knowledge w ould includ e an un derstand ing of:

the manda te of the agency and the a reas being aud ited w ith in theagency;

objectives of the agency and relevant programs;

programs and per formance goals of the agency;

organisational and accountabil ity relationships w ithin the agency;

t h e in t e r n a l a n d e x te r n a l e n v ir o n m e n t of t h e ag e n cy a n d t h e

stakeholders;

external constraints affecting program delivery;

Initiating the Performance Audit


24/82


agency management processes and operat ions; and

resources of the agency.

Appendix C de ta i l s a r ange of key f ea tures , o f the audi tee and i t s

environment, that need to be understood by the auditor.

3.12 Obtaining the required k now ledge of the bu siness is a continu ous

and cumulative process of gather ing and assessing information, and

relating the resultant knowledge to audit evidence at all stages of the

aud i t . I t i s impor tant that au di tors weigh u p the costs of obta ining

information and the addit ional value of the information to the audit

outcomes, to ensure that m aximu m value is being obtained from th e aud it

resources expended.

3.13 Sources of information to assis t in understanding the agency

include:

enabling legislation and legislative speeches;

ministerial statements , government submissions and decisions;

r ecen t aud i t r epo r t s , r ev iews , eva lua t ions and inqu i r ie s i n to t he

agency;

the agencys st ra tegic and corporate plans , mission sta tement and

annu al report;

b ud g et statem en ts;

agency pol icy fi le s, man agement comm it tee and execut ive board

minutes;

agency organisation charts , internal guidelines and op erating manuals;

the agencys program evaluation and internal aud it plans and reports;

confe rence r eport s and minutes ;

discussions with agency management and key stakeholders; and

management in formation systems .

3.14 Past audits and other reviews can provide an extremely useful

source of in format ion . They can he lp avoid un necessa ry work in

examining areas that have been under recent scrut iny and highl ight

d eficiencies tha t ha ve not yet been remed ied. There is, how ever, no

substi tute for discussions with senior agency management to gain an

ove r a ll p r og r am pe r s pec t ive aga ins t t he backgr ound o f t he above

information.


25/82

25

3.15 Perform ance mon itoring, accou ntability and eva luation p rocesses

w ithin the pu blic sector are generally agency-based. How ever, there

could a lso be a range of inform ation, wh ich crosses agencies, w hich ma y

also assist in the information-gathering stage, such as:

studies by indu stry, professional or special interest groups;

inquiries or previous reviews by the legislature;

informat ion held by coordinat ing agencies or by interdepar tmental

committees;

r esearch by academics;

work under taken by other governments overseas ; and

m ed ia cov era ge.

Role of preliminary study3.16 Once a topic has been selected for audit , the audit team may

conduct a preliminary study to fur ther understand the activity under

aud it and id entify fun dam ental issues. The preliminary stud y aims to

p rovid e sufficient ju stification to p roceed to a p erform ance au d it or else

to conclud e any fur th er work and report findings. At the end of the

p r e l i m i n a r y s t u d y t h e r e w i l l n o r m a l l y b e a r e p o r t t o s e n i o r S A I

man agement that sum marises the findings of the study an d recommend s

future action.

3.17 If a performance audit is recommended, the preliminary study

report should identify the major issues to be pursued, define the audit

objectives, scope and focus, estimate potential impacts and develop a

timetable and resource bud get to conclud e a timely and defensible audit

r epor t .

3.18 If a p erformance aud it is not recomm ended , the preliminary stud y

repor t should su mm arise the stud y conclusions. To finalise the preliminary

s tudy f ie ldwork, an exi t conference should be held with the agency

management.

Audit criteria

3.19 A u d i t c r i t e r i a a r e r e a s o n a b l e a n d a t t a i n a b l e s t a n d a r d s o f

performance against which economy, eff iciency and effectiveness of

activities can be assessed. They reflect a nor m ative (ie. d esirable) contro l

model for the subject matter under review. They represent good practicea

reasonable and informed persons expectation of what should be.



26/82


3.20 When cr i ter ia are compared with what ac tual ly exis ts , audi t

find ings are generated . Meeting or exceeding th e criteria might ind icate

b e s t p r a c t i c e , b u t f a i l i n g t o m e e t c r i t e r i a w o u l d i n d i c a t e t h a t

improvements can be made.

3.21 Typically, an audit will commence, following its identification bythe SAI p lann ing process, w ith one or more issues to be explored . To

explore these issues then requires the formulation of criteria, followed

by development of test programs to assess whether the cr i ter ia have

been met . But , in prep ar ing these programs and car rying them ou t ,

aud itors may realise that the issues previously d efined m ay need to be

extended or m odified.

3.22 The auditor should be satisfied that suitable criteria have been

identified to enable the auditor to assess the activities subject to audit

and to achieve the au d it objectives. Som e cha racteristics of suitab le criteria

include:

Reliability:Reliable criteria result in consistent conclusions w hen u sed

by an other au ditor in th e same circum stances.

Objectivity: Objective criteria are free from any bias of the au d itor or

management.

Usefulness: Useful criteria result in find ings and conclusions that m eet

users information needs.

Understandability: Understandable criteria are clearly stated and are

not subject to significantly different interpretations.

Comparability: Comparable criteria are consistent with those used in

performance audits of other similar agencies or activities and with

those used in p revious performance aud its within the agency.

Completeness:Com pleteness refers to the dev elopm ent of all sign ifican t

criteria appropriate to assessing performance in the circumstances.

Acceptabi l i ty: Acceptable cr i ter ia are those to which the audi tedagency, legislatu re, med ia and general p u blic are generally agreeable.

The high er th e d egree of acceptan ce of the criteria, the m ore effective

the performance audit.

3.23 Criter ia can perform a ser ies of important roles to assis t the

cond uct of a performance aud it as they can:

form a common basis for commu nication within the aud it team and

w ith SAI management concerning the n ature of the aud it;


27/82

27

form a basis for commu nication with the agency management in that

the audit team will often solicit agency management understanding

of, and concu rrence with th e criteria and eventu al accep tance of au d it

find ings in light of those criteria;

l ink the objectives to the aud it test programs carr ied out d uring theimplementation phase;

form a basis for the data collection ph ase of the aud it, providing a

basis on w hich to bu ild pr oced u res for the collection of aud it evidence;

an d

provide the basis for audit findings, helping to add form and structure

to audit observations.

3.24 The degree to which criteria are successful in serving these uses

is of ten determined by their level of deta i l and the form they take.General aud it criteria are developed d uring the p reliminary stud y. As

the p reliminary stud y p rogresses these criteria are u sually expand ed and

ma d e more specific. By the end of the p reliminary stu d y, the criteria

s h o u l d b e s u f f i c i e n t l y d e t a i l e d t o g i v e c l e a r g u i d a n c e f o r t h e

implementation stage of the audit, particularly in the development of

specific audit programs to test the criteria.

3.25 I t is unrealis t ic to expect that activit ies , systems or levels of

p erform ance relating to econom y, efficiency an d effectiveness will alway s

fully meet the criteria. I t is imp ortan t to app reciate that satisfactoryperformance does not mean perfect p erformance but is based on w hat a

reasonab le person w ou ld expect, taking into account agency circumstan ces.

The general a im would, however , be high level per formance within

resource constraints.

3.26 SAIs should note the implications for performance auditing of

the emerging shift in the role of many governments from a provider of

services to a facilitator an d regulator and the increasing u se of private

operator s by the pu blic sector. The performan ce au d it focu s, on the

economy, efficiency and effectiveness of public sector activity, may have

to be modi f ied to take account of the inc reas ing pr iva t i sa t ion and

liberalisation of pu blic sector op erations in m any countries. SAIs will be

called upon to devise new approaches, techniques and criteria for the

per formance audi t of programs that may not be direct ly funded by

governments, but may affect the public at large by way of quality of

service, cost and equ ity of access. In this environm ent, p erforman ce

au d iting n eeds to take into account th e shift from a govern men t-centred

to a more people-oriented approach.



28/82


3.27 It is generally useful to obtain agency management input to the

d evelopmen t of criteria. Au d it criteria w ould typ ically be exposed to

the agency at the start of the main p erformance aud it. Any d isagreement

w ith agency man agem ent abou t criteria can then be identified, discussed

and, if possible, resolved at an early stage without impacting adversely

on au dit independence.

Sources of audit criteria

3.28 Each SAI w ill need to d evelop au d it criteria that a re valid for the

natu re of the activity und er review. These may includ e quanti tat ive

and/ or qualitat ive measures.

3.29 To avoid the necessity to create criteria from first principles for

each audit, the audit team should investigate the following sources of

existing criteria:

cr iter ia used previous ly in s imilar audi ts ;

cr iter ia publ ished by other SAIs ;

performance standards used by the agency, or previous inquiries by

the legislature;

agencies carrying out s imilar activities;

professional organisations and standard-sett ing bodies; and

general management and subject mat ter l itera ture .

3.30 These sources provide a basis for the development of suitable

criteria for the audit, but may require interpretation and modification to

ensure their relevance to the agency.

3.31 Criteria m ust b e realistic and take into account th e context of the

agen cy. Som e key criteria relate directly to the agency itself, for exam ple:

enabling and re la ted legis la t ion;

agency operat ing and procedures manuals ; and

central agency p olicies , s tand ards, directives and guidelines.

Agency performance information

3.32 Informat ion about the per formance of an agency underp ins

perform ance au d it ing. Perform ance information, ei ther qu anti tat ive

measures or qua l i t a t ive as sessments , i s fundamenta l to eva lua t ing

econom y, efficiency an d effectiveness. Criteria relating to sat isfactory

performance can be derived from the agencys own objectives or from

accepted indu stry and / or government standard s of performance.


29/82

29

3.33 In examining p erformance information, aud itors shou ld:

consider w hether the agency has sufficient and reliable procedures in

place to measure and report on p erformance;

ascer ta in w hether the per formance measures in p lace are complete ,

relevant a nd ju stified on a cost-benefit basis;

e xa m i n e p r o ce d u r e s t o d e t er m i n e if th e y r e la t e t o th e a g e n cy s

corporate goals; and

consider whether the performance measures are incorporated into the

ma nagem ent d ecision-mak ing processes; that is, are they report ed an d

used within the agency.

3.34 These issues draw the performance auditor into a consideration

o f q u a n t i t a t i v e a n d q u a l i t a t iv e p e r fo r m a n c e i n f o r m a t i o n . Su c h

consideration should be an essential element in all performance audits.

3.35 Appendix D conta ins a l i s t o f impor tan t i s sues tha t can be

considered in a performan ce au d it of any gov ernm ent program or activity.

As such they serve as a useful checklist for performance auditors at the

start of a performan ce au d it. The qu estions are designed to help iden tify

potential issues and problems that can be explored in m ore detail.



30/82


4. Implementing the Performance

Audit

Introduction

4.1 The audit objectives and criteria will normally be tested by an

audit program of procedures that include:

observing, interviewing and documenting;

tes ting and checking; and

analysing.

Developing the audit program

4.2 In developing the au dit p rogram, it is important that the criteria:

relate to the aud it objectives; that is, enable relevant eviden ce to be

collected on issues w hich will maximise the imp act of the au d it;

are clear ly s tated and include sufficient detail to enable them to be

readily understood by those carrying out the audit;

are organised in a logical manner so that the aud it examination can be

conducted as efficiently as possible; form an efficient method of gather ing suf ficient evidence without

superfluous testing; and

t ake accoun t o f any ea r li er r ela t ed aud i t wor k and / o r pu b li shed

research on the topic.

4.3 Performan ce aud it programs n eed to be customised for each au d it.

Furth er factors to be considered w hen d eveloping th e program s includ e:

sizeaud it programs generally increase in s ize and complexity (more

detailed procedures, questionnaires and checklists) with increases inthe scope of the au d it;

geographic d i sper sionthe d i sper sion and loca t ion of s it e s to be

visited can marked ly affect the au d it pr ogram . Detailed procedu res

may be required to ensure consistency w hen different personnel are

carrying out the same audit at different locations;

aud it environmentmanagement receptiveness to being audited and

the sensitivity of the area in the agency will affect the way in which

procedures are developed and applied;


31/82

31

the components of the program/ act ivity or the system to be aud ited,

eg. its inputs, processing activities, outputs and outcomes; and

w hether broad issues only have been identified, or specific cr iter ia

are available for audit examination.

4.4 Au dit man agers shou ld ensure that m eans of collecting app ropriate

eviden ce are listed aga inst each au d it criterion. The aud it ma nager shou ld

also consid er th e costs vis-a-vis the benefits of collecting evidence.

4.5 In some cases, a cr i ter ion that appears reasonable may not be

able to be tested at all. For examp le, a criterion that appropriate priority

be given to the development of an information system could n ot be assessed

w ithout kn owing all other priorities and their justifications. Thus the

criterion shou ld be revised. Criteria m ust also be reviewed for consistency

w ith th e legislation, for p racticality an d for comp rehension by field staff.4.6 Furth ermore, once preliminary research/ fieldw ork is un der way,

new issues may arise that warrant reconsideration and revision of the

initial criteria. H ow ever, before add ing new issu es, the likely imp act on

the audit budget and timetable must be considered.

4.7 In developing an aud it program it will not be p ossible to anticipate

all contingencies. In the early stages of an au d it, there is a need to retain

flexibility and to review the aud it pr ogram for ap pr opr iateness. I t is

preferable to start with a program outlining the approach to the audit

issues and revise and extend it as the aud it develops.

4.8 One approach to developing an audi t program is an i tera t ive

process such as the one outlined in Figure 3:

Figure 3The Audit Program Revision Cycle


4.9 In light of observations on the ou tcomes of this process, plan ning

f o r t he nex t s t age o f t he aud i t p r og r am can be m od i f i ed i . e . t he

interpretations/ conclusions arising from one stage of the aud it program

can be used to review and mod ify the next stage.


32/82


Audit approaches

4.10 There are a number of analytical approaches to undertaking an

au d it prog ram and mak ing assessments against criteria. These ap proaches

include:

analysis o f p rocedures ;

use of exist ing data or evidence provided by the agency;

analysis o f r esu lt s;

case stu d ies;

su rveys; an d

quantitat ive analysis.

Analysis of procedures

4.11 An analys is of procedures is of ten a s tar t ing point for audi t

analys is. The au d i tor w ould review sys tems in place for planning,

condu cting, checking and m onitoring the activity being aud ited . This

would involve interviews of managers and examination of documents

such as budgets, financial reports, program guidelines, annual or other

plans , procedures manuals , delegat ions and repor t ing requirements .

Procedures would be tested against established criteria or a desirable

control mod el. This would typ ically mean that procedu res w ould be

checked, among other things, for completeness, relevance against thelegislation, internal consistency and practicability.

Use of existing data

4.12 It is impo rtant for au d it staff to investigate the dat a held by agency

man agement and by o ther r e levant sources . This may inc lud e the

m a n a g e m e n t i n f o r m a t i o n s y s t e m s u s e d t o m a n a g e a g e n c y

program s/ activities and / or the data collected on ind ividu al program s.

Analysis of results

4.13 Analysis of resu lts from examin ing a nu mber of instan ces of agency

activity in a particular area w ill help decide w heth er agency performan ce

in that area conforms to audit criteria and is generally satisfactory.

Case studies

4.14 The case study is a method for learning about a complex issue,

based on a comprehensive und erstand ing of the particular instance. The

case stud y involves an extensive descrip tion and analysis of the p articu lar

issue within the context of the whole area under review.


33/82

33

Surveys

4.15 Anoth er meth od of obtaining in sight into an agencys activities is

by the cond u ct of a surv ey. This is a m ethod of collecting information

from members of a population (such as the agencys staff, suppliers or

clients) to assess the incidence, distribution, and interrelation of eventsand conditions.

Quantitative analysis

4.16 Where pract icable , an ent i re populat ion should be analysed.

Where this is not feasible, due to cost and time constraints, sampling

techniques should be used . The natu re of the popu lation should be

examined to d ecide the most app ropriate sampling method ology. When

u sing either a statistical or a non -statistical samp ling ap proa ch, the au d itor

should select an ap prop riate aud it samp le, perform au dit procedu res on

the samp le and evaluate sample results so as to provide su fficient au dit

evidence.

4.17 When selecting an au dit samp le, the aud itor shou ld consid er the

specific au d it objectives and the attribu tes of the p opu lation from w hich

the aud itor wishes to draw th e samp le. In determining the sample size,

the au ditor should consider wh ether sampling risk w ill be redu ced to an

accept ably low level. The au d itor shou ld select samp le items so as to

have a reasonable expectation that all sampling units in the population

hav e an equ al chan ce of selection.

4.18 To each item selected th e aud itor shou ld ap ply au dit p rocedu res

ap pr opr iate to the pa rticu lar au d it objective. The au d itor shou ld consid er

what conditions would constitute an anomaly or error by reference to

the aud it objectives. The aud itor should consider the natu re and cause

of any errors identified and their possible effect on the particular audit

objective and on other areas of the audit.

4.19 Errors foun d in the samp le shou ld be p rojected to th e pop ulation.

The auditor should consider the effect of the projected error on theparticular audit objective and on other areas of the audit.

Refining the audit program

4.20 The initial au d it program shou ld be refined by d eveloping action

lists that reorgan ise the p rogram by au d it activity. For exam p le, for the

au d it activity interviews, an au d itor wou ld list all criteria/ issu es that

are to be ad d ressed in this way. At the same time, questions for the

interviews can be dev eloped from th e criteria/ issu es. Similarly, a list of

audit s teps should be established against other activit ies such as f i le

searches.



34/82


4.21 The result of this process is that for every criterion there is at

least one audit step (sometimes several will be necessary; for example,

interviews alone rarely p rovide ad equa te eviden ce). As well, for every

audit step there is a relevant issue related to the criterion.

4.22 It is also likely tha t the initial aud it progr am w ill require revisionto take accoun t of external factors. For examp le, the p recise natu re of

the d ata held (whether in d atabases or on files) may not be know n, nor

the likely difficulty of access. A decision is then necessary on w heth er to

aban d on th e test, m ake an in terim find ing of lack of satisfactory record-

keeping, or pu rsue an alternative means of gathering evidence.

4.23 I n c a r r y i n g o u t p l a n n e d a u d i t s t e p s a d d i t i o n a l r e l e v a n t

information, not explicitly covered by the audit program, may come to

light. In this case, the aud itor shou ld identify the issues highlighted by

the new information.

4.24 One of the p urp oses of the preliminary stu dy is to refine the au dit

pr ogram . It is also ad visable to review the prog ram early in the field w ork

s tage of the main aud i t to ensure it remains app ropr ia te. H owever,

revisions at this stage may lead to incompatibility of results from the

beginning to the end of the aud it. This w ill need to be consid ered by the

audit manager against the advantages of improving the program.

4.25 The level of detail of the final audit program will depend on a

number of factors:

the complexity of the aud it issues to be tested;

the extent of the aud it; for example, a large audit carr ied ou t in several

locations w ould need a d etailed program to ensure consistency; and

the level of the s taff carrying ou t the aud it; wh ere jun ior s taff have

responsibility for carrying out fieldwork, a more detailed program

would normally be appropriate.

4.26 For multiple-agency audits , the main issue in the program is

w heth er each au d it step need s to be carried ou t in each agency. Different

requ irements for samp le sizes ma y lead to some au d it steps being carried

ou t in on ly certain agen cies.

4.27 For single-agency audits, the auditor needs to be careful that the

instances chosen for analysis can be extrapolated to form a fair p ictu re of

the agency or pr ogram u nd er review as a w hole. In the case of cross-

agency audi t s , the re i s a s imi la r concern tha t agenc ies chosen for

examination are suff iciently representative or that the results can be

reasonably inferred across the public sector.


35/82

35

Audit test programs

4.28 Audit test programs form an integral part of the audit program.

An audit test program refines audit criteria into a series of procedures

and / or activities (tests) to obtain relevant an d r eliable eviden ce up on

w hich conclusions may be d rawn .

4.29 Audit test programs are the key link between the development

of aud it objectives/ criteria and the condu ct of an au dit leading to credible

and d efensible find ings. This is illustrated in Figure4.

Figure 4The Development of Audit Test Programs

AuditObjective

TestMethodology

TestQuestions/Analysis

Examine findings and reassesscriteria if necessary.

Test Programs

FindingsAudit

Criteria


4.30 Test programs should not be prescr iptive but should have the

followin g characteristics:

Clear purpose: The purpose of the test program should be clear ly

defined from the outset to show relevance to the audit topic and to

effectively focus the audit.

Easily understood: The test program should be easy to understand,

w ith any jargon and acronym s explained.

Sound logic: A logical link should exist between the objectives of the

audit, the audit criteria and the audit test program.

Good layout/design: Page layout and design should be kept simple,

preferably in a working paper style format for easy referencing.

Good co-ordination: An au d it test program shou ld be used to clearly

define the roles of each team member to avoid duplication of work.

Flexibility: Test programs should remain flexible to allow for the

introdu ction of new evidence/ criteria, and the exclusion of outd ated

or irrelevant evidence/ criteria.

Cost effectiveness: Test programs must ultimately be cost effective;tha t i s , the t ime and r esources used on a t e s t p rogram mus t no t

outw eigh th e likely benefit such a p rogram w ill prod uce.


36/82


Developing findings and recommendations

4.31 Audit findings are identified by relating audit observations to

au d it criteria. Au d it observations are based on th e analysis of information

collected du ring the aud it . Aud it findings should be d eveloped an d

eva lua t ed t h r oughou t t he va r ious phas e s o f a pe r f o r m ance aud i t .P o t en t i a l f i nd ings i den t i f i ed i n t he p l ann ing s t age o r du r ing t he

preliminary study should be followed up in the detailed examination

ph ase of the aud it.

4.32 I n s t ances w he r e ag ency p e r fo r m an ce exceeds t he expec t ed

per formance (as infer red f rom the audi t cr i ter ia) are good pract ice

findings, provided the targets/ benchmarks are realistically d etermined ,

and s hou ld a l s o be r epo r t ed . The p r oce ss o f ana lys ing ev idence ,

d eveloping findings and pr odu cing recommen d ations to resolve identified

areas of poor p ractice is su mm arised in Figure 5.

Figure 5The Process of Developing Recommendations

Audit criteria (what should be)

Audit evidence (what is)

Audit findings ('what is' compared with 'what should be')

Determine the causes and effects of the finding

Develop audit conclusions and recommendations

Estimate likely impacts of the recommendations wherever possible

4.33 The detailed evaluation of audit findings is generally completed

du ring the p reparation of discussion pap ers for d istinct segments of the

au dit or near the conclusion of the aud it f ieldw ork. H ow ever, some

evaluation m ay extend into the reporting p hase, as find ings are challenged

and further evid ence is obtained . It is at this latter stage that a final

decision is reached on the findings and recommendations that will be

reported. Once an aud it finding has been identified , two comp lementary

forms of assessmen t tak e p lace; an assessment of the significance of thef inding and the determination of the causes of good performance or

under-performance (where it is below that expected).


37/82

37

4.34 The effect of a find ing m ay be qu antifiable in m any cases. For

e x a m p l e , t h e c o s t o f e x p e n s i v e p r o c e s s e s , e x p e n s i v e i n p u t s o r

u np rodu ctive facilities can u sually be estim ated. Ad d itionally, the effect

of inefficient p rocesses, for exam ple id le resources or p oor m anag emen t,

may become apparent in terms of time delays or wasted physical and

other resour ces. Qu alitative effects, as evid enced by a lack of control,

poor decisions or lack of concern for service, may also be significant.

The identified effect should demonstrate the need for corrective action.

The effect can also have occurred in the past, be occurring now or may

p ossibly occur in the futu re. If the effect occur red in the p ast the aud it

finding will stand only if the situation h as not already been remedied to

prevent it from recurring.

4.35 The aud itor shou ld id entify the un derlying cause of a find ing, as

this form s the basis for the recommen d ation. The cau se is that w hich, if

changed, would prevent similar findings. The cause may be outside the

control of the agency under audit, in which case the recommendation

shou ld d i r ect a t t en t ion ou ts id e the agency. H ow ever , the agency

concerned should be provided with the relevant part of the report for

commen t. This factor should also be taken into consideration w hile

developing the timeline for reporting.

4.36 The auditor may identify a cause-and-effect chain and have the

op tion of report ing the find ings at different po ints in the chain. In this

situation the auditor should highlight the most critical deficiency in the

chain.

4.37 Findings m ay be p resented in d iscussion pap ers for comment by

agency management. Agency responses can then be documented and

analysed. Where the agency d isagrees w ith the aud i t f ind ings and

recommendations, the reasons for such disagreement should be fully

analysed.

4.38 The recommendations made by the SAI regarding performance

aud its have to be argued in a logical mann er. Recomm endations should

indicate broadly what issues might be examined by management whenseeking solutions an d shou ld focus on the m ore significant issues requiring

at tent ion . The less s ignificant issues should b e refer red to agen cy

man agement for action.

4.39 Recomm end ations requ ire careful review to ensure that they are

practical and add value. The aud itor should ensure, for examp le, that

the recommendation addresses the objectives of the audit; ie. economy,

efficiency, effectiveness or accou ntability as app rop riate. A good test for

the auditor is to consider how the recommendation would be followed

up ; how the implementation of the recomm endation could be tested; andwha t s pec i f i c a c t i ons t he agency can unde r t ake t o im p lem en t t he

recommendation.



38/82


5. Evidence and Documentation

Audit evidence5.1 Aud it evid ence is information collected an d used to sup port au dit

findings. The conclusions and recomm end ations in the aud it report stand

on the basis of such evidence. Consequ ently, perform ance au d itors m ust

give careful thou ght to the n ature, quality and amou nt of evid ence they

collect.

5.2 All field w ork shou ld be plan ned from the persp ective of acquiring

evid ence intended to sup port th e findings ap pearing in the final report.

The p lan for condu cting fieldw ork is the aud it program wh ich, in turn , isbased on the audi t object ives and on cr i ter ia where they have been

developed.

Competence, relevance and sufficiency of evidence

5.3 Th e INTOSAI Auditing Standards13 state Competent, relevant and

reasonable evidence should be obtained to support the auditors judgement and

conclusions regarding the organisation, program, activity or function under audit .

Competence of evidence5.4 Evid ence is comp etent (valid an d reliable) if it actua lly rep resents

w hat it pu rp orts to repr esent. The reliability of eviden ce can be ensu red

and assessed by considering the following:

cor robora t ion of ev idence is a pow er fu l t echnique for inc reas ing

reliability. This involves the au d itor looking for different typ es of

evidence from different sources;

evidence sourced from outside the agency is normally viewed as more

reliable for au di t p urp oses than informat ion generated within th e

agency;

documentary evidence is usually considered to be more reliable than

oral evidence;

evidence generated through d irect aud itor observation or analysis is

more reliable than indirectly obtained evidence;

the reliability of agency-generated information will be a fun ction of

the reliability of th e agen cys intern al control systems;

13 Auditing Standards, Auditing Standards Committee, INTOSAI, June 1992, pp. 50, 42


39/82

39

ora l evidence that is cor roborated in w r it ing is more re liable than

oral evidence alone; and

original docum ents are more reliable than p hotocopies (if or iginals

are photocopied the auditor should note the source of original and

the date cop ied) . Photocop ies of eviden ce regard ed as being of s ignif icant impor tance to an audi t should, whenever poss ible , be

certified by relevant authorities.

Relevance of evidence

5.5 Relevance requires that the evidence bear a clear and logical

relationship to the aud it objectives and to the criteria. One app roach to

planning for data collection is to list, for each issue and criterion, the

nature and locat ion of evidence that is needed, as wel l as the audi t

procedure that is to be implemented.

Sufficiency of evidence

5.6 The aud itor shou ld obtain su fficient ap prop riate au dit evidence

to be able to draw reasonable conclusions on which to base the audit

r ep or t . Su f ficiency i s the measu re of qu ant i ty of au d i t ev iden ce .

Ap pr opr iateness is the measu re of qu ality of au d it evid ence, its relevance

to particular criteria and its reliability.

5.7 Evid ence is sufficient w hen there is enou gh r elevan t and reliable

information to p ersuade a reasonable person that the p erformance aud itfindings, conclusions and recomm endations are warranted and supp orted.

In determining whether documentary evidence is sufficient, the auditor

m ust take accou nt of the status of the d ocum ent; eg. d raft intern al agency

d ocum ents ma y hav e little statu s in terms of the agencys intentions and

decisions.

5.8 The factors that dic ta te the s t rength of evidence required to

sup port an observation in performance auditing includ e:

level of materiality or significance of the observation; d egree of risk associated w ith coming to an incorrect conclusion;

exper ience gained in previous audi t examinat ions on the degree of

reliability of the agencys records and representations;

known agency sens it ivity to an issue; and

cos t of obta ining the evidence rela t ive to the benefi ts in terms of

supporting the observation.

Evidence and Documentation


40/82


5.9 E v i d e n c e g a t h e r e d d u r i n g a p e r f o r m a n c e a u d i t m a y b e

predominant ly qua l i t a t ive in na ture and r equi r es ex tens ive use of

professional jud gemen t. Accord ingly the aud itor w ould ord inarily seek

corroborating evid ence from d ifferent sou rces or of a d ifferent n atu re in

making assessments and forming conclusions.

5.10 When p lanning the aud it, the aud itor w ould id entify the probable

natu re, sources and a vailability of au d it eviden ce requ ired. The au d itor

should consider such factors as the availability of other audit reports or

stud ies and the cost of obtaining th e aud it evid ence.

Characteristics of performance audit evidence

5.11 I n pe r f o r m ance aud i t t e r m s , aud i t ev idence i s t he f ac t s o r

information used:

to conclud e whether an agencys management and employees have

accepted and carried out app ropriate actions to conform to operational

pr inciples , pol ic ies or s tandards for us ing resources ef fect ively,

efficiently and economically; and

to demonstrate to a third p ar ty that the aud itors findings are credible

and defensible.

5.12 Auditors need to be aware of potential problems or weaknesses

w ith perform ance au d it eviden ce. Potential pr oblems includ e:

evidence based on a s ingle source (this may imp act on re liabil ity,

validity and sufficiency);

o r al ev id e n ce n o t s u p p o r t a ble b y d o cu m e n t a t io n o r o b se r va t io n

(reliability);

evidence is not time-sensitive, ie. outd ated and does not reflect changes

(relevance);

evidence too expensive to obtain relative to benefits (relevance and

sufficiency);

source of evidence has a vested interest in outcome (reliability);

s a m p l es co l le ct e d a r e n o t r e p r e s en t a t i v e (r e le v a n c e, v a li d i t y,

sufficiency);

evidence may be related to an isolated occu rrence (validity, sufficiency);

evidence is incomplete, ie . does not dem onstra te a cause or effect

(reliability, sufficiency); and

evidence is conflicting (reliability).

5.13 Evidence can be ca tegor i sed as to i t s typephys ica l , o ra l ,documentary or analytical.


41/82

41

Physical evidence

5.14 Physical evidence is obtained by observing people and events or

examining prop erty. The eviden ce can take the form of ph otograp hs,

charts, ma ps, graph s or other pictorial representations. A picture of an

unsafe condition is far more compelling than a written description.5.15 When the ob servation of p hysical cond ition is critical to achieving

the au d it objectives it should b e corroborated . This ma y be achieved by

h a v i n g t w o o r m o r e a u d i t o r s m a k e t h e o b s e r v a t i o n , i f p o s s i b l e

accompanied by representatives of the agency.

Oral evidence

5.16 Oral evidence takes the form of statements that are usually in

respon se to inqu ir ies or interview s. These s ta temen ts can pr ovid e

important leads not always obtainable throu gh other forms of aud it work.They can be m ad e by em p loyees of the ag ency, beneficiaries and clients

of the program being audi ted, exper ts and consul tants contacted to

provid e corroborating evidence in relation to an aud it, and by m embers

of the general p ub lic.

5.17 Corroboration of oral evidence is needed if it is to be used as

evidence rather than simp ly as backgroun d. This could be:

by wr it ten confirmat ion by the interviewee;

by weight of mu ltiple ind ependent sources revealing the same facts;or

by checking records la ter.

5.18 In assessing the reliability and relevance of oral evidence the

auditor needs to have regard to the credibility of the interviewee; that

is, the position, knowledge, expertise and forthrightness of the person

being interviewed .

Documentary evidence

5.19 Docum entary eviden ce in p hysical or electronic form is the most

common form of audit evidence. It may be external or internal to the

a g e n c y. E xt e r n a l d o c u m e n t a r y e v i d e n c e m a y in c l u d e l e t t e r s o r

m em

asosai performance audit guidelines

Documents