performance auditing outcomes: a comparative study

Financial Accountability C? Management, 4(1) Spring 1988, 0267-4424 $2.50

PERFORMANCE AUDITING OUTCOMES: A COMPARATIVE STUDY

D A V I D J . HATHERLY AND LEE D . PARKER’

Published research and professional literature on performance auditing to date has been largely concerned with outlining general international experi- ence and developments, considering relevant concepts and describing single - system processes. This observation also applies to audits entitled ‘value- for-money’ or ‘operational’ but which employ the same key concepts and pursue the same objectives (Parker, 1986). Indeed the 1986 International Congress of Supreme Audit Institutions (1986) selected ‘performance audit’ as the common future term to be applied to audits examining economy, eff- ciency and effectiveness of operations.

Little by way of research results are available however, with respect to the results or impacts of this type of audit. This paper presents the results of a n exploratory study designed to provide preliminary evidence as to certain apparent outcomes of performance audits through the examination of a sample of reports produced in Australia over recent years. Performance auditing in Australia is in its infancy but has been undertaken by a number of Auditor- Generals in the Commonwealth and different states of Australia. Reports produced for the Commonwealth by the Australian Audit Office (AAO), in Victoria by the Victorian Auditor-General (VAG) and in South Australia by the South Australian Auditor-General (SAAG), are examined with respect to concepts employed, report content, report format and auditee responses. These variables are also examined in the light of different approaches to performance auditing employed by the three Auditors-General.

DEVELOPING VIEWS

In attempting to assess the relationship between views of performance auditing which have been developing in the auditing literature and its application in practice, this study focuses upon four aspects. These are:

* The authors are respectively, Professor of Accounting at the University of Edinburgh; and Professor of Accounting at the School of Administration, Griffth University, Brisbane, Australia. They are grateful for data and advice supplied by the Auditor-Generals’ offices of South Australia, Victoria and the Australian Audit Office and for the assistance of Greg Pound of the Australian Account- ing Research Foundation. This paper has also benefited from comments by a Griffith University Accounting Group Workshop, Professor Dale Flesher (University of Mississippi), James Guthrie (University of New South Wales), Greg Pound (AARF), Roger Simnett and Dr Allan Farley (Monash University).

22 HATHERLY AND PARKER

1 The extent of agreement between conceptually preferred constituent concepts expounded in the performance audit literature and those stated by the Auditors-General.

2 The goodness of fit between concepts stated by Auditors-General and the concepts they actually employ.

3 The degree of conformity of Australian audit report formats to those recommended in the international performance audit literature.

4 The type of auditee responses to Australian reports compared with some observed responses recorded in international literature.

Performance Audit: Constituent Concepts

As reported by Parker’s (1986) review of international published audit literature, a high level of agreement exists in the audit literature concerning the preferred conceptual definition of performance auditing, value-for-money auditing, operational auditing and management auditing. A performance audit is generally defined as the independent examination and evaluation of the economy and efficiency of an entity’s operations as well as the effectiveness of its programmes (Knighton, 1973; Cabnet, 1976; Kaiel, 1976; Fountain and Lockridge, 1976; and Rothwell, 1984). Similarly, a value-for- money audit has been generally defined as an audit examination designed to determine whether the organization in question is performing economically, efficiently and effectively in its use of resources, operating procedures and pursuit of objectives (Flint, 1978; Fielden and Robertson, 1981; Langhout, 1984; Special Report, 1984; and McSweeney, 1985). Operational auditing has been generally defined as an examination of an organization’s financial and non-financial operations in order to ascertain the degree of economy and efficiency of its operations and resource utilisation, and to determine the effectiveness of its operations in achieving stated programme and organizational objectives (Phyrr, 1969; Morse, 1983 and 1976; Garrido, 1980; Lane, 1981; and Ponder, 1984). Finally a management audit has been generally defined as an evaluation of management and the organization’s functioning and performance with respect to economy, efficiency and effectiveness of operating areas, activities and results (Smith, Lanier and Taylor, 1972; Santocki, 1976; Vangermeersch, 1976; Tanaka, 1978; Ellis and Melrose, 1980; and Siege1 and Milich, 1985).

The above conceptual definitions are clearly common across the terms employed. This overlap has been well recognised in the literature (Flesher, 1977; Hara, 1976; Crockett, 1980; Allard, 1981: Lane, 1983; Boys, 1984; Rothwell, 1984; Cowperthwaite, 1985; Chorlton and Mills, 1985; and Parker, 1986) and by the International Congress of Supreme Audit Institu- tions (1986) in its adoption of ‘performance audit’ as the most appropriate common term. There is also fairly clear agreement as to the definition of the

PERFORMANCE AUDITING OUTCOMES: A COMPARATIVE STUDY 23

constituent concepts of economy, efficiency and effectiveness. Economy is a resourse acquisition concept emboyding a least cost notion and has been defined as the acquisition of human and material resources of appropriate quality and quantity at the lowest reasonable cost (Dilley, 1975; Chandler, 1984; and Special Report, 1984). Efficiency is a resource usage concept which can be defined as the case of a given set of resources that maximise associated outputs at minimum total cost o r the use of minimum input resources for a predetermined level of output (Malan, 1979; Chandler, 1984; and Special Report, 1984). Effectiveness is an ends-oriented rather than a means-oriented concept that has been defined as the degree to which predetermined entity objectives for a particular activity or programme are achieved (Malan, 1979; Drebin, 1980; Chandler, 1984; and Special Report, 1984).

Adherence to Conceptual Objectives

It is clear therefore that performance auditing has conceptually preferred objectives. These are the examination of the economy and efficiency of a n entity’s operations and resource usage as well as the effectivness of its programmes. These objectives were compared with the concepts formally defined by each Auditor-General as his performance audit objectives.

The AAO states its conceptual objectives in its publication Elements of EJi- ciency Auditing: A Basic Guide (1986). It claims to follow ‘the comprehensive audit approach’ citing the GAO’s approach in the USA as an example.’ Efficiency audit is ‘the AAO term applied to the performance audit and its responsibilities cover examination of efficiency and economy of the operations or functions of an audited organisation, and the procedures of that organisation for reviewing the efficiency and economy of its own operations in achieving the objectives specified by government’ (1986, p. 2). The AAO further states that it is concerned with assessing the efficiency of administrative systems and processes employed in the deployment and use of financial, staff- ing and other resources. Audits are directed at any area where wasteful use of money, personnel or other resources is suspected. These conceptual objectives are said to be defined by Australian Federal Government legislation and the AAO alleges that it has been precluded from examining effectivness by the legislature (AAO, 1986).‘

It is clear then that the AAO’s stated conceptual objectives prior to September 1986 did not adhere to the generally accepted conceptual objectives of performance auditing. While encompassing economy and efficiency, they excluded consideration of effectivness.

The VAG states its conceptual objectives in the comprehensive audit report on government stores operations and departmental cash management (Waldron, 1984). It claims to embrace the Canadian model of comprehensive auditing including ‘an independent and objective assessment of the monitoring processes used within organisations to ensure that resources are

24 HATHERLYANDPARKER

economically and efficiently utilised and objectives of programs are achieved’ (Waldron, 1984, p. 1). This section of the comprehensive audit is described as value-for-money auditing. The VAG describes the monitoring of efficiency as evaluating both the auditee’s information and systems designed to facilitate efficiency. Monitoring effectiveness is described as identifying major operational objectives of the auditee, ensuring their consistency with legislative/ministerial directives and higher level objectives, and assessing the adequacy of auditee mechanisms in place for monitoring achievement of operational objectives.

The VAG’s stated conceptual objectives fall short of conceptually preferred objectives for performance auditing. They examine economy and the auditee’s own mechanisms for monitoring efficiency and effectiveness, whereas conceptually preferred performance audit objectives include an examination of actual efficiency and effectiveness. In addition the VAG states that economy is only reviewed ‘in some circumstances’ (Waldron, 1984, p.

The SAAG states its conceptual objectives in its annual report for the year 1984-1985 (Sheridan, 1985). It claims that its traditional financial and compliance audit has always contained ‘an element of value for money auditing’ (Sheridan, 1985, p. 5). This is defined as an audit that ‘identifies matters of efficiency and economy that often arise during the course of the normal audit examination’ and ‘then sets about confirming whether the operation, or the programme, or the function can be undertaken in a more efficient and economical way’ (Sheridan, 1985, p. 5). Therefore the SAAG argues that this form of auditing is an extension of the financial and compliance audit.

Once again it is readily apparent that the SAAG’s conceptual objectives diverge from conceptually preferred objectives for performance auditing. They include examination of economy and efficiency of operation but exclude consideration of effectiveness.

When the three cases are compared, it is evident that all three fail to adhere to the conceptually preferred objectives of performance auditing. It is important to recognise, however, that this failure is in relation to the Auditors’-General stated objectives. The AAO and SAAG exclude any consideration of effectiveness. The VAG claims not to consider actual efficiency and effectiveness but only auditee mechanisms designed to facilitate efficiency and effectivness. The relationship between the Auditors’-General stated conceptual objectives and those actually evidenced in their reports will be considered shortly.

2).

Perfrmance Audit: Report Formats

Parker’s (1986) survey of the audit literature found virtual unanimity in the advocacy of the long-form audit report. This form was generally advocated for both private and publicly available reports. Reasons cited, included the


degree of subjectivity involved in the audit process, the broad scope of the audit, problems of potential legal liability of auditors, measurement difficulties, interdisciplinary audit teams, variety of evidence forms and non-financial as well as financial dimensions (Caldwell, 1975; Flint, 1978; Flesher, 1980; Knoll and Howard, 1983; and Siege1 and Milich, 1985).

In terms of general long-form report format, the auditing literature offers a number of normative proposals. The AICPA’s Special Committee on Oper- ational and Management Auditing suggested the following:

1 Introduction (including objectives, scope, general approach and procedures employed).

2 Summarised list of critical findings. 3 Recommendations. 4 Detailed supporting data (where appropriate).

Ponder ( 1984)

From a UK perspective, Flint (1978) suggested that any performance audit report introduction should include the scope of work done, the activity examined, the purpose of the examination, the extent and nature of the examination and measurement/evaluation criteria. Further normative guidelines can be distilled from the GAO’s reporting standards for public sector audits in the USA:

1 Reports should be concise, clear and complete. 2 Factual matter should be presented accurately, completely and fairly. 3 Findings and conclusions should be presented in as simple language as

the subject permits. 4 Reports should include auditor’s opinions on causes of problem areas

and recommendations for improvements. 5 Emphasis should be placed upon improvement possibilities rather than

criticism of the past. 6 Noteworthy accomplishments and improvements should be recognised. 7 Views of auditee officials should be recorded. 8 Audit scope and objectives should be clearly explained.

(Comptroller General, 1974)

More detailed performance audit report formats are to be found in Bromage (1971 and 1972) and Flesher (1984).

Auditee Responses

The question of auditee responses to performance audits has barely been can- vassed in the audit literature. Parker (1986) has referred to possible political resistance (particularly to the notion of auditing ‘effectiveness’) which has emerged in public debates in Australia. Hammel (1977) and McSweeney


(1985) have warned against the dysfunctional effects of any attempt to force auditee performance standards into any preconceived ‘cookie cutter’ mould. Indeed Hammel has gone so far as to warn that such an audit may risk being perceived as a frontal assault upon the satisfaction, needs and motivation of management. Auditor recommendations may be interpreted as questioning or challenging managers’ judgement or decisions. Auditees may also perceive that uninformed outsiders are making recommendations that encroach upon their management prerogatives and threaten their organizational status (Doades, 1978: and Haribhakti, 1982).

Evidence of auditee resistance to performance audits has already been observed in Canada. A survey of municipal officials in Canadian cities con- ducted by the Canadian Comprehensive Auditing Foundation (with a view to ascertaining views of elected representatives) found that auditees considered these audits to be expensive, time-consuming and of limited value and that local politicians were generally cynical in their attitude to them (Bens, 1983; and Cowperthwaite, 1985). This broad scope audit has also been resisted by some prominent Crown Corporations and their directors. Organizations such as Air Canada, Petro Canada and Canadian National Railways consider private sector performance criteria to be adequate and appear to have seen the auditor as infringing on directors’ prerogatives and second-guessing management decisions with the benefit of hindsight (Allard, 1981; and Boisclair, 1984).

ADDRESSING RESEARCH QUESTIONS

While the performance audit literature subscribes to common concepts, definitions and objectives, practical audit approaches do vary. As Parker (1986) has observed in his review of international developments, US audits have focused upon economy, efficiency and effectiveness of operations. Canadian audits have focused upon economy, efficiency and the audit of controls and procedures designed to facilitate effective operations. UK audits have tended to focus upon economy and efficiency of operations. These observations prompt the question as to whether differences in approach to performance auditing cause different audit outcomes.

The primary hypothesis of this study can therefore be stated in its null form as:

H, Variant approaches to performance auditing make no significant difference to actual audit outcomes.

The actual audit outcomes can, of course, be numerous. The variables selected for this study as representing key outcomes, can also be stated in null hypothesis form as follows:


H, Variant approaches to performance auditing make no significant difference to the goodness of fit betweeen declared audit objectives and objectives employed in the audit report.

H, Variant approaches to performance auditing make no significant difference to audit report format.

H, Variant approaches to performance auditing make no significant difference to auditee responses.

These hypotheses are based upon the foregoing analysis and discussion in the paper.

METHODOLOGY

The sample of performance audit outcomes comprised reports of three Auditors-General in each of the Commonwealth of Australia, Victoria and South Australia. The reports examined were as follows:

Australian Audit Office (AAO) 1 Report of the Auditor-General on an Efficiency Audit - Defence

Science and Technology Organisation: task cost management, March 1986.

2 Report of the Auditor-General on an Efficiency Audit - Department of Primary Industry: administration of meat inspection services by the Export Inspection Service, June 1986.

Victorian Auditor-General (VAG) 1 Report of the Auditor-General - Works Contracts Overview: Second

Report, June 1983.

2 Report of the Auditor-General - Comprehensive Audits: Government Stores Operations, Departmental Cash Management, October 1984.

South Australian Auditor-General (SAAG) 1 Report of the Auditor-General for the Year Ended 30 June 1984.

2 Report of the Auditor-General for the Year Ended 30 June 1985.

Sample selection was made on the following basis. AAO, VAG and SAAG reports were chosen as being from Australian audit offices with significantly different approaches to the performance audit (as explained in the folowing paragraph). The two most recent available reports from each office as at July 30, 1986 were selected for examination. The differences in report publication dates were unavoidable given the variable and intermittent issue dates (except for the SAAG annual report) of these reports. In order to compare more than


one report from each office, this necessitated comparing reports across the 1983 - 1986 period.

Each office pursues a different approach to performance auditing. The Australian Audit Office, as evidenced by the title of its special reports, has to date pursued efficiency auditing. Audits of effectiveness before September 1986 were beyond its officially designated scope of authority (being vested in the duties of Department of Prime Minister and Cabinet). The Victorian office, as evidenced by the title of the second report in the sample, has formally stated that it follows the Canadian model of Comprehensive Auditing. Its performance audits therefore include economy and inspection of controls and procedures designed to facilitate efficiency and effectiveness. The South Australian office audits efficiency but not as a separate special audit. Its Auditor-General emphasised that his staff are required to identify efficiency problems during the normal course of the traditional financial and compliance audit. The South Australian report is therefore the traditional annual audit report that encompasses financial, compliance and performance audit findings. This reflects the SAAG's view that performance auditing should be integrated with the financial and compliance audit p r o c e ~ s . ~ The SAAG reports do define the performance audit concept employed and include performance audit findings.

Reports produced by three variant approaches to performance auditing are therefore examined. These are classified in Table 1. One is a special purpose efficiency audit, one is a special purpose comprehensive audit, and one is a subset of a traditional financial/compliance audit.

Case study method has been employed for the purposes of this study. Its recent advocates for use in accounting research include Hagg and Hedlung (1979), Post and Andrews (1982) and Kaplan (1984). This form of analysis

Table 1

Variant Performance Audit Approaches

Office Performance Audit Title Report Type Audit Approach

AAO Economy and Efficiency Efficiency Audit

Separate Special Purpose

VAG Economy and Comprehensive Separate Controls for Audit Special

Efficiency and Purpose Effectiveness

SAAG Economy and Financial1 Traditional Eficiency Compliance Annual

Audit


is predominantly qualitative and does suffer some difficulties with respect to precision and internal/external validity. However it is most suitable for deal- ing with the research questions posed in this study because of its holistic perspective and its high level of relevance. The holistic perspective allows the researchers to comprehend the problem as a whole, taking into account a variety of factors and influences. A high level of relevance offers the prospect of valuable insights offered by research findings to a significant area of audit application (namely the performance audit) (Buckley, Buckley and Chiang, 1976). Thus case study analysis facilitates the consideration of context and allows the researchers to illustrate aspects of a functioning process while providing rich descriptions of the research subject (Kaplan, 1984).

The reports of each Auditor-General are therefore examined as comparative case studies of performance audit outcomes. The outcomes are the organizational activity identified as the focus for study using publicly available reports as the primary source of evidence. While the research questions addressed in this study are stated in hypothesis form, the investigation can be best typified as exploratory. This study therefore seeks to identify issues, problems and unique aspects of performance auditing. It addresses the overall questions of how these audits took place and how the parties involved acted and reacted. (Post and Andrews, 1982). O n this basis the study is aimed at building an empirical base for generating major research propositions in relation to performance auditing.

GOODNESS OF FIT

In order to consider whether variant approaches to performance auditing make a difference to the goodness of fit between declared objectives and actual objectives employed (H,), the recommendations given in the selected audit reports were analysed into those concerned with improving (i) economy (ii) efficiency and (iii) effectiveness. Such a process of classification must of neces- sity involve a degree of subjective interpretation particularly where some recommendations may include elements of both efficiency and effectiveness. The process of classification was further complicated when reports did not always give clearly stated recommendations or did not always contain a clear statement of auditee activity or programme objectives. Some of these difficulties occurred in the 1985 SAAG report, whilst some other reports gave clearly stated ‘audit findings and comments’ rather than ‘recommendations’. Effec- tiveness measures the extent to which predetermined auditee objectives are achieved, and it follows that the assignment of recommendations to the effectiveness category is assisted by a clear statement of auditee objectives. These difficulties resulted in a greater element of subjectivity entering into the analysis for the reports affected. The results of the analysis of the recommendations are shown in Table 2, which gives the number and percent of the

30 HATHERLY A N D PARKER

Table 2

Analysis of Recommendations

ECON EFFIC EFFEC TOTAL No 7% No % NO 7% No

AAO March 86 0 0 0 0 8 100 8 June 86 4 13 15 50 1 1 37 30

AAO Total 4 1 1 15 39 19 50 38

V A G June 83 14 24 23 40 21 36 58 Oct 84' 0 0 36 92 3 8 39 Oct 84" 0 0 1 7 13 93 14

V A G Total 14 13 60 54 37 33 1 1 1

SAAG June 84 6 20 23 7 7 1 3 30 SAAG June 85 3 27 7 64 1 9 1 1 SAAG Total 9 22 30 73 2 5 41

Government Stores Operations * * Departmental Cash Management

recommendations per report addressing (i) economy, (ii) efficiency and (iii) effectiveness.

The AAO reports demonstrate a readiness to comment on effectiveness in addition to the AAO's declared objectives of economy and efficiency. This departure from declared objectives is evident in the June 86 report (37 per cent of recommendations depart) but it is most noticeable in the March 86 report which wholly addresses the effectiveness of the Australian export meat inspection and control system in maintaining the integrity of Australian meat exports. Hence, the AAO reports clearly indicate that the AAO has gone beyond its declared objectives and has examined effectiveness (even before September 1986).

The VAG reports indicate a modest interest in economy (13 per cent of all VAG recommendations) and rather more interest in efficiency (54%) and effectivness (33%). This is consistent with the VAG's assertion that it is concerned with efficiency, effectiveness and, in some circumstances, economy. However, the total figures mask wide variations between different reports. For instance the report on government stores operations is almost wholly (92 %) concerned with efficiency and pays scant regard to whether those operations are able to meet their stated objectives of providing the required goods 'in the right quality, in the right quantity, at the right price and at the right time.' (Waldron 1984, p. 8). Indeed these issues have largely been excluded from the terms of reference of the report.

In the case of the VAG reports it is of interest of examine whether audit observations were in line with declared objectives, and limited to the auditee's


mechnnisms for monitoring and improving efficiency and effectiveness. Such an examination reveals that the VAG does not limit himself to comments on mechanisms but is willing to make observations directly on whether efficiency or effectiveness has been achieved. Some examples are:

Duplication of stores information has resulted in additional operating costs and inefficient use of manpower resources.

(Waldron 1984, p. 35)

The completed project is not in accordance with the original submission made to, and approved by the Premier in 1978, in that it does not conform to Olympic and international competition standards.

(Waldron 1983, p. 24)

The actions of school councils in applying eductional allowances . . . has resulted in additional costs to councils and parents of up to flOM during 1983-4 . . .

(Waldron 1984, p. 38)

As many as fifteen (38%) of the audit observations contained in the government stores operations report comment directly on efficiency. Although it is perhaps understandable that the VAG should wish to use examples of actual inefficiencies in order to provide weight to the recommendations, such direct observations do mean that the VAG is exceed- ing his stated intentions.

As previously explained, difficulties were experienced in analysing the SAAG reports. Table 2 , however, includes the results of an analysis of the SAAG recommendations. For 1984, it shows 29 recommendations classified as economy or efficiency and one as effectiveness. The comment on effectiveness stated (Sheridan 1984, p. 182) that non participation in the car pool scheme operated by the Department of Services and Supply will reduce the economic viability and overall effectiveness of the pool operation. This is a comment which directly relates to the effectiveness of a programme (scheme). There are however other comments (five in total) which relate to the effectiveness of management systems or techniques. These have been classified as efficiency since they are concerned with the effectiveness of systems primarily designed to achieve the efficient use of programme resources. For 1985 Table 2 shows ten recommendations concerned with economy/efficiency and just one concerned with effectiveness. In the comment on effectiveness the SAAG asserted that the Health Commission was failing to meet its (presumed) objective of providing a timely approval process for computing developments at health units (Sheridan 1985, p. 13).

When the observations and recommendations from each of the AAO, VAG and SAAG reports are considered together, it can be concluded that the AAO and VAG have departed from their declared audit objectives in a major respect but that the SAAG has not. The AAO departed because it considered effectiveness in addition to economy and efficiency and the VAG departed because it commented on substantive matters of efficiency and effectiveness as well as auditee mechanisms for achieving these aims. The SAAG, apart


from the very occasional comment on effectiveness, fulfilled its declared objectives of economy and efficiency. There is therefore support for the view that variant audit approaches are associated with differences in the goodness of fit between declared audit objectives and objectives employed in the audit report. The result is two poor fits (AAO and VAG) and one good fi t (SAAG). Thus for the audit reports studied, the evidence tends to reject H,.

REPORT FORMAT

In order to consider whether variant approaches to performance auditing make a difference to audit report format (H2), reports produced by the AAO, VAG and SAAG were examined. Criteria for comparing formats were developed from the available literature as discussed earlier in this paper. Accordingly, criteria employed were:

1 2 3 4 5 6 7 8 9

10

Audit objectives specified. Audit scope specified. Audit procedures employed. Problems identified. Causes of problems identified. Future-oriented recommendations. Auditee accomplishments and improvements recognised. Views of auditee officials published. Clear use of language. Concise and clear presentation.

The results of a qualitative analysis of the reports are shown in Table 3.

Table 3

Comparative Performance Audit Report Formats

AAO VAG SAAG

March 86 June 86 83 Report 84 Report 84 Report 85 Report Report Report

Objectives Yes

Procedures No Problems Yes Causes Some Recommendations Yes Improvements Rare Responses Yes Language Fair Presentation Confusing

Scope Yes No Yes No Yes Some Yes Rare Yes Fair Confusing

Yes No No Yes Some No No No Good Good

Yes Yes Yes Yes Some Yes No Yes Good Good

No No Yes Yes Some No Some No Poor Poor

No No Yes Yes Some No Some No Poor Verv Poor


Both AAO reports studied specified audit scope but curiously it was the March 1986 report that specified audit objectives while the June 1986 report did not. Neither report described or summarised audit procedures employed. Both identified problems in the auditees’ operations but only sometimes did the reports allude to apparent causes. Future-oriented recommendations for improving auditee operations were made in both reports but accomplishments and improvements that auditees had achieved themselves were rarely recognised. Auditee responses to auditor recommendations were published. Language and general presentation tended to be somewhat confusing. For example, the March report labelled audit scope as ‘audit approach’ while the June report switched labels to ‘audit scope’. Recommendations were titled as such in the March report but in the June report they were only identifiable by the italicised work ‘recommend’ appearing within a related paragraph. In general, the March report was not clearly written and tended to overwhelm the reader with a mix of background data and findings. T h e June report improved its layout with smaller paragraphs and point form summaries. Reference to possible causes of problems more often only took the form of general supporting detail on findings. In general, the AAO reports fell well short of meeting the criteria for reporting specified in the performance audit literature.

The VAG reports showed a pronounced improvement between 1983 and 1984, when assessed against the established criteria. Both reports specified audit objectives, but only the 1984 report specified audit scope and procedures employed. Both identified problems in auditee operations but only discussed causes sometimes. Unlike the 1983 report, the 1984 report did recommend improvements for auditee operations but neither report gave recognition to any accomplishments and improvements already made by auditees themselves. Auditee responses were not published in the 1983 report but did appear in the 1984 report. Language and presentation was clear in both reports. In general, report sections were clearly labelled and audit objectives were specified in detail. The 1984 report summarised audit procedures at a general level, but neither report contained any favourable observations or commendations whatever. Whether this reflects a reporting failure or the entire lack of positive achievement by auditees is difficult to determine. Both reports were very clearly presented with intelligent use of headings, paragraphs, and point form material. Unlike the AAO reports studied, these VAG reports employed relatively consistent format. Thus, while the 1983 VAG report did fall well short of specified reporting criteria, the 1984 VAG report shows significant improvement and appears largely to have conformed with criteria specified in the performance audit literature.

The SAAG report for 1985 did specify audit objectives including financial, compliance, economy and efficiency at the very front of the report covering all its general audit work in South Australia. However objectives for performance audit aspects of particular auditees that attracted comment in the report


were not stated. Audit scope was not stated. Some very general references were made to audit procedures but only with respect to financial and compliance audits. Problems in auditee operations were identified and causes were discussed for about half the problems identified. Clearly stated recommendations for improvement were not observed. Auditor observations were limited to statements such as ‘attention is drawn to’, ‘I am concerned by’ and ‘I expect that’. Some qualified approvals for auditee improvements achieved or being undertaken were included in the report but were not highlighted but ‘buried’ in the general text of discussion and observations. No auditee responses to audit findings were published although some were implied in the auditor’s comments. Language and presentation of the report was generally considered to be poor. Headings only reflected audit subjects and problems; background was not separated or highlighted. Performance audit report sections were succinct but tersely written, with brief (sometimes one line), disconnected paragraphs and minimal explanation or detail provided. By and large the inadequacies of the 1985 SAAG report were also evident in the 1984 SAAG report although, overall, the presentation of the 1984 report was rather better. It was therefore readily apparent that the SAAG report fell well short of meeting the criteria for reporting specified in the performance audit literature.

When reports from each of the AAO, VAG and SAAG are considered together, it can be concluded that the AAO and SAAG reports differ significantly from the criteria for report format specified in the performance audit literature. On the other hand, the VAG reports improved markedly, so that the 1984 VAG report largely conformed with those criteria. The significant difference between the VAG and AAO/SAAG reports therefore refutes H,. It appears that variant approaches to performance auditing are associated with significantly different audit report formats in the cases examined.

AUDITEE RESPONSES

In order to consider whether variant approaches to performance auditing make a significant different to auditee responses (H3), the auditee responses contained in the AAO and 1984 VAG reports were categorised into acceptances and rejections of the audit recommendations. Unfortunately the SAAG and 1983 VAG reports did not contain sufficient auditee responses to justify inclusion in the analysis. Table 4 shows the percentage acceptance rates according to audit approach (AAO v VAG) and according to whether the underlying recommendation relates to efficiency/economy or effectiveness. Percentage acceptance rates are the number of acceptances expressed as a percentage of total responses.

Table 4 indicates that the AAO acceptance rate (57%) is lower than the VAG acceptance rate (69%) primarily as a result of a lower acceptance rate


Table 4

Acceptance Rates Indicated by Auditee Responses

E CON/EFFEC EFFEC TOTAL

Accept % Accept % AAO Total 78% 37 % 57% VAG Total' 74% 54% 69 % TOTAL 76% 45 % 64 %

* Excludes 1983 Report

for recommendations regarding effectiveness (37 % v 54%). A statistical assessment of the lower AAO rate can be made by reference to the following 2 x 2 table categorising the audit responses:

Accept Reject

AAO 17 13 30 VAG 33 15 48

50 28 78

The Phi coefficient for this 2 x 2 table is - 0.12 indicating little association between acceptances and the audit approach. Moreover this value of Phi is not significant at the 0.05 level.4 This statistical assessment is not wholly persuasive in view of the non-randomness of the sample of responses. Never- theless, given the lack of statistical significance, it would be unwise to reject hypothesis H,. It appears that variant approaches to performance auditing may not make a significant difference to auditee acceptance of the audit recommendations.

Table 4 also indicates that the acceptance rate for effectiveness is lower than the efficiency acceptance rate for both the AAO (37% v 78%) and the VAG (54% v 74%). A statistical assessment of this lower rate can again be made by reference to the relevant 2 x 2 table which is as follows:

Accept Reject

Econ ./Effic . 37 12 49 Effectiveness 13 16 29

50 28 78

The Phi coefficient for this 2 x 2 table is + 0.31 indicating moderately low association (Champion 1981, p. 302) between the nature of the recommendation (economy/efficiency or effectiveness) and acceptances. The value of Phi


Table 5

Analysis of Auditee’s Reasons for Rejecting Audit Recommendations

Reasonable given the circumstances Audit recommendations are impractical Failed to recognise compensation controls Contrary to efficiency Contrary to effectiveness Disputed size of potential savings Other disagreements

9 3 4 5 4 1 2 -

28 -

is significant at the 0.05 levels providing tentative support that there is a difference between an efficiency and an effectiveness recommendation when it comes to the likely acceptance of that recommendation. This may indicate greater auditee resistance in respect of effectiveness recommendations.

In order to examine whether there is any predominant reason for the auditees’ rejections of audit recommendations, the frequency of reasons for rejection are analysed in Table 5.

The most frequent case put forward by the auditee argues explicitly or implicitly that the existing situation is reasonablegiven the circumstances which are outside the control of the auditee. In most cases this is stated explicitly. For example:

Given the complexity of the changes and the need to negotiate them with a number of groups, the slippage is reasonable . . .

(Monaghan, June 1986, p. 19)

Where the audit feels that the audit recommendations are impractical a rather more robust response is generally forthcoming:

It is not practical to control the illegal production of stamps if determined dishonest parties so choose this course to substitute meat products.


In four responses the auditee argues that the auditor has failed to recognise compensating controls which exist within the system as a whole. This type of response is typified by the following:

The Tyden seal when taken with other measures, is an effective control as used by EIS to secure premises and vehicles since 1981. The EIS does not accept Audit’s comments and does not intend abandoning its use . . . EIS agrees that the final con-


trol is only as effective as the controls that precede it but reiterates that container sealing is only part of a total security system.


There is one instance of the auditee disputing the size of the savings which the auditor claims would follow an audit recommendation. However, the most significant responses are those which argue either (i) that implementation of an audit recommendation targeted at effectiveness would cause loss of efficiency or (ii) that an audit recommendation targeted at greater efficiency would cause objectives to be missed with consequent loss of effectiveness. in Table 5 these responses are described as ‘contrary to efficiency’ and ‘contrary to effectiveness’. An example of a ‘contrary to effectiveness’ respnse is contained in the VAG report on Government Stores Operations (Waldron, 1984) which consists predominantly of efficiency recommendations. In this report the Gas and Fuel Corporation respond to VAG criticisms of inefficiently high applicance inventory levels by stating that the levels are necessary to meet Corporate objectives of (i) delivering 95% of appliances sold within five days, (ii) presenting to customers in the showrooms a comprehensive display of models and (iii) maintaining buffer stocks to satisfy ‘on the spot’ sales. (Waldron, 1984, p. 27). The Corporation maintained that its policies were designed to ensure customer satisfaction. However, the VAG did not examine customer satisfaction and was not therefore in a position to comment fully on both the efficiency and effectiveness implications of its recommendation to rationalise stocks. Conflicts between efficiency and effectiveness are also implicitly the subject of comment by the Department of Defence in response to the auditor finding examples of ineffective task management:

The Department noted that staff of the DTSO had been reduced by 250 since . . . 1980. Conversely, the workload of the organisation had increased . . . the pressure to achieve the required productivity increases had contributed to some of the problems identified in the Report . . .

(Monaghan, March 1986, p. 25)

‘Contrary to effectiveness’ and ‘contrary to efficiency’ responses are significant because they suggest the auditor may not have fully considered the effectiveness implications of efficiency recommendations and vice versa.

These analyses of published auditee responses cannot support more than limited inferences at this stage. They are based upon comparisons between only two of the three offices included in the study. They may reflect the influence of the subject of each audit, the manner in which the audit process was undertaken, general socio-economic and political pressures of concern to the auditees, audit team composition and other variables. Indeed auditees may be prompted towards defensive postures simply because their economy, efficiency and/or effectiveness is being made subject to public scrutiny. The above discussion therefore represents a first-level analysis of manifest auditee responses only.


CONCLUSIONS AND FURTHER RESEARCH

The Auditor-General’s reports examined in this study appeared to be consistent in the divergence of their stated objective concepts from the constituent concepts of performance auditing. The AAO and SAAG declared that their objectives did not include the audit of effectiveness of operations while the VAG only purported to audit controls for efficiency and effectiveness. In certain instances referred to, their substantive audit concepts agreed much more closely with the constituent concepts of performance auditing.

With respect to the primary hypothesis of this study, the evidence is somewhat mixed. There did appear to be a significant difference in goodness of fit between declared audit objectives and objectives employed in the audit report under different approaches (H,). The SAAG adhered to its declared objectives while the AAO and VAG did not, but included reference to substantive matters of efficiency and effectiveness. A significant difference between audit approaches was also evident with respect to their adherence to normative report formats recommended in the performance audit literature (H2). Only the most recent VAG report examined adhered to the recommended format. No significant difference between audit approaches however was evident with respect to auditee responses (H-,). In general, greater resistance was dis- played towards auditor recommendations concerning effectiveness of operations than those concerning efficiency. The most common rationales professed by auditees were the influence of uncontrollable external variables and the ‘unavoidable’ trade off between efficiency and effectiveness. O n balance, it can therefore be concluded that variant approaches to performance auditing do appear to make some difference to actual outcomes, though not with respect to all variables considered in this study. The lack of difference encountered in auditee responses stands as a significant exception in this regard.

A number of qualifications must be made with respect to these findings. It must be stressed that the cases examined constitute an exploratory foundation study, the scope of which is necessarily limited by the relative infancy of this form of auditing in the Australian public service context, differences in timing of reports’ publication dates, and the inherent limitation of case study method. The reports examined also related to a wide variety of audit subjects and hence this study’s findings may be influenced to a degree by the differences in audit subjects reported upon by the Auditors-General. With respect to auditee responses, the SAAG and earlier VAG reports did not pub- lish such responses and so conclusions in this regard must be qualified by the small sample of the reports examined. It is also recognised that the audit report contents were treated as surrogates for audit processes employed and actual audit outcomes that might subsequently occur (e.g. parliamentary or auditee management actions). The degree to which they are valid surrogates may be influenced by the extent to which the reports are employed as part


of a legitimising process or as constructors of a social reality. Finally it is important to recognise the dynamic environment of Australian performance auditing. This study presents observation of activity over a short period of time. Since July 1986, further audit reports have been published by some of these offices. The AAO has gazetted its own performance audit standards and the VAG is in the process of producing a performance audit manual. Concep- tual and practical developments in Australian performance are thus an ongoing process.

Given the paucity of available research results relating to performance audit operations and impact, this study represents but a first step in an urgently needed field of investigation. In itself it suggests a need for international replication to discover whether the above findings are valid in such countries as the UK, Canada and USA. Longitudinal study of concept application, report format and auditee response may indicate whether development of performance auditing follows a consistent pattern over time. Finally, audit outcomes may be further investigated by surveys and inter- views of auditees as well as observation of subsequent auditee actions. Such research offers the prospect of a clearer understanding not only of the manifest forms and objectives of performance auditing, but of the extant nature of its operation and its impact upon auditee behaviour.

NOTES

I

2

The implied linking of this ‘comprehensive’ terminology to the GAO is somewhat misleading since that terminology was primarily deveoped by the Canadians (Parker, 1986). An apparent change in the AAO’s attitude was evidenced by its promulgation on 8th September, 1986, of a set of performance audit standards (Commonwealth, 1986) that extended its audit scope to include examination of auditee effectiveness in achieving objectives. The reports included in this study were published prior to that AAO policy change. This information was supplied directly by the South Australian Auditor-General, Mr Tom Sheridan. 0 = -0.12 givesX = Ne2 = 1 .12 (Champion, p. 316). With 1 d.f. a P e q u a l to or larger than 2.706 would be significant at the 0.05 level (one sided test). 0 = +0.31 gives ,V = N0* = 7.5. With 1 d.f. a ,V equal to or larger than 2.706 is significant at the 0.05 level (one sided test).

3

4

5

REFERENCES

Allard, J .C. (l981), ‘Comprehensive Auditing in Crown Corporations: A Stronger Handle on

Australian Audit Office (1986), Elmmts of Ejicimcy Auditing: A Basis Guide, (Canberra: Australian

Bens, C.K. (1983), ‘Comprehensive Auditing: A Management Approach For Improving

Boisrlair, J.P. (1984), ‘The Challange of Comprehensive Auditing: Thriving O n Value for

Boys, P. (1984), ‘Value-For-Money Test Spreading in Sector’, Accountancy (July 1984), pp.

the Public Purse’, CA Magazine, Vol. 114 (February 1981), pp. 38-43.

Government Publishing Service, 1986).

Accountability’, Covmmental Finance, Vol. 12 (March 1983), pp. 9-12.

Money’, CA Magazine (January 1984), pp. 24-29.

118- 120.


Bromage, M.C. (1971). ‘Reporting on Management Audits’, The Federal Accountant, Vol. 20 (December 1971). pp. 71-82. - (1972), ‘Wording the Management Audit Report’, The Journal of Accountancy, Vol. 133

(February 1972), pp. 50-57. Buckley, J.W., M.H. Buckley and H-F. Chiang, (1976), Research Methodology and Busmess

Decisions, (New York: National Association of Accountants and The Society of Industrial Accountants of Canada, 1976).

Cabnet, B. (1976). ‘Performance Auditing in Metropolitan Dade County, Florida’, Governmental Finance, Vol. 5 (November 1976). pp. 44-47.

Caldwell, K.S. (1975). ‘Operational Auditing in State and Local Government’, Gouernmental Finance. Vol. 4 (November 1975), pp. 36-43.

Champion, D.J. (1981), Basic Statisticsfor Social Research (New York, Macmillan, 1981). Chandler, R . (1984), ‘A VFM Auditor Must Give Clients Value For Money’, Accountancy Age

(August 16 1984), p. 19. Chorlton, C . R . and D. Mills, (1985). ‘Operational Auditing in Ontario Hydro: Off to a Good

Start’, C A Magazine, Vol. 118 (January 1985), pp. 40-43. Commonwealth of Australia, ‘Notice by Auditor-General Setting Auditing Standards’, Gazette:

Special, No. S451, Canberra (Monday, 8 September, 1986). Comptroller General of the United States (1974), ‘Excerpts from the “Standards for Audit of

Governmental Organizations, Programs, Activities and Functions” ’, Public Management (February 1974), pp. 8-10.

Cowperthwaite, G . H . (1985), ‘The Comprehensive Audit or Value For Money Review’, Australian Accountants Centenary Congress, Adelaide (November 17-20, 1985).

Crockett, J .R . (1980), ‘Modelling the Operational Audit’, The Internal Auditor, Vol. 37 (June

Dilley, S.C. (1975), ‘Expanded Scope Audits - Untapped Opportunities?’, The CPA Journal, Vol. 45 (December 1975), pp. 30-35.

Doades, R . (1978). ‘The Mentality of Management Auldits’, Public Utilities Fortnightly, Vol. 101 (February 16, 1978), pp. 25-28.

Drebin, A.R. (1980). ‘Criteria For Performance Measurement in State and Local Government’, Governmental Finance, Vol. 9 (December 1980), pp. 3-7.

Ellis, R.L. and J.P. Melrose (1980), ‘Auditing the Management Process’, Auditor, Vol. 37 (August 1980), pp. 5364.

Fielden, J. and D. Robertson (1981), ‘Value for Money and Performance Reviews’, The Manage- ment Accountant, Vol. 16 uuly 1981), pp. 317-318.

Flesher, D.L. (1977), ‘Operations Auditing: For the Independent Auditor’, The CPA Journal, Vol. 47 (July 1977). pp. 17-21. - (1980), ‘Operational Auditing’, TheJournal of Accountancy, Vol. 150 (November 1980),

pp. 110, 112, 114. ___ (1984), ‘Writing the Operational Audit Report’, The Internal Auditor, Vol. 41 (February

1984), pp. 41-43. Flint, D. (1978), ‘Economy, Efficiency and Effectiveness - The “Value-For-Money’’ Audit’,

The Accountants Masarinc (June 1978), pp. 245-249. Fountain, J .R . and R . Lockridge (1976), ‘Implementation and Management of a Performance

Auditing System’, Gournmental Finance, Vol. 5 (November 1976), pp. 12-21, Garrido, A. (1980), ‘Operational Auditing - A Systematic Approach’, International Journal of

Gounnmmt Auditing (April 1980), pp. 11, 13. Hammel, L.G. Jr . (1977), ‘Regulatory Directed Management Audits: Some Behavioural

Implications’, Public Utilities Fortnightly, Vol. 100 (July 7, 1977), pp. 11-16. Hara, L.F. (1976). ‘Performance Auditing: Where Do We Begin?’, Governmental Finance, Vol. 5

(November 1976). pp. 6-10. Haribhakti, S.V. (1982). ‘Humanising the Operational Audit’, The Chartered Accountant, Vol. 31

(October 1982). pp. 33-35. International Congress of Supreme Audit Institutions (1986), General Statement o f X I I Zncosai on Per-

fomanccliudit, Audit ofpublic Entcrpriics and Audit Quality, (Canberra: Australian Audit Office, 1986).

Kaiel, M.L. (1976), ‘Performance Auditing in Portland, Oregon’, Governmental Finance, Vol. 5 (November 1976), pp. 38-43.

1980), pp. 66-76.


Kaplan, R. (1984), ‘The Case for Case Studies in Management Accounting Research’, Haruard Business School Working Paper (No. 55, 1984).

Knighton, L.M. (1973), ‘Performance Auditing in Better Perspective’, The Internal Auditor, Vol. 30 (March 1973), pp. 40-49.

Knoll, R.J. and Howard, T . N . (1983), ‘What is Operational Auditing?’, Topics in Health Care Financing, Vol. 10 (Winter 1983). pp. 1-11.

Lane, D.C. (1981), ‘The Operational Audit - For Improved Operations and Higher Profita- bility’, 7he Accountant’s Magazine, Vol. 85 (February 1981), pp. 44-46.

Lane, D.C. (1983), ‘The Operational Audit: A Business Appraisal Approach to Improved Operations and Profitability’, Journal ofthe Operational Research Socicfy, Vol. 34 (October 1983), pp. 961-973.

Langhout, J . (1984), ‘Comprehensive Auditing’. Accountancy SA (March 1984), pp. 315-316. Malan, R.M. (1979), ‘Trends in Performance Auditing’, GouernmentalFinance, Vol. 8 (September

McSweeney, B. (1985), ’Does the Theory Accord with Municipal Practice?’, Certified Accounfanf (April 1985). pp. 24-25.

Monaghan, J .V. (1986). Reporf of the Auditor-General on an Efficiency Audit: Defence Science and Technology Organisation: Task Cosf Management (Canberra: Australian Government Publishing Service, March 1986).

~ (1986), Report of the Auditor-General on an Efficiency Audit: Departmmf of Primary Industry: Administration of Meat Inspection Services by the Export Inspection Seruice (Canberra: Australian Government Publishing Service, June 1986).

Morse, E.H. J r . (1973), ‘Operational Auditing and Standards for the Public Sector’, G A O Reuirw (Winter 1973), pp. 30-36.

~ (1976), ‘Operational or Performance Auditing of Government Activities’, The Government Accountants Journal, Vol. 25, No. 3, (Fall 1976), pp. 1-9.

Parker, L.D. (1986), Value for Money Auditing: Conceptual, Deuelopmmf and Operafional Issues (Melbourne: Australian Accounting Research Foundation, 1986).

Ponder, E.H. (1984), ‘Operational Auditing by CPA Firms’, The CPA Journal, Vol. 54 (October 1984), pp. 38, 40, 42, 44, 46, 48, 50.

Post, J . and B. Andrews (1982), ‘Case Research in Corporation and Society Studies’, Research in Corporate Social Perfononce and Policy, Vol. 4 (1982). pp. 1-33.

Pyhrr, P.A. (1969), ‘Operational Auditing: A Run For Daylight’, Financial Executive, Vol. 37 (May 1969), p. 19-20.

Rothwell, W. (1984). ‘How to Conduct a Real Performance Audit’, Training, Vol. 21 , No. 6 (June 1984), pp. 45, 47-49.

Santocki, J (1974), ‘Management Audit From the Inside’, The Accountant, Vol. 28, (March 28, 1974). pp. 381-383.

Sheridan, T.A. (1984), South Australia: Report ofthe Audifor-Generalfor the Year Ended 30th June 1984 (Adelaide: Woolman, Government Printer South, Australia, 1984).

~ (1985), South Australia: Report of the Audifor-General For The Year Ended 30 June 1985 (Adelaide: Woolman, Government Printer South Australia, 1985).

Siegel. J . G . and M.F. Milich (1985), ‘Auditing Corporate Management’, Managerial Planning, Vol. 33 (March/April 1985). pp. 8-10.

Smith, C . H . , R.A. Lanier and M.E. Taylor (1972), ‘The Need For and Scope of the Audit of Management: A Survey ofAttitudes’, Thc Accounting Reuinu, Vol. 44 (April 1972), pp. 270-283.

Special Report (1984), ‘Canadian Comprehensive Auditing Foundation and Comprehensive Auditing’, C G A Magazine (October 1984), pp. 37-38, 40-41, 62.

Tanaka, N.R. (1978), ‘Management Audits in the Utility Industry’, C A Mafazinc (February 1978), pp. 45-47.

Vangermeersch, R. (1976), ‘Management Auditing is For You’, The National Public Accountant, Vol. 21 (May 1976). pp. 21-22.

Waldron, B.J. (1983), Report of the Audifor-General: Work Confracts Oueruiew - Second Report (Melbourne: Atkinson, Government Printer, June 1983). - (1984), Reporf of thc Auditor-General: Comprehcnsiue Audifs - Government Stores Opera-

tions, Departmental Cash Management (Melbourne: Atkinson, Government Printer, October 1984).

1979), pp. 23-27.

performance auditing outcomes: a comparative study

Documents