Understanding Performance Audits

Deborah V. Loveless, CPA

November 10, 2015

Objectives

Present Overview of Government Auditing Standards Pertaining to Performance Audits

Highlight the Similarities and Differences Between Financial and Performance Audits

Explain Phases of a Performance Audit Illustrate Use of Performance Audits in

Tennessee

Comptroller of the Treasury Tennessee Comptroller of the Treasury elected

by General Assembly for two-year term Audit Divisions

State Audit Local Government Audit

Broad statutory authority to audit Broad statutory access to records Audit authority in all grants and contracts Fraud Hotline

Performance Audit Definition Government Auditing Standards definition:

Performance audits provide findings and conclusions based on an evaluation of sufficient, appropriate evidence against criteria. Performance audits provide objective analysis to assist management and those charged with governance and oversight in using the information to improve program performance and operations, reduce costs, facilitate decision making by parties with responsibility to oversee or initiate corrective action, and contribute to public accountability.

Matters that may be assessed include the effectiveness, efficiency, and economy with which the program function, system, or process operates. Performance audits are conducted in accordance with accepted standards.

Performance Audit Objectives Performance audit objectives vary widely and

include assessments of Effectiveness, economy and efficiency Internal controls Compliance Prospective analyses

Government Auditing Standards

General Standards Independence Professional Judgment Competence Quality Control and Assurance

Chapter 6 - Performance AuditField Work Standards Introduction (6.01) Reasonable Assurance (6.03) Significance in a Performance Audit (6.04) Audit Risk (6.05) Planning (6.06 – 6.51) Supervision (6.53 – 6.55) Obtaining Sufficient, Appropriate Evidence (6.56 – 6.78) Audit Documentation (6.79 – 6.85)

Reasonable AssuranceThe ultimate goal is to obtain reasonable assurance that evidence is sufficient and appropriate to support the auditors’ findings and conclusions in relationship to the audit objectives.Reasonable assurance varies:

Findings and conclusions Objectives Evidence (sufficiency and appropriateness) Professional judgment is critical

Significance GAGAS definition: relative importance of matter

within the context in which it is being considered, including quantitative and qualitative factors.

Use significance to Decide type of audit work to perform Decide extent of audit work to perform Evaluate results Develop conclusions, findings, & report

Significance v. Materiality

Audit Risk Possibility that the auditors’ findings,

conclusions, recommendations, or assurance may be improper or incomplete

Contributions to increased risk: Evidence is not sufficient or appropriate Inadequate audit process Intentional omissions Misleading information

(last 2 can be due to misrepresentations or fraud)

Planning Auditors must adequately plan and document

the planning of work necessary to address the audit objectives.

Define objectives, scope and methodology based on assessment of: Risk - of something going wrong Significance - of whatever going wrong

Planning is ongoing May have to adjust as we go along

Planning - Risk In planning, we must assess risk and

significance by understanding: Nature of program Potential report user’s needs Internal controls IS controls Legal, regulatory, contract provisions Fraud Ongoing investigations/legal proceedings Prior audit results

Plan the audit to reduce audit risk

Planning – Criteria Use criteria to evaluate performance GAGAS lists examples:

Laws, regulations Contracts, grant agreements Standards Measures, expected performance, benchmarks Defined business practices

Planning - Criteria Attributes of Suitable Criteria

Objectivity – Should be free from bias. Measurability – Should permit reasonably consistent

measurements, qualitative or quantitative, of subject matter.

Completeness – Should be sufficiently complete so that those relevant factors that would alter a conclusion about subject matter are not omitted.

Relevance – Should be relevant to the subject matter.

Planning - Criteria Other considerations:

Criteria identify the required or desired state or expectation with respect to the subject matter that is being measured or evaluated

Criteria provide a context for evaluating evidence and understanding the findings, conclusions, and recommendations included in the auditor’s report

In all cases, the criteria must be identified in the report

Planning - Evidence Evaluate whether information to be used as

evidence will be sufficient, relevant, etc. If available evidence will be insufficient:

Modify scope and methodology Identify additional sources Is that a finding in itself? - i.e., lack of internal

controls

Planning – Using the Work of Others Others = auditors, specialists Auditors should identify the work others have

done Avoid duplication Identify problem areas, recommendations for further

work, ideas, sources, etc.

If using others’ work – document their qualifications, independence, etc.

Planning – Assigning Staff Staff, other resources should be assigned so

that: Team has collective knowledge, experience Sufficient number of staff and supervisors On-the-job training can be provided Specialists available if needed

Must document their scope, etc.

Planning - Communicating Communicating with management, those

charged with governance and others Communicate overview of

Objectives Scope Methodology Timing

Communicate to Auditee management Those charged with governance Individuals contracting/requesting audit If audit based in law, legislative committee

Planning – Written Audit Plan Preparing audit plan

Auditors must prepare a written audit plan for each audit.

Form and content can vary Update plan with significant changes Allows supervisors to monitor:

Likelihood of obtaining sufficient, appropriate evidence Staffing sufficiency

Supervision Audit supervisors…must properly supervise

audit staff Problem with defining “properly” Evidence of supervision:

Supervisory sign-off, ticks Timeliness of sign-off Writing and clearing of coaching notes/points Documentation of team meetings

Obtaining sufficient, appropriate evidence Auditors must obtain sufficient, appropriate

evidence to provide a reasonable basis for their findings and conclusions

Evidence is evaluated as a whole Appropriateness

Relevance Validity Reliability

Hierarchy of Evidence Some evidence is stronger than other evidence

Obtained when controls are strong Obtained through auditor’s direct physical exam,

observation, or computation Obtained from original documents Obtained from interview with no interference Obtained from interview with staff who have direct

knowledge of program Auditor conducted surveys Auditor sampling

Evidence Sufficiency Sufficiency guidelines

As risk increases, need for quality and quantity of evidence increases Stronger evidence might allow for less evidence

Large volume of evidence does not compensate for lack of relevant, valid, or reliable evidence

Overall Assessment of Evidence

Must determine and document overall assessment of evidence for sufficiency and appropriateness used to support findings and conclusions

Evaluate in context of significance Evidence is not sufficient when:

High risk Carries significant limitations Does not address objectives or support findings

Elements of a Finding Finding Elements

Criteria Condition Cause Effect Recommendations

Early Communication

Should inform those charged with governance Significant and urgent deficiencies:

Compliance with laws, grants, contracts Abuse, fraud Internal controls

Early communication does not eliminate need for inclusion of issues in audit report

Audit Documentation Auditors should prepare audit documentation in

sufficient detail to enable an experienced auditor, having no previous connection to the audit, to understand from the audit documentation the nature, timing, extent, and results of audit procedures performed, the audit evidence obtained and its source and conclusions reached, including evidence that supports the auditors’ significant judgments and conclusions.

Documentation Documentation is important

Provides principal support for the report Aids auditors in conducting and supervising Allows for review of audit’s quality

GAGAS requires documentation of Audit’s objectives, scope, and methodology Work performed to support “significant judgments &

conclusions” Evidence of supervisory review

BEFORE report released

Documentation When auditors do not comply with GAGAS

standards Should document departure and impact on audit and

on audit conclusions Sometimes occurs due to

Law, regulation Scope limitations Records access limitations

Chapter 7 - Performance AuditReporting Standards Auditors must issue audit reports

communicating the results of each completed performance audit.

Form of the report Report contents Report issuance and distribution

Reporting Form Form of report can differ based on users’ needs Retrievable Acceptable forms include:

Written reports Letters Briefing slides Presentation materials

Reporting - Purpose Communicate results to

Those charged with governance Auditee officials Oversight officials

Make results less susceptible to misunderstanding

Make results available to public (as applicable) Facilitate follow-up

Report Contents Objectives, scope, and methodology

Provide context and perspective Including any limitations and constraints Detail items tested, populations, period covered,

sampling, etc. Any major assumptions

Report Contents Audit Findings

Including sufficient evidence to support findings Scope of internal control work and any identified

deficiencies Place findings in perspective, include background

information for context Describe report limitations or uncertainties Fraud occurred or is likely to have occurred

If management fails to report to appropriate external parties, auditor must report

Report Contents Conclusions

Based on audit objectives and findings Include logical inferences about the program based

on findings (not just a summary)

Report Contents Recommendations

Correct problems identified during the audit Addressed to those with authority Specific, practical, cost effective, and measurable

GAGAS compliance statement Auditee’s response

Can provide evaluation of response Can obtain oral comments due to time If auditee refuses to provide, indicate such

Report Contents Confidential or sensitive information

Legally protected Classified Public safety and security

Disclose omission and reason for omission Assess impact of public working papers access

Consider handling verbally Example: undercover license plates

Report Distribution Distribution to

Those charged with governance Auditee officials Oversight bodies/organizations arranging for audits

(i.e., legislative requesters) Legal authority or otherwise responsible for acting

on reports Anybody authorized as appropriate

Document any limitation on report distribution

Comparison of Financial Audits and Performance Audits

Similarities: Professional competence, due professional care,

and adequate planning and supervision are required Independence is required Materiality or significance is always considered Auditor may provide recommendations for correcting

a deficiency or improving a condition Auditor forms a conclusion or opinion about the

subject matter’s conformity with specified criteria

Comparison of Financial Audits and Performance Audits More similarities:

In both engagement types, the auditor: Gains an understanding of the subject matter Gains an understanding of internal control if relevant to the

subject matter Obtains reasonable assurance Identifies the criteria in the report Forms a conclusion based on evidence obtained Considers potential independence impairment

Comparison of Financial Audits and Performance Audits Major differences:

Financial Performance

Criteria are defined by the audit standards YES NO Auditor ordinarily decides on the applicable

criteria and may seek agreement from the responsible party about the appropriateness of the criteria for the engagement NO YES

Responsible party asserts as to conformance of the subject matter to the criteria YES NO

Responsible party always makes representation to the auditor YES NO

Comparison of Financial Audits and Performance Audits Reporting Requirements:

Financial Performance Identification of criteria used YES YES Description of significant limitations or

uncertainties that could affect the validity of the auditor’s opinion, conclusion, or findings YES YES

Description of the auditor’s responsibility YES YES Detailed description of work performed NO YES Information to provide context for and

perspective on what is being reported NO YES Engagement can result in a conclusion

rather than an opinion NO YES

Phases of a Performance Audit Audit Assignment/Audit Selection Planning Scoping Detailed Audit Report Writing Distribution

Audit Planning Identify major issues surrounding the agency

Problems? Is the program producing desired outcomes? Answer to a specific question? Achievements?

In order to identify the issues, we need basic background information

Sources Used in Planning Agency website Agency literature News search Prior performance

audits Other recent audits Statutes Rules Legislative history

Interview staff & management

Advocacy groups and stakeholders

Board members Professional literature Statistics/Performance

Measures

Planning During planning keep in mind the five elements

of findings: Condition – What is going on? Cause – Why is it happening? Effect – Why should we care? Criteria – What are we measuring against? Recommendation – What do we want them to do

about it?

Scoping Process of deciding which of the identified

issues to pursue in-depth What makes an issue a high priority?

High dollar High impact – public, other agencies, individuals High profile Agency audit history Inherently high risk Auditor experience

Detailed Audit

Perform fieldwork testing and research In-depth research on only the selected topics Identify rest of the elements Document what we already know or suspect Conduct surveys

Methodology - Interviews Differences between Performance and

Financial Interviews Rarely based on standardized checklists Starting “from scratch” Why and How are as important as What Generally not just updating last audit Different topics/Different ultimate purpose Can be broad vs. specific, known audit topic

Methodology - File Reviews Systematic review of agency records to collect

data Hard copy records Computerized records Both! – Understand overlap

Start analysis = evaluate mid-way Working paper minimum:

Methods – i.e., How selected sample File-by-file results Conclusion/Analysis

Report Writing What makes it a finding?

Almost all findings have recommendations Observations/Comments?

Can have recommendations, but less often the case Positive – agency didn’t do anything “wrong” Legislative interest Event updates Very minor issues Slightly outside of the scope

Agency must respond to findings, but not observations/comments

Report Writing Clear communication is an underlying

Government Auditing Standards principle Audience is diverse and busy Present enough information to…

Convince legislators and agency Understand major topics Answer likely objections

Performance Audits in Tennessee Governmental Entity Review Law

(1977 Sunset legislation) provides a responsible method for reviewing state

government entities to ensure the state regulation is beneficial rather than detrimental to the public interest

assigns a termination date to all state entities created by statute

authorizes program review audits (performance audits) by the Comptroller of the Treasury to aid in legislative review of entities

Performance Audits in Tennessee Sunset Audits

222 entities in initial legislation 260 entities in current legislation Average of 15 public hearings per year Average of 75 entities reviewed per year Average of 4-year extension (rarely the 8-year

extension allowed by statute) 166 terminations (7 in 2014)

Primary reasons for termination: inactive, fulfilled mission, or transfer of function to another entity

Tennessee Government Services Helps 452,000 job seekers and 10,000 veterans find

jobs

Helps 800 children find permanent homes

Answers over 284,000 child abuse, drug abuse, and mental health hotline calls

Administers 1,854,540 immunizations, 42,600 TB tests, and 16,500 lead blood level assessments

Tennessee Government Services Serves more than 1,700 schools, 930,000 students, and

70,000 teaching professionals

Ensures safety on 87,000 miles of state and federal highways

Controls 2,100 wildfires and manages the conservation of 185,000 acres

Disposes of more than 1 million pounds of hazardous household waste

Tennessee Government Services Routinely inspects:

92,014 fuel pumps

22,000 elevators

19,000 bridges

73 public airports

2,933 miles of rail

11,000 school buses and childcare vans

All X-ray equipment

101 credit unions

155 banks

Examples of Performance Audits in Tennessee

Department of Intellectual and Developmental Disabilities

Department of Children’s Services Tennessee Film, Entertainment and Music Commission Department of Labor and Workforce Development Douglas Henry State Museum Commission Tennessee State Board of Accountancy Tennessee Rehabilitative Initiative in Correction Board

Department of Intellectual and Developmental Disabilities (October 2013) Objective: To determine whether the department had

established sufficient services for individuals with developmental disabilities in accordance with statute

Findings: The department had not established sufficient services for individuals with

developmental disabilities obtained data on the number of Tennesseans with

developmental disabilities coordinated with other state agencies to offer services for

individuals with developmental disabilities pursued funding to provide services for individuals with

developmental disabilities

Department of Children’s Services (January 2014) 22 objectives dealing with departmental operations in

Child Safety & Child Health, Juvenile Justice, Child Programs, and Computer Operations

12 findings and 13 observations addressing Child Safety investigation thoroughness child protective investigative operations tracking of child abuse and neglect referrals background checks on individuals providing services to

children in state custody

Department of Children's Services (Continued)

probation and aftercare supervision requirements sufficiency of residential treatment options calculation and analysis of recidivism assessment of foster care placement and monitoring of private

provider placement practices assessment and provision of long-term medical care to

methamphetamine-exposed children in state custody analysis of call wait times for Tennessee Child Abuse Hotline Child Abuse investigative caseloads

Tennessee Film, Entertainment and Music Commission (January 2013) Objectives: to evaluate the effectiveness of the film

incentive program and to evaluate the incentive award process

Findings The Department of Economic and Community Development and the

Department of Revenue have disregarded their statutory responsibility and exercised poor management and administrative oversight of the state’s headquarters film incentive program

The Tennessee Spend, which is eligible expenses used to calculate the incentive payments, is likely to be significantly overstated for reasons including poor internal controls, insufficient policy, and lack of management accountability among the departmentsinvolved with its determination

Department of Labor and Workforce Development (October 2014) Objective

To determine whether there is adequate and appropriate oversight of inspectors and inspections in the Elevator, Boiler, and Amusement Device Units

Finding The department does not yet have a viable amusement device

regulatory unit six years after jurisdiction was transferred from the Department of Commerce and Insurance

Department of Labor and Workforce Development (Continued) Objective

To determine whether there are adequate controls and plans to ensure that adequately trained mine rescue teams are in place as required by statute

Finding The department’s Mine Safety Unit is out of compliance with

state statute in regard to its mine rescue teams’ distance from underground mine operations

Department of Labor and Workforce Development (Continued) Objective

To determine whether the department has made a detailed contingency plan to deal with federal government shutdowns

Observation The department has only minimal guidance to address actions

in case of a federal government shutdown

Douglas Henry State Museum Commission (September 2015) Objectives

To determine if staff had taken adequate steps to preserve its collection

To determine controls over alcohol inventory

Findings Water problems in the museum pose a threat to artifacts on

display or in storage Management lacked internal controls to ensure security of

alcohol stored on site

Tennessee State Board of Accountancy (September 2015) Objectives

Determine the effect of an inactive Peer Review Oversight Committee on the board’s ability to oversee accounting professionals

Determine other states’ requirements governing the confidentiality of peer review results

Finding Under its current structure, the Tennessee State Board of

Accountancy is limited in its ability to oversee the peer review program in Tennessee

Tennessee Rehabilitative Initiative in Correction Board (October 2015) Objectives

Document the controls over the accounts receivable and accounts payable processes, review aging report, and determine process for collecting past-due receivables

Ensure the accuracy of the fiscal year 2014 account balances that were reported to the board of directors, as well as the amounts reported in the state’s CAFR

Determine if TRICOR entered into formal agreements with its Cook Chill customers and assess the Cook Chill program’s financial position

With respect to TIMS, determine if TRICOR management followed information systems’ industry best practices regarding system controls

Tennessee Rehabilitative Initiative in Correction Board (Continued) Findings

TRICOR’s executive director failed to ensure fiscal staff performed key fiscal and financial reporting functions, which led to a pervasive breakdown of control that resulted in material financial misstatements and board decisions that were based on inaccurate financial information

TRICOR and the Department of Correction managements continue to operate the Tennessee Cook Chill program without executed agreements, resulting in unmet expectations and a program net loss of more than $4 million as of March 31, 2015

TRICOR did not provide adequate internal controls in four specific areas

Understanding Performance Audits

Deborah V. Loveless, CPA

November 10, 2015