peer review techniques - southeastern actuaries conference
Post on 12-Feb-2022
2 Views
Preview:
TRANSCRIPT
1/22/2008
1/22/2008 1
1
Peer Review Techniques to
Optimize Quality
Darryl Wagner
Southeastern Actuaries
Conference
November 18, 2010
2
Sunset in CharlestonI am very happy to be here!
2
1/22/2008
1/22/2008 2
3
Agenda
• Overview/relevance
• Example of process
• Reviewee perspective
• Reviewer perspective
• Closing thoughts
• Questions
3
4
“The actuarial profession is achieving increasing visibility. The
debate on health care reform, recent problems in the life
insurance industry, and catastrophic losses in the property/
casualty industry have focused more attention on actuaries and
the work that they do…Peer review is seen as another possible
method to enhance our professionalism.”
1/22/2008
1/22/2008 3
5
“The actuarial profession is achieving increasing visibility. The
debate on health care reform, recent problems in the life
insurance industry, and catastrophic losses in the property/
casualty industry have focused more attention on actuaries and
the work that they do…Peer review is seen as another possible
method to enhance our professionalism.”
Peer Review, Concepts on Improving Professionalism
American Academy of Actuaries, Council on Professionalism ,
Discussion Paper, 1997, No. 1
*Note: Much of this presentation draws heavily from this
discussion paper.
66
Overview/Relevance
1/22/2008
1/22/2008 4
7
What’s in a Name?
• Peer review focuses on the “who”
• Quality review (QR) shifts focus to “what”
• A definition…”Evaluation by a qualified professional of a
professional work product (or portions thereof)”
7
8
Why is QR Important?
• Improving quality of the QR subject
• Meeting the needs and expectations of the users of our work
• Expanding future opportunities
• Both parties gain knowledge and experience
• Continuous improvement in quality
• Compliance with standards and other requirements
• Risk/cost management
8
1/22/2008
1/22/2008 5
9
Relevant Guidance
• Academy Discussion Paper
on Peer Review, 2005
• Code of Professional Conduct
• Qualification Standards
• Actuarial Standards of Practice
9
10
• Proposals• Engagement letters/contracts• Technical work • Reports/memoranda/etc.• Significant correspondence• Presentations• System implementation• Model conversion• Quarterly financial reporting production• Product launch/repricing• Feasibility studies• Other?
Subjects of Quality Review
10
1/22/2008
1/22/2008 6
11
Who Should Do the Reviewing?
• Key-Requisite skills and experience
• “Independent”
• May be more than one person
• Risk profile may warrant
• Multi-disciplines involved?
• Different aspects of review
11
12
When Should QR Happen?
• Pre-release vs. post-release
• Step approach vs. draft work product approach
• Appropriate to the targeted phase(s) of work
• “Contracting”
• Planning
• Technical work
• Work products
• Documentation
12
1/22/2008
1/22/2008 7
13
• Consistency
• “Big picture” and detail
• Planning for QR
• Documentation of QR
• Feedback loops/confidentiality
• Cost/benefit and QR alternatives
• Standardization/support
• Resolution of differences
Other Considerations
13
1414
Example of Process
1/22/2008
1/22/2008 8
15
Pre-Release Quality Review
Example of Process
Nature of Assignment Level of Review
Simple letter with commentary 1 Proofreading
(no conclusion)
Straightforward calculation 2 Checking arithmetic
result to client
Analysis, commentary, simple 3 Second opinion and read-over
recommendation, or conclusion
Substantial analysis or numersous 4 Pre-release peer review of
calculation of routine nature - procedures and report
significant financial implications
Report including analysis and 5 Substantial concurrent
conclusions of uncertain or peer review or parallel
non-routine nature that has calculations and analysis
substantial financial implications
Level
of
Co
mp
lexit
yL
evel o
f Inte
nsity
of R
evie
w
Peer Review, Concepts on Improving Professionalism, AAA Discussion Paper, 1997
16
Pre-Release Quality Review
Example of Process
Establish the process for getting peer review done
• Allow time in the work plan for peer review
• Make peer review a priority
• Make peer review a habit
• Make sure that all staff understand that peer review
is an important step to be completed before a
project is finished
• The use of forms and checklists may be helpful
1/22/2008
1/22/2008 9
17
Post-Release Procedural Review
Example of Process
The emphasis is not so much on the quality of a specific work
product, but rather on the work procedures that are used by the
practitioner to produce and ensure a quality work product.
1. Final Reports Calrity, completeness, applicability of analysis
to client issues, compliance with laws and
professional standards, use of appropriate caveats,
all reports signed by qualified professional
2. Files Ability to track work and to find all necessary items
3. Work Procedures Work flow and monitoring, adequate checking
recording of client communications, appropriate
supervision, pre-release peer review
4. Client/Business Issues Avoidance of conflict of interest, clear
understanding of scope of assignment and
authority to proceed, consideration of associated
business risk
Peer Review, Concepts on Improving Professionalism, AAA Discussion Paper, 1997
18
Post-Release Procedural Review
Example of Process
Suggested approaches to the review
• Determine the frequency and intensity of post-release
procedural reviews
• Prepare in advance for the on-site visit
• Identify projects in advance that will be reviewed
• Interview staff to understand how large projects are
coordinated
• Reference project documentation and files during discussion
with staff
• Reviewer’s report should summarize findings and include a
section on opportunities for improvement
• An exit interview should be conducted to discuss findings
1/22/2008
1/22/2008 10
1919
Reviewee Perspective
20
Reviewee Perspective
• Begin with the end in mind
• Think through overall QR process
• Recognize what you may not know-consult
• Manage as part of the project
• Identify needs/roles
• Line up resources
• Manage execution
• Keep your eye on the prize: Quality
20
1/22/2008
1/22/2008 11
2121
Reviewer Perspective
22
Reviewer Perspective
• The review should be conducted by a qualified
(ideally independent) professional, i.e. someone
who is a “peer” of the professional who is primarily
responsible for the project
• Both the work product and the work file should be
peer reviewed from:
• The perspective of the reader
• A professional standpoint
• An actuary’s standpoint concerning the business risk issues
1/22/2008
1/22/2008 12
23
Reviewer Perspective
• Review the work product from the perspective of the reader:
Is it clear? Is it right? Is it complete? Does it meet the needs of the user?
Do the conclusions flow logically from the data and numerical analysis performed?
Are all assumptions and methods specified?
• Review the work product from a professional standpoint:
Does it meet actuarial standards of practice or other applicable professional
standards?
Does it satisfy pertinent laws and regulations?
Would another actuary reading this work product be inclined to disagree with
methods used, assumptions employed, or conclusions reached?
24
Reviewer Perspective
• Review the work product from a business risk standpoint:
- Is the scope of the assignment clearly spelled out and adequately covered?
- Are any limitations on the intended user of the work product clearly delineated?
- Are the sources of data and extent of reliance identified?
- Is potential variability of results adequately discussed?
- Is there any statement that might be misinterpreted and lead to wrong conclusions?
• Review the work file:
- Can work be tracked from input to conclusion?
- Has appropriate checking occurred throughout?
- Have key methods and assumptions been documented?
- Has the work flow been appropriately monitored?
- Has compliance with ASOPs been adequately documented?
1/22/2008
1/22/2008 13
25
Technical Actuarial Review Framework
• Data
• Methodology
• Assumptions
• Process
• Results
2626
Closing Thoughts
top related