peer review process trottie kirwan chair apa peer review committee span meeting 24 april 2009
TRANSCRIPT
Peer review process
Trottie KirwanChair APA Peer Review
Committee
SPAN Meeting 24 April 2009
Original process
Original Peer Review Process• APA has run interdepartmental peer review since 1999• Started in the big specialist children’s hospitals• Aim
– raise standards of clinical practice within a paediatric anaesthetic department. Allow for experiences to be shared, with the dissemination of good or innovative practice (Review document)
– to assess departments in relation to standards which should characterise anaesthetic departments throughout the country (Peter Crean in Anaesthesia 2003)
• Based on Good Practice – a guide for departments of anaesthesia. RCA and AAGBI. London, 1998
• 2 day visit by a team of four including a lay member• All departments, visitors and visited, found it valuable
BUT• The process is very demanding of both the
visited department and the visitors– 2 day visit including interviews with 16 different staff
members as well as parents– Detailed report averaging 30 pages
• APA decided to extend the process to be available to all anaesthetic departments with paediatric practice
• Original process too great a commitment for smaller departments with fewer paediatric anaesthetists, particularly as visitors to write the reports
New Process
Simplified Peer Review Scheme
• New scheme developed by Peer Review Committee– Alistair Cranston, Peter Crean, Trottie Kirwan, Madeleine
Wang
• Intended as a tool for any department treating children to– Check compliance with existing criteria of RCA, NSF,
HCC/CQC– Demonstrate that standards of practice are good– Help in discussions with management and colleagues
• Simplify the process for departments with a smaller paediatric workload and fewer paediatric anaesthetists
• Reduce the demands on the reviewing team
Simplified Peer Review Scheme
• Provide a structure for local self-appraisal to help departments in raising standards of clinical practice. – Self assessment– Peer verification
• Based on – Good Practice Guide 2006 (JCGP RCA & AA), – RCA guidelines for the provision of anaesthetic
services 2004– NSF for Children Standard for Hospital Services 2003– Criteria in the proposal come from RCA guidelines
Self-assessment
Self-assessment
• Departments self check against criteria in a freely available template
– Department’s perceived strengths and weaknesses compiled by the paediatric anaesthetists
– Check-list of criteria for good practice (from RCA guidelines)
– 360° appraisal by colleagues and patients/families
• Tool to monitor their own practice, without any external input
Self-assessment Checklist(from RCA guidelines)
• Organisation• Theatres and day surgery
facilities• Recovery • Equipment • Emergency department• Support services• Critical care• Transfer • Staff• Training • Clinical governance• Child and family experience
Self-assessment ChecklistRCoA guidelines
Equipment
2.1 A full range of monitoring devices, paediatric anaesthetic equipment and disposable items for general and regional anaesthesia should be available in theatres and all other areas where children are anaesthetised. This should include a full range of disposable equipment including the following which should be appropriate for use in children of all sizes and ages:blood pressure cuffsintravenous cannulaetemperature probespulse oximetry.
2.2 Resuscitation drugs and equipment, including an appropriate defibrillator, should be routinely available at all sites where children are to be anaesthetised.
2.3 Anaesthetic machines should incorporate ventilators, which have controls and bellows permitting their use over the entire age range together with the facility to provide pressure controlled ventilation.
2.4 There should be appropriate thermostatic control of the operating room; temperature monitoring and patient warming devices should be available in both the operating room and recovery area.
2.5 Intravenous fluids should normally be administered only by volumetric infusion pumps.
Checklist
Wherever children are anaesthetised:
i. There is appropriate paediatric anaesthetic equipment
ii. Monitoring equipment meets the current standards of the Association of Anaesthetists
iii. Ventilators are suitable for children and can deliver pressure controlled mode
iv. There are resuscitation drugs and equipment
v. There are suitable temperature monitoring, and warming devices
There are volumetric infusion pumps for IV fluid administration
Strengths and Weaknesses
• The paediatric anaesthetists, separately and collectively, list of those aspects of their work which they perceive as good and those they think need improving – The aspects which need improving are considered, any
which can be fixed from within the department are addressed
– The process is repeated until eventually the list consists only of strengths, and of weaknesses over which the group has no control
• Found to be a useful tool for discussing problems people have got used to but could actually deal with
360° Appraisal by Colleagues• Colleagues are invited to help in the department’s self-
review – Chief Executive – Medical Director– Head of Department of Anaesthesia, anaesthetic trainees – Consultant paediatricians, paediatric surgeons, ED, PICU &
transfer – Theatre Manager, Senior ODP / anaesthetic nurse– Directorate Business Manager, Anaesthetic Secretary– Senior paediatric ward nurses, play specialist, pain team
• Explanatory letter to say that this is a voluntary process
• Process focussed on the department’s organisation and practice – concerns about individuals should be reported through the normal channels
360° Appraisal by Colleagues• How the paediatric anaesthetic department is
perceived – Concerns with the management of the department or the
service
• Anaesthetists working as a team– Formal working relationships with colleagues– Informal collaboration for lists and individual patients
• Effective management of the service – Theatre time, emergency support – Cover for absent colleagues
• Staffing – Staff levels, recruitment and retention problems– System to deal sensitively with colleagues below an
acceptable level
• Effective planning for equipment, staffing and budget
Patient/Parent Appraisal• Areas covered
– Pre-admission – Pre-operative preparation– Anaesthetic room– Recovery– Analgesia
• Design of questionnaire– Open questions elicit more information– Closed questions give more consistency
• Timing– At discharge easy and cheap– After return home more informative
Peer review
Peer Review• Department completes self-assessment and
volunteers for peer verification• Visiting team
– 2 doctors and a lay visitor, one from outside the region – Lead reviewer with peer review experience– Could include an observer
• Self-assessment paperwork sent to the visiting team
• One day visit – Review the self-assessment evidence– meet paediatric anaesthetists – visit to clinical areas – debriefing session
• Review team send a report within a month
Peer Review
• External APA report – confirmation of good work– suggestions for consideration to improve organisation &
practice– recommendations for use in negotiating plans with
colleagues and management
• Problems arising at visit– visiting team cannot substantiate the self-assessment: this
will be stated at the debriefing meeting and no report sent – a potentially serious problem is unearthed during a review:
the review team leader will contact the regional co-ordinator and an APA peer review committee member immediately
Peer review process
Organisation of the Process• Peer review is organised on a regional basis
– within local paediatric anaesthetic networks where they exist
– regional co-ordinator arranges teams of reviewers – each team has an out-of-region member – national co-ordinator for hospitals not in a local
network
• Extending the process nationally includes– smaller departments of fewer paediatric anaesthetists – less experience of review visiting and report writing– more support is needed
• Standard job description documents– task lists for regional co-ordinators, review teams, and
departments being reviewed– criteria for eligibility for Peer Review Team
membership, anaesthetic and lay
Peer Review
APA peer review version2009
Regional co-ordinator
1. Compile list of anaesthetists to be local review team members. Criteria for eligibility: a. consultant anaesthetist b. member of APA c. experience of peer review d. for team leaders, experience of 2 reviews
2. Recruit new team members. Criteria: a. consultant anaesthetist b. member of APA
People who are interested in getting involved but have no experience of peer review should attend one review as an observer
3. List of local people to be lay visitors. Criteria for eligiblilty: a. Experience of hospital visiting, receiving hospital paediatric and/or anaesthetic care, or working with
children and understanding their needs b. Ability to be objective – should not work within the visited Trust, a neighbouring Trust, PCT or SHA c. Willing to contribute to the report writing as a team member – so not to be visiting in the role of a
report writer for another institution Suitable people might be found from voluntary groups listed with PALS, with former Public Patient Involvement Forum administrators or from the new networks of voluntary organisations known as Local Involvement Networks (LINKs). Local teachers or other workers with children who have experience with their own children in hospital have been enlisted.
4. Receive self assessment from departments wanting to be reviewed, and set up a reviewing team consisting of: a. a lead, who has at least experience as a reviewer, b. another consultant anaesthetist who has at least experience as an observer c. a lay member At least one team member should be from outside the region and there may also be observers from other regions and the APA peer review committee
5. The co-ordinator must ensure that the reviewing team members can write a fair report. Colleagues with a professional interest in a good or bad report cannot serve as review team members for that visit
6. Be aware of communication and arrangements between the department to be visited and the review team, and advise in case of problems
7. Receive the report from the review team, and the feedback on the process from the visited department and the review team, and chase them up if they are not done in 6 weeks
8. Forward the report and feedback to the APA committee chair and discuss any problems in either the report or the feedback
Peer Review
Guidance for departments being reviewed
1. The department needs to agree that peer review will be useful, and carry out the self-evaluation, using it as a tool for self-improvement (unless you are perfect already)
2. Carry out the SWOT process, getting all the paediatric anaesthetists to agree to a consensus document
3. Once the self-evaluation is completed and you are ready to be reviewed, contact the local paediatric anaesthesia peer review co-ordinator, where there is one, or peer review committee chair.
4. Make contact with the review team leader
5. Inform relevant individuals, wards and departments in the Trust, emphasising that this is a voluntary process undertaken by the anaesthetic department to recognise and improve the quality of the service, and not because of a complaint.
6. Collate the short portfolio, self evaluation documents and supporting evidence paperwork and send a copy to each member of the reviewing team, as much as possible in electronic format, 4 weeks before the visit to send to the visiting team:
Paperwork to send to the review team members
• The short portfolio
• SWOT consensus document
• The results of the colleague appraisals
• The self-assessment check-list
Check-list criteria evidence which should be sent:
Reports of incident reviews
Department policies and guidelines
Summaries of completed audits from the last 2 years
Minutes of meetings which contain relevant information
Copies of patient information
Evidence of any research – facilities, support (ethics), trainee input
Critical incident reports and presentations to M&M meetings
List of any complaints and the management of them
Peer Review
Organisation of the Process
• Confidentiality– Reports, and the information they are based on, are
confidential and will not be passed to anyone by the visiting team members
– The visited department may use the report as they choose
• Quality control– individual performance monitoring by formal feedback
after each review by both the visitors and visited– consistency monitoring by an external visitor
• Support for the process – peer review committee– e-network on APA website – meetings at APA ASM and linkmen meetings
Feedback Self-assessment Surprisingly useful. We all thought that we knew the problems in
our hospital but when we discussed our thoughts we found some important concerns that I had not considered before. Additionally it encouraged us to have a detailed meeting at which I could disseminate useful information that others paed anaesthetists had been unaware.
Arranging the review Time consuming. Very difficult to arrange reviewers. People need a lot of notice.
Clear requirements for evidence
I may not have read the documentation closely enough but I missed the requirement for evidence. Would have been helpful to give a long list of possible sources.
Visit discussions useful and professional
Visit discussion was very useful, there were no great surprises
Report received promptly, and contents as expected from visit discussions
Not received yet ( only 2 weeks ago)
Process helpful and fair Process very helpful. Stimulated discussion and action within the hospital before the review and added political muscle to help implement long overdue change in some areas.
Other comments Time consuming but completely worthwhile process. A continuous cycle of review of the process will help to make it less time consuming and more rewarding
Process development
• Peer review intended to be a collaboration with users – Proposals on APA website – colleagues invited to feed back – SWACA pilot
• Changes to self-assessment – drop case studies – flexibility in training and experience criteria
• Changes to process – better guidance on how to do it– standard templates for questionnaires, reports, feedback– addition of out-of-region review team member
• Good ideas – children included in reviewing team
Benefits
• Visited department– Incentive to look critically at local practice and make
changes before writing self-assessment– Team building within the department and with other
colleagues– Reports have been useful in getting organisational
change and improvements in facilities and equipment• Visitors
– Good ideas to manage common problems: meeting life support criteria, managing scheduling of lists
• Region – Clinical links reinforced– Recognition that our colleagues in other hospitals are
good doctors and friendly humans
Misgivings
• Conspiracy– HCC approach – choose inappropriate criteria and
stop good departments doing good work– Predator approach – local big hospital stealing work– Forced marriage approach – departments merge and
one gets the cream
• Cock-up– Huge amount of work for no outcome– Report which will be ignored– Review will have no power to compel change
• May be overcome if some hospitals make a start while others wait to see how it turns out
SPAN
• Some interest• Some cynicism
• Risk/benefit balance in favour of giving peer review a try
• Perhaps start with the psychopathically keen
References
• Draft proposal for revised peer review scheme http://www.apagbi.org.uk/docs/APA_PeerReview(Draft).pdf
• Good Practice: A Guide for Departments of Anaesthesia. The Royal College of Anaesthetists and Association of Anaesthetists of Great Britain and Ireland, 3rd edition 2006 http://www.rcoa.ac.uk/docs/goodpractice(oct2006).pdf
• Department of Health (2003) Getting the right start: National Service Framework for Children - A Standard for Hospital Services http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_4006182
• Guidelines for the Provision of Anaesthetic Services. Royal College of Anaesthetists. 2004 http://www.rcoa.ac.uk/docs/GPAS.pdf
• Quality in paediatric anaesthesia: a pilot study of interdepartmental peer review. Crean PM, Stokes MA, Williamson C, Hatch DJ Anaesthesia. 2003 Jun;58(6):543-8.