development of quality system assessment program final...

Development of Quality System Assessment ProgramFinal Report

Health & Human Services

AUDIT / TAX / ADVISORY / LINE OF BUSINESS

2© 2006 KPMG, an Australian partnership, is part of the KPMG International network. KPMG International is a Swiss cooperative. All rights reserved.

The KPMG logo and name are trademarks of KPMG.Liability limited by a scheme approved under Professional Standards Legislation.

Contents

Introduction 3

Project Methodology 3

The context of the Quality System Assessment Program 4

Review of the Literature 6

Synthesis of the literature review findings 12

Stakeholder consultations 14

Objectives and qualities of the Quality System Assessment Program 18

The assessment framework 19

The assessment process 20

The Quality System Activity Statement 24

Conclusion 25

Attachment A - The AHS Quality System Activity Statement

DisclaimerThis report has been prepared using information provided to KPMG by the Clinical Excellence Commission (CEC). KPMG has relied on that information being accurate. KPMG has not undertaken any audit or other forms of testing to verify the accuracy, completeness, or reasonableness of information provided. Accordingly, KPMG can accept no responsibility for any errors or omissions in the information shown in this report where it is based upon that information provided.This report has been prepared at the request of the CEC in accordance with the terms of KPMG’s engagement contract dated 14th March 2006 Other than our responsibility to the CEC neither KPMG nor any member or employee of KPMG undertakes responsibility arising in any way from reliance placed by a third party on this report. Any reliance placed is that party’s sole responsibility.



Introduction

The Clinical Excellence Commission (CEC) contracted the services of KPMG to develop the methodology, conceptual framework and a model assessment tool for the Quality System Assessment Program (QSAP).The project methodology utilized a review of the literature and consultation with stakeholders and experts to help formulate a methodology and assessment framework.The objectives, constraints and ideal qualities of the QSAP were identified and utilized to evaluate the methodological options available. A self-audit of the patient safety and clinical quality system with external verification of the internal audit process supported by an improvement framework was determined to be the most suitable approach for the QSAP.A set of criteria was developed to underpin the Safety and Clinical Quality Program Standards and to inform the development of the assessment tools.A self-assessment tool suitable for assessing the clinical quality and patient safety systems at Area Health Service (AHS) level was developed. A draft of the self-assessment tool, the Draft AHS Quality System Activity Statement (the Activity Statement) will be piloted after the completion of this project.The draft AHS Activity Statement and the assessment criteria were presented to the Directors of Clinical Governance and expert reference group to provide comment. This comment was incorporated into the final draft of the tool. Advice was provided to the CEC on aspects of the pilot process and further development work that needed to complete the QSAP development.This document provides an overview of the project, a brief summary of the literature review and the emerging methodology and framework.

A core project team was established comprised of the KPMG Project Director, KPMG Project Manager and the CEC Director of Quality System Assessments. The project methodology utilised the following elements.

A desktop review of the literature;Key stakeholder consultations throughout the project. The primary modes of consultation were:

consultation workshops with an expert reference group convened for the purpose of the project;an assessment of the expectations of the Deputy Director Generals, Chief Executives and Directors of Clinical Governance Units; andA workshop with Clinical Governance Unit Directors and NSW Health staff on the assessment framework and incorporation of their feedback into the draft AHS assessment tool.

Identification of relevant NSW Health policies and standards appropriate for consideration in developing the assessment criteria and assessment tool;Development of assessment criteria; andDevelopment of an AHS self assessment tool for piloting.

Project methodology



The context of the QSAP

The NSW Patient Safety and Clinical Quality Program was developed in response to the Walker Inquiry into events at the Macarthur Health Service Campbelltown and Camden Hospitals. The NSW Patient Safety and Clinical Quality Program was launched in May 2005 by the NSW Health Department and sets the agenda for a comprehensive clinical quality program, including the establishment of:

the Clinical Excellence Commission;Clinical Governance Units in each Area Health Service;a new incident information management system;a process for the systematic management of incidents and risks; and a quality system assessment program for all public health organisations, the QSAP.

The QSAP will be positioned within a broad range of activities already in place that aim to assess , improve or provide assurance on the safety and quality of patient care in NSW. These activities are represented in Figure1 and include:

Improvement initiatives eg Collaborative projectsAccreditationPolicy developmentCredentialing proceduresRegulation of:

Health service provider organizationsHealth professionals

In developing the methodology for the QSAP it is important to contextualise its application to ensure that it provides a unique and value adding role and does not duplicate effort.It is recognised that the Clinical Excellence Commission (CEC) is still in its early days of developing its role within the NSW health system. The QSAP will be a critical element in the suite of CEC activities and will play a significant role in forming the perception, particularly among health service staff, of the benefits to the system provided by the CEC.

QSAP

CEC

NSW Health

Other Stakeholders

Registration Boards

Professional colleges

Consumers

Accreditation Agencies

International safety &

quality groups

Insurers

Individual health

professionals

AHS’s

Ambulance Service

Corrections Health

Health Systems

PerformanceQ & S

SIP

SCIDUA & SCIDAWS

Other Improvement

Programs

Other Statutory

health corporations

(Statutory Health

Corporation) QSAP

Figure 1: The context of the QSAP



Review of the literature



Review of the literature

A comprehensive desktop review was performed of literature identified by the CEC and from our own research. Material was identified from health, financial and other service industries. This assisted in identifying elements valuable in assessing quality, safety and risk management systems and processes that have not previously been utilised in the health industry. Articles were identified either by the CEC, experts from within the project team or expert group or through a search of selected databases, journals and search engines (Pubmed, Medline, MJA, BMJ, Australian Health Review, Google). The following list provides a brief summary of the scope of articles reviewed:

Existing approaches in health to patient safety & clinical quality assessment (28 references)

AustraliaUnited KingdomEuropean Union United StatesCanadaNew Zealand

Assessment Tools used in various jurisdictions for evaluation of patient safety and clinical qualities (23 references) General safety & policy issues including literature and research about safety and policy in health environments (8 references)Reporting & rating methodologies In health related settings (2 references)Other industries (10 references)

PetroleumAviationMiningFinance sector

Methodological options for the QSAP were synthesized from the literature. The following outlines some of the key references and summarizes the learnings from the literature for the developing QSAP model.



A recent Australian example - Victorian Auditor GeneralIn 2003 the Victorian Auditor General was asked to evaluate the patient safety systems of the Victorian public hospitals. In order to do this the Auditor General undertook a survey of 99 hospitals. This survey involved quantitative and qualitative responses based on self assessment. The second component was audit fieldwork at 5 health services. The audit was carried out in two large regional and three large metropolitan health services.The approach was based on asking four questions about the safety and quality system of the Victorian hospitals and reviewed particular performance areas to answer the questions posed. The four questions are listed in Table 1 and the corresponding areas of assessment are outlined.

The audit was performed in accordance with Australian auditing standards including tests and proceduresThe Auditor General recommended the priority development of:

Consistent definitions of patient safety and clinical quality termsDevelopment of minimum datasets relating to adverse events and other indicators of patient safety and clinical quality Performance review criteria for safety and quality systems in health careDevelopment of information management systems and standards to support the patient safety and quality systems

Are clinical RM systems rigorous? Risk management framework, policies, data & reporting clinical incidents

Are clinical RM systems effective? Incident reporting and response, risk rating

Are people issues managed effectively? Training & policies

Is performance monitoring and reporting effective?

Data reporting

Table 1: Victorian Auditor General questions regarding safety and quality systems and areas of assessment.

Table 2: OECD Indicators for patient safety



United Kingdom: Controls Assurance Standards and NHS Standards for Better Health

This corporate governance based program was the centrepiece of the drive towards improving the standard of health care in the NHS in recent years. The Controls Assurance Standards were used by the Commission for Healthcare Audit and Inspection (CHAI) in their assessment of the quality, safety and overall performance of NHS Trusts. It aimed to provide an integrating mechanism for existing audit and risk management processes and incorporated self-assessment and independent verification of self-assessment within its methodological approach.Assessment of compliance against individual standards were evaluated using three performance levels

Minimal ModerateExpected

The Controls Assurance Standards were deemed to be prescriptive and the criterion too rigid. Evaluation of the Controls Assurance Standards demonstrated that they were not seen to adequately account for difference in organisation type.The NHS Controls Assurance Standards evolved into the NHS Better Health standards of which there are:

24 Core standards that were mandated10 Developmental standards which were used as improvement goals

The Standards specified the methodology for self assessment which was varied and offered a triangulated approach. For example - Workshop with employee teams and management using specialist facilitator to analyse strengths and obstacles, develop action plan. This was deemed to be an onerous process which was often used as an exercise to identify people to blame – in 1 Trust it took 8 people over a month each to extract and collate evidence required for the assessment process. There was widespread antagonism regarding the CHAI process and the perception amongst staff that the audit process was used to find people to blame and did not support staff in systems improvement.

This is a voluntary program that is run by the NHS Litigation Authority (NHSLA). The NHSLA is a Special Health Authority, responsible for handling negligence claims made against NHS bodies in the UK. In addition to dealing with claims when they arise, they have an active risk management programme to help raise standards of care in the NHS and hence reduce the number of incidents.Trusts are rewarded for participation in the program with discounts on insurance premiums where they can demonstrate compliance with the CNST RM Standards.Trusts are provided with the standards, supporting evidence, verification directions and the scoring system utilised. Standards are assessed progressively at 3 levels:

Level 1 – Basic elements of CRM frameworkLevel 2 – Clinical Risk Management integrated into policies procedures Level 3 - Integration of Clinical Risk Management into clinical workplace is monitored

NHS Litigation Authority assessors perform assessment site inspections every 2 years with 1 assessor performing a 2.5 day visit and a pre-assessment visit.

United Kingdom: Clinical Negligence Scheme for Trusts



United States – Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators

Three sets of health quality indicators are produced by AHRQPrevention quality indicatorsInpatient quality indicatorsPatient safety indicators

The three sets of indicators are used primarily for quality improvement. Comparative indicator performance reports are published and available for benchmarking on the web. Hospital performance on AHRQ indicators is incorporated into payment schemes by health service purchasers. This is a data driven approach with each indicator defined by supporting technical specifications. Any validity issues and level of evidence available to inform assessments. Benchmarked adverse event rates are available and can be accessed by member organisations through the Web.The Patient Safety indicators focus on screening for adverse events. Indicators occur at two levels with adverse event rates available for benchmarking

the provider level ( 20 indicators) the area/catchment level indicators ( 7 indicators)

There are limitations to the utility of the indicators. They rely on the completeness and accuracy of documentation in the medical records and the sensitivity of the coding processes. Also there are varying levels of evidence supporting the validity of indicators with some being more robust than others.

OECD Patient safety Indicators – Health Systems levelIn 2004, as one in a series of technical papers on methodological studies and statistical analysis the OECD published a set of 21 patient safety indicators to assist in the drive towards a more standardised approach to measuring various health metrics in OECD countries. The patients safety indicators are listed in Table 2.

Hospital acquired infections Ventilator pneumoniaWound infectionInfection due to medical careDecubitus ulcer

Operative and post-operative complications Complications of anaesthesiaPost-op PE or DVTPost-op sepsisTechnical difficulty with procedure

Sentinel events Transfusion reactionWrong blood typeWrong-site surgeryForeign body left in during procedureMedical equipment-related adverse eventsMedication errors

Obstetrics Birth traumaObstetric trauma- vaginal and caesarean

Other care-related adverse events Patient fallsIn-hospital hip fracture

Table 2: OECD Indicators for patient safety



United States – Veterans Affairs Assessment for Patient Safety

This is a voluntary self assessment tool which is improvement focussed rather than a compliance model. The tool sits within a well developed organisational safety management system of which it is one of many components. The system has a very strong no-blame culture and encourages self evaluation. Elements in the self assessment tool are priority rated to assist organisations to focuss on high risk areas initially. The priority ratings are as follows:

A = Highest priority impacting fundamental program initiativesB = Important to program areasC = Strongly recommended

It is anticipated that organisations will follow a program of gradual development and imrpovedperformance as they progress further on into the implementation of the program. The assessment tool Incorporates JCAHO Standards with items being able to be directly translated to selected Standards.

ECRI (formerly the Emergency Care Research Institute) is a US based not for profit health services research agency whose mission is to improve the safety, quality, and cost-effectiveness of healthcare. It has a range of services one of which is a Web-based risk assessment tool that assesses risk exposure in healthcare organisations.This risk-assessment tool provides instant automated, aggregate reports that detail areas needing improvement and provide comparison data suitable for benchmarking. It also produces targeted recommendations including information on the evidence base for recommendations and information on supporting standards that may be of assistance in improvement work. Participants are provided with performance scores to assist in prioritising areas for improvement. The assessment can be used in hospitals medical practices and long-term care facilities and can assist organisations to achieve compliance with regulatory requirements. Pricing starts at $US3,500 per facility. Performance assessments may be used by funding bodies to determine levels of payment for services.

United States – ECRI INsight Program



Managing safety in other industries

In response to public concerns regarding the performance of peak organisations providing public services in London the Greater London Authority Group implemented a program for assessment of performance of its services which comprises the Greater London Authority, Transport for London, the Metropolitan Police Authority, the London Development Agency and the London Fire Brigade.Measurement of compliance is via a methodology derived from assessment of financial controls compliance. Organisations perform a self assessment which includes a review of internal risk controls and self monitoring mechanisms. This is supported external audit by the Audit CommissionThe assessment aims to answer the following questions:

What is the organisation trying to achieve? How has the organisation set about delivering its priorities for improvement? What improvements has the organisation achieved/not achieved to date? In light of what the organisation has learnt so far, what does it plan to do next?

A five point scale performance assessment ranging from 'excellent' (the highest) through 'good', 'fair' and 'weak', to 'poor' (the lowest) is given at the conclusion of the assessment cycle.

The ATO sets requirements of the tax assessment process through guidelines which are provided with the self assessment forms. Self assessment undertaken by individuals and organisations.Thematic approach to targeting with annual in depth assessment of different areas of the Tax self-assessment.Benchmark performance levels established to guide assessment and the interpretation of performance.Outliers are selected for external verification of the self assessment by ATO officers. A random audit of those within the normal range of performance also supports this process.Where non-compliance with guidelines is identified an escalation and sanction strategy may be utilised. In some instances of non-compliance or event of outlier performance such as fraud site visits are conducted to view sources of documentation.Training materials and manuals provides comprehensive guidance to officers, covering both law and likely practical issues when they are accessing information on site as part of verification of self assessment.

Companies operating off-shore facilities must comply with safety regulations as set out by NOPSA. The NOPSA requires assurance of compliance through submission by the operator of a “Safety Case”. Safety case describes means by which the operator will ensure adequacy of the design, construction, installation, maintenance or modification of the facility.The NOPSA takes systems view of safety with operators self enforcing and self monitoring compliance. The NOPSA works to build a relationship with operator with a NOPSA officer working closely with operators within his portfolio to ensure a focus on improvement.NOPSA must accept a Safety Case if “there are reasonable grounds” for believing that the operator has complied with the relevant regulation. It also allows for partial or total exemption from some aspects where appropriate.The NOPSA model has the following overarching set of principles.

All information will be treated as confidential, within the limits of FOIEach assessment will be fair and technically competentThere will be consistency between different assessmentsAssessment processes will be transparentGood project management practices will be appliedGood quality management practices will be appliedThe detail of assessments will be proportional to the level of riskThe results of assessment will be presented to relevant stakeholdersActions taken in response to findings will be graduated, and proportionate to the risk

Australian Tax Office

Essential Services - Greater London Authority Group

Australia - National Offshore Petroleum Safety Authority (NOPSA)



Synthesis of the literature review findings

The following is a synthesis of the major findings of the literature that have relevance for the developing QSAP methodology, assessment framework and tools. A systems view should be taken with evaluation of systems and processes rather than individual performances. The focus should be on learning and improvement and the assessment process should provide recommendations for improvement in safety systems management and assessment of the system. The assessment should have a focus on clinical risk management and incorporate a process improvement model. The QSAP should encourage ownership of risk and control of risks at all organisational levels. The concept of ownership of risk control points has achieved a high level of engagement of staff particularly with self assessment models. In depth focus targeting specific aspects of the patient safety and clinical quality system on an annual basis will assist in delivering the improvement aims of the program. A strong focus on self assessment is a cost effective method for delivering parts of the QSAP. Site visiting a representative sample will assist in validation of the self assessment. A paper based self assessment survey may be utilised with development of a web-based data collection and reporting tool at a later date. The self-assessment tool should be simple to use and allow open-ended responses.A structured approach to the criteria for assessment which allows organisations flexibility to exclude criteria not appropriate or of low significance assists in minimising redundancy and ensure that the detail of the assessment is proportional to risk presented. This should also allow organisations some scope to focus on local strategic directions.The following model has potential for application with two levels of criteria:

Core criteria which is mandatedDevelopmental criteria which are utilised as best practice improvement goals

Rationale for the assessment criteria and explanatory notes on the self-assessment questions including direction on assessment methods should be developed and provided to organisations to assist in completion of the assessment and achieving compliance. A graded assessment of compliance levels may also be used and consideration given to rewards depending on level of compliance. There should be a clear format of the criteria for assessment with the evidence base and direction provided on sources of verification. Requirement for organisations to extract large amounts of documentation for desk-top review be avoided. Wherever possible synergy with accreditation standards should be recognised and accounted for to prevent duplication of effort.Where clinical indicators are used ensure that there is supporting technical specifications accurately defining the indicator. Any validity issues and level of evidence regarding indicators used should be published. Where possible make adverse event rates available for benchmarking purposes.Attention should be given to providing clarity around definitions of patient safety and clinical quality terms. Wherever possible use widely accepted definitions. It was identified that the WHO was in the process of conducting a project on Healthcare safety and quality definitions and that this work could be utilised in the future. In the interim wherever possible the Generic Reference Model be used to define terms.



Stakeholder consultations



Consultation on emerging framework and assessment tools

The project methodology utilised a series of workshops with an expert group of key stakeholders to provide advice and validate the emerging methodology, framework and tools. Four workshops were held at key stages of the project on March 3 2006, March 24 2006, April 12 2006 and May 1 2006. The following group of experts attended the workshops and provided email or verbal comment on the emerging body of work:

Professor Cliff HughesDr George BearhamMs Bernie HarrisonDr Maree BellamyDr Michael SmithDr Philip HoyleProfessor Bruce BarracloughMs Liz Forsyth (KPMG)Ms Louise Kershaw (KPMG)

In evaluating the proposed approach and designing the methodology for the QSAP the project team’s expertise in risk management and risk assessment was utilised.The Directors of the Clinical Governance Units and the Deputy Director Generals were consulted as to their expectations of the QSAP and to identify gaps and areas of mismatch between the developing concept and the opinions of key stakeholders. A workshop was held with the Directors of Clinical Governance and senior NSW Health staff on June 9, 2006 at which the assessment framework and AHS Activity Statement was presented. This group was asked to provide comment on the material presented at the workshop and comment on the documents.



AHS Chief Executives & Directors of Clinical Governance Units

As part of the development process for the QSAP a stakeholder analysis of Chief Executives of Area Health Services and Directors of Clinical Governance Units was undertaken between 29th March – 7th April 2006. All interviewees were provided with the pre-set questions, a copy of the QSA Assessment Framework document from NSW Health and a Brief relating to the QSA program. The questions were as follows:

What outcomes are you expecting to see from the QSAP?What do you think the QSAP can deliver for you (i.e. what is the role of QSA from your perspective?)What do you think is the biggest risk for you and your organisation from the QSAP? Are there particular areas or issues that you think should be excluded from the assessment process? Are there particular areas of focus that would add value and should be included to the QSA Program? Which Area Directorates would you envisage working with CEC to conduct the QSA Program?

The following is a summary of responses to each of the five questions and is collated in terms of what the QSAP must deliver for NSW PHOs. The Quality System Assessment must:

be based on improvement model and incorporate knowledge management;provide networking opportunities;allow for benchmarking;Include peer review;provide the impetus to get safety and quality on the AHS agenda;link to CEO performance agreement;provide tools, templates and best practice models; andinclude improvement strategy linked with planning process.

The Quality System Assessment must not:be a “tick box” exercise against criteria. Need evidence of effectiveness;generate new data to collect but wherever possible use existing data;duplicate other quality processes;use a review process as per accreditation. It should use an audit process rather than an extensive review; andjust utilise self-assessment. There needs to be some process of verification of internal findings.

Feedback from stakeholders has highlighted a range of other factors for consideration during the further development of the QSAP including:

evaluation of the sustainability of improvements that have been implemented through programs such as the Clinical Services Redesign Program;review of other functions of Clinical Governance Units such as training and education.review of Clinical Practice Improvement projects and collaborative improvement programs;evaluation of the use of clinical process indicators in managing patient safety and clinical quality performance such as measurement of the use of beta blockers after AMI;measure of culture through operational systems; and evaluation of the organisational commitment to open disclosure.



Deputy Directors General

It was determined that the Deputy Director Generals (DDGs) of Health be consulted to ensure that the developing framework and methodology was within their expectations. Interviews were secured with the DDG Performance and the DDG Strategic Development. The main issues raised and discussion points are outlined below.

The QSAP is a mechanism for the CEC to audit the safety and quality systems that are in place in NSW Health system on behalf of NSW Health. In doing the QSAP CEC is acting on behalf of NSW Health. The outcomes of the audit will be reported to NSW Health for it to act on if necessary.The focus of the QSAP should be on improvement not blame.The proposed method of self audit with independent verification is an acceptable model to NSW Health. Verification should involve a combination of targeted interviews with key stakeholders and, where it exists, analysis of data should be used for verification.At AHS level the assessment will primarily be of the effectiveness of governance by the Chief Executive and Director of the Clinical Governance Unit.The QSAP should be reviewing incident/adverse event rates and provide opportunity for PHOs, facilities, and clinical units to benchmark.The imperative for the QSAP at this stage is to set deadlines for pilot and roll-out of the AHS Activity Statement. Rather than having an extensive consultation and refining of the tools the CEC should move quickly to complete the first round of assessments and fine tune the assessment tools at a later date.There was a concern expressed that the QSAP would have too heavy a focus on the acute sector and hospitals in particular. The QSAP needs to encompass assessment of:

community servicesmental health services; andoutreach services.

There is a need to investigate innovative ways of measuring the quality of community care and potential to utilise work being done in the areas of preventative medicine and chronic care. One possible measure to use as a springboard for measurement of this sector of the health service is ambulatory care sensitive admissions. The QSAP needs to encompass the quality dimension in more than one way. Justice Health have developed output measures however these do not assess the quality of the output. For example the percentage of inmates who receive a detox program is measured however no evaluation is done on the appropriateness of the program selected.Justice Health is keen to work with the CEC to develop an effective assessment tool for it’s patient safety and clinical quality systems.



The assessment framework



Objectives and qualities of the QSAP

Guided by consultation with stakeholders, relevant policies and the review of the literature a consensus was reached on the broad objectives of the QSAP. The objectives of the QSAP are to:

provide assurance of compliance with patient safety and clinical quality policies, standards and guidelines;assess the level of development of the patient safety system and clinical quality improvement;support improvement at a local, facility and systems level; andidentify current and future risks to patient safety within the NSW health system.

In order to deliver an informed evaluation of hospital safety to NSW Health and the community the assessment methodology needs to identify and adequately manage the significant risks inherent in implementing this type of program in the current environment. Perceived risks of the QSAP include the potential use of information by NSW Health or the organisations being assessed to blame individuals. Also, the misuse of information by the media and the potential for a political backlash to what is perceived to be unacceptable standards of patient safety and clinical quality. There is also the risk that the QSAP will not achieve its objectives and that it will not be able to accurately measure the status of patient safety and clinical quality. These factors have the potential to disengage management and clinicians whose cooperation will be pivotal to its success.It was identified through the literature and consultation with stakeholders that management of the identified risks will be enhanced if the QSAP has the following qualities. The QSAP should:

be patient centred and clinician reflected;be credible, transparent and rigorous;utilise a systems approach;have an improvement focus with no pass/fail outcomes;ensure active participation of executive, management and clinicians;be underpinned by a 5-7 year strategy;be cost effective with regard to:

direct and indirect costsdirect and indirect benefits;

include objective assessment and provision of proof of compliance;include a limited number of core, established dimensions to measure; andensure that any actions other than implementation of improvement plans such as an escalation or sanction strategy is managed by NSW Health.

There are a variety of methodological options for the QSAP to assess the safety and clinical quality in the NSW Health system. These include models from the Australian and international health arena as well as assessment and audit methods utilised in other industries such as finance, public sector services and aviation discussed previously. The options to be considered is informed by a review of the literature and information provided by key stakeholders.



The assessment framework

There are some underlying assumptions that are made when developing the assessment methodology. While these assumptions may not currently be met in all PHO’s it is anticipated that the appropriate systems will be developed as the QSAP is implemented. These assumptions are that the:

organisation has a system for management of patient safety and clinical quality;

organisation self-enforces and self –monitors compliance with policies, protocols and guidelines;

organisation continuously improves its performance; and

QSAP provides an external assessment and assurance on the patient safety and clinical quality system checking self enforcement and self-monitoring of improvement.

Different aspects of the Patient Safety and Clinical Quality System will be assessed at different levels. The assessment process will ultimately occur at the following three levels:

AHS or PHOFacility/Clinical Stream/Service Clinical Unit

It is also anticipated that area for improvement in the performance of NSW Health in respect to policy and program gaps will also be identified through the QSAP. The assessment of responses at different levels of the organisation will also be able to be correlated to assess effectiveness of governance and reporting structures. Implementation of policies and the effectiveness of performance monitoring and risk controls will be able to be evaluated (Figure 2).

AHS-Establishment of systems,

processes and guidelines

-Performance monitoring

FACILITY/CLINICAL STREAM-Implementation and local

adaptation of AHS systems, processes and guidelines- Performance monitoring

CLINICAL UNIT-Day to day application of processes and guidelines

- Risk control- Performance monitoring

Figure 2: Multi-level assessment and correlation of findings to evaluate governance system



The assessment process

Targeted focus on specific aspects of the patient safety and clinical quality system will occur on an annual basis. (Figure 3) It is anticipated that there will be a thematic approach to targeted areas of assessment which may be customised for specific AHS issues identified in previous Quality System Assessment. Alternatively, issues may be identified for targeted in depth assessment state-wide. It is anticipated that subjects for targeted in depth assessment may include systems and processes of peer review, incident management systems or death reviews. The QSAP will utilize a two staged assessment process, a self-assessment followed by external audit to verify the internal assessment process (Figure 3).The first stage self-assessment process is performed by the organisation through the completion and submission of the Quality System Activity Statement. The initial activity statement will be used to establish a base-line measure through a combination of directed response and open-ended questions. The base-line measure will:

identify characteristics of safety management system;identify differences in approach between PHO’s, facilities and clinical units;be used to help establish improvement aims;Identify or confirm targeted areas of assessment;be used to further develop criteria and questions for subsequent Activity Statements; andidentify existing risk process control points.

The second stage of the assessment process, the external audit will undertake the following activities to support the self-assessment phase:

an audit of outliers and a representative sample of Activity Statements will be performed to verify responses provided; andsite visits of a sample of respondents which will include:

interviews of key informants; anda review of sources of verification identified in the Activity Statement responses.

ReviewActivity

Statement criteria

Year 1 Year 5

Independentbaseline

data collection

Statement of activity Statement

of activityStatementof activity

targeted area of assessment



External Audit External Audit External Audit External Audit

Figure 3: Representation of the QSAP assessment framework



Closing the quality loop

There are a number of elements of the developing QSAP framework that will assist in ensuring the delivery of its improvement aims. These include the incorporation of an integrated risk management approach encouraging process improvement through risk ownership and control. Responses between organisational levels will be correlated to assist in the evaluation of governance structures. At the conclusion of the assessment process recommendations for improvements in the organisations patient safety and clinical quality systems will be provided. This will include information identifying exemplar sites, evidence based best practice models and opportunities for networking. In the event that an action is taken in response to findings these actions should be graduated and proportionate to the risk presented. Any action other than improvement strategies will be managed by NSW Health.Where possible a portfolio arrangement should be developed between external auditors and PHOs in order to develop relationships and foster a team approach. This will facilitate the improvement role of the QSAP and encourage coaching & mentoring of PHO staff.Results of the assessment will be reported to relevant stakeholders. The avenues for reporting will be as follows.

Reporting to PHO/AHS/Facility/Unit: The first report of assessment will be provided to the PHO/AHS/Facility/Unit that has been assessed. An interim report will be provided to allow the PHO/AHS/Facility/Unit to validate and comment on the report. The report will provide feedback on performance against assessment criteria, identify areas for improvement and provide an assessment of standing against other organisations including stratification to allow a comparison within peer groups.Reporting to NSW Health system to provide information on systems issues and identify strategies to improve patient safety & clinical quality systems. This may include identification of a need for specific initiatives, policy development or resourcing issues.Reporting to the public to provide an assessment of the state of the safety and quality management systems of NSW Health organisations. Aggregated results will be reported to the public. Some options for public reporting include web based reporting of performance against other organisations stratified for peer group and reporting of individual factors using statistical techniques that provide meaningful comparison and address issues of relative risk.



Scope and content of the assessment

Guidance on the scope and content of the QSAP was provided through two NSW Health documents that were developed in 2004, the NSW Patient Safety and Clinical Quality Program (PD2005_608) and the Quality System Assessments- assessment framework (NSW Health, May 2005). These documents provide specifications for the umbrella program, the NSW Patient Safety and Clinical Quality Program of which the QSAP is one element. Within the policy documentation outlining the NSW Patient Safety and Clinical Quality Program it was identified that the QSAP should include the following elements.

An annual review of AHSs to identify, analyse, advise on systemic issues related to:Patient safety; and Clinical quality.

A focus on compliance with standards, policy and guidelines.Compliance with the Patient Safety and Clinical Quality Program Standards.

The NSW Patient Safety and Clinical Quality Program identifies seven quality and safety standards to which all Area Health Services are required to comply. These Standards outline requirements for:

systems to monitor and review patient safety;effective clinical governance;incident management systems;mitigation of clinical risk;systems to assess core adverse event rates by medical record review;processes for performance review of clinicians by their peers in order to maintain best practice and improve patient care; andaudits of clinical practice.

It has been identified that some of the elements of the Safety and Clinical Quality Standards do not have any policy framework or existing policies and guidelines are either out of date or not robust enough to be assessed against. These include peer review, medical record audit and clinical audit. These elements were managed by identifying associated criteria as developmental. The initial base-line assessment is non-specific and focussed on fact finding and establishing a picture of the status quo. It is anticipated that there will be significant variation in the process and nature of these activities within the PHOs and that they may become a targeted area of improvement in the future.



Development of the assessment criteria and activity statement

It is anticipated that development of the assessment criteria will be an iterative process being informed by the findings of the assessment sand as best practice models emerge that are suitable for the NSW context. An initial set of assessment criteria were developed as part of this project and were used to develop questions in the AHS Activity Statement. The following policies and NSW Health documents were used to develop the criteria and AHS Activity Statement questions:

HPA 990024 1999 A Framework for Managing the Quality of Health Services in NSWCorporate Governance and accountability compendium for NSW HealthPD2005-608 NSW Patient Safety and Clinical Quality ProgramNSW Clinical Governance Directions Statement (2005)PD2005_609 Clinical Governance Implementation PlanPD2006_030 Incident Management PolicyPD2005_219 Reporting Maternal Deaths to the NSW Department of HealthPD2005_404 NSW Incident Information Management SystemGL2005_061 Complaints Handling Frontline – better practice guidelinesGL2006_002 Managing complaints or concerns about a clinicianPD2006_007 Complaint or Concern about a clinician – Principles for actionPD2005_333 Model Policy for Safe Introduction of New Interventional ProceduresCircular No 2004/56 Preventing wrong patient, wrong site, wrong procedurePD2005_247 Infection Control PolicyPD2005_203 Management of reportable infection control incidentsPD2005_261 Management of fresh blood componentsPD2005_206 Medication Handling in NSW Public Hospitals.Quality System Assessment – Assessment framework

In developing the Activity Statement the following data was identified as being potential sources of information for PHO’s in completing the Activity Statements or for use in the external audit and validation of the self-assessment.

IIMS – complaints and incidentsRCA recommendations & actionsHR information (credentialing, performance management system, staffing skill mix and quantum)Performance data (HIE or feeder systems)

Mortality reports, access block, wait lists and ICD adverse events codesTMF incident dataPoliciesGovernance structuresTraining/education schedulesQuestionnaires, culture survey, cross sectional surveyMedical record reviewRisk registersExtraordinary internal or external reviewsMinutes of meetingsInterview of key informants



The Quality System Activity Statement

Throughout the development process a set of requirements and characteristics of the Activity Statement and self- assessment process emerged. It was determined that the self-assessment tool that was developed should:

avoid a requirement for organisations to extract large amounts of documentation for desk-top review;allow open-ended responses to many questions particularly where there was heterogeneity of approaches within PHOs;include provision of rationale and information regarding assessment methods utilized;be paper based in the short term with consideration given to development of a web-based self assessment tool in the future;allow for partial or total exemption from some aspects where appropriate;ensure that the detail and effort of the assessment process is proportional to the level of risk presented by the issues being assessed; andbe simple to use.

The Activity Statement provides an assessment of different dimensions of the patient safety and clinical quality systems with questions relating to:

structure including governance systems, policies and any risk management framework that is in place within the organisation;process including risk control, quality improvement, review and policy implementation processes; andoutcomes including performance management of patient safety and clinical quality indicators.

The Activity Statement for assessment at AHS level and that is suitable for piloting was developed (Attachment A). The AHS Activity Statement primarily focuses on the structure, governance, policy and risk management frameworks in place. It is anticipated that the Activity Statements developed for self assessment at facility/stream or clinical unit level will incorporate more questions relating to processes in place to control risks, quality improvement processes, performance monitoring and management of clinical outcomes.



Conclusion

This project to develop a framework, methodology and self-assessment tool for the Quality System Assessment Program has provided a solid foundation for the continued development of further elements of the program. The methodology developed will allow for an iterative approach to the content areas for assessment and has been demonstrated to have strong support from key stakeholders in the process.The project utilized an approach that built on previous learning from the Australian and international experience from health and from other industries which have well developed safety management and audit systems. Frequent consultations with experts and stakeholders as the methodology and framework developed ensured that the approach developed had consensus support from key players who may act as champions and leaders for the QSAP in the future.Work subsequent to the completion of this project should build on outcomes to date and incorporate:

pilot of the AHS Activity Statement;development of the methodology and structure for the external audit process;development of Activity Statements at the level of facility/clinical stream and clinical unit;development of survey tools for Justice Health, the Ambulance Service, the Children’s Hospitals, mental health facilities, outreach services, community health services and miscellaneous services within the NSW Health jurisdiction;evaluation and refinement of the Activity Statements;verification of the methodology;costing for roll-out of the QSAP;development of tool to enable online lodgement of Patient Safety Activity Statement; anddevelopment of reporting formats and tools.



Attachment A - The AHS Quality System Activity Statement

development of quality system assessment program final...

Documents