Validation of a Toolkit to Assess the Communication Climatein Health Care Organizations

Matthew Wynia, MD, MPH,1 Megan Johnson, MA,1 Thomas P McCoy, MS,2 Leah Griffin, MS2

1Institute for Ethics, American Medical Association, Chicago; 2Wake Forest University School of Medicine, Winston-Salem NC.

Statement of Problem

• Effective communication is the foundation for quality health care

• Most measures of communication focus on the patient-clinician dyad.

• Communication might be affected by organizational factors, including a hospital or clinic’s internal climate and infrastructure.

• If organizations could assess the extent to which they promote (or hinder) effective communication, such assessments would be useful for quality monitoring, to focus QI interventions and, perhaps, for public reporting and benchmarking.

Objectives of Program

• To validate a 360° organizational assessment toolkit, designed to measure organizational supports for effective, patient-centered communication.

• Validation of the toolkit through the development process (content validity), and by examining correlations between each of the 9 domains assessed (construct validity), and comparing results in each domain with several related patient-reported measures of quality (convergent validity).

Description of Program

• Multi-stakeholder consensus on what organizations should do to ensure effective, patient-centered communication

• All performance expectations voted on by 21-member Oversight body

• 1-10 scales, assessing feasibility, importance and measurability

• Any mean score <7 or individual score ≤3 eliminated the performance expectation

• Agreement on 9 domains for assessment

• Toolkit developed to assess performance on these 9 domains through:

• Patient surveys

• Clinical staff surveys

• Non-clinical staff surveys

• Executive surveys

• Organizational policy workbook

• Focus group protocols (optional)

• 5 hospitals and 4 clinics nationwide were assessed using the toolkit

• 1,763 patient surveys, 651 staff surveys, 29 executive surveys

Data Analyses

• Correlations and Coefficient alphas of performance scores within and between sub-domains

• Associations of performance scores in each sub-domain with patient-perceived quality (including org. ethics) using Logistic Regression

• Comparisons of performance within and between the 9 organizations

RESULTS

This research was supported by the Ethical Force Program. The Institute for Ethics at the AMA initiated the Ethical Force Program ® in 1997 as a collaborative to develop health care system–wide performance measures for ethics. The protocol, surveys, and focus group questions were approved by Western Institutional Review Board. Funding was provided by a grant from The California Endowment.

Collect information

Evaluate performance

Understandorganizational commitment

Engage communities

Develop workforce

Engage individualsa) cultureb) languagec) health literacy

Key lessons learned

• Organizations can undertake a valid 360° assessment of performance with regard to supporting effective communication, with discrete results in 9 important domains. Assessment results may be useful for tracking organizational performance, benchmarking, and to inform tailored quality improvement interventions.

Summary of findings to date

• Survey domains are reliable (average Chronbach’s alpha = 0.83) and correlate to other quality measures. Considerable within and between site variability: all sites show strong results (score >80) in some domains and weak (score <60) in others – no domain received consistently high scores at all sites.

Figure 1Correlations of performance in the 9 domains

Table 2Correlations* between performance on each domain and several patient-perceived measures of quality

Figure 2Performance variability within and between organizations

Communication Quality Improvement:The Domains Assessed by the Toolkit

Table 1Coefficient Alpha of measures within each domain

Content area Coefficient AlphaPatient surveys

Coefficient AlphaStaff surveys

Org. Commitment 0.87 0.91

Data Collection 0.65 0.90

Develop Workforce n/a 0.93

Engage Community 0.64 0.78

Engage Individuals 0.90 0.82

Health Literacy 0.88 0.86

Language 0.83 0.96

Culture 0.59 0.88

Perf. Monitoring n/a 0.84

CommunicationDomain

I receive high quality medical care from

the health care system

My medical records are kept

private

If a mistake were made in my health

care, the system would try to hide it

from me

  OR (95% CI) OR (95% CI) OR (95% CI)

Org. Commitment 1.34 (1.22-1.54) 1.22 (1.05-1.40) 0.73 (0.66-0.86)

Data Collection 0.95 (0.90-0.95) 1.00 (0.95-1.05) 1.0 (1.00-1.05)

Develop Workforce 1.47 (1.28-1.69) 1.28 (1.10-1.47) 0.73 (0.62-0.86)

Engage Community 1.54 (1.28-1.76) 1.28 (1.10-1.54) 0.73 (0.59-0.86)

Engage Individuals 1.40 (1.22-1.61) 1.22 (1.05-1.40) 0.73 (0.62-0.86)

Health Literacy 1.40(1.22-1.61) 1.28 (1.10-1.47) 0.73 (0.62-0.86)

Language 0.90 (0.82-0.95) 1.05 (0.95-1.16) 1.0 (0.90-1.16)

Culture 1.28 (1.16-1.40) 1.16 (1.05-1.28) 0.82 (0.73-0.90)

Perf. monitoring 1.40 (1.22-1.54) 1.22 (1.05-1.40) 0.73 (0.66-0.86)

*Correlations between a 5-point change in scale score and patient responses to trust and quality items.