stakeholder engagement in a patient-reported outcomes · pdf filepurposes.37 at the time of...

ORIGINAL RESEARCH

Stakeholder Engagement in a Patient-ReportedOutcomes (PRO) Measure Implementation: A Reportfrom the SAFTINet Practice-based Research Network(PBRN)Bethany M. Kwan, PhD, MSPH, Marion R. Sills, MD, MPH, Deborah Graham, MSPH,Mika K. Hamer, MPH, Diane L. Fairclough, DrPH, K. E. Hammermeister, MD,Alicyn Kaiser, PA-C, MMSc, Maria de Jesus Diaz-Perez, PhD, andLisa M. Schilling, MD, MSPH

Purpose: Patient-reported outcome (PRO) measures offer value for clinicians and researchers, althoughpriorities and value propositions can conflict. PRO implementation in clinical practice may benefit fromstakeholder engagement methods to align research and clinical practice stakeholder perspectives. Theobjective is to demonstrate the use of stakeholder engagement in PRO implementation.

Method: Engaged stakeholders represented researchers and clinical practice representatives fromthe SAFTINet practice-based research network (PBRN). A stakeholder engagement process involvingiterative analysis, deliberation, and decision making guided implementation of a medication adherencePRO measure (the Medication Adherence Survey [MAS]) for patients with hypertension and/or hyperlip-idemia.

Results: Over 9 months, 40 of 45 practices (89%) implemented the MAS, collecting 3,247 surveys(mean � 72, median � 30, range: 0 - 416). Facilitators included: an electronic health record (EHR)with readily modifiable templates; existing staff, tools and workflows in which the MAS could be inte-grated (e.g., health risk appraisals, hypertension-specific visits, care coordinators); and engaged lead-ership and quality improvement teams.

Conclusion: Stakeholder engagement appeared useful for promoting PRO measure implementation inclinical practice, in a way that met the needs of both researchers and clinical practice stakeholders. Limita-tions of this approach and opportunities for improving the PRO data collection infrastructure in PBRNs arediscussed. (J Am Board Fam Med 2016;29:102–115.)

Keywords: Comparative Effectiveness Research, Hypertension, Medication Adherence, Patient Outcome Assess-ment, Patient-centered Outcomes Research, Practice-based Research

Two key elements of patient-centered outcomesresearch (PCOR) are (1) stakeholder engagement

throughout the research process, and (2) selectionand measurement of “outcomes that the populationof interest notices and cares about and that informdecision making relevant to the research topic.”1

This article was externally peer reviewed.Submitted 26 April 2015; revised 28 August 2015; ac-

cepted 9 September 2015.From the Departments of Family Medicine (BMK), Pediatrics

(MRS), and Emergency Medicine (MRS), and Divisions of Car-diology (KEH) and General Internal Medicine (LMS), Depart-ment of Medicine, and Adult and Child Consortium for HealthOutcomes Research and Delivery Science (ACCORDS) (BMK,MRS, MKH, DLF, KEH, LMS), University of Colorado Schoolof Medicine, Aurora, CO; Children’s Hospital Colorado, Aurora,CO (MRS); The DARTNet Institute, Aurora, CO (DG); Depart-ment of Biostatistics and Informatics, Colorado School of PublicHealth, Aurora, CO (DLF); Metro Community Provider Net-

work, Englewood, CO (AK); Salud Family Health Centers, FortLupton, CO (MdJD-P).

Funding: This work was supported by the Agency forHealth care Research and Quality grant 1R01HS019908(Scalable Architecture for Federated Translational InquiriesNetwork; principal investigator, Lisa M. Schilling).

Conflict of interest: none declared.Corresponding author: Bethany M. Kwan, PhD, MSPH,

University of Colorado School of Medicine, ACCORDS,13199 E. Montview Blvd, Ste 300, Mail Stop F443 Aurora,CO 80045 �E-mail: [email protected]).

102 JABFM January–February 2016 Vol. 29 No. 1 http://www.jabfm.org

The implication is that, historically, outcomes mea-sures in clinical research (eg, blood pressure, sur-vival, laboratory test results) do not reflect out-comes that matter to stakeholders in that research(eg, patients, clinicians, researchers).2 Outcomesthat matter may reflect more subjective states ofpatient well-being, experience, or behavior that arenot directly observable by another person. Patient-reported outcomes (PROs), defined as “outcome[s]reported directly by patients themselves and notinterpreted by an observer,”3 have been widely ad-opted as one way to assess outcomes that matter toPCOR stakeholders.4 Hence, PCOR protocols of-ten call for the measurement of PROs, and forstakeholders in that research to be engaged in mak-ing decisions about which PRO to measure andhow to measure it (ie, PRO selection), and how tosystematically collect PRO measures from the pop-ulation of interest (ie, PRO measure implementa-tion).5–8

In practice-based research, PRO measures areoften meant to be implemented in the clinical prac-tice setting, administered to patients in the contextof their care. Thus, both the researchers and thoseinterfacing with clinical practice (eg, patients, cli-nicians, staff, operations, and practice and organi-zational leadership) are stakeholders whose diverseperspectives should be considered in a PRO selectionand implementation effort. In addition to increasingrelevance and interest, engaging both researchers andclinical practice stakeholders in PRO selection andimplementation may reduce barriers to implementa-tion and enhance data quality and value.5,8–15 Whilethere are several frameworks and methods for stake-holder engagement,15 there is currently limited liter-ature on the application of these methods in PROselection and implementation. The objective of thisarticle is to describe the application of a stakeholderengagement methodology to PRO implementation inclinical practices involved in a practice-based researchnetwork (PBRN). The PRO measures implementedas a result of this effort are then intended to be usedas outcomes data for our broader PCOR efforts con-cerning the effects of care in a patient-centered med-ical home (PCMH) on outcomes for a cohort ofpatients with hypertension and/or hyperlipidemia.

PRO measures and UsesPROs are often measured using questionnaires ad-ministered directly to patients, either verbally or in awritten or electronic survey.16 PRO measures include

measures of a patient’s health status, ability to func-tion, symptoms, quality of life, or experience ofcare.17 Information provided by PRO measures canbe used to inform individual patient care, populationsurveillance, adverse event monitoring, quality im-provement, performance monitoring, and re-search.18,19 In clinical practice, uses of PRO measuresinclude monitoring symptoms, disease progress, andresponsiveness to treatment; augmenting patient–provider communication and shared decision making;and providing feedback to health care professionals asa performance measure.6,20–23 Systematic reviews ofthe effects of implementing PRO measures in clinicalpractice show potential for improved care processes(eg, patient–provider communication, diagnosis),whereas findings related to health outcomes (eg, pa-tient health and well-being, satisfaction) aremixed.6,7,22,24,25

PRO ImplementationWhile PRO measures have value for both clinicaland research purposes, there are scientific and lo-gistical barriers to efficient and effective implemen-tation and use of the results—a longstanding prob-lem in practice-based data collection efforts.26,27

Beyond common concerns such as data validity andquality (eg, data completeness),28–30 researchersand clinicians may have conflicting priorities thatmake it difficult to collect PRO measures to serveresearch and clinical purposes. The most valid mea-sures and rigorous data collection methods from aresearch perspective may not have sufficient clini-cal, quality improvement, or operational value; assuch, practices may not be able to justify the timerequired to modify workflows, implement data col-lection tools, and train personnel. Yet, when indi-vidual practices implement PRO measures in waysthat best fit their environment and workflow (apriority for clinicians), this introduces practice-level inconsistencies in how data are collected andpossible selection bias, creating threats to validityfrom a research perspective.31 Thus PRO imple-mentation in clinical practice may benefit fromstakeholder engagement methods that facilitatealignment of research and clinical practice stake-holder perspectives.

Stakeholder Engagement MethodsDeverka and colleagues15 define stakeholder en-gagement as “An iterative process of actively solic-iting the knowledge, experience, judgment and val-

doi: 10.3122/jabfm.2016.01.150141 Stakeholders Implement a PRO: From the SAFTINet PBRN 103

ues of individuals selected to represent a broadrange of direct interests in a particular issue, for thedual purposes of: 1) Creating a shared understand-ing; and 2) Making relevant, transparent and effec-tive decisions.” Early steps in engagement includeidentifying relevant stakeholders, establishing rolesand responsibilities of the various stakeholders, andselecting an engagement strategy.14 This engage-ment strategy should build trust, garner the com-mitment of the stakeholders and researchers, andelicit and align diverse perspectives. Stakeholderengagement methods go beyond elicitation of theperspectives of stakeholders—as exemplified by fo-cus group or key informant interview methods—toencompass full participation and collaboration.15,32

Through such collaboration, research and clinicalstakeholders seek to select PRO measures that are“actionable, efficient, interpretable, obligatory, anduser-friendly,” aspects that are thought to be keyfeatures of successful PRO implementation.33 Thuswe expected that stakeholder engagement strategieswould allow us to select and implement PRO mea-sures in a way that met both research and clinicalpractice needs.

MethodsSettingThe Scalable Architecture for Federated Transla-tional Inquiries Network (SAFTINet) is a multi-state, safety-net focused PBRN. PBRNs are builton a foundation of stakeholder engagement and area vital laboratory for real-world research.34,35

SAFTINet is also a distributed data network,36

with locally controlled databases of administrative,clinical, Medicaid claims and enrollment, and PROdata—including the PRO data gathered throughthe stakeholder engagement process describedhere—which can be used for a broad range ofresearch, quality improvement, and care deliverypurposes.37 At the time of this project, SAFTINethad 4 partnering clinical practice organizations,with 54 participating primary care practices (feder-ally qualified health centers or federally qualifiedhealth center “look-alikes”) in Colorado and Ten-nessee, caring for approximately 260,000 uniquepatients per year. An estimated 65,000 patients hadhypertension and 39,000 patients had hyperlipid-emia. Specialties include family medicine, internalmedicine, pediatrics, and behavioral health. Prac-tices range in location from urban to rural; all havean electronic health record (EHR).

Stakeholder Engagement MethodsExisting models and frameworks for stakeholder en-gagement in PCOR include a taxonomy of stake-holder engagement proposed by Concannon and col-leagues,8 and a conceptual model for stakeholderengagement from Deverka and colleagues.15 Bothsuggest that the first step is identifying relevant stake-holders, which may include researchers, clinicians,health care providers (the institutions), and patients,among others. According to the analytic-deliberativemodel for stakeholder engagement presented by De-verka et al, the stakeholders then undertake a processof gathering and analyzing the evidence (the inputs,which include stakeholder values, experience, and re-view of the literature), deliberating (the methods forcombining evidence), and decision making (the deci-sions, including topic generation, study designs, andimplementation strategies). Although this model waspublished after our work was completed, it closelymirrors our process and we refer to it here as anorganizing framework.

Identifying Relevant StakeholdersThe 7P stakeholder engagement framework high-lights 7 stakeholder groups to engage in developingand implementing research protocols: patients,providers, “principal investigators” (ie, research-ers), policymakers, product makers, payers, andpurchasers.2 Our network’s first research protocolswere broadly concerned with studying the effects ofhealth care delivery models (eg, the PCMH) onchronic disease control. While all “7Ps” of stake-holder groups are relevant to this research, wechose to focus primarily on providers and research-ers for our first attempt at stakeholder engagementas a nascent PBRN (we have since engaged pa-tients, payers, policymakers, product makers, andpurchasers in our research).

The stakeholder group labeled “providers” is abroad category that includes not just the cliniciansthemselves but also others who work for clinical prac-tice organizations. This includes nursing staff, medi-cal assistants, quality improvement teams, health in-formation technology staff, and clinical operationsand administration personnel. Upon recruitment toSAFTINet, each clinical practice organization part-ner named an internal site lead (typically someone ina leadership position, such as a clinical director,funded at 0.10 full-time equivalent (FTE) under asubcontract of our infrastructure development grant)and a site coordinator (typically someone with mas-


ter’s- or doctoral-level training in a clinical or publichealth–related field; funded at 0.20 FTE) to overseeand manage the scope of work required to implementthe network infrastructure and research protocols(work much broader than that required to undertakethe activities described in this article). The site leadseach were asked to identify a person in their organi-zation who could represent the clinical practice stake-holder perspective in developing research protocols,then serve as a champion for implementing the re-search protocols within their organization (also sup-ported at 0.10 FTE by our grant). The selected “clin-ical practice representatives” were typically cliniciansor other doctoral-trained individuals with interests inhealth services research or quality improvement; theywere often those directly involved in internal PCMHimplementation and evaluation.

Analysis, Deliberation, and Decision MakingAfter identifying relevant stakeholders, the nextstep in stakeholder engagement is to undertake aniterative process of analysis, deliberation, and deci-sion making.15 We primarily used facilitated groupdiscussions for this process. The partner represen-tatives, along with the site coordinators, severalPCOR investigators, the network project manager,and community engagement leads, comprised our“partner engagement community” (PEC), whichmet twice monthly via Web conference for over 3years to discuss the developing research protocols,including the process of PRO selection and imple-mentation described in this article (which tookplace over 10 months of this 3-year period). Thecommunity engagement leads, who were jointlyaffiliated with the American Academy of FamilyPhysicians National Research Network and theDARTNet Institute (http://dartnet.info), coordi-nated and led the PEC meetings. During thesemeetings, we identified stakeholder needs, priori-ties, and concerns; reviewed relevant literature ormet with experts to ensure our work was guided bythe evidence; proposed and debated options; andultimately reached a consensus. Detailed agendasand PowerPoint slides were used to organize think-ing during PEC meetings; follow-up calls and E-mails and meeting summaries highlighting key de-cisions and next steps were used to communicateplans (especially important when PEC representa-tives periodically missed the conference calls), andproject plans were used to monitor progress. ThePEC meetings also helped to foster trust and rela-

tionships among the stakeholder representatives,based on mutual respect and understanding.

Between PEC meetings the partner represen-tatives would meet with others in their organiza-tion who had a vested interest in whatever topicwas currently being discussed to vet options andideas generated during PEC meetings. They thenbrought these perspectives back to subsequentPEC meetings to inform decision making. Forinstance, they would meet with quality improve-ment teams, individual providers in participatingpractices, or working groups dedicated to certainhealth conditions (eg, asthma, hypertension).Similarly, the researchers on the PEC wouldmeet with the larger research team to discuss andprovide input back to the PEC. We also con-ducted local partner site pilot testing and itera-tion before reaching final decisions. While wehad an open invitation for partners to bring oth-ers from their organization to participate in thePEC discussions, this rarely if ever occurred be-cause of busy schedules.

Implementation PlanningThe clinical partner representatives served as liaisonsbetween the PEC and their local sites, and took respon-sibility for local decision making regarding specific im-plementation strategies. We provided a structured PROplanning worksheet to guide these local decision-mak-ing processes (see Online Appendix). The PRO plan-ning worksheet was designed for this project based onpublished recommendations from the International So-ciety of Quality of Life (ISOQOL).38 The ISOQOL“User’s Guide to Implementing Patient-Reported Out-comes Assessment in Clinical Practice” articulates 9questions to be answered before implementation.39 Weidentified 3 additional implementation decisions specificto our objectives, which were necessary because of thescale of the project (54 practices) and the need to inte-grate the data into the SAFTINet databases, which areObservational Medical Outcomes Partnership commondata model version 4 schemas (http://OHDSI.org), us-ing a standardized data format to support interoperabil-ity. We addressed these decisions in 3 high-level steps, asshown in Table 1. Our overarching goals were to (1)collect data of sufficient scale and quality that itcould be used for rigorous and relevant research,(2) improve care for patients at risk for cardiovas-cular disease, and (3) implement the data collectionin a feasible, minimally disruptive, and sustainablemanner.


http://OHDSI.org

Implementation Step 1: Determine PRO Content andSelect a Measurement ToolPRO data collection goals from the researcher andclinical partner perspectives are listed in Table 2. Thespecific context for the PRO implementation workpertained to our research on the effects of receivingcare in a PCMH on outcomes for patients with in-creased cardiovascular risk as a result of hypertensionand/or hyperlipidemia. The types of outcomes con-sidered by the group to be relevant to this researchcontext included general health status, patient activa-tion, medication adherence, cardiovascular risk per-ception, readiness for change, and self-efficacy. Weultimately agreed on medication adherence and bar-riers to medication adherence as the content area.Notably, medication adherence is (1) consistent with

existing clinical partner reporting requirements, (2)an important factor in reducing cardiovascularrisk,40,41 and (3) potentially influenced by receivingcare in a PCMH.42,43

In general, attributes of appropriate instrumentsfor measuring PROs include consistency with theconceptual and measurement model for the con-struct of interest, reliability, validity, ability to de-tect change, ease of interpretation of the results,low administrative and respondent burden, and ex-istence of cultural and language adaptations andtranslations.44 Among the existing brief, validatedinstruments for medication adherence,45 we se-lected a 1-item instrument developed by Gehi andcolleagues46 because of its simplicity and discrimi-nant and face validity. This tool asks respondents to

Table 1. Implementation Steps and Decisions for Patient-Reported Outcome Measures

Implementation Step Decisions

1: Determine PRO content and select ameasurement tool

1a: Identify stakeholder goals, available resources and barriers for collectingPRO measures1b: Determine which questionnaire(s) to use

2: Establish local implementation plans 2a: Select the patients to be assessed2b: Determine the setting and timing of assessments2c: Choose a mode for administering and scoring the questionnaire2d: Design processes for reporting results to providers and/or patients2e: Identify aids to facilitate score interpretation2f: Develop strategies for responding to issues identified by the questionnaires2g*: Identify local champions and key personnel, and prepare tools andmaterials for training and implementation across practices2h*: Establish systems for electronic data capture of the PRO results instructured, discrete fields

3: Implementation and evaluation ofthe PRO measure

3a*: Facilitate and track implementation across practices to ensure adequatedata quality, and assess and address barriers3b: Evaluate the impact of the PRO implementation on the practice

*SAFTINet project additions to the International Society for Quality of Life Research patient-reported outcome guidance.39

PRO, patient-reported outcome.

Table 2. Patient-Reported Outcome Measure Data Collection Goals

Research Partner Goals Clinical Partner Goals

• Relevant to patients with condition ofinterest (eg, hypertension andhyperlipidemia)

• Align with the clinical partners’ currentreporting requirements and quality improvementinitiatives, such as PCMH and MU

• Potentially influenced by the care in aPCMH

• Brief (�4 items), valid measure exists in bothEnglish and Spanish and at an appropriatereading level

• Valid and reliable measure exists • Is of utility to the providers (ie, related to acommon patient issue, amenable to intervention)

• Content should not be redundant withexisting data sources (eg, the EHR)

• Informs day-to-day interactions with patients

EHR, electronic health record; MU, meaningful use; PCMH, patient-centered medical home.


indicate how often in the last month they had takentheir medication as prescribed. We slightly modi-fied the instructions to state “as instructed” ratherthan “as prescribed,” to include over-the-counter aswell as prescription medications. We adapted anexisting, validated medication adherence barrierschecklist at the request of the clinicians and prac-tice representatives, focusing on barriers that areamenable to intervention and commonly encoun-tered in clinical settings.47 The tool was translatedinto Spanish by a certified translator.

Before full-scale implementation, 3 of the clini-cal practice representatives asked 1 to 2 clinicians intheir practices to pilot-test the tool’s readabilityand clinical utility during real clinical encountersover the course of 2 weeks. The clinical practicerepresentatives reported back to the PEC that thebarriers question helped to engage patients in aconversation about barriers to medication adher-ence, which the clinician pilot-testers considered tobe clinically useful. To support evaluation of thetool’s utility, we added 2 items to assess the fre-quency of such conversations. After 2 more weeksof pilot testing, with no further suggestions, wedeemed the tool appropriate for full-scale imple-mentation. The final, single-item medication ad-herence measure and barriers checklist was referredto as the Medication Adherence Survey (MAS).

Implementation Step 2: Establish Local ImplementationPlansUsing our PRO planning worksheet, each of theclinical partner organizations established local im-plementation plans, considering existing workflow,personnel, resources, structures, and processes, anddetermined the best way in which a new PROmeasure could be integrated into their practices(Table 3). Each clinical partner representative sep-arately convened appropriate others in their orga-nizations to discuss and complete the worksheetbased on what they expected would work best intheir practices. (Note that those “appropriate oth-ers” varied by organization, ranging from organi-zation-level quality improvement teams or workinggroups tasked with quality improvement in cardio-vascular risk reduction, to the collective of primarycare clinicians within the organization). After thislocal decision-making process occurred, membersof each clinical partner organization reviewed theirplan with members of the central research team,the feasibility of the plan was discussed, and revi-

sions based on input from both researchers andclinicians were made.

To optimize response rates, data quality, andsustainability, we allowed for flexibility across part-ner organizations. The timelines for implementa-tion varied across organizations, as PRO imple-mentation needed to be in sync with other plannedorganizational activities (eg, EHR upgrades). Weprioritized consistency across partner organizationsin terms of (1) patients targeted, (2) frequency ofadministration, and (3) the format for the struc-tured capture of results (ie, discrete fields for vari-able labels and values). Table 3 describes the im-plementation plans made by the 4 participatingclinical organizations and the preferences and rec-ommendations from the research team; each sec-tion of the table reflects a major section of the PROplanning worksheet.

Implementation Step 3: Implementation and Evaluationof the PRO measureEach of the clinical partner representatives workedwith his or her respective practices to implement,evaluate, and adapt their PRO data collection pro-cesses according to plan and using the internalresources at their disposal. The network projectmanager and the community engagement leadsprovided high-level project management over theimplementation process using project managementtools, such as project plans and regular check-incalls with each site coordinator. Implementationwas an iterative process, akin to continuous qualityimprovement, including auditing the implementa-tion processes, following up with providers to en-sure consistent data collection, and addressing anybarriers and concerns about impact on workflow,patient satisfaction, and quality of care. Partnersdocumented in their planning worksheets anychanges over time to their data collection processesso that we could maintain a record of what changesoccurred and when.

We report on the approach to and results of theplanning and implementation process based on de-tailed notes from conference calls and team meet-ings, project plans, reports from clinical partnersshowing the total number of surveys administeredat each participating practice, slide presentations,and E-mails from throughout the process. Theserecords were used to ensure we accurately de-scribed the process and numbers of surveys col-lected, and to summarize the lessons learned, which


Table 3. Patient-Reported Outcome Measure Implementation Plan Details By Organization

PRO PlanningWorksheet Sections

Research TeamRequest

Decisions

Organization 1 Organization 2 Organization 3 Organization 4

Patients targeted All adult patients withhypertension orhyperlipidemia

All adult patientswithhypertension orhyperlipidemia

All patients withhypertension orhyperlipidemia

All adult patients All patients withhypertension

Identification ofpatients

No request made Data query andlist generationby IT staff

Medical assistantchecksmedication list,gives the tool toanyone takingblood pressure orcholesterolmedications

All adult patientsasked tocomplete thetool uponcheck-in forvisit

Rooming medicalassistantreviewsproblem listand reason forvisit

Frequency ofadministration

At least annually; allvisits preferred

Annually At all visits At all visits At hypertensionvisits (2 times/year)

Mode of administration(by whom, in whatsetting, with whatmaterials)

No request made Care coordinatorsadministered bytelephone aspart of largerpatient survey

Administered bymedical assistant;responsesrecorded onpaper uponcheck-in for visit

Administered aspart of writtensurvey beforetheappointmentupon check-infor visit

Electronicsurvey;provider askedthe questionsverbally andrecorded theanswers in theEHR template

Data capture Responses recorded instructured fields inan EHR or otherdatabase

Care coordinatorsenteredresponsesdirectly intodiscrete fieldsin the EHRtemplate

Responses manuallyentered intoancillarydatabase; paperforms scannedinto the EHR

Medical assistantentered resultsin the EHRtemplate

Responsesentered intothe EHRtemplate forhypertensionvisits

Presentation andinterpretation ofresults (to whom,when, in what form?)

Recent/timely patientresponses areavailable to theprovider at the timeof the visit

Care coordinatorsscheduled anappointmentwith a providerfor patientswith barriers;patientresponses areavailable to theprovider in theEHR summary

Paper form isavailable to theprovider at thetime of the visit

Available to theprovider at thepoint of carethrough theEHR

Part of theproviderinterview withpatients duringhypertensionvisits

Training PowerPointpresentation madeavailable

Medicalproviders,nurses, andcarecoordinatorstrained inadministration,utilization, andinterpretation;led by clinicianchampions

Clinicianchampionstrained otherproviders;practice managertrained themedical assistants

Training atprovidermeeting onpurpose oftool,interpretation,process/workflow;Web-basedtraining formedicalassistants

Business analysttrains medicalassistants onworkflow forhypertensiontemplate;medicaldirectortrainedproviders onuse of thetemplate;training doneremotely withancillary in-person supportfortroubleshooting

Continued


were explicitly discussed during PEC meetings andcaptured in the minutes from the meetings. Be-cause the PRO was implemented as a clinical qual-ity improvement project, informed consent of pa-tients who completed the PRO was deemedunnecessary by 2 institutional review boards, solong as secondary use of clinical data for researchpurposes was described in the organizations’ pri-vacy policies. Use of the PRO data in our PCORprotocols (results to be reported elsewhere) thus isconsidered secondary use and will be provided tothe research team under a data use agreement.

ResultsStakeholder meetings began in December 2011,and we reached consensus on implementation plansby September 2012. The mutually agreed uponstart date for use of the MAS was January 1, 2013.Each partner’s local site coordinator was taskedwith coordinating execution of the plan, trainingstaff, monitoring progress, and assessing and ad-dressing barriers to the implementation and use ofthe survey. Site coordinators reported monthly tothe project manager the number of surveys col-lected at each participating practice. Given the fo-cus on adult patients, 45 of the 53 network primarycare practices were expected to participate in thisPRO implementation (8 were pediatrics-only prac-tices). Note that organization 3 ended up imple-menting the MAS in all practices and with all ages,including their 3 pediatrics-only practices, since theMAS was expected to be useful for the care ofchildren taking medications for chronic conditionssuch as attention deficit hyperactivity disorder orasthma. Because this was unique to organization 3and not relevant to our PCOR protocol, the num-

bers reported herein do not include surveys admin-istered to those under age 18 years.

Initial adoption of the MAS was slow. After 5months, 22 of 45 practices (49%) had adopted theMAS (ie, collected at least 1 survey). Commonbarriers to implementation cited by partners in-cluded competing organizational priorities andworkflow changes. At that time we discussedthese challenges as a group and opted to adopt abenchmark. Each practice would attempt to col-lect at least 25 surveys by the end of the grant inSeptember 2013. This 25-survey benchmark wasbased on a recommendation from the study bio-statistician, who indicated that 25 surveys perpractice (the unit of analysis in the multilevelmodels specified in our PCOR protocol) wouldprovide adequate power for testing our primaryhypothesis (ie, that patients receiving care in aPCMH have higher medication adherence thanthose who do not). By the end of September, 4months later, 40 of 45 practices (89%) had im-plemented the MAS and collected a total of 3247surveys, an average of 72 per practice (median, 30surveys; range, 0 – 416 surveys). Among the prac-tices, 58% (26 of 45) met the 25-survey bench-mark. The distribution of survey totals acrosspractices and organizations is shown in Table 4.Of note, the 22 practices that had already ad-opted the MAS at the time the benchmark wasimposed outpaced those who had not yet adoptedthe MAS (mean � 119 surveys; median � 91surveys). Conversely, the 23 practices that hadnot yet adopted the MAS approached the 25-survey benchmark and little more by the end ofthe project (mean � 27 surveys; median � 23surveys). Thus, a benchmark seems to be helpfulfor late adopters.

Table 3. Continued

PRO PlanningWorksheet Sections

Research TeamRequest

Decisions


Required resources Time for localsitecoordination

Care coordinatortime; EHRteam time tobuild template;IT staff timeto generate acontact list ofeligiblepatients

Time fortraining; timefor manualdata entry

Time formodificationsto the EHR;time fortraining

Time for EHRtemplatebuilding;personnel timefor trainingand monitoringprogress

EHR, electronic health record; IT, information technology; PRO, patient-reported outcome.


Lessons LearnedAt a PEC meeting clinical practice representativeswere asked to reflect on barriers and facilitators toimplementation of the MAS across their practices.Partners reported that the following factors facili-tated implementation of the MAS in clinical prac-tice: the presence of an established EHR with read-ily modifiable templates (a critical factor present inorganizations 1, 3, and 4, but not 2); the availabilityof care coordinators to facilitate data collection,interpretation, and triage (present in organization1); audit and feedback reports; established tools andstructures and associated workflows in which theMAS could be integrated (eg, health risk appraisals,hypertension-specific visits; present in organiza-tions 3 and 4); and engaged leadership and qualityimprovement teams.

The clinical practice stakeholders reported sev-eral challenges during this process. A major chal-lenge was that maintaining momentum can be dif-ficult when there is staff turnover, requiringadditional training and reorientation to the project.To the extent that the data collection can be inte-grated into standard workflows and processes of

care in which new staff will be trained, this issuecould be mitigated. Organization 2 specificallycited the lack of available incentives to encouragepractice staff and providers to administer the MASas a barrier. Organization 1 cited lack of priorexperience with practice-based research as a chal-lenge (all other organizations have been involvedwith practice-based research for �10 years). Orga-nization 2 cited competing priorities as a majorbarrier, mainly a system-wide effort to implement anew EHR, which prevented the MAS from beingincorporated into EHR templates as was done atthe other organizations. In addition, as described inthe quality improvement and practice transforma-tion literature,48 engaged leadership and a willingchampion within each individual practice (eg, qual-ity improvement leader or office manager) helpedto maintain momentum, to demonstrate the valueof the data for improving quality of care, and toprovide audit and feedback to providers and staff.Practices lacking their own local champion (evenwith engaged leadership at the organization level)were thought to have struggled the most with im-plementation of the MAS.

Table 4. Participating Organizations and Clinical Practices and Data Collection Results over the 9-Month StudyPeriod


Clinical practicerepresentative

Director of behavioralhealth

Internal medicinephysician

Associate medicaldirector

Research and datamanager

Participatingpractices, n

14 9 13 9

Practice specialties, nIMH 0 0 5 0FM 6 4 7 9IM 1 2 1 0FM � IM 0 3 0 0FM � IM �

pediatrics7 0 0 0

Practice location(urban/rural), n*

6/8 9/0 5/8 7/2

Medical providers perpractice, median(range)

4 (1–9) 8 (6–15) 2 (1–5) 7 (2–15)

Patients per practice,median (range)

2,925 (554–10,359) 6,644 (5,539–10,199) 2,765 (1,869–16,237) 8,689 (2,375–15,058)

PracticesimplementingMAS, n (%)

10 (71) 8 (89) 13 (100) 9 (100)

Surveys per practiceMedian (range) 24 (0–136) 23 (0–149) 69 (3–416) 164 (16–311)

*Based on 2010 census designation; some practices designated “rural” by the census are better characterized as suburban.FM, family medicine; IM, internal medicine; IMH, integrated mental health; MAS, Medication Adherence Survey.


This approach to engagement required signifi-cant compromise and investment of effort from allparties. The researchers had to tolerate the lack offidelity to a specific data collection protocol, vari-ability across practices in commitment to the workand its implementation, and a prolonged rolloutperiod. Even with a considerable amount of finan-cial support from our infrastructure grant, the clin-ical organizations had to commit considerable timeand effort to discussion and planning, and theninvest internal resources in a full-scale implemen-tation of the PRO. Given that their highest prioritywas providing care to patients, this investment wasat times difficult to justify. We estimate this workrequired approximately 2000 person-hours for theplanning ([5 people � 100 hours each] � ]10 peo-ple � 60 hours each]) and coordination of local sitetesting and implementation (4 organizations � 2–3people per organization� 80 hours each).

Ultimately, the clinical partner stakeholders per-ceived the MAS as having marginal clinical utility;although the barriers checklist at times drove con-versations with patients, the single-item adherencemeasure was perceived as invalid because most pa-tients reported taking their medications exactly asinstructed �90% of the time. Although our stake-holder engagement process was designed to pro-mote sustainable data collection, most SAFTINetpractices elected to discontinue use of the MAS atthe project’s end, primarily because of the desire tominimize the data collection burden to patients andproviders.

DiscussionIn this article we describe our method of engag-ing researchers and clinical practice stakeholdersin the implementation of PROs relevant to thecare of patients with hypertension and/or hyper-lipidemia. We undertook this approach with theexpectation that aligning research and clinicalpractice stakeholder perspectives would yieldhigh-quality, complete, clinically useful, and sus-tainable data collection processes. Our processmirrored the analytic-deliberative model ofstakeholder engagement developed by Deverkaand colleagues,15 such that stakeholders analyzedinputs (stakeholder values, personal experience,and research evidence), deliberated using facili-tated discussions, and decided on implementa-tion strategies using planning worksheets in-formed by the ISOQOL User Guide.39

Our approach to decision making and imple-mentation can be likened to a cascading hub-and-spokes model (Figure 1). In summary, at the hub(the PEC), the site coordinator and clinical practicerepresentative from each provider organization metwith the research and governance personnel forhigh-level analysis, deliberation, and decision mak-ing. These representatives then convened sepa-rately with others in their local organizations andindividual clinical practices to again analyze, delib-erate, and make more specific, context-appropriatedecisions locally, which were communicated backto the hub. The clinical practice representativesand site coordinators were then responsible formanaging local implementation of decisions, in-cluding determining which specific individualswould take on various roles and responsibilitieswithin the organization and within each practice inthe organization.

We learned several lessons about the utility of astakeholder engagement approach to PRO selec-tion and implementation. Using this approach, wesuccessfully selected and implemented PROs thatmet the needs of both researchers and clinical prac-tice stakeholders, although there was only a looselydefined protocol, and tolerance for flexibility andadaptations across organizations was required.While resource intensive, a notable benefit of thisapproach was that clinical partners were willing tocoordinate adoption of PRO measures across allprimary care practices in their organizations. An-other benefit of an approach in which PRO mea-sures are implemented within the context of clinicalcare and quality improvement (ie, not just for re-search purposes, but rather to support clinical de-cision making at the patient level) was that thetraditional research concept of subject recruitmentdid not apply. Thus our institutional review boardswaived informed consent (comparable to secondaryuse of electronic health records data), which can bea critical logistic barrier.

There are limitations to what stakeholder en-gagement can achieve, however. We expected thatengaging clinical practice stakeholders would en-hance the selected PRO measure’s clinical utility(informs clinical decision making) and sustainabil-ity (use of the PRO measure continues after thestudy period ends). Ultimately, the clinical utilityand sustainability of the MAS was predicated on theperceived performance characteristics of the MASin practice. Key PRO performance characteristic


constructs include psychometric soundness, per-son-centered, meaningful, amenable to change, andfeasible to implement.49 Unfortunately, the clinicalpractice stakeholders did not perceive the MAS asvalid because an unexpectedly large majority ofpatients reportedly indicated perfect medicationadherence. The barriers checklist seemed to havethe greatest utility from a clinical practice perspec-tive. The finding that asking patients about barriersserved as an entree to a conversation about medi-cation regimens is consistent with a proposedframework for evaluating the effects of using PROmeasures to support chronic illness care.50 The lowperceived validity of the MAS may have been due toour slight modifications to the measure by Gehi etal,46 thus invalidating the previously establishedscale. Alternatively, it may be that a broad measureof general medication adherence (ie, not specific toa particular medication) is unclear or unacceptable

to patients. Furthermore, while the measure byGehi et al had been validated in a research context,it was not, to our knowledge, tested in a real-worldclinical practice context. Finally, it may be thatmedication adherence is not a suitable PRO forpatients with increased cardiovascular risk. In thetime since we conducted this work, the AmericanHeart Association released a statement indicatingthat appropriate PRO measures for assessing car-diovascular health may include patient-reportedhealth status with respect to symptoms (burden ofangina, dyspnea, depression), functional status, andhealth-related quality of life—topics that may res-onate with patients as meaningful to their quality oflife.51

LimitationsOur experiences with this approach may not gen-eralize beyond this group of partners or beyond this

Figure 1. Cascading hub-and-spokes model of implementation. IT, information technology; PCOR, patient-centeredoutcomes research; QI, quality improvement.


particular clinical context (ie, PROs for a cohort ofpatients with hypertension/hyperlipidemia); thetimeline may have been much shorter and the levelof interest at the practice level may have beengreater if there was a well-established recommen-dation for a PRO for this cohort. While we con-sidered researcher and clinical practice stakeholderperspectives in PRO selection and implementation,we did not engage patients beyond asking for theirfeedback on the readability of the survey duringpilot testing. By engaging patients, we might haveselected a more valid and reliable measure of ad-herence, or we might have selected a different PROaltogether. We did not formally evaluate data col-lection using this stakeholder engagement ap-proach versus another approach (or no stakeholderengagement at all); thus we can make no conclu-sions regarding the relative effectiveness of thisapproach. However, the clinical partners would nothave considered undertaking this broad PRO im-plementation effort if the researchers had not ap-proached it from a collaborative perspective. In thatrespect, we are confident that the adoption ofPROs at the organizational level was greater be-cause of this approach.

Finally, this description of our collective experi-ence as researchers and clinical practice represen-tatives is not research or a formal evaluation of ourprocess, and no qualitative analysis was conducted.Rather, this is a methods and reflection article,coauthored by both the researcher and clinicalstakeholder representatives.

Implications and Future ResearchSince the advent of the EHR, there has been anearly overwhelming increase in the amount ofdata that providers are expected to collect anddocument in structured fields and checkboxes inthe EHR.52,53 When adding PRO measures tothis data collection burden, the value gained by aPRO is weighed in the context of the effortrequired to collect and interpret the data. Addi-tional PRO data collection will eventually be-come untenable for both patients and care pro-viders. Potential solutions include targeting datacollection to those patients for whom the data aremost relevant (thus reducing the total burden toany individual patient); establishing systems thatallow for a larger involvement of the care team inmaking clinically relevant information availableto the clinical provider at the point of care;

adopting the use of patient portals, kiosks, andsmartphone applications to collect informationbefore the visit; and weeding out data collectionthat does not prove to be useful. Therefore, fur-ther work is needed to identify methods andinfrastructure that can be useful for rapid adop-tion of PROs for research purposes.

The authors acknowledge the efforts of the SAFTINet Part-ner Engagement Community and the comparative effective-ness research team, and the work done by all SAFTINetpartner organizations, including providers, staff, and patients.The authors thank Elizabeth Staton, technical writer, for herassistance with editing this article. They also thank the sub-ject matter experts they consulted in the process of selectingthe PRO content area and measurement tool, including Drs.Michael Ho, Stacie Dougherty, Raymond Estacio, and TomMaddox.

References1. Hickam D, Totten A, Berg A, Rader K, Goodman S,

Newhouse R. The PCORI methodology report.Washington, DC: Patient-centered Outcomes Re-search Institute; 2013.

2. Concannon TW, Meissner P, Grunbaum JA, et al. Anew taxonomy for stakeholder engagement in pa-tient-centered outcomes research. J Gen Intern Med2012;27:985–91.

3. Calvert M, Blazeby J, Altman DG, Revicki DA, Mo-her D, Brundage MD. Reporting of patient-reportedoutcomes in randomized trials: the CONSORTPRO extension. JAMA 2013;309:814–22.

4. Reeve BB, Wyrwich KW, Wu AW, et al. ISOQOLrecommends minimum standards for patient-re-ported outcome measures used in patient-centeredoutcomes and comparative effectiveness research.Qual Life Res 2013;22:1889–905.

5. Greenhalgh J. The applications of PROs in clinicalpractice: what are they, do they work, and why? QualLife Res 2009;18:115–23.

6. Greenhalgh J, Meadows K. The effectiveness of theuse of patient-based measures of health in routinepractice in improving the process and outcomes ofpatient care: a literature review. J Eval Clin Pract1999;5:401–16.

7. Valderas JM, Kotzeva A, Espallargues M, et al. Theimpact of measuring patient-reported outcomes inclinical practice: a systematic review of the literature.Qual Life Res 2008;17:179–93.

8. Concannon TW, Meissner P, Grunbaum JA, et al. Anew taxonomy for stakeholder engagement in pa-tient-centered outcomes research. J Gen Intern Med2012;27:985–91.

9. Marshall S, Haywood K, Fitzpatrick R. Impact ofpatient-reported outcome measures on routine prac-tice: a structured review. J Eval Clin Pract 2006;12:559–68.


10. Rose M, Bezjak A. Logistics of collecting patient-reported outcomes (PROs) in clinical practice: anoverview and practical examples. Qual Life Res2009;18:125–36.

11. Donaldson MS. Taking PROs and patient-centeredcare seriously: incremental and disruptive ideas forincorporating PROs in oncology practice. Qual LifeRes 2008;17:1323–30.

12. Fung CH, Hays RD. Prospects and challenges inusing patient-reported outcomes in clinical practice.Qual Life Res 2008;17:1297–302.

13. Lohr KN, Zebrack BJ. Using patient-reported out-comes in clinical practice: challenges and opportuni-ties. Qual Life Res 2009;18:99–107.

14. Concannon T, Fuster M, Saunders T, et al. A sys-tematic review of stakeholder engagement in com-parative effectiveness and patient-centered outcomesresearch. J Gen Intern Med 2014;29;1692–701.

15. Deverka PA, Lavallee DC, Desai PJ, et al. Stake-holder participation in comparative effectiveness re-search: defining a framework for effective engage-ment. J Comp Eff Res 2012;1:181–94.

16. Cella D, Riley W, Stone A, et al. The Patient-Reported Outcomes Measurement Information Sys-tem (PROMIS) developed and tested its first wave ofadult self-reported health outcome item banks:2005–8. J Clin Epidemiol 2010;63:1179–94.

17. U.S. Department of Health and Human ServicesFDA Center for Drug Evaluation and Research;U.S. Department of Health and Human ServicesFDA Center for Biologics Evaluation and Research;U.S. Department of Health and Human ServicesFDA Center for Devices and Radiological Health.Guidance for industry: patient-reported outcomemeasures: use in medical product development tosupport labeling claims: draft guidance. Health QualLife Outcomes 2006;4:79.

18. Snyder CF, Jensen RE, Segal JB, Wu AW. Patient-reported outcomes (PROs): putting the patient per-spective in patient-centered outcomes research. MedCare 2013;51(8 Suppl 3):S73–9.

19. Basch E. New frontiers in patient-reported out-comes: adverse event reporting, comparative effec-tiveness, and quality assessment. Annu Rev Med2014;65:307–17.

20. Boyce MB, Browne JP. Does providing feedback onpatient-reported outcomes to healthcare profession-als result in better outcomes for patients? A system-atic review. Qual Life Res 2013;22:2265–78.

21. Rothwell PM, McDowell Z, Wong CK, Dorman PJ.Doctors and patients don’t agree: cross sectionalstudy of patients’ and doctors’ perceptions and as-sessments of disability in multiple sclerosis. BMJ1997;314:1580–3.

22. Espallargues M, Valderas JM, Alonso J. Provision offeedback on perceived health status to health careprofessionals: a systematic review of its impact. MedCare 2000;38:175–86.

23. Jagsi R, Chiang A, Polite BN, et al. Qualitativeanalysis of practicing oncologists’ attitudes and ex-periences regarding collection of patient-reportedoutcomes. J Oncol Pract 2013;9:e290–7.

24. Varni JW, Burwinkle TM, Lane MM. Health-relatedquality of life measurement in pediatric clinical prac-tice: an appraisal and precept for future research andapplication. Health Qual Life Outcomes 2005;3:34.

25. Gilbody SM, House AO, Sheldon T. Routine ad-ministration of Health Related Quality of Life(HRQoL) and needs assessment instruments to im-prove psychological outcome–a systematic review.Psychol Med 2002;32:1345–56.

26. Balasubramanian BA, Cohen DJ, Clark EC, et al.Practice-level approaches for behavioral counselingand patient health behaviors. Am J Prev Med 2008;35(5 Suppl):S407–13.

27. Pace WD, Staton EW. Electronic data collectionoptions for practice-based research networks. AnnFam Med 2005;3(Suppl 1):S21–9.

28. Fernald DH, Froshaug DB, Dickinson LM, et al.Common measures, better outcomes (COMBO): afield test of brief health behavior measures in pri-mary care. Am J Prev Med 2008;35(5 Suppl):S414 –22.

29. Peters M, Crocker H, Jenkinson C, Doll H, Fitzpat-rick R. The routine collection of patient-reportedoutcome measures (PROMs) for long-term condi-tions in primary care: a cohort survey. BMJ Open2014;4:e003968.

30. Weenink JW, Braspenning J, Wensing M. Patientreported outcome measures (PROMs) in primarycare: an observational pilot study of seven genericinstruments. BMC Fam Pract 2014;15:88.

31. Brundage M, Blazeby J, Revicki D, et al. Patient-reported outcomes in randomized clinical trials: de-velopment of ISOQOL reporting standards. QualLife Res 2013;22:1161–75.

32. Bogart LM, Uyeda K. Community-based participa-tory research: partnering with communities for ef-fective and sustainable behavioral health interven-tions. Health Psychol 2009;28:391–3.

33. Spertus J. Barriers to the use of patient-reportedoutcomes in clinical care. Circ Cardiovasc Qual Out-comes 2014;7:2–4.

34. Hartung DM, Guise JM, Fagnan LJ, Davis MM,Stange KC. Role of practice-based research net-works in comparative effectiveness research. J CompEff Res 2012;1:45–55.

35. Mold JW, Peterson KA. Primary care practice-basedresearch networks: working at the interface betweenresearch and quality improvement. Ann Fam Med2005;3(Suppl 1):S12–20.

36. Maro JC, Platt R, Holmes JH, et al. Design of anational distributed health data network. Ann InternMed 2009;151:341–4.

37. Schilling LM, Kwan BM, Drolshagen CT, et al.Scalable Architecture for Federated Translational


Inquiries Network (SAFTINet) technology infra-structure for a distributed data network. EGEMS(Wash DC) 2013;1:1027.

38. Snyder CF, Aaronson NK, Choucair AK, et al. Im-plementing patient-reported outcomes assessment inclinical practice: a review of the options and consid-erations. Qual Life Res 2012;21:1305–14.

39. Aaronson N, Choucair A, Elliott T, et al. User’sguide to implementing patient-reported outcomesassessment in clinical practice. November 11, 2011.Milwaukee, WI: International Society for Qual-ity of Life Research. Available from: http://www.iso-qol.org/UserFiles/file/UsersGuide.pdf. AccessedNovember, 12, 2014.

40. Baroletti S, Dell’Orfano H. Medication adherencein cardiovascular disease. Circulation 2010;121:1455– 8.

41. Ho PM, Bryson CL, Rumsfeld JS. Medication ad-herence: its importance in cardiovascular outcomes.Circulation 2009;119:3028–35.

42. Moczygemba LR, Goode JV, Gatewood SB, et al.Integration of collaborative medication therapy man-agement in a safety net patient-centered medical home.J Am Pharm Assoc (2003) 2011;51:167–72.

43. Epstein RM, Fiscella K, Lesser CS, Stange KC.Why the nation needs a policy push on patient-centered health care. Health Aff (Millwood) 2010;29:1489 –95.

44. Aaronson N, Alonso J, Burnam A, et al. Assessinghealth status and quality-of-life instruments: attri-butes and review criteria. Qual Life Res 2002;11:193–205.

45. Garfield S, Clifford S, Eliasson L, Barber N, WillsonA. Suitability of measures of self-reported medica-

tion adherence for routine clinical use: a systematicreview. BMC Med Res Methodol 2011;11:149.

46. Gehi A, Ali S, Whooley M. Self-reported medicationadherence and cardiovascular events in patients withstable coronary heart disease: The Heart and SoulStudy. Circulation 2007;115:E584–4.

47. Gellad WF, Grenard JL, Marcum ZA. A systematicreview of barriers to medication adherence in theelderly: looking beyond cost and regimen complex-ity. Am J Geriatr Pharmacother 2011;9:11–23.

48. Studer Q. Making process improvement ‘stick’.Healthc Financ Manage 2014;68:90–4, 96.

49. Patient reported outcomes (PROs) in perform-ance measurement. Washington, DC: NationalQuality Forum; 2013. Available from: https://www.qualityforum.org/Publications/2012/12/Pa-tient-Reported_Outcomes_in_Performance_Mea-surement.aspx. Accessed November 24, 2015.

50. Santana MJ, Feeny D. Framework to assess the ef-fects of using patient-reported outcome measures inchronic care management. Qual Life Res 2014;23:1505–13.

51. Rumsfeld JS, Alexander KP, Goff DC Jr, et al. Car-diovascular health: the importance of measuring pa-tient-reported health status: a scientific statementfrom the American Heart Association. Circulation2013;127:2233–49.

52. Tipirneni R, McMahon LF Jr. Measuring value inprimary care: enhancing quality or checking the box?Health Serv Res 2014;49:1724–8.

53. Poissant L, Pereira J, Tamblyn R, Kawasumi Y. Theimpact of electronic health records on time effi-ciency of physicians and nurses: a systematic review.J Am Med Inform Assoc 2005;12:505–16.


https://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx




Appendix 1

doi: 10.3122/jabfm.2016.01.150141 Stakeholders Implement a PRO: From the SAFTINet PBRN E1

E2 JABFM January–February 2016 Vol. 29 No. 1 http://www.jabfm.org

doi: 10.3122/jabfm.2016.01.150141 Stakeholders Implement a PRO: From the SAFTINet PBRN E3

E4 JABFM January–February 2016 Vol. 29 No. 1 http://www.jabfm.org

stakeholder engagement in a patient-reported outcomes · pdf filepurposes.37 at the time of...

Documents