evAluATinG innovATion in PRAcTice:

28 / Canadian Government Executive // December 2017

evaluation

The Backdrop: Advancing Health Care Throughout Canada, health care systems are challenged by population growth, ag-ing, and physician retirement and relo-cation. These, among other factors, have added demands to and limited the capac-ity of the health care system. Such challenges require a combina-

tion of initiatives, including community-driven efforts to first understand and then respond to local and regional needs. In British Columbia, theMinistry ofHealthandDoctorsofBCworkedinpartnershipthrough the General Practice Services Committee (GPSC), to foster community action of this sort, through a three-year initiativenamedAGPforMe. Launchedin2013,AGPforMehadthreegoals:• Confirmandstrengthenthecontinuous

doctor and patient relationship, includ-

AGPforMeing better support for the needs of vul-nerable patients.

• Increase the capacity of the primaryhealth care system.

• Enablepatientswhowantafamilydoc-tor to find one.

FundingforAGPforMewasprovidedintwo ways: the creation of incentive fees to facilitate practice-level change with physicians; and investments in projects delivered by more than thirty Divisionsof Family Practice, groups of family physi-cians in over 230 communities across Brit-ish Columbia.Attheheartofit,AGPforMewastruly

a community-driven, broad undertaking to foster innovation, quality and continuityin patient access and care.

The Task: Undertaking a Provincial Evaluation of A GP for Me Evaluatingan initiative like thatofAGPforMeiscomplicated.Afewofthetwistsand turns of this endeavour centred on the following.

Relating Results: AGPforMewasoneof many efforts in British Columbia to fur-ther health care system transformation. Additionally, divisions of family practice had been formed around the province, providing an alternative way for fam-ily physicians to collaborate with health authorities and other partners to identify and address local health care challenges. OverthecourseofAGPforMe,about150projects were started by divisions, each furthering its own form of innovation. This made the direct attribution and sum-maryofresultstoAGPforMechalleng-ing.

harnessing efforts: The purpose of the provincial evaluation was to not only gain an understanding of the effectiveness of A GP forMe from theexperiencesof thoseinvolved, but also to share lessons learned from innovations that sought to enhance thequalityofcare.

In addition to the provincial evaluation, divisions were also completing assess-

Bill Reid

JAsminA fATic

PeTRA lolic

ments of the effectiveness of supported projects, to inform their own learning and innovation. The provincial evaluation had to harness these local efforts to allow for an integration of findings, while also not obstructingtheuniqueworkthatwaspro-ceeding.

dealing with unknowns: In 2015, a substantial Provincial Evaluation Planwas prepared for A GP forMewith theinvolvement of staff from theDoctors ofBC and the Ministry of Health, a GPSCEvaluation Working Group, contractedspecialists, a Divisions Reference Group,and evaluators hired by various divisions.

Following the completion of this Pro-vincial Evaluation Plan, and by the timethe provincial evaluation commenced, it became apparent that the awareness of stated outcomes and indicators of perfor-manceforAGPforMehadsincedimin-ished such that partners were challeng-ing concepts, describing intentions and using measures in a variety of disparate ways. This complicated the ability to draw

December 2017 // Canadian Government Executive / 29

Allowances had to be made for adaptations to the Provincial Evaluation Plan to address gained insight, respond to any gaps in information, and ensure key perspectives were accounted for.

comparisons and roll-up findings across the province. In addition, as the provin-cial evaluation progressed, it became clear that baseline data was not always available for some of the indicators and planned data sources (e.g. a Core Pro-vincial Patient Survey) would not be ob-tained in time.

The Answer: A Collaborative and Adaptive Approach to the EvaluationThe success of the provincial evaluation ofAGPforMewasfoundedonthefollow-ing key principles that guided the work of what was a hybrid evaluation team, made upofstafffromtheDoctorsofBCasthecontractingauthorityandMNPLLPasthecontracted evaluator.

collaboration: Members of the Doc-torsofBCteamandtheMNPteamworkedclosely together, such that the provincial evaluation was a truly collaborative un-dertaking. This extended beyond fre-quent, jointcheck-inmeetingstoincludethe co-facilitation of consultations as well as the assigning of work across the com-bined teams. It was also critical to coordi-nate with the work of other evaluators, be it the ones engaged by divisions or those working on components of the Provincial EvaluationPlan.

integration: The provincial evaluation had to integrate the foundational work of the Divisions of Family Practice and

that of several other external evaluators, as well as administrative data from the MinistryofHealth.Withthedivisions,atiered, thematic analysis was carried out of thirty “Final Evaluation and ProjectImplementation Close-Out Reports” inthe form of a high-level summary, divi-sion highlights, and direct report extracts.

In addition to these close-out reports and what was referenced above, there were the results of surveys (e.g. of family physicians, with the GPSC, and the Cana-dian Community Health Survey), inter-views with health authorities and the BC Ministry ofHealth, and facilitated focusgroup sessions.

Adaptation: Allowances had to be made for adaptations to the Provincial Evaluation Plan to address gained in-sight, respond to any gaps in informa-tion, and ensure key perspectives were accounted for.

It was also critical to follow an iterative approach to validating early findings and interpretations. This took the form of a staged confirmation of the findings with a variety of stakeholders, covering the vari-ous components of the analyses (e.g., in-terpretations of the administrative data).

The Takeaways: Reflecting on the Experiences of the EvaluationJust asAGP forMehas illustrated thatthere is not one solution to address the complex challenges of BC’s health care system, the provincial evaluation high-lighted opportunities for future practice.

There are clear benefits to be gained from adopting hybrid evaluation teams, bringing together experienced staff and third-party contractors. Yet, it is impor-tant to also keep in mind the mainte-nance of harmony and balance in teams, be it with respect to size, well-defined and agreed upon roles, and contributed areas expertise.

Having a well-thought-out plan, or framework, for an evaluation is another factor of success. The value of this plan-ning though, is nicely summed up by Vol-taire, “don’t let the perfect be the enemy

of the good.” Ongoing socialization of an evaluation plan and allowing for moments for adaptation along the way should be embraced.

With initiatives that serve to foster inno-vation, especially across geographic and other communities of interest, there can be a concern of imposing structure, which can come in the form of evaluation. Just as innovation depends on some arrange-ment for implementation, evaluators need to consider how findings will be combined in a coherent way.

Finally, the pursuit of an all-inclusive, retrospective evaluation may be better replaced with an iterative strategy that al-lows for the staging of evaluative studies over the course of a major long-term ini-tiativelikeAGPforMe. Inthisway,theevaluation approach will naturally evolve with the initiative, and the latter stages of evaluation will yield findings that are in-formed by refined priorities, strategic di-rection and focus. During thefinal stages of theAGP for

Meevaluation,theGPSCwaslookingfor-ward and updating its strategic direction to include the patient medical home as the foundation of an integrated system of care. The provincial evaluation informed the evolution of the primary health care system in British Columbia, by providing important insights into local health care challenges; raising awareness of commu-nity-driven innovations that build on the important work of physicians, divisions of family practice, and health authorities; and furthering an environment for con-versation on how to continue to improve patient access and care.

Bill reid, Partner, Advisory Services, MNP and Past President, BC and Yukon Chapter, Canadian Evaluation Society, bill.reid@mnp.caJAsminA fAtic, Manager, Advisory Ser-vices, MNP, and Past Vice President, BC and Yukon Chapter, Canadian Evalua-tion Society, jasmina.fatic@mnp.caPetrA lolic, Executive Coordinator, GPSC, Doctors of BC, plolic@doctor-sofbc.ca

evaluation