implementation of electronic medical records and ... · implementation of electronic medical...
TRANSCRIPT
Implementation of Electronic Medical Records and Preventive Services: a Mixed Methods Study
by
Michelle Greiver
A thesis submitted in conformity with the requirements for the degree of Master of Science
Health Policy and Management Evaluation University of Toronto
© Copyright by Michelle Greiver 2011
ii
Implementation of electronic medical records and
preventive services: a mixed methods study
Michelle Greiver
Master of Science Degree
Health Policy and Management Evaluation University of Toronto
2011
Abstract
The implementation of Electronic Medical Records (EMRs) may lead to improved quality of
primary health care. To investigate this, we conducted a mixed methods study of eighteen
Toronto family physicians who implemented EMRs in 2006 and nine comparison family
physicians who continued to use paper records. We used a controlled before-after design and
two focus groups. We examined five preventive services with Pay for Performance incentives:
Pap smears, screening mammograms, fecal occult blood testing, influenza vaccinations and
childhood vaccinations.
There was no difference between the two groups: after adjustment, combined preventive
services for the EMR group increased by 0.7% less than for the non-EMR group (p=0.55, 95%
CI -2.8, 3.9). Physicians felt that EMR implementation was challenging.
iii
Acknowledgments
I would like to thank my thesis supervisor, Dr Jan Barnsley, and my thesis committee, Drs Rick
Glazier, Rahim Moineddin and Bart Harvey, for their support and mentorship during the
planning, conduct and analysis of this study. I am also very grateful to my colleagues in primary
care who generously provided access to their charts (both paper and electronic) for the study.
iv
Table of Contents Acknowledgments.......................................................................................................................... iii
Table of Contents ........................................................................................................................... iv
List of Tables ............................................................................................................................... viii
List of Figures ..................................................................................................................................x
List of Appendices ........................................................................................................................ xii
Chapter 1 : Background ..................................................................................................................1
Health care quality and performance ..........................................................................................1
Electronic Medical Records in the context of system-wide computerization .............................2
Primary Care Reform in Ontario .................................................................................................5
Pay for Performance ....................................................................................................................7
Research questions ....................................................................................................................11
Chapter 2 : Theoretical frameworks applicable to the implementation of Electronic Medical Records ......................................................................................................................................13
Background ...............................................................................................................................13
Innovation Attributes ................................................................................................................14
Relative advantage .............................................................................................................14
Compatibility .....................................................................................................................15
Complexity .........................................................................................................................16
Observability ......................................................................................................................17
Reinvention ........................................................................................................................17
Process of implementation ........................................................................................................18
Individual characteristics and interactions between implementers ...........................................19
Organizational Attributes ..........................................................................................................20
Summary ...................................................................................................................................21
Chapter 3 : Methods ......................................................................................................................23
v
Participants ................................................................................................................................23
EMR cohort ........................................................................................................................23
Non EMR cohort ................................................................................................................24
Determination of physician and practice characteristics ...........................................................29
Study Design .............................................................................................................................31
Primary and Secondary Outcomes .....................................................................................31
Variable Measurement .......................................................................................................33
Sample size calculation ......................................................................................................37
Quantitative Analysis .........................................................................................................38
Qualitative design ..............................................................................................................39
Qualitative Analysis ...........................................................................................................40
Chapter 4 : Results, Quantitative Analysis ...................................................................................41
Characteristics of the study physicians .....................................................................................41
Comparison between EMR and non-EMR cohorts ...................................................................44
Service provision in EMR cohort ..............................................................................................51
Comparison of service provision in EMR cohort by level of EMR use ...................................55
Comparison of the two FHNs ...................................................................................................61
Chapter 5 Results, Qualitative Analysis ........................................................................................68
Barriers ......................................................................................................................................68
Lack of compatibility and high complexity .......................................................................69
Technological barriers: lack of interoperability, lack of technical support and infrastructure failures .............................................................................................70
Lack of on-going training and education ...........................................................................72
Facilitators and benefits ............................................................................................................73
Availability of an EMR Champion ....................................................................................73
Increased efficiency for some practice processes ..............................................................74
vi
Perception of time for eventual pay-back of the investment .............................................74
Physicians’ perceptions of patient reaction ........................................................................75
Perceived improvements in quality ....................................................................................76
Chapter 6 : Integrating theory, qualitative and quantitative results ..............................................78
Chapter 7 : Discussion and Conclusions .......................................................................................82
Discussion .................................................................................................................................82
Limitations ................................................................................................................................88
Policy suggestions .....................................................................................................................91
Improving practice IT infrastructure ..................................................................................92
Supporting EMR implementation ......................................................................................92
Promoting the measurement and improvement of outcomes .............................................93
Practice suggestions ..................................................................................................................93
Research suggestions ................................................................................................................94
Conclusions ...............................................................................................................................96
References ......................................................................................................................................97
Appendices ...................................................................................................................................109
Appendix A ..................................................................................................................................109
Case study of a Quality Improvement project using EMR .....................................................109
Organization of preventive care in FHN2 ........................................................................109
Description of processes ..................................................................................................110
Appendix B ..................................................................................................................................114
Literature search strategy ........................................................................................................114
Appendix C ..................................................................................................................................116
Letter of invitation and survey of AHC physicians ................................................................116
Appendix D ..................................................................................................................................118
Baseline survey of participants ...............................................................................................118
vii
Appendix E ..................................................................................................................................121
Data recording form ................................................................................................................121
Appendix F...................................................................................................................................123
Example of electronic audit: Pap smears ...............................................................................123
Appendix G ..................................................................................................................................124
Inclusion and exclusion criteria for administrative cohorts ....................................................124
Pap smears .......................................................................................................................124
Mammograms ..................................................................................................................124
Influenza vaccinations .....................................................................................................125
Fecal occult blood testing ................................................................................................125
Appendix H ..................................................................................................................................127
Focus Group Questions ...........................................................................................................127
Appendix I ...................................................................................................................................128
Characteristics of physicians and patients in FHNs and FHGs in Ontario .............................128
Appendix J ...................................................................................................................................130
Research Ethics Board Approval ............................................................................................130
Appendix K ..................................................................................................................................131
Physician Focus Group Consent Form ....................................................................................131
Copyright Acknowledgements.....................................................................................................132
viii
List of Tables Table 1: Timeline for changes offered to primary care physicians ................................................ 9
Table 2: Service provision before and after P4P .......................................................................... 10
Table 3: Physicians included and excluded in non-EMR cohort (N=23) .................................... 26
Table 4: Changes over time for the two cohorts and planned comparisons within and between
groups ............................................................................................................................................ 34
Table 5: Self reported characteristics of physicians in EMR and non EMR cohorts .................. 41
Table 6: Physician and practice characteristics in EMR and non EMR cohorts, derived from
administrative databases ............................................................................................................... 42
Table 7: Service provision in EMR and non-EMR cohorts ......................................................... 45
Table 8: Comparison of changes in overall service provision between EMR and non-EMR
cohorts ........................................................................................................................................... 46
Table 9: Comparison of overall composite process score in EMR cohort (excluding FOBT) by
year ................................................................................................................................................ 53
Table 10: Comparison of chart audits and administrative data for EMR cohort, composite score
for mammography, Pap smears and influenza vaccinations ......................................................... 55
Table 11: Self reported characteristics of physicians by usage of EMR ..................................... 57
Table 12: Physician and practice characteristics by usage of EMR, derived from administrative
databases ....................................................................................................................................... 58
Table 13: Comparison of changes in overall service provision (excluding FOBT) .................... 61
Table 14: Self reported characteristics of physicians by FHN .................................................... 62
Table 15: Physician and practice characteristics by FHN, derived from administrative databases
....................................................................................................................................................... 63
ix
Table 16: Differences in overall service provision between FHNs (FHN2 - FHN1), excluding
FOBT ............................................................................................................................................ 66
Table 17: Number of patients being reminded by letter about overdue service .......................... 66
Table 18: Characteristics of physicians in FHNs (capitation) and FHGs (enhanced fee-for-
service) in Ontario ....................................................................................................................... 128
Table 19: Characteristics of patients in FHNs and FHGs in Ontario ........................................ 129
x
List of Figures
Figure 1: Theoretical factors affecting implementation of an innovation ................................... 22
Figure 2: Flow diagram detailing the recruitment of the non-EMR cohort ................................. 28
Figure 3: Composite process score for EMR and Non EMR cohorts, using administrative data
and chart audit data ....................................................................................................................... 47
Figure 4: Influenza vaccination for EMR and Non EMR cohorts, using administrative data and
chart audit data .............................................................................................................................. 48
Figure 5: FOBT for EMR and Non EMR cohorts, using both administrative data and chart audit
data ................................................................................................................................................ 49
Figure 6: Mammography for EMR and Non EMR cohorts, using both administrative data and
chart audit data .............................................................................................................................. 50
Figure 7: Pap smears for EMR and Non EMR cohorts, using both administrative data and chart
audit data ....................................................................................................................................... 51
Figure 8: Individual preventive services for EMR cohort, using chart audit data ....................... 52
Figure 9: Service provision in EMR cohort, derived from administrative data ........................... 54
Figure 10: EMR usage 18 months after EMR installed ............................................................... 56
Figure 11: Overall service provision by EMR usage (excluding FOBT) .................................... 60
Figure 12: Service provision by FHN (excluding FOBT) ........................................................... 65
Figure 13: Documented service provision by FHN, with investigator removed ......................... 67
Figure 14: Field for adding the presence of a preventive test .................................................... 111
Figure 15: List of patients used to generate reminder letters ..................................................... 112
xi
xii
List of Appendices
Appendix A: Case study of a Quality Improvement project using EMR…………….…. 109
Appendix B: Literature search strategy…………………………………………………….114
Appendix C: Letter of invitation and survey of AHC physicians…………………………. 116
Appendix D: Baseline survey of participants……………………………...…………….... 118
Appendix E: Data recording form...………………………………………………………..121
Appendix F: Example of electronic audit: Pap smears……………...…...…….………......123
Appendix G: Inclusion and exclusion criteria for administrative cohorts………………….124
Appendix H: Focus Group Questions………………………………………………………127
Appendix I: Characteristics of physicians and patients in FHNs and FHGs in Ontario........128
Appendix J: Research Ethics Board Approval…………………….………....….…….……130
Appendix K: Physician Focus Group Consent Form…………….…………..……………..131
1
Chapter 1 : Background
“Paper records are increasingly becoming obsolete and inadequate.
They limit the flow of information, insufficiently document patient
care, impede the integration of health care delivery, create barriers
to research, and limit the information available for administration
and decision making.”
Roy Romanow, 20021
The quality of care provided to patients matters.2 "If you can not measure it, you can not
improve it" (Lord Kelvin); the ability to repeatedly measure and monitor the care that is provided
is a prerequisite for quality improvement efforts.3 Such an undertaking can be laborious and time
consuming if data are recorded on paper. Computerized data entry, as part of an Electronic
Medical Record (EMR), may overcome some of these difficulties by automating data collection
procedures. EMRs have been specifically identified as critical to quality improvement, and are
targeted for funding under the American Recovery and Reinvestment Act of 2009.4 Various
policies favouring the establishment of EMRs have been implemented in most Canadian
provinces. However, despite much hope for improvement, the effect of the implementation of
EMRs on the quality of care provided to patients is currently unclear. In Ontario, family practice
has been exposed to several new policies affecting funding since the new millennium: new
payment mechanisms (primary care reform), Pay-for-Performance incentives and funding for
EMRs have been introduced in rapid sequence within the same practices. This study followed a
group of physicians as they responded to these new incentives and began the implementation of
an EMR system.
Health care quality and performance
There are serious gaps in the quality of health care;5 barely half of recommended services are
provided to adults.6 These problems have led to calls for systematic efforts to reform health care
delivery.2 As a result, various policy measures to address the gaps in quality have recently been
adopted.
2
The implementation of policies, especially in complex and chaotic systems such as health care is
rarely straightforward;5 all significant changes can produce unintended consequences.7 An
examination of the effects and consequences of policies can benefit health care policy-makers by
adding evidence that can be used through policy analysis to inform the planning of future
programs. As well, this evidence can be used by those impacted, to improve the uptake of
potentially useful aspects of a policy, to manage implementation of new programs, and to
mitigate potentially harmful aspects of the change: “planning for the future requires knowing
where you have been”.8 (p 129)
This thesis examined the implementation of an innovation (Electronic Medical Records in
primary care practices) within the context of two other simultaneous changes in health care
delivery and payment in Ontario—primary care reform and Pay for Performance. We measured
the association between EMR implementation and the provision of preventive services covered
by Ontario’s Pay for Performance program, and explored how physicians experienced EMR
implementation.
Electronic Medical Records in the context of system-wide computerization Information technology (IT) is pervasive in our society; computers, the Internet and IT-enabled
communication tools such as smart phones are used on a daily basis by millions of people. It
seems intuitive that IT usage should be routine in health care, since there is so much information
and communication being managed in this setting.
However, the Ontario health care system is currently still largely paper-based. Patient records
are kept in paper charts by various providers and in various locations. Information is exchanged
by fax, by mail or through the phone; in some cases, reports are simply carried by patients.
Despite the fact that health care is so critically dependent on accurate information, patients have
no guarantee that their data will follow them: medical errors, service duplication and unnecessary
waits due to missing or incomplete information are common.2, 5, 9 Coordination of care amongst
different providers is poor.10, 11 Systematic collection of data for quality improvement or
research is limited due to the difficulties inherent in manual chart reviews.1, 3
3
Electronic Records have been proposed as a solution to these issues.10, 12 They are searchable,
they may improve patient care,13 and they have the potential to improve the efficiency of our
health care system.14, 15 Quality improvement projects become possible, as on-going
performance data can be obtained from the electronic databases in practices.16 System
integration (patient data being shared across different electronic databases, electronic
prescribing) is feasible once data are kept in an electronic form.17, 18
The Kirby and Romanow reports have both recommended the establishment of electronic health
records for all Canadians.1, 19 The First Ministers committed to accelerating the implementation
of these electronic records as part of their 2003 Accord on a 10 year plan to transform health
care.20
There is some evidence that the introduction of computerized systems has impacted care: a large
US Veteran’s Administration study found significant improvements in quality due to system re-
engineering, including integrated Electronic Medical Records (although the specific effect of
EMRs was not directly studied).21 A recent systematic review found that computerized decision
support systems improved practitioner performance, especially with regards to immunizations.13
However, most of the software systems reviewed were stand-alone programs, and not
commercial off the shelf EMRs similar to those commonly found in primary care practices.
Several studies have also found that EMRs may not improve care,22-26 or may even be a source of
errors: a study of diabetes care in family practices found that EMR-based practices had poorer
quality than paper-based practices.27 The introduction of computerized records led to medication
errors and worsened hospital-based mortality.28, 29 Many of the studies on the introduction of
EMRs have been descriptive, with very few evaluating outcomes.30 A recent systematic review
noted that much of the evidence on the effectiveness of EMRs came from only four benchmark
research institutions, with limited generalizability, and very little data coming from standard
commercial systems.31 At the present time, the effect of the large scale implementation of
commercial off-the-shelf EMRs on quality of care in small, community-based family practices is
not known.
What is known is that very few providers in North America have currently implemented
Electronic Medical Records.32-35 A recent survey found that only 4% of US physicians
practicing in ambulatory care used a fully functional electronic system, and 13% had a basic
4
system.34 Only 12.3% of Canadian family physicians use electronic records instead of paper
records.35 The US and Canada were ranked last out of 10 countries in terms of practice
computerization.33
The most significant barrier to acquiring EMR has been identified as the cost of these systems: 36-38 providers pay for the electronic records in their practices, while most of the benefits accrue
to patients and to the health care system.39, 40 This barrier results in slow adoption at the practice
level. As one physician said, “None of the many beneficiaries of our investment—patients,
insurance companies, our specialist colleagues, health plans, our liability carrier—have directly
shared in the cost of implementing an electronic health record system”.41(pg 225) Countries that
fund EMRs for physicians (such as the UK or Denmark) have very high levels of adoption, while
Canada and the US lag far behind.33
Clearly, subsidies are needed in order for physicians to adopt these systems.42 In order to further
EMR adoption, the Ontario government decided to offer financial support for physician office
computerization. In October 2001, the ePhysician Project, jointly managed by the Ontario
Medical Association and the Ministry of Health and Long Term Care, was created to oversee the
establishment of this project.43 This eventually led to the formation of OntarioMD, a corporation
owned by the Ontario Medical Association.
In 2003, the government transferred $150 million (the Primary Care Information Technology
Fund) to OntarioMD.44 OntarioMD was given responsibility for certifying EMR software
applications, thus ensuring that they included necessary components and met standards, and for
administering the physician subsidy. In order to receive funding, physicians had to buy their
EMR software from the list of vendors certified and approved by OntarioMD; the subsidy
amounted to $28,600 per physician, estimated as being about 70% of the cost of adoption. The
funding consisted of a $4,500 readiness grant paid once the contract with the vendor was signed,
followed by $600 per month for three years, starting once the practice used the new software for
scheduling and billing, and a further subsidy of $2,500 when the physician declared that he or
she had completed data entry on 600 patients or 2/3rd of the practice (whichever was less) in the
electronic patient summary page.44
The stage was now set for a significant number of Ontario family physicians to adopt EMRs.
However Ontario’s EMR strategy has been intimately tied to primary care reform: only
5
physicians participating in primary care reform projects, such as Family Health Networks45
(described below) were eligible for funding. Therefore, the launch of EMRs in Ontario primary
care occurred concurrently and was tied closely to system-wide changes in the payment and
organization of primary care.
Primary Care Reform in Ontario
Much of Ontario’s health care is provided within primary care; each day, about 137,000 visits
are made to family physicians.46 In 2004, there were 10,287 general practitioners/family
physicians in Ontario;47 about 70% were in solo practice.47
Family medicine had been in trouble during the 90’s: it was increasingly viewed as the “poor
cousin” to (better paid) specialties, and fewer medical students were choosing to enter this field,
in part due to high debt burdens after medical school. The number of family physicians
accepting new patients was rapidly declining. Hospitals were closed and the number of inpatient
beds reduced, with no increase in community-based medical services. Patients had to be seen in
overcrowded emergency rooms, due to the lack of available family physicians. Thus, out of a
sense of crisis, long awaited and discussed primary care reform projects rose to the top of the
political agenda.48
In 2001, the Ontario Medical Association and the Ontario government offered family physicians
the opportunity to take part in primary care reform by forming new groups, called “Family
Health Networks”, or FHNs.49 To form a FHN, physicians signed a contract with a group of
colleagues (a minimum of five physicians), and agreed to change their method of payment. The
largest proportion of physician payment was now derived from capitation (a set fee per enrolled
patient, per year, based on the patient’s age and sex), with a smaller component of incentives
from pay for performance and fee-for-service funding. Patients rostered with a family physician
by formally enrolling. This allowed the identification of a practice population; the capitated
payment was based on a physician’s roster. Physicians agreed to provide some after-hours
services, and to share call. The original FHN contracts included a clause specifying that IT
support would start once this program was ready for implementation. The $150 million Primary
Care IT fund was specifically tied to, and targeted to support, the new networks; it did not
include primary care physicians choosing not to participate in networks, nor specialists.
6
FHNs were notified in January 2005 that the IT program and subsidy were ready to be
implemented, and that the process of choosing a certified vendor could begin. By the end of the
program (August 31st 2008) 2,700 physicians had received approval for the subsidy.50 This
represents most of the eligible physicians—about 1,900 physicians participating in FHNs and
other eligible primary care reform projects (from a total of about 2,000 eligible physicians), and
800 other physicians, randomly chosen from a pool of 2,400 non-eligible physicians who applied
for funding through an “EMR lottery”.
In 2003, primary care reform was expanded, with family physicians being offered the
opportunity to join Family Health Groups, or FHGs. These groups rostered their patients and
received enhanced fee-for-service payments: they were paid an additional 10% for providing
selected services to their rostered patients.51 The majority of fee for service physicians joined
these FHGs, but they were not offered subsidies for EMRs. Instead, these physicians could
apply for the subsidy through the “EMR lottery” mentioned above, knowing they would be
randomly selected for funding.
There is no doubt that subsidies led to the adoption of EMRs as the majority of physicians
eligible for a subsidy actually bought a system; subsidies therefore appear to be an important part
of health system computerization. However, EMRs are complex and difficult to learn; simply
buying an EMR does not necessarily mean that it will be implemented.52 The subsidy likely
captured some physicians who were not ready to change; they purchased an EMR because they
were part of a FHN—all physicians in a group had to agree to purchase a common system.
Surveys had shown that partial implementation (hybrid paper and electronic records) was more
common than full computerization.34, 35 Financial subsidies in Ontario targeted adoption and
had led to widespread purchases of EMRs (a necessary first step); however, only a small and
highly targeted amount of this funding was dedicated to rewarding actual use (implementation)
of the EMR ($2,500 for completing 600 patient summaries).
There are very few empirical studies that specifically address implementation of innovations in
health care settings, and even fewer addressing EMR implementation.53 Evidence on EMR
implementation using commercial off the shelf (as opposed to home-grown) software in small
community-based practices is very sparse.31 There is literature on business IT implementation:
for example, a large study of complex software systems in business organizations found that
7
31% of all implementations fail, at a median cost of US$2.3 million; the majority of those that
did not fail had time and cost overruns.54 Studies on implementation will be reviewed further in
Chapter 2.
The implementation of EMRs may be influenced by other factors and incentives. In the next part
of this chapter, I review an important set of financial incentives occurring concurrently with, and
possibly affecting, EMR implementation: the Pay for Performance movement.
Pay for Performance
EMR systems are complex, and their implementation involves changes in many possible
processes; only some changes will be implemented in the early stages of computerization. We
chose to focus on preventive services targeted by Pay for Performance (P4P) bonuses as markers
of progress during the early stages of EMR implementation because of the financial rewards
attached to meeting practice targets for these services, as well as for the following additional
reasons.
1. Physician agreement that the preventive services targeted by P4P represent good care.
Physicians want to improve the quality of services that they deliver to their patients; providing
good care is an integral aspect of their personal and professional values.5, 37 P4P, if linked to
measures that are recognized as benefiting patients, can be well aligned with physician values.
Preventive care can reduce the incidence and severity of some of the leading causes of mortality
and morbidity in North America.55, 56 The preventive services used in this study are core
components of primary care,57 are well accepted as indicators of quality of care,58 and have been
used as measures of the quality of services provided in primary care.21, 59 Services that are
perceived as having high value may be more likely to be implemented.60 In our previous study
on Pay for Performance, physicians indicated that they would like to implement more effective
processes for the preventive services affected by P4P,61 indicating readiness to proceed from
contemplating to preparing for change.62
2. Availability of EMR tools targeting P4P
In order to be eligible for funding in Ontario, an EMR system must include the ability to measure
and manage the provision of preventive services; all physicians implementing funded EMRs
8
therefore have access to electronic tools targeting P4P services. These tools are integrated with
the EMR system, and have been validated through conformance testing, as part of OntarioMD’s
funding approval process. Therefore, physicians now have access to new tools to manage
preventive services that they regard as important and that are targeted by P4P.
The P4P movement is a recent addition to the culture of quality.63, 64 Older payment systems are
often viewed as part of the problem; fee-for-service rewards quantity of services, but not quality.
P4P has been proposed as a method of re-aligning payment towards quality.5, 64 In the UK, P4P
incentives comprising about 30% of primary care payment have recently been implemented, and
have led to very high levels of target achievement in the first year of implementation.59 P4P is
increasingly used in the United States as well, with over 50% of Health Maintenance
Organizations having implemented this method of payment.65 Congress has mandated the
introduction of P4P for Medicare and Medicaid.63
However, there is much debate on the effect of these incentives: studies have found that P4P has
a weak and inconsistent effect on quality of care;66-68 the greatest gains occurred where baseline
quality was the poorest, but most of the funding accrued to providers who were already
providing high quality care.66 Thus, the incremental effect of P4P was limited, in large part due
to the difficulties and costs inherent in the implementation of programs designed to increase the
services targeted by P4P.69 Small community-based practices have limited funds, knowledge
and resources devoted to measuring and improving quality.3 Additional problems with P4P can
include unintended consequences, such as increasing health care disparities, discouraging care
coordination, limiting the number of “challenging” patients accepted into practices64 and
decreasing the provision of unfunded services,70 as well as the challenges of choosing
appropriate measures: some services may be difficult to measure.64 P4P funding cannot be
allocated to all qualifying services (leaving some patient groups at a disadvantage).70
In Ontario, the provincial government implemented P4P incentives in 2001 for selected
preventive services, as part of the primary care reform “package”. Participating family
physicians received payments based upon the percentage of their rostered patients meeting
targets for five preventive services: Pap smears within 30 months for women age 35 to 69,
screening mammograms within 30 months for women age 50 to 69, influenza vaccines in the
previous Fall for patients age 65 and over, and primary immunizations for children under two.71
9
Fecal occult blood (FOB) screening for eligible patients age 50 to 75 was added in 2006.72
These measures were recommended by both the US Preventive Services Task Force and the
Canadian Task Forces on Preventive Health Care, with either an A (good evidence for inclusion)
or B (fair evidence for inclusion) level recommendation,55, 56 and were widely accepted within
the profession as representing good quality preventive care. The maximal amount of incentive
was $2,200 per service, representing a total of $11,000 annually per physician if all five targets
are reached. Physicians also received a contact payment ($6.86) for reminding patients who
were overdue.
The incentives were initially offered only to FHN physicians; contract negotiations between the
Ontario Medical Association and the Ministry of Health opened Pay for Performance incentives
to physicians in FHGs, starting in 2007.72 A timeline for the changes occurring in primary care is
shown in Table 1. The fact that a new model was offered does not mean that it was immediately
adopted or implemented by physicians. In Ontario, uptake of FHNs was initially very slow,
while the FHG model proved more popular as it involved a smaller degree of change in
physician payment mechanisms.
Table 1: Timeline for changes offered to primary care physicians
2001 2001 2003 2005 2006 2007
Family Health Networks (capitated payment) offered to physicians
P4P for four preventive services available to physicians participating in FHNs
Family Health Groups (reformed fee for service payment) offered to physicians
EMR subsidy available to physicians participating in FHNs
P4P for FOB screening available for physicians participating in FHNs
P4P for all five services available to physicians participating in FHGs (April 2007)
To determine the effect of these P4P incentives in our local setting, we recently conducted an
observational before-after study of 18 community-based family physicians in sub-urban Toronto,
Ontario.61 These physicians formed two FHNs of 9 members each at the end of 2004 and
became eligible for the incentives at that time. We sampled charts from lists of rostered patients
using a random number table, for the year before (2004) and the year after (2005) the
introduction of incentives. We recorded whether a service was provided within the time frame
10
recommended for each incentive, as documented on the chart; a note that a service had been
done by another provider was also acceptable. Only patients who were registered at that practice
for at least two years were included. We audited 50 charts per service before and after
incentives, except for children’s vaccinations; due to the low numbers of eligible children per
physician, all charts for eligible children were audited. We compared the provision of each
service before and after incentives using chi-square statistics. We reported rate ratios (RR); due
to the high baseline provision of services, the RR may be inflated and was corrected using
poisson regression.73
None of the physicians were using EMRs at the time of that study. Mirroring studies in the
US,66, 67 we found that the incentives were associated with small increases in the provision of
services (Table 2).61
Table 2: Service provision before and after P4P
Service % serviced before incentives (2004)
% serviced after incentives (2005)
% difference Rate Ratio, 95% CI (p value)
Influenza vaccination
76.3 83.3 +7 1.10, 1.06 – 1.15 (<0.0001)
Mammograms 81.9 85.4 +3.5 1.05, 1.01 – 1.10 (0.025)
Pap smears 84.4 86.1 +1.8 1.02, 0.98 – 1.02 (0.28, NS)
Children’s vaccinations
93.2 95.8 +3.4 1.04, 0.99 – 1.09 (0.17, NS)
The physicians had high levels of service provision prior to incentives; other studies have
recorded much lower baseline rates of service provision.74-77 Ceiling effects were almost
certainly present: some patients will refuse services, or a service may not be needed for other
reasons (for example, for a terminally ill patient), so reaching 100% provision is not appropriate.
High baseline provision of services has been found to lead to less increase after P4P.66 All of the
11
physicians within the two FHNs purchased an EMR system by March 2006, thus providing a
natural group to study the effect of EMR implementation after the introduction of P4P.
In summary, P4P in Ontario has focused attention on certain preventive services. However,
physicians using paper-based records may not have the tools to implement the processes needed
to improve service provision, limiting the effect of P4P. These tools include routine audits and
feedback, point of care reminders, and recalls of overdue patients.3, 40, 41, 78-80 Ontario’s EMR
systems must include these tools in order to be certified and be eligible for a subsidy. We
therefore reasoned that physicians implementing certified EMRs would have access to new
automated tools that could help them increase their provision of preventive services. Our
research questions address the degree of change in these preventive services associated with
EMR implementation.
The research that forms this thesis followed the cohort of 18 family physicians mentioned above,
as they implemented EMRs in their practices. We examined the change in the provision of
preventive services with P4P incentives for the two years prior to EMR, and for the first two
years of implementation. To provide a temporal comparison, we also studied a contemporaneous
cohort of nine physicians who were not using EMRs. To frame and explain our quantitative
results, we explored possible reasons underlying changes related to EMR implementation in this
context through qualitative methods. We used focus groups to explore the perceptions of study
physicians using EMR about the implementation of their new electronic systems. In other
words, while quantitative results provide information about what happened, qualitative findings
help to explore and understand why it happened.81
The second chapter in this thesis presents theories relevant to the implementation of innovations
in health care settings. The third chapter outlines the quantitative and qualitative methods we
used. The fourth and fifth chapters present the quantitative and qualitative results respectively.
The results are combined and integrated in the sixth chapter. The seventh and final chapter
outlines conclusions and discusses the results. We also present a case study describing process
changes for preventive services for one group of physicians implementing EMRs in Appendix A.
Research questions
The research questions are:
12
1. Quantitative:
a. Primary question: was there a difference in the change in preventive services targeted
by Ontario’s P4P incentives between community-based family physicians
implementing EMRs and those using paper-based records?
b. Secondary questions: was there a change in preventive services over time in a group of
physicians who purchased an EMR? Was the degree of EMR implementation
associated with a difference in the change of service provision? Were there differences
in the change between two physician groups implementing the same EMR at the same
time? Did physicians with EMRs implement reminder letter mailings to overdue
patients?
2. Qualitative: what factors were perceived by physicians as influencing their EMR
implementation?
13
Chapter 2 : Theoretical frameworks applicable to the implementation of Electronic Medical Records
The research questions in this study address the first two years of implementation of an
innovation, the EMR, in a group of primary care practices. To provide a theoretical context for
the changes occurring in these practices, I review theory underlying diffusion of innovations,7, 53,
82, 83 with attention to studies relevant to implementation in health care settings.
Background Theory can be defined as “a system of ideas or statements held as an explanation or account of a
group of facts or phenomena”.84 In this chapter, I use theory to frame the possible changes
occurring during EMR implementation, and will use theory-driven reflection85 to discuss the
findings of this study in Chapter 6, the integrative chapter.
While there are many empirical studies of adoption, few studies specifically address the
implementation of an innovation in health care.53 In particular, research relevant to theories
underpinning EMR implementation in primary health care practices is an understudied area.
Most studies of implementation draw on Rogers’ Diffusion of Innovations theory,7 a well tested
framework that can provide a theoretical basis for some of the processes underlying EMR
implementation.7, 14, 86 However, several processes precede implementation: potential users
must first be aware of the innovation;87 this may be followed by adoption, which is the decision
to use the innovation.7 The decision to adopt may be collective (made by group consensus),
optional (individuals decide independently), or authority-based (one individual decides for the
group).7 Once the innovation has been adopted, implementation, which is the process of actually
putting the innovation into use in a particular setting, can be initiated.7
Theories and studies on health care innovations do not always discriminate between diffusion,
adoption and implementation. As previously noted, the EMR is often adopted once financial
barriers are removed; all physicians in the intervention group in this study bought systems.
EMRs were widely adopted by funded primary care groups in Ontario; however, implementation
may not necessarily follow adoption.
14
For this study, I have applied Rogers’ Diffusion of Innovations framework. Rogers’ innovation
theory addresses a wide variety of factors that could affect implementation of an innovation;
these include attributes of the innovation, the process of implementation, individual
characteristics of, and interactions between, the implementers, and organizational factors.7, 53, 88 I
will discuss each of these attributes, along with their theorized effects on implementation of
innovations.
Innovation Attributes What adopters think about the characteristics of an innovation, or its attributes, has been found
to be a strong predictor of its adoption and implementation.86 For example, in a study of
implementation of different health-related technologies in hospitals, the attributes of the
innovations explained 37% of the variance in implementation.89 Perceptions of the attributes are
dynamic and change during implementation;53, 90 as well different attributes may be important at
different stages of implementation. In health care settings, the attributes that have most
consistently been associated with variation in adoption and implementation are relative
advantage, compatibility, complexity and observability.91 Another attribute, reinvention, has not
been well studied but may be an important factor in EMR implementation.53
Relative advantage
Relative advantage is the degree to which the innovation is perceived as being better than the
previous state.7 The dimensions of this attribute are related to costs and benefits including cost
advantages (expected profit, low initial cost), increased status, increased efficiency (less time or
effort needed to attain goals), and decreased discomfort. As well, the shorter the time lag
between initiating the innovation and obtaining benefits, the greater the perceived relative
advantage.7
A perception of high relative advantage is one of the strongest predictors of the rate of
adoption.7, 86 In a study of endoscopic procedures in hospitals, implementation did not begin if
there was no perceived relative advantage (that is, endoscopy was not viewed as improving
efficiency).90 This attribute may therefore be a pre-condition for implementation; however,
there are few studies of relative advantage in relation to implementation.53
15
A critical need in many primary health care practices is efficiency, especially during
consultations. The mean length of a primary care visit has been reported as being 10 to 17
minutes.92-94 However, physicians report spending 50% more time charting during the first 6
months of EMR implementation.52 The EMR is not monolithic; it consists of hardware and
software features in addition to patient charting (for example, electronic communication,
prescriptions, electronic lab results). Some aspects of the EMR may be perceived as having
greater relative advantage and therefore may be implemented earlier or more completely. In a
study of EMR implementation, the ability to receive lab results electronically and to generate
consultation requests electronically were mentioned most frequently as time saving by 18 months
of implementation.52 A survey of US physicians found that some aspects of the EMR (electronic
prescriptions) were more commonly used than others (e.g. electronic generation of diagnostic
imaging requests).95
Older physicians may be less familiar with and, therefore, less efficient with computers than
younger physicians who grew up with the technology. The impact of physician age is borne out
in the 2007 National Physician Survey:35 almost 14% of physicians under the age of 35 use
EMR instead of paper records, while only 6.4% of those over 65 do so, with clear age gradients.
The perception of relative advantage may differ by age, which may be a proxy for experience
and efficiency in using information technology.
Incentives are believed to increase the degree of relative advantage at the point of adoption. In
other words, the provision of financial incentives for EMRs can make their purchase more
attractive to physicians. While EMRs may be perceived as having a high relative advantage
during adoption because of incentives and/or efficiency expectations, this can change to a less
positive view during implementation, leading to a mismatch in expectations.96
Compatibility
Compatibility is the degree to which the innovation fits with users’ values, needs and past
experiences.7 More compatible innovations are more likely to be adopted91, 93 but this attribute
may have different effects during implementation. In one survey-based study, clinicians’ belief
that the EMR improves quality of care (that is, it was compatible with values) was the factor
most strongly correlated with high EMR implementation.97 However, in a study of guideline
implementation in gynecology, guidelines that were initially less compatible with physicians’
16
values were associated with a greater degree of change in process performance after audit and
feedback provided opportunities to improve care.98
Community-based practices include staff and allied health professionals; the distribution of
benefits and problems accruing from EMR implementation in a practice and, therefore, the
EMR’s compatibility may vary amongst different practice members.99 For example, nurses in
both hospital and primary care practices perceived implementation of a system-wide
computerized physician order entry system to be more compatible with their workflows than
physicians did.100
Implementation of four different health care innovations occurred faster when the innovation
mapped onto the distribution of interests, values and power of those involved.101 For example,
laparoscopic cholecystectomy was rapidly adopted in Montreal hospitals;101 it led to shorter
hospital stays, so that patients began demanding it and insurance companies refused to cover
longer stays; surgeons were highly motivated to quickly learn the new technique, or risk losing
part of their practice.101 Champagne found that micropolitical power (the ability to control things
locally) had the most effect on implementation;102 if the users who have control perceive a
greater compatibility and relative advantage during implementation, it may well proceed faster or
more completely. As with other attributes, perceptions of compatibility may change during the
process of implementation, as the organization evolves along the innovation.
Complexity
Complexity reflects the perception that the innovation is difficult to learn and use, and is
negatively associated with implementation.7, 103 Grilli and Lomas, for example, found that less
complex clinical guidelines were more likely to be implemented by physicians.103 In one
survey, physicians reported that simplicity and ease of use of an EMR application was a
significant positive factor in implementation.104
Several empirical studies have reported that technological barriers were prevalent during EMR
implementation.85, 105 The complexity of EMR during implementation was possibly increased
due to the need to manage complex interconnecting hardware and software systems.85
17
Innovations that incorporate a large departure from previous routines with few prior
implementations can be thought of as being both highly complex and not compatible with past
experiences. Rogers calls these “radical” or “disruptive” innovations.7 In a study on innovation
implementation in industry, Dewar and Dutton found that more radical innovations are less likely
to be implemented.106 These “radical” innovations require a greater degree of investment in
implementation activities than less complex and disruptive innovations.106
A study of business IT systems implementation found better initial performance with IT systems
that were less complex and involved a smaller degree of change from current workflows.
However, teams implementing a more radical IT system improved more as implementation
continued past the initial stages: they had a greater degree of organizational change and
adaptation to the new system.107 Radical innovations may thus entail a greater degree of initial
risk and may require a greater degree of organizational change to be implemented successfully.
Observability
Observability refers to the ability of others to see the results of the innovation. Rogers has noted
that technology includes both hardware and software. Hardware is easier to observe than
software, leading to slower diffusion for software-dominant innovations.7
The majority of family physicians in Ontario were solo practitioners,47 (although this is
changing, especially for younger physicians),35 so the opportunity to observe the EMR being
implemented may sometimes be low within individual practices. The effect of observability on
implementation is uncertain; for example, Grilli and Lomas did not find a correlation between
observability and implementation of guidelines.103
Reinvention
Reinvention is the extent to which the innovation can be modified to fit the organization and
local context as it is implemented.7, 53 This malleability can be thought of as an attribute of the
innovation, but is also part of the process of implementation. Innovations that cannot be
modified without making them less effective are less likely to be implemented.108 Gladwin, for
example, found that a health information management system needed to be extensively
reinvented during implementation to meet the needs of local health units in Uganda.109
18
However, if implementers do not perceive a relative advantage during implementation, they may
modify the innovation to ensure its failure and discontinuance, as nurses110 and physicians111 did
in case studies of hospital-based EMR implementation. For example, in one implementation,
surgeons refused to conform to some prescription patterns in the EMR, while pharmacists
insisted on compliance. Conflicts between physicians, nurses and pharmacists escalated and
surgeons eventually asked their colleagues not to admit patients to that unit—leading to
discontinuance of the system.111
Process of implementation There is a time dimension to implementation, as it occurs through stages.89 Rogers has defined
these stages as redefining/restructuring, clarifying and routinizing.7 At the
redefining/restructuring stage, the innovation is reinvented so that it fits the organization and the
organization changes as well; these processes require a significant amount of organizational
investment for the innovation to be implemented successfully. Clarifying involves more clearly
defining the role of the innovation within the organization. During routinization, the innovation
becomes part of usual operations; innovations that are successfully reinvented to fit their context
are more likely to be sustained.
This process of co-evolution during implementation has been researched by Denis, Hebert,
Langley and colleagues.101 In a study of the implementation of four different health care
innovations, they described innovations as being “composed of a hard core that was relatively
fixed and a soft periphery related to the various ways in which it might be implemented.”101(pg 66)
A study of the implementation of an innovation (quality standards) in business organizations
found that improvements in performance were correlated with both successful reinvention of the
innovation and the ability of the organization to transform itself to take advantage of the
innovation.112 As Denis and colleagues observe, “clear evidence about the appropriateness and
conditions for good practice rarely emerges until the innovation has been experimented with for
some time precisely because learning is required to optimize it”.101(pg 72) Because of this
complexity, some aspects of implementation will always be unpredictable, and will depend on
local factors.113
The process of implementation may not occur continuously or smoothly. In a study of new
processes in manufacturing and service industries, Tyre and Orlikowski found that there may
19
only be a brief window for reinvention to occur before the innovation becomes routinized;
further reinvention then occurs in fits and starts as problems arise.114
Individual characteristics and interactions between implementers Rogers has categorized adopters by the timing of adoption of an innovation, as innovators, early
adopters, early majority, late majority and laggards.7 While these categories are applicable to
adoption of innovations, their effect on implementation is less certain. An important group
during implementation may be the early adopters who are also “opinion leaders”.53 Opinion
leaders are well integrated in their social systems and exert considerable influence on their peers. 7 The presence of local opinion leaders has been found to increase the implementation of
evidence based practice115 and to affect prescriptions for antibiotics.116 While opinion
leadership means that an individual has influence on peers, it does not indicate the direction of
that influence (in favour or opposed to an innovation).53 If an opinion leader is enthusiastic
about the innovation, and becomes dedicated to supporting it, he or she may then become its
“champion”.117 Ideally, champions are technically knowledgeable about the innovation, have
good interpersonal skills and have strong connections within their peer groups enabling them to
overcome perceived barriers as they arise. 14, 118 In a case study of four hospital-based EMR
implementations, failures were due to issues such as physician concerns about loss of status
(having to do nurses’ work), which the implementation leaders were unable to successfully
address.111 In successful implementations in the same study, champions were able to solve
problems. For example, a physician tagged all diagnostic imaging requests as “bullet wound”
because this was the first item in a drop down list; the champion replied by sending a request for
a psychiatric consultation.111 The presence of a champion may be one of the most important
factors leading to a successful implementation—possibly even more so for very radical
innovations.96, 117, 119, 120
The presence of supportive leadership at the organizational level may be a factor in improving
implementation. Effective leadership has been found to improve IT implementation in business
organizations;121 supportive CEOs of hospitals had positive influences on the implementation of
innovations.89 Leadership style may need to change from adoption to implementation:
participatory leadership may be more effective at the selection stage, while more decisive
leadership may be needed during implementation, so as to quickly solve problems.122 Effective
20
leadership and the presence of a champion both had a positive effect on EMR implementation in
primary care practices123 and on the implementation of shared medical records.85
Organizational Attributes The attributes of organizations may affect the implementation of innovations. It has been
suggested that lower organizational complexity, higher formalization (emphasis on following
rules), and higher centralization (decision making being concentrated in one place) favour
implementation.124, 125 However, a meta-analysis of innovations in organizations found that
higher organizational complexity was associated with more effective implementation.126 The
literature on the effect of organizational complexity on implementation appears inconclusive.
The size of an organization may affect implementation of innovations. A meta-analysis found
that larger size favours implementation;127 however, a study in health care organizations found
that size had a negative association with implementation,102 and a meta-analysis of
innovativeness in organizations found that the impact of size was not significant.126 A survey of
EMR use in US ambulatory practices found that large groups (>50 physicians) were more likely
to have a “fully functional” EMR.34 The size of an organization appears to have an inconsistent
association with implementation.
The presence of “slack resources” (or resources beyond those required for the management of
daily tasks)7, 53 is significantly associated with implementation of innovations. Rogers theorizes
that larger organizations may have more slack and that it may be slack, rather than organizational
size, that is related to innovation implementation.7 Dijkstra, in a study of diabetes guideline
implementation in hospitals, found that lack of slack (too little time, not enough nursing support)
was associated with perceived barriers to implementation.128 Slack may be associated with the
availability of training and resources dedicated to implementation (for example, technical
support for complex IT implementations). In a randomized controlled trial, intensive teacher
training was associated with a doubling of implementation of a health program in schools.129 In
another study, the availability of technical support and training was associated with
implementation of a shared health record.85
Cooper and Zmud studied IT-based materials requirement planning (a system for predicting what
to order for manufacturing goods) in industry. They found that integrated use of the technology
21
to obtain maximal gains, or “infusion” was difficult to achieve.130 Infusion required buy in from
different departments, not all of which understood or immediately realized benefits from the new
technology, leading to “bureaucratic resistance”.130 Senior management support and training
could be used by the organization to overcome the resistance.130
Summary Perceptions of the attributes of an innovation have been associated with its degree of
implementation. Perceptions of attributes may differ according to the position and
characteristics of the implementer, and can change during the process of implementation. As
shown in Figure 1, high relative advantage likely has a positive association with implementation.
High compatibility and low complexity may also be associated with a greater likelihood of
implementation, but with a lesser degree of change in processes (such as the provision of
preventive services) if implementation is successful. The role of observability is unclear.
The organization and the innovation co-evolve during implementation; re-invention of the
innovation allows this co-evolution to occur. The presence of an enthusiastic opinion leader (or
champion) and effective organizational leadership appear to have a positive effects on
implementation. Organizational size and complexity have inconsistent effects; however, the
availability of slack resources may be correlated with successful implementation
While I have not generated formal theory-based hypotheses to test as part of this thesis, the
concepts examined in this chapter will be applied to the interpretation of findings described in
Chapter 5.
22
Figure 1: Theoretical factors affecting implementation of an innovation
Innovation implementation over time
Relative advantage (++)
Complexity (-)
Compatibility (+)
Organizational slack (+); investment in
resources, training (+)
Reinvention (flexibility of innovation and organization) (+)
Presence of champion (++);
supportive leadership (+)
Observability (0)
Innovation Attributes positively (+), negatively (-), or inconclusively (0) affecting implementation
Organizational Attributes positively (+) or inconclusively (0) affecting implementation
Organizational size (0)
Adopter characteristics (exam
ples: age, position)
affecting perception of innovation attributes
23
Chapter 3 : Methods
Participants
This study followed two cohorts of physicians: a group of eighteen physicians using EMRs, and
a second group of nine physicians using paper records (non-EMR cohort). These physicians
were community based, were all affiliated with an urban general hospital and were located in the
north Toronto area. All physicians participating in this study were members of the local After
Hours Clinic (AHC), which was a large cooperative after hours service: 143 family physicians in
the local community took turns providing services at the clinic on week-ends and evenings.
Physicians in the AHC had their own family practices, but directed their patients to the clinic for
after hours care, knowing that it would be staffed by one of their local colleagues and that they
would receive a report outlining the care their patients received. Eighteen AHC physicians
belonged to one of the two FHNs in this study, while the other 125 AHC physicians, including
the nine physicians using paper records, belonged to a single Family Health Group (FHG).
EMR cohort
The EMR cohort was composed of two FHNs, each with nine physicians. Both FHNs formed at
the end of 2004, and subsequently started implementing the EMR at similar times: FHN1 in
January and February of 2006, and FHN2 in March and April of 2006. They were using the
same EMR software and the same hospital-based server. The two FHNs used separate databases
within the server; they were able to share patient data and EMR processes within but not across
FHNs. All physicians in both FHNs consented in writing to participate in this study.
As previously mentioned, FHN physicians received the majority of their fees from capitation,
and were eligible for the EMR subsidy.
The principal investigator was a member of FHN2, and was also a participant in this study. She
implemented an EMR within her own practice; she was involved in and directly observed many
of the processes described in this study. She kept a chronological record of the progress of EMR
implementation, available at http://drgreiver.blogspot.com.
24
Non EMR cohort
The non-EMR physicians were recruited through a letter of invitation sent to the membership of
the AHC from the principal investigator. Peer to peer recruitment has been shown to be the most
successful method of enrolling family physicians in studies.131 All AHC physicians who were
not in one of the two FHNs in this study belong to one FHG.
As previously described, physicians belonging to a FHG were mostly paid through fee for
service, with a supplement for rostered patients. FHG physicians were not eligible for the EMR
supplement.
These physicians were all affiliated with the same urban general hospital and practiced in the
same geographic area of the city as the EMR cohort. The Board of the AHC provided the
principal investigator with the names and addresses of the FHG physicians. We sent a letter of
invitation and a survey asking these physicians to report the approximate number of patients in
their practice, number of days per week that they worked, and whether they had or were planning
to implement an EMR within the year (see questionnaire in Appendix C). Twenty four
physicians responded (19%). Since none of the EMR physicians practiced part time, one
respondent was excluded on the basis of part-time practice (fewer than 2.5 days per week).
Characteristics of the AHC respondents are shown in Table 3 and corresponding characteristics
for the EMR physicians are shown in Table 5. The study budget included data collection costs
for a maximum of 10 non-EMR physicians.
For purposes of comparison, the two cohorts should be as similar as possible. Thus, the
following inclusion/exclusion criteria were employed in the selection of the non-EMR
physicians. The principal investigator reviewed the size of the practices, physician gender, intent
to implement an EMR in the next year and number of days worked reported in the returned
questionnaires. Physicians who reported panel sizes or number of working days beyond the
ranges for EMR physicians were excluded. The EMR physicians were all community-based
with panel sizes ranging from 630 to 2200 patients. Their working hours ranged from 30 to 60
hours, representing approximately three to six days per week. Five physicians who were
currently implementing an EMR or planning to start one in the near future were excluded. Four
physicians with panel sizes larger or smaller than those of the EMR physicians were excluded.
One physician worked in a hospital-based academic practice and was excluded. This left 12
25
physicians, of which nine (75%) were male. 56% of EMR physicians were male; to approximate
the gender frequency of the EMR cohort, two male physicians were excluded, leaving ten
physicians. After being contacted by phone, one physician declined and nine agreed to
participate. This physician was not replaced to allow some budget flexibility.
26
Table 3: Physicians included and excluded in non-EMR cohort (N=23)
Size of practice (number of patients)
Physician Gender
Number of days worked
Inclusion Reason
400 Female 3 Excluded Less than 630 patients
500 Female 3 Excluded Less than 630 patients
700 Female 4 Excluded Hospital teaching practice
850 Male 4 Included
850 Male 4 Excluded Planning EMR
850 Female 2.5 Excluded Less than 3 days in practice
1000 Male 4 Included declined
1000 Female 5 Included
1000 Female 4 Excluded Planning EMR
1262 Male 5 Included
1300 Female 4 Included
1350 Male 5.5 Excluded Male gender
1450 Female 3 Included
1500 Female 5 Excluded Planning EMR
1600 Male 4 Included
1600 Male 4 Included
1620 Male 4 Included
1800 Female 5 Excluded Starting EMR
1900 Male 4.5 Included
27
Size of practice (number of patients)
Physician Gender
Number of days worked
Inclusion Reason
2000 Female 5 Excluded Starting EMR
2000 Male 5 Excluded Male gender
2400 Female 5 Excluded More than 2200 patients
3000 Female 6 Excluded More than 2200 patients
The nine physicians in the non-EMR cohort continued to use paper-based records throughout the
study period. As noted above, the number of physicians recruited for the non-EMR cohort (nine)
and the number of years selected for the chart audit (two) were limited by funding constraints.
A flow chart detailing the process of recruitment for the non EMR cohort is shown on Figure 2.
28
Figure 2: Flow diagram detailing the recruitment of the non-EMR cohort
125 AHC members that
were not part of EMR
cohort
Letters
mailed
24 physicians willing
to participate (19%)
Reviewed by PI
One physician excluded:
spends less than 2.5 days
per week in practice
23 physicians
eligible
Thirteen physicians were excluded:
-planning/starting EMR: 5
-Panel size out of range: 4
-Practicing less than 3 days per week: 1
-Hospital based practice: 1
-Male gender: 2
Ten physicians called
by PI
-One declined
-Nine agreed to participate and were
included in the study
Reviewed by PI for
similarity to EMR cohort
29
Determination of physician and practice characteristics We obtained data about physician and practice characteristics through a questionnaire
administered to each physician consenting to participate in the study (see Appendix D).
Following the methods of Glazier et al,132 we also obtained and examined aggregated,
anonymized data derived from linked administrative databases at the Institute for Clinical
Evaluative Sciences (ICES), after review and approval from ICES and Sunnybrook’s Research
Ethics Boards. ICES was provided with the participating physicians’ College of Physician and
Surgeon’s registration number; this allowed identification of the physicians and their practice
populations in the databases. The information provided to ICES indicated whether the physician
was in the EMR or non-EMR cohort, which FHN the physician was a member of, and whether
the physician was using the EMR during encounters. To decrease risks to privacy, only cell sizes
of greater than five were reported. We collected information on physician and patient factors
that could impact the provision of preventive services:
• Number of years the physician had been in practice: physician performance may decline
with increasing years of service.133
• Physician characteristics (gender, Canadian vs foreign medical school graduation,
certification as a family physician): female physicians134, 135, Canadian graduates136 and
certifiants in family medicine (CCFP)137 may be more likely to deliver the recommended
preventive services.
• Size of practice: low volume practice may be associated with higher quality care.138, 139
• Size of physician group: solo practice may be associated with less preventive care.136
• Comprehensiveness of care: obtaining more primary care services from the same
physician may be associated with greater provision of preventive services.140
• Patient age: increasing patient age may be associated with the provision of more
screening mammograms, but fewer Pap smears.141
30
• Recency of immigration and patient incomes by neighborhood of residence: recent
immigration and low income has been found to be associated with a lower provision of
Pap smears142 and mammograms.75, 143
• Associated morbidities and co-morbidities: increasing burden of illness may be
associated with fewer preventive services.144
The ICES data were derived from the following sources:
• The ICES Physician Database (IPDB) for information on physician country of
graduation;
• The Corporate Provider Database (CPDB) for information on physicians OHIP billing
number;
• The Ontario’s Registered Persons Database (RPDB) for patient age (as of August 31st
2007), gender and immigration recency by date of OHIP registration;
• The Client Agency Program Enrolment (CAPE) tables for information on patient
enrolment in each physician’s roster;
• Statistics Canada data on neighborhood income, linked to patients’ residential postal code
for estimates of income quintiles;
• The Canadian Institute for Health Information’s Discharge Abstract Database for hospital
discharge diagnoses;
• The Ontario Health Insurance Plan for billing and diagnostic data to identify patient visits
and diagnoses;
• The Ontario Diabetes Database (ODD) for diabetics;
• The Ontario Asthma Database (OASIS);
• The Ontario Congestive Heart Failure Database;
• The Ontario Chronic Obstructive Pulmonary Disease Database;
• The Ontario Hypertension Database;
• The Ontario Myocardial Infarction Database
Comprehensiveness of care was determined by measuring the percentage of bills for 21
commonly provided services that were provided by the patient’s own family physician.132, 145
We report morbidity and co-morbidity using the Johns Hopkins Adjusted Clinical Groups (ACG)
31
software,146 available at ICES, which uses administrative data to categorize patients in terms of
morbidity, co-morbidity and resource use.147 This software assigns diagnostic codes to one of 32
Aggregated Diagnosis Groups (ADGs);148 ADGs are measures of co-morbidity and expected
resource utilization.147 Another measure, Resource Use Bands (RUBs) aggregate ACGs with
similar estimated health care resources utilization, from low (0) to high (5);147 this serves as a
proxy for morbidity. We also obtained validated ICES data on several important chronic
conditions (diabetes,149 congestive heart failure,150 hypertension,151 myocardial infarction,152
asthma,153 chronic obstructive pulmonary disease,154 mental health problems155) as an additional
measure of morbidity.
Study Design This study used a concurrent quantitative-qualitative mixed method design. In this design, the
emphasis was on the quantitative method, with a limited embedded qualitative method156 to
explore factors described in Chapter 2 that have been found to be related to the implementation
of innovations. The integration of the qualitative and quantitative findings is reported in
Chapter 6.
We used a parallel prospective and retrospective observational cohort design (controlled before
and after study) for the quantitative aspect of the study. We concurrently conducted two focus
groups of physicians implementing the EMR for the qualitative aspect. We also recorded a case
study of one group’s implementation of new EMR processes for the management of preventive
care.
We already had data for 2004 and 2005 for the EMR cohort as part of a previous study on Pay
for Performance.61 We retrospectively collected data for 2006, and prospectively collected data
for 2007 for both cohorts. The independent variable was the introduction of EMR in one cohort
and not in the other one; the dependant variables were measures of the change in the selected
preventive services provided to patients and documented in the charts over time.
Primary and Secondary Outcomes
The primary outcome was whether or not a preventive service was provided and documented in
the chart for an eligible patient within the time frame recommended by the P4P program. The
target patient population consisted of all eligible rostered patients in the study physicians’
32
practices. Rostered patients had formally identified a physician as their family doctor by signing
an enrollment form; all physicians in this study had rostered practices.
An eligible patient had been in the practice for at least two years (defined as having a
documented encounter with the physician in the chart two years or more prior to the audit).
Having been in the practice for two years or more helped to ensure that patients who were
transient or who had recently joined a practice (and may not have had a chance to have a service
yet) were excluded. We defined a service as having been provided if the chart documented the
provision of:
• A Pap smear within 30 months for rostered women age 35 to 69. Women with a
hysterectomy were excluded.
• A screening mammogram within 30 months for rostered women age 50 to 69. Women
with a history of breast cancer were excluded.
• An influenza vaccination in the Fall (October 1st to December 31st) for rostered patients
age 65 and over.
• Five completed primary immunizations (four diphtheria-polio-tetanus-
pertussis/haemophilus vaccinations and one measles-mumps-rubella vaccination) for
rostered children, by the age of 30 months.
• A fecal occult blood test (FOBT) within 30 months for rostered patients age 50 to 75.
Patients with a history of Colorectal Cancer or Inflammatory Bowel Disease were
excluded. Patients who had a colonoscopy within the past five years were excluded.
Documentation that the patient received the service through another health care provider, within
the same time period, was acceptable.
To provide a measure of performance that included all preventive services studied, we calculated
a “composite process score”.67, 157 The score is calculated by using the total number of charts
audited for eligible patients as the denominator, and the total number of services documented on
those charts as the numerator. We used this score to compare service provision between groups
and within groups over time.
The secondary outcome was a measure of process change. We audited records for
documentation of a reminder letter having been sent to patients who were overdue for a service.
33
Reminder letters have been found to be effective in increasing the provision of immunizations
and cancer screening services.80, 158, 159 These were collected using the EMR software when
possible (see Appendix F), or with chart audits (EMR or paper).
Variable Measurement
We tracked preventive services and reminder letters for four years in the EMR cohort:
• 2004: pre Pay for Performance (P4P)
• 2005: post P4P
• 2006: first year of EMR usage (EMR transition year)
• 2007: second year of EMR usage
The non-EMR cohort was tracked for two years:
• 2006: pre P4P
• 2007: post P4P
This second cohort provided a temporal comparison for the first two years of EMR
implementation.
The changes over time and an outline of planned comparisons within and between the two
cohorts are shown in Table 4.
34
Table 4: Changes over time for the two cohorts and planned comparisons within and between
groups
Year 2004 2005 2006 2007 Comparison
EMR cohort (N=18)
No P4P Incentives
P4P Incentives
P4P Incentives +EMR
P4P Incentives +EMR
Change over time
Non-EMR cohort (N=9)
No P4P Incentives for FHG physicians
P4P Incentives
Comparison Change with EMR vs no EMR
We sought to explore changes in preventive services between physicians at different stages of
EMR implementation. We examined physicians’ patterns of usage at 18 months after adoption
of the EMR52 by calculating the ratio of number of signed off patient encounters in the EMR
with the number of encounters booked in each physician’s schedule during October 2007. We
reasoned that physicians who used the EMR consistently to record patient encounters would
have approximately the same number of signed off encounters as booked encounters (a ratio of
1).
We explored comparisons between the two FHNs that implemented the EMR at the same time,
reasoning that social and group interactions may differ between these two FHNs. We sought to
explore differences in the changes in preventive services between the two groups implementing
EMR at the same time. Lists of the nine physicians belonging to each FHN were available from
the previous study.61
The Ministry of Health and Long Term Care provided lists of patients eligible for each service to
family physicians. We selected charts for audits from these lists using a random numbers table.
We recorded the presence or absence of a service within the required time period, the patient’s
35
date of birth, and their gender, as well as whether a letter was sent to remind a patient overdue
for a service. A representative copy of a data recording form is shown in Appendix E. We
entered the data into an Epi Info database.160
We audited paper charts for physicians in the non-EMR cohort and obtained data from EMR
charts for physicians in the EMR cohort. When data were missing in the EMR (example: a
mammogram done prior to the start of EMR), we retrieved and audited the paper chart as well.
Two physicians in the EMR cohort had transferred their paper charts to a commercial storage
company; we sought and were granted permission by the physicians and the storage company to
access those charts.
Five data auditors abstracted data from the paper charts. Two of these auditors had participated
in our previous study on Pay for Performance,61 and were already familiar with the practices of
the physicians in the EMR cohort; the auditing forms and process were similar to those used in
the previous study. The research coordinator initially audited ten charts for each service in two
practices (five charts in each practice); this was reviewed with the principal investigator to
ensure consistency and to address questions. Once both coordinator and principal investigator
were satisfied with the consistency and accuracy of the process, the coordinator then trained each
data auditor, and reviewed at least ten charts for each service prior to sending the auditor into the
field. The research coordinator held meetings with the principal investigator and auditors at least
every six weeks to review progress and address concerns; team members regularly
communicated via email, and the study coordinator met regularly with each auditor. The data
were collected by the auditors on paper forms, and then entered independently in the Epi Info
database by two data entry clerks. Each clerk entered a training sample of at least ten charts for
each service, and this was reviewed with the coordinator. In order to assess auditing and data
entry, a randomly selected 10% sample of data for each service, each year and each physician
was re-audited and entered in the database; we used the Kappa statistic to compare the two
audits.
We planned to sample data electronically from the entire practices of EMR physicians. The
EMR can automatically generate lists of eligible patients, based on age, gender, and rostered
status. Physicians could electronically flag patients ineligible for a service, so that they were not
included in the denominator. A software application to extract data anonymously from each
36
practice was programmed by the EMR company, and was pilot tested in the principal
investigator’s practice, prior to the study. An example of an electronic audit is shown in
Appendix F. Once a service was provided and recorded in the appropriate location of the EMR,
the numerator (number of patients having received the service), denominator (number of eligible
rostered patients), and percentage having received the service can be automatically generated by
the system; if data are entered correctly, these numbers should be identical to the numerator and
denominator obtained using individual chart audits. The data were entered in that manner in
FHN2’s charts but not in FHN1’s; only FHN2’s data were retrieved using the software
application. We retrieved FHN1’s data using individual chart audits. We reviewed the data
obtained from the electronic audits of FHN2; some out of range service dates had been
incorrectly added by the software; these charts were individually audited, and the service (if
present) was manually added. Physicians had to indicate that a service was done by clicking an
electronic indicator field in the EMR (see Appendix A); this was not always consistently done,
and the research coordinator audited charts for the presence of the service if no information was
present. Once the data review and cleaning process was completed, 10% of the charts were
manually re-audited for reliability purposes; no errors were found.
To validate our chart abstraction data, we obtained administrative data on the same services for
the two cohorts. These data were obtained from ICES, using the same datasets outlined above;
we added data from the Ontario Cancer Registry (OCR) and the Ontario Breast Screening
Program (OBSP), after approval from ICES. Children’s vaccinations were not examined, as
billing codes include vaccinations other than the five used in the study.
The denominators were the number of patients eligible for each service alive and rostered to the
physicians in each cohort by March 31st of each fiscal year (for example, March 31st 2005 for the
2004 fiscal year). Physicians report the performance levels they have achieved on March 31st to
the Ministry of Health and Long Term Care.
Exclusion criteria for administrative data were the same as for chart audit data. To exclude
women with hysterectomies or previous cervical cancer, we obtained diagnostic codes for
cervical cancers or procedural codes indicating a hysterectomy. To exclude women with breast
cancers, we obtained the diagnostic code indicating this. To exclude patients with previous
inflammatory bowel disease, colorectal cancer or a colonoscopy in the previous five years, we
37
looked for a diagnostic code indicating one of these conditions above, or a procedural code for a
colonoscopy.
The numerators were the number of eligible rostered patients having received a service in the 30
months prior to March 31st of each year for Pap smears, mammograms or FOB testing, or an
influenza vaccination from October 1st to December 31st of the prior year. For Pap smears, we
obtained physician billing codes and laboratory billing codes for this service. For mammograms,
we obtained data from OBSP as well as from radiology billing codes for mammography. For
influenza vaccinations, we obtained physician billing codes specific to influenza vaccination, as
well as general vaccination codes for patients age 65 and over during the fall (as influenza
vaccination may have been miscoded). The rate of service was defined as the proportion of
eligible patient receiving a service at least once in the past 30 months (Paps, mammograms,
FOBT) or in the previous fall (October 1st to December 31st) for influenza vaccination.
A detailed description of the inclusion and exclusion criteria and codes for the administrative
data is provided in Appendix G.
Sample size calculation
To calculate the sample size, we first needed to decide on a minimum clinically meaningful
change. A change of 5% in a year is often used in the literature; for example, a large study in the
US found a 5% increase in services after the introduction of EMR (although this was in the
context of system-wide re-engineering).21 We assumed a similar increase after the introduction
of EMR, and calculated the sample size in order to have a power of 80% to detect an absolute
increase in service provision of 5% or higher (using rates for influenza vaccination, from 83% to
88%) in the year before EMR compared with the year after EMR, with a 5% type I error. We
needed to sample 724 charts per service per year (40 charts per service per provider); to increase
power, we oversampled, and audited 50 charts per year, per service, per physician. The entire
practices of FHN2 physicians were automatically audited. To avoid overweighing this group,
the number of patients recorded for this study was reduced to 50 per year, per physician, per
service, by using the statistical software to randomly sample 50 entries for each
year/service/physician from the available data.
38
Physicians who practice in groups may influence each other, which may lead to a clustering
effect161. Based on our previous study61, the estimated Inter-Class Correlation162 (ICC) was 0.01.
If we assume that, on average, the recruited physicians are clustered within groups of size 4, and
ICC=0.01, the inflation factor would be (1+(4-1)x 0.01)=1.03 which would have a negligible
effect on the sample size, by increasing the number of charts needed to 41. We oversampled by
25% (from 40 to 50 charts per service per year), and this would account for the possible
clustering effect.
According to our previous study,61 family physicians in the EMR cohort look after a mean of 13
eligible children per practice resulting in the audit for children’s vaccinations being
underpowered for both the EMR and Non-EMR audits. We could not randomly sample 50
charts per year, per physician, so we audited every child’s chart. We did not audit children’s
vaccines for the non-EMR cohort due to an inability to obtain lists of eligible children for 2006.
Quantitative Analysis
We compared the change in provision of services between the EMR and non-EMR cohorts. We
first compared the change in the composite process score for each group using the chi-squared
test. We then used multivariable logistic regression to simultaneously adjust for patient age,141
physician gender,61 physician experience (time since graduation)133 and CCFP vs non-CCFP
status.137 We used the Generalized Estimating Equation (proc Genmod in the SAS statistical
software application) to adjust for the clustering structure of the data in regression models.
We calculated service provision in the EMR cohort for each year. We used the chi-squared test
to compare the composite process score with the score for the previous year. We then adjusted
for differences in patient age163 using logistic regression.
We analyzed the differences in the change in the composite process score between physicians
within the EMR group who were or were not using EMR for encounters by 18 months using
analysis of variance with repeated measures (random effect model).73
We compared the provision of preventive services by FHN, for each year, using the chi-squared
test. We adjusted for patient age,141 physician gender,61 physician experience (time since
graduation)133 and CCFP vs non-CCFP status137 using multivariable logistic regression.
39
ICES data could not be used for adjustment, as the data were not collected at the patient level.
We compared the year over year change in administrative data for each cohort using chi-squared.
We did not adjust for physician or patient data, as individual level data were not available. The
principal investigator was also a participant in the study. We reanalyzed the data within FHN2
after removing her practice’s data.
Analyses were performed with the use of SAS software, version 9.1 (SAS Institute). All tests
were two sided and p values less than 0.05 were considered statistically significant.
Qualitative design In order to obtain qualitative data on the factors that promote or impede the implementation of
EMRs , we conducted two focus groups with members of the EMR Cohort. Focus groups are
particularly suited for collecting information on people’s attitudes and experiences, “how they
think and why they think that way”, within a particular context.164 Each focus group was
composed of the members of one FHN only in order to capitalize on interactions that would
naturally be occurring within each of the two groups.164 We invited all physicians in both EMR
cohorts to participate. The principal investigator is a member of FHN2; to avoid introducing
bias, she did not conduct or participate in either focus group. The interviews were conducted by
one of the researchers (JB), who had extensive experience in qualitative and focus group studies
in primary care, along with the research coordinator. The focus groups lasted approximately 1
hour each, and were conducted in February 2008. To maximize ease of participation, the
sessions were held after office hours or at lunch time in one of the participating physicians’
office. No compensation was offered to participants for attending the group; a light meal was
provided. We used a semi-structured guide based on our previous study61 (which the participants
did not see in advance); the guide is shown in Appendix H. We did not specifically ask about
preventive care, as the focus group took place during the study, and we did not want to bias
practice behaviour by introducing suggestions about preventive processes. The interviewer
introduced the topic by stating that the discussion would explore participants’ experiences with
EMRs; the initial question was whether participants used only electronic records or a
combination of paper and EMR. The interviewer then encouraged participants to talk to each
other and guided the discussion.164 She also encouraged participants to share opposing views on
the implementation of EMR in their practices.156, 164 All physicians signed an additional consent
40
form to permit the focus group recording, transcription and analysis (Appendix I). The
interviews were audio-taped and transcribed verbatim.
Qualitative Analysis
Two members of the research team (JB, MG) initially independently read and coded the
transcripts. The constant comparative method,165 a method of checking and comparing data to
identify categories,166 was used to identify key words and themes describing the participants’
views about, and experiences with the EMR system. We also searched the data for alternative
explanations.167 Key words and themes were provisionally classified into categories. The coders
then met to compare and contrast findings, category mapping and interpretations;166 we resolved
disagreement by consensus. We selected verbatim participant quotes to demonstrate that our
findings were grounded in the data.165
The study was approved by the University of Toronto’s Research Ethics Board; the Sunnybrook
Research Ethics Board approved the use of ICES data. All physicians provided written informed
consent. The approval allowed chart review and anonymized data collection without individual
patient consent.
The study was funded by the Ministry of Health and Long Term Care of Ontario through a health
system linked research grant.
41
Chapter 4 : Results, Quantitative Analysis
Characteristics of the study physicians
The 18 family physicians in the two FHNs that made up the EMR cohort were located in 9
offices, comprised of two solo physicians, five two-physician practices, a three-physician
practice, and a six-physician group practice. Three of the five practices with two physicians
were composed of one FHN physician and one non-FHN physician. The non-FHN physicians in
these hybrid practices did not use EMR. They did not participate in this study, other than for a
single practice where both partners participated in this study, with one physician as part of the
EMR cohort and the other as part of the non-EMR cohort. The nine non-EMR physicians
practiced in nine different offices, ranging from two solo practices, to a group of six physicians.
The characteristics of the study physicians are presented in Table 5 and Table 6.
Table 5: Self reported characteristics of physicians in EMR and non EMR cohorts
EMR (n=18) Non-EMR (n=9)
Year of graduation: Mean, median (range)
1977,1977 (1964-1992)
1983,1984 (1966-1993)
Male, n (%) 10 (56) 6 (66)
CCFP, n (%) 11 (61) 7 (77)
Number of MDs in practice: mean, median (range)
3, 3 (1 to 6) 4, 4 (1 to 6)
Number of hours worked per week: mean, median (range)
43.7, 42 (30 to 60) 47.5, 44 (28 to 80)
Number of patients per physician: mean, median (range)
1323, 1206 (630-2200)
1295, 1200 (850-1600)
42
Table 6: Physician and practice characteristics in EMR and non EMR cohorts, derived from
administrative databases
VARIABLE VALUE EMR Non-EMR
Physicians N=18 N=9
Canadian vs foreign graduate 16/18 8/9
Patients (mean) N=23,514 (1,306) N=10,591 (1,177)
Age as of August 31st 2007 Mean ± SD 43.8 ± 22.1 47.1 ± 21.3
Median
(IQR) 45 (27-60) 47 (31-63)
Patient Gender M 10,106 (43.0%) 4,767 (45.0%)
Neighborhood income quintile*132 Unknown 51 (0.2%) 31 (0.3%)
Lowest: 1 3,084 (13.1%) 1,594 (15.1%)
2 3,643 (15.5%) 1,438 (13.6%)
3 4,345 (18.5%) 1,951 (18.4%)
4 5,091 (21.7%) 2,414 (22.8%)
Highest: 5 7,300 (31.0%) 3,163 (29.9%)
Recent Immigrant132 1,398 (5.9%) 1,148 (10.8%)
Comprehensiveness of Care†132, 145 Mean ± SD 0.54 ± 0.35 0.50 ± 0.34
Median
(IQR) 1 (0-1) 1 (0-1)
Overall Morbidity (Resource Use Bands) ‡146 Mean ± SD 2.73 ± 1.02 2.81 ± 1.14
Median 3 (2-3) 3 (2-3)
43
VARIABLE VALUE EMR Non-EMR (IQR)
0 1,047 (4.5%) 657 (6.2%)
1 1,480 (6.3%) 616 (5.8%)
2 4,778 (20.3%) 1,720 (16.2%)
3 12,567 (53.4%) 5,344 (50.5%)
4 2,783 (11.8%) 1,614 (15.2%)
5 859 (3.7%) 640 (6.0%)
Overall comorbidity (Aggregated Diagnosis Groups) ¶146 Mean ± SD 4.77 ± 3.04 5.43 ± 3.48
Median
(IQR) 4 (3-7) 5 (3-8)
0 1,046 (4.4%) 657 (6.2%)
1-4 11,189 (47.6%) 3,962 (37.4%)
5-9 9,502 (40.4%) 4,615 (43.6%)
10+ 1,777 (7.6%) 1,357 (12.8%)
Diabetes149 1,934 (8.2%) 1,041 (9.8%)
CHF150 386 (1.6%) 300 (2.8%)
Hypertension151 5,594 (23.8%) 2,823 (26.7%)
MI152 311 (1.3%) 193 (1.8%)
Asthma153 3,143 (13.4%) 1,500 (14.2%)
COPD154 1,120 (4.8%) 626 (5.9%)
44
VARIABLE VALUE EMR Non-EMR
Mental Health155 4,937 (21.0%) 2,391 (22.6%)
* Statistics Canada data on neighborhood income, linked to patients’ residential postal code for estimates of income
quintiles
† Comprehensiveness of care was determined by measuring the percentage of bills for 21 commonly provided
services that were provided by the patient’s own family physician
‡ Resource use bands indicate morbidity and expected health care system use, from 0 (lowest) to 5 (highest)
¶ Aggregated diagnosis groups indicate comorbidity, from 0 groups (lowest level of comorbidity) to 10+ groups
(highest level)
Physicians in the non-EMR cohort were slightly younger and less likely to be female, but worked
similar hours and had a similar size of practice. Non-EMR physicians looked after a population
with incomes similar to that of the EMR cohort, but saw a greater percentage of recent
immigrants; their patients were slightly older. The practices of the non-EMR physicians were
also characterized by a greater proportion of patients with associated morbidities and co-
morbidities. Factors affecting service levels which were collected at the physician or patient
level were entered in the regression model, and we adjusted for physician gender, CCFP status,
and years of practice experience, as well as patient age.
To estimate auditing and data entry quality, we re-audited a randomly chosen 10% sample of
charts for all audits, (both electronic and paper-based), and compared the two audits. Overall
agreement was excellent (kappa 0.954).
Comparison between EMR and non-EMR cohorts We used chart audits to compare the cohort of physicians using EMR with those not using EMR
over a two year period (2006 and 2007).
We present results for individual services on Table 7. We combined the overall results for all
services (composite process score) and these are shown on Table 8. P4P incentives for FOBT
45
were introduced for FHNs in October 2006, for FHGs in April 2007, concurrent with a public
health campaign funded by Cancer Care Ontario. FOBT had a much lower rate of provision than
the other services. The change in the provision of that service may have been influenced by
different factors than for the other services, due to these contemporaneous issues.
Due to the differences between FOBT and the other preventive services we studied, we also
calculated results for the two cohorts with FOBT excluded.
Table 7: Service provision in EMR and non-EMR cohorts
Service Cohort 2006 2007 Difference Difference in change between EMR and non-EMR
Influenza vaccine
EMR 70.7% 69.8% -0.9% 8.3% less in EMR cohort
Non-EMR 59.2% 66.5% 7.4%
Pap smears
EMR 76.1% 79.7% 3.6% 1.1% more in EMR
Non-EMR 75.6% 78.1% 2.5%
FOBT
EMR 28.7% 32.1% 3.4% 1.4% less in EMR
Non-EMR 41.1% 45.9% 4.8%
Mammograms
EMR 75.2% 80.9% 5.7% 6.3% more in EMR
Non-EMR 78.3% 77.7% -0.6%
V
46
Table 8: Comparison of changes in overall service provision between EMR and non-EMR
cohorts
Change in composite score between 2006 and 2007
EMR Non-EMR Difference in change between the two groups
Adjusted difference between the two groups
Including FOBT From 63.0% to 65.7%: 2.7% increase (95% CI 0.6% - 5.0%)
From 63.6% to 67.1%: 3.5% increase (95% CI 0.5% - 6.6%)
0.8% less increase in EMR, (95% CI -3.0 , 4.6)
0.7% less increase in EMR (p=0.55, 95% CI -2.8 , 3.9)
Excluding FOBT From 74.0% to 76.8%: 2.8% increase (95% CI 0.5% - 5.1%)
From 71.0% to 74.1%: 3.1% increase (95% CI -0.2% - 6.4%)
0.3% less increase in EMR (95% CI -3.7 , 4.4)
0.3% less increase in EMR (p=0.53, 95% CI -3.0 , 3.6)
The intracluster correlation was 0.017, which was small and did not affect our results. We
examined physician gender, CCFP vs non-CCFP status, physician experience (years since
graduation) and patient age, as these variables are predictors of preventive service provision and
were collected at the patient and physician level as part of our study (see Appendix D and
Appendix E) . Female physician gender and younger patient age were positively associated with
the likelihood of receiving a service; there was no association with physician experience or
CCFP status. There was no clinically important or statistically significant difference between the
two groups with respect to the change of service provision.
In order to validate our results, we also obtained administrative data for the services. A
comparison of the composite scores for Pap smears, mammograms and influenza vaccinations
for the two cohorts is shown on Figure 3. To be consistent with the chart audits, each service in
the administrative dataset was assigned an equal weight in the calculation of the composite score.
47
Figure 3: Composite process score for EMR and Non EMR cohorts, using administrative data
and chart audit data
50%
55%
60%
65%
70%
75%
80%
85%
90%
Prev
entiv
e se
rvic
e
EMR cohort,administrative data
78.2% 78.0% 75.7% 76.7%
Non EMR cohort,administrative data
76.9% 77.0% 74.4% 74.7%
EMR cohort, chart audits 80.7% 84.9% 74.0% 76.8%
Non EMR cohort, chartaudits
71.0% 74.1%
2004 2005 2006 2007
In the administrative dataset, the EMR and non-EMR cohorts paralleled each other closely.
Using administrative data, there was a statistical difference in the change in services between the
two cohorts (data not shown); however, this was likely due to the large numbers of patients
included and the difference in the change was less than 5%. As such, it was not clinically
relevant. The decrease in overall provision seen in both administrative cohorts in 2006 was
driven by a decrease in influenza vaccinations (Figure 4).
Comparisons for each service for both chart audits and administrative data are shown in Figure 4,
Figure 5, Figure 6 and Figure 7.
48
Figure 4: Influenza vaccination for EMR and Non EMR cohorts, using administrative data and
chart audit data
50%
55%
60%
65%
70%
75%
80%
85%
90%
Influ
enza
vac
cina
tions
EMR cohort,administrative data
74.2% 69.5% 62.6% 64.3%
Non EMR cohort,administrative data
72.2% 71.7% 62.8% 65.2%
EMR cohort, chart audits 76.2% 83.3% 70.7% 69.8%
Non EMR cohort, chartaudits
59.2% 66.5%
2004 2005 2006 2007
The decrease in influenza vaccination observed from 2005 to 2006 in the EMR cohort with chart
audits was paralleled by a decrease with administrative data. A similar decrease was found in
administrative data for the non-EMR cohort.
49
Figure 5: FOBT for EMR and Non EMR cohorts, using both administrative data and chart audit
data
10%
15%
20%
25%
30%
35%
40%
45%
50%
FOBT
EMR cohort,administrative data
21.7% 23.2% 23.6% 26.4%
Non EMR cohort,administrative data
22.2% 27.8% 28.2% 27.1%
EMR cohort, chart audits 27.1% 28.7% 32.1%
Non EMR cohort, chartaudits
41.1% 45.9%
2004 2005 2006 2007
FOBT had year over year increases in the EMR cohort, evident in both administrative data and
chart audits. The non EMR cohort had inconsistent changes over time and the chart audits for
that cohort showed an increase between 2006 and 2007 that was not evident in the administrative
data.
50
Figure 6: Mammography for EMR and Non EMR cohorts, using both administrative data and
chart audit data
50%
55%
60%
65%
70%
75%
80%
85%
90%
Mam
mog
ram
s
EMR cohort,administrative data
79.3% 81.3% 82.0% 82.8%
Non EMR cohort,admnistrative darta
80.3% 80.2% 80.7% 80.6%
EMR cohort, chart audits 81.9% 85.4% 75.2% 80.9%
Non EMR cohort, chartaudits
78.3% 77.7%
2004 2005 2006 2007
The EMR cohort had small increases in mammography following the introduction of P4P in
2005, while introduction of the EMR in 2006 was associated with a decrease in documented
mammography in chart audits. There was no decrease in the provision of this service in
administrative data (p=0.48). Provision of mammography in the non-EMR cohort did not
change. Based on administrative data, the difference in the change between the EMR and non-
EMR cohorts was significant, but this may not be of clinical significance as there was less than a
5% difference: there was a 0.7% increase in the EMR cohort between 2005 and 2006, and a
0.5% increase in the non EMR cohort in the same year, a difference of 0.2%.
51
Figure 7: Pap smears for EMR and Non EMR cohorts, using both administrative data and chart
audit data
50%
55%
60%
65%
70%
75%
80%
85%
90%
Pap
smea
rs
EMR cohort,administrative data
81.0% 83.1% 82.7% 82.9%
Non EMR cohort,administrative data
78.2% 78.9% 79.8% 78.4%
EMR cohort, chart audits 84.2% 86.1% 76.1% 79.7%
Non EMRcohort, chartaudits
75.6% 78.1%
2004 2005 2006 2007
Administrative data showed little change in the provision of Pap smears in the EMR and non
EMR cohorts. The decline in Pap smear provision in the EMR chart audits associated with the
introduction of EMR in 2006 was not reproduced in the administrative data (p=0.52).
Two physicians in the non-EMR cohort switched out of the FHG in 2006, and one additional
physician did so in 2007. A reanalysis of the administrative data which included only physicians
in a FHG in each year did not change our results (data not shown).
Service provision in EMR cohort Chart audit results for the percentage of eligible patients receiving each preventive service in the
EMR cohort are shown in Figure 8.
52
Figure 8: Individual preventive services for EMR cohort, using chart audit data
20%
30%
40%
50%
60%
70%
80%
90%
100%
Prev
entiv
e se
rvic
es
Influenza vaccinations 76.2% 83.2% 70.7% 69.8%
FOB 27.1% 28.7% 32.1%
Mammograms 81.9% 85.4% 75.2% 80.9%
Pap smears 84.2% 86.1% 76.1% 79.7%
Children's vaccinations 93.1% 95.7% 66.7% 89.8%
2004: pre P4P 2005: post P4P 2006: EMR transition
2007: EMR
The intracluster correlation for each service was generally small, at 0.036 for influenza
vaccination, 0.0197 for FOB screening, 0.0189 for mammography, and 0.009 for Pap smears.
Results for children’s vaccinations should be interpreted with caution, due to the very small
numbers of eligible children in each practice. We examined physician gender, CCFP vs non-
CCFP status, physician experience (years since graduation) and patient age. Only increasing
patient age was a significant factor for the provision of a preventive service. The results within
the EMR cohort were adjusted for patient age; no adjustment was needed for physician factors in
the year to year comparison, as we followed the same physicians longitudinally.
The changes in the composite score for the EMR cohort (excluding FOBT) are shown on Table
9.
53
Table 9: Comparison of overall composite process score in EMR cohort (excluding FOBT) by
year
Year Patients eligible for services
Patients receiving service: n, % (95% confidence interval)
Unadjusted change from previous year (95% CI)
Unadjusted comparison with previous year, p value
Adjusted comparison with previous year, p value
2004 (pre P4P)
3039 2480, 81.6% (80.2% - 83.0%)
2005 (post P4P)
2950 2520, 85.4% (84.2% - 86.7% )
3.8% (1.9% - 5.7%)
<.0001 0.0002
2006 (EMR transition)
2759 2039, 73.9% (72.3% - 75.5%)
-11.5% (-13.6% - -9.4%)
<.0001 <.0001
2007 (EMR) 2870 2239, 78.0% (76.5% - 79.5%)
4.11% (1.9% - 6.3%)
<.0003 0.02
The initial year of EMR implementation was associated with a statistically significant decrease
in the composite score, followed by an increase in the second year which did not reach baseline
levels.
To validate our chart audits, we obtained administrative data on the same services; data on
children’s vaccinations were not available. The provision of each service derived from
administrative data is presented in Figure 9.
54
Figure 9: Service provision in EMR cohort, derived from administrative data
20%
30%
40%
50%
60%
70%
80%
90%
100%
Year
Prev
entiv
e se
rvic
e
influenza vaccinations 74.2% 69.5% 62.6% 64.3%
FOB 21.7% 27.1% 28.7% 32.1%
Mammograms 79.3% 81.3% 82.0% 82.8%
Pap smears 81.0% 83.1% 82.7% 82.9%
2004 2005 2006 2007
Administrative data show a decline in influenza vaccinations from 2004 to 2006, and an increase
in FOBT, but no change in Pap smears or mammograms. There was no significant change
between 2005 and 2006 for Pap smears (p=0.52) or mammography (p=0.48); there was a change
in influenza vaccination (p<0.0001) and FOBT (p<0.0001). While there was a decline in Pap
smears and mammography associated with EMR implementation (2006) in the chart audits, this
was not present in administrative data.
Composite process scores per year for chart audit and administrative data (mammography, Pap
smears and influenza vaccinations) are shown on Table 10. These three services were used as
we had chart audit data and administrative data for the EMR cohort for all four years. To be
consistent with the chart audit data, each service in the administrative dataset was assigned an
equal weight.
55
Table 10: Comparison of chart audits and administrative data for EMR cohort, composite score
for mammography, Pap smears and influenza vaccinations
Year Percentage of patients receiving service, chart audits
Percentage of patients receiving service, administrative data
Difference within year
Difference in change from previous year
2004 (pre P4P) 83.1 78.2 4.9% greater with chart audits
2005 (post P4P) 84.9 78.0 6.9% greater with chart audits
2% greater increase with chart audits
2006 (EMR transition)
74.0 75.7 1.7% less with chart audits
8.6% greater decrease with chart audits
2007 (EMR) 76.8 76.7 0.1% greater with chart audits
1.8% greater increase with chart audits
There was a decline in the composite score for preventive services in 2006, using administrative
data (p<0001). The decline in the administrative dataset was driven by a decrease in influenza
vaccinations. A greater proportion of patients were found to have received services with chart
audits than with administrative data for every year except for 2006 (EMR transition).
Comparison of service provision in EMR cohort by level of EMR use
Figure 10 demonstrates the patterns of usage observed for the month of October 2007. This was
approximately 18 months after the EMR system was installed in all offices. As detailed in the
methods, these patterns of usage were obtained by taking the number of encounters recorded in
the EMR and dividing by number of appointments booked in each physician’s schedule in
October 2007. A ratio of 1 meant that the physician was entering an encounter in the EMR for
each patient booked in their schedule.
56
Figure 10: EMR usage 18 months after EMR installed
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Sign
ed o
ff e
ncou
nter
s/bo
oked
app
oint
men
ts,
Oct
ober
200
7
Three patterns of usage emerged by 18 months. Two physicians were using the EMR
extensively (the “super-users”), one in each FHN. The “super-users” had a ratio of EMR
encounters to booked appointments exceeding 1, meaning that they were entering data in
encounters outside of booked appointments (phone calls, other EMR related data entry). The
“users” had a ratio approximating 1, meaning that they were using the EMR and entering data
consistently for most encounters. Six “users” were in FHN1 and four were in FHN2. The “non-
users” were seldom using the EMR to record encounter data. Two of the “non-users” were in
FHN1 and four were in FHN2.
The characteristics of these groups are shown in Table 11 and Table 12.
Super-users: 2
Users: 10
Non-users: 6
57
Table 11: Self reported characteristics of physicians by usage of EMR
Non User (n=6) User/super user (n=12)
Year of graduation: mean, median (range)
1978, 1979 (1969-1985)
1976, 1975 (1964-1992)
Male, n (%) 4 (66) 6 (50)
CCFP, n (%) 3 (50) 8 (67)
Number of MDs in practice: mean, median (range)
3, 2 (1 to 6) 4, 3 (1 to 6)
Number of self-reported hours worked per week: mean, median (range)
42.4, 40 (30 to 60) 44.3, 45 (32 to 60)
Number of patients per physician (self-reported): mean, median (range)
1140, 1206 (780 to 1425)
1415, 1350 (630 to 2200)
58
Table 12: Physician and practice characteristics by usage of EMR, derived from administrative
databases
VARIABLE VALUE EMR-nonusers EMR-users
Physicians N=6 N=12
Canadian vs foreign graduate 5/6 11/12
Patients (mean) N=6,837 (1,140) N=16,677 (1,390)
Age as of August 31, 2007 Mean ± SD 48.1 ± 22.0 42.1 ± 21.8
Gender M 3065 (44.8%) 7,041 (42.2%)
Income quintile 16 (0.2%) 35 (0.2%)
1 1005 (14.7%) 2079 (12.5%)
2 1052 (15.4%) 2591 (15.5%)
3 1252 (183%) 3093 (18.5%)
4 1479 (21.6%) 3612 (21.7%)
5 2033 (29.7%) 5267 (31.6%)
Recent Immigrant 556 (8.1%) 842 (5.0%)
Comprehensiveness of Care Mean ± SD 0.52 ± 0.35 0.55 ± 0.35
Median
(IQR) 1 (0-1) 1 (0-1)
Overall Morbidity (RUB) Mean ± SD 2.80 ± 1.05 2.69 ± 1.01
Median
(IQR) 3 (2-3) 3 (2-3)
0 317 (4.6%) 730 (4.4%)
59
VARIABLE VALUE EMR-nonusers EMR-users
1 348 (5.1%) 1132 (6.8%)
2 1274 (18.6%) 3504 (21.0%)
3 3643 (53.3%) 8924 (53.5%)
4 894 13.1%) 1889 (11.3%)
5 361 (5.3%) 498 (3.0%)
Overall comorbidity (ADGs) Mean ± SD 5.05 ± 3.20 4.65 ± 2.96
Median
(IQR) 5 (3-7) 4 (2-6)
0 316 (4.6%) 730 (4.4%)
1-4 2939 (43.0%) 8250 (49.5%)
5-9 2883 (42.2%) 6619 (39.7%)
10+ 699 (10.2%) 1078 (6.5%)
Diabetes 681 (10%) 1253 (7.5%)
CHF 167 (2.4%) 219 (1.3%)
Hypertension 2134 (31.2%) 3460 (20.7%)
MI 127 (1.9%) 184 (1.1%)
Asthma 832 (12.2%) 2311 (13.9%)
COPD 497 (7.3%) 623 (3.7%)
Mental Health 1534 (22.4%) 3403 (20.4%)
60
The group of physicians that had EMR but did not use it for encounters by 18 months had fewer
female physicians, fewer CCFPs, fewer years in practice and smaller practices. They looked
after older patients with greater morbidity and co-morbidity levels, and more recent immigrants.
Composite scores for chart audits for both groups are presented in Figure 11, and comparisons
are presented in Table 13.
Figure 11: Overall service provision by EMR usage (excluding FOBT)
50%
55%
60%
65%
70%
75%
80%
85%
90%
Prev
entiv
e se
rvic
es
Non-Users 79.2% 83.8% 70.1% 72.2%Users 82.7% 86.2% 75.7% 80.1%
2004 2005 2006 2007
61
Table 13: Comparison of changes in overall service provision (excluding FOBT)
Non-users Users/super-users
Difference in change between the two groups
Adjusted p value for difference between the two groups
Change after EMR introduction (from 2005 to 2006)
-13.7% (95% CI -9.9% - -17.5%)
-10.5% (95% CI
-8.0% - -13.0%)
3.2% greater decrease in non-users
0.46
Change in second year of EMR (from 2006 to 2007)
2.1% (95% CI
-2.1% - 6.2%)
4.5% (95% CI 1.8% - 7.1%)
2.4% smaller increase in non-users
0.16
Overall change 2005-2007
-11.6% (95% CI
-7.9% - -15.4%)
-6.1% (95% CI
-3.7% - -8.4%)
5.5% greater overall decrease in non-users
0.04
We adjusted for physician gender, CCFP status, physician experience (years since graduation),
and patient age. The decrease in services found in chart audits associated with EMR
implementation affected physicians who used the EMR to record patient encounters and those
who didn’t. There was a greater decrease (which was non-significant) and less increase in the
second year of EMR for the physicians who did not use EMR to record encounters. Overall,
there was a statistically significant greater decrease in preventive services for the non-user group.
Comparison of the two FHNs
FHN1 was composed of a large (6 physician) office and a three physician office, while FHN2
was scattered across several small solo or two physician practices. The Principal Investigator
was a member of FHN2. Physician and practice characteristics for the two FHNs are presented
in Table 14 and Table 15.
62
Table 14: Self reported characteristics of physicians by FHN
FHN1 (n=9) FHN2 (n=9)
Year of graduation: Mean, median (range)
1975, 1972 (1965 to1992) 1978, 1980 (1969 to 1984)
Male, n (%) 6 (66) 4 (44)
CCFP, n (%) 6 (66) 5 (56)
Number of MDs in practice: mean, median (range)
5, 6 (3 to 6) 2, 2 (1 to 2)
Number of self-reported hours worked per week: mean, median (range)
43.1, 42 (30 to 60) 44.4, 43 (35 to 60)
Number of patients per physician: mean, median (range)
1336, 1160 (780 to 2200) 1311, 1211 (630 to 2200)
63
Table 15: Physician and practice characteristics by FHN, derived from administrative databases
VARIABLE VALUE FHN1 FHN2
Physicians N=9 N=9
Canadian vs foreign graduate 8/9 8/9
Patients (mean) N=12,147 (1,350) N=11,367 (1,263)
Age as of August 31, 2007 Mean ± SD 42.2 ± 21.5 45.6 ± 22.5
Median
(IQR) 43 (25-58) 46 (28-62)
Gender M 5,492 (45.2%) 4,614 (40.6%)
Income quintile 27 (0.2%) 24 (0.2%)
1 1,749 (14.4%) 1,335 (11.7%)
2 2,071 (17.0%) 1,572 (13.8%)
3 2,287 (18.8%) 2,058 (18.1%)
4 2,562 (21.1%) 2,529 (22.2%)
5 3,451 (28.4%) 3,849 (33.9%)
Recent Immigrant 699 (5.8%) 699 (6.1%)
Comprehensiveness of Care Mean ± SD 0.55 ± 0.36 0.53 ± 0.35
Median
(IQR) 1 (0-1) 1 (0-1)
Overall Morbidity (RUB) Mean ± SD 2.68 ± 1.01 2.78 ± 1.04
Median
(IQR) 3 (2-3) 3 (2-3)
0 550 (4.5%) 497 (4.4%)
64
VARIABLE VALUE FHN1 FHN2
1 838 (6.9%) 642 (5.6%)
2 2,600 (21.4%) 2,178 (19.2%)
3 6,486 (53.4%) 6,081 (53.5%)
4 1,341 (11.0%) 1,442 (12.7%)
5 332 (2.7%) 527 (4.6%)
Overall comorbidity (ADGs) Mean ± SD 4.64 ± 2.95 4.90 ± 3.13
Median
(IQR) 4 (2-6) 4 (3-7)
0 549 (4.5%) 497 (4.4%)
1-4 5,985 (49.3%) 5,204 (45.8%)
5-9 4,813 (39.6%) 4,689 (41.3%)
10+ 800 (6.6%) 977 (8.6%)
Diabetes 977 (8.0%) 957 (8.4%)
CHF 166 (1.4%) 220 (1.9%)
Hypertension 2,687 (22.1%) 2,907 (25.6%)
MI 135 (1.1%) 176 (1.5%)
Asthma 1,638 (13.5%) 1,505 (13.2%)
COPD 511 (4.2%) 609 (5.4%)
Mental Health 2,721 (22.4%) 2,216 (19.5%)
65
As compared to FHN2, FHN1 physicians were working in larger groups, had more CCFPs,
fewer female physicians, more years in practice and larger practices. They had younger patients
with lower morbidity and co-morbidity levels, and fewer patients in the upper income stratum.
We adjusted for physician gender, CCFP status, and years of practice experience, as well as
patient age.
The overall rate of services documented in chart audits for each FHN is presented in Figure 12,
and the differences between the two FHNs are presented in Table 16
Figure 12: Service provision by FHN (excluding FOBT)
Service provision by FHN
65%
70%
75%
80%
85%
90%
Serv
ice
prov
isio
n
FHN1 81.4% 83.3% 73.3% 73.7%
FHN2 81.8% 87.6% 74.4% 81.5%
2004 2005 2006 2007
66
Table 16: Differences in overall service provision between FHNs (FHN2 - FHN1), excluding
FOBT
Year Unadjusted difference between FHN1 and FHN2, % (95% CI)
P value (adjusted)
2004 0.4% (-3.1%, 2.4%) 0.75
2005 4.3% (1.7%, 6.8%) 0.02
2006 1.2% (-4.4%, 2.1%) 0.50
2007 7.9% (4.9%, 10.9%) 0.003
The intracluster correlation was 0.0112, which was low and did not impact the results. The p
value for the results was adjusted for physician gender, CCFP status, years of practice experience
and patient age. There was a statistically significant and clinically important difference between
the two FHNs in 2007.
We audited charts of patients who were overdue for a service for the presence of a note
indicating that a reminder letter had been sent; the number of patients being reminded by group
and by year is shown in Table 17. Some patients who were overdue had been sent more than one
reminder letter; each patient was treated as a single count, regardless of the number of reminders
they had been sent.
Table 17: Number of patients being reminded by letter about overdue service
Year 2004 2005 2006 2007
FHN1 2 9 0 0
FHN2 6 23 265 677
Total EMR 8 32 265 677
Non EMR 1 0
67
Few reminder letters were sent by FHN1 or by the non-EMR physicians. FHN2 sent out an
increasing number of reminders following EMR implementation (2006 and 2007). Seven to 82
patients per physician were reminded in 2006 and 66 to 109 patients per physician in 2007.
Some patients were sent more than one reminder letter if they did not respond to the first letter;
FHN2 sent a total of 393 letters in 2006 and 977 letters in 2007.
We reanalyzed the results with the Principal Investigator (who was also a participant in the
study) removed, and the results remained unchanged (Figure 13).
Figure 13: Documented service provision by FHN, with investigator removed
Service by FHN, without NY2
65%
70%
75%
80%
85%
90%
Serv
ice
prov
isio
n
FHN1 81.4% 83.3% 73.3% 73.7%
FHN2 81.0% 87.0% 73.7% 81.0%
2004 2005 2006 2007
68
Chapter 5 Results, Qualitative Analysis Five out of nine eligible members of FHN1 and seven of eight eligible members of FHN2
participated in a focus group. The principal investigator was a member of FHN2, and did not
participate in the interviews. It was clear from the quantitative data that some physicians were
“non-users” of the EMR; however, it could not be determined whether the EMR was never
implemented or was implemented and then abandoned. Physicians mentioned that the decision
to adopt was tied to the EMR subsidy, and was collective (made as a group); the decision to
implement may have been optional, and may have been responsive to peer influences.
“I think one of our driving forces was the availability of funding”.
“It was part of our agreement as a group to go ahead. Initially I think some of us thought we
would just do billing and scheduling on the EMR, but gradually we sort of pulled each other
into it.”
While the majority of the physicians had implemented aspects of the EMR, participants in the
focus groups agreed that they continued to use both paper and EMR, and were in effect running
hybrid systems.
“I carry the chart to look things up because not all patients are fully integrated into the EMR
system”.
Participants also indicated that, despite the difficulties inherent in EMR implementation, they
were not willing to go back to paper-based records.
Several themes associated with the first two years of EMR implementation emerged from the
focus groups and were categorized as barriers, and facilitators or benefits of EMR
implementation.
Barriers
Participants described problems during EMR implementation; we grouped these into the
following themes
• Lack of compatibility and high complexity
69
• Increased workload and costs; delayed benefits
• Technological barriers: lack of interoperability, lack of technical support and
infrastructure failures
• Lack of ongoing training and education
Lack of compatibility and high complexity
Participants perceived the EMR system as complex (difficult to use) and inflexible, and thus, not
highly compatible with their current needs. Some of this was felt to be due to software interface
issues and perceived software immaturity. Participants recognized that these were not issues
isolated to their setting.
“It (the EMR software) is not intuitive and none of them out there are. I was at a big
meeting, and other people are using other systems and they have the same complaints as we
do”
“If you flip back in business and look at the programs 20 years ago they didn’t have excel
spread sheets and this and that and the other and I think we have to evolve. It has to be
intuitive and have the flexibility and that is just not in the existing software. There won’t be
until they have the volume of people”
“When you go and put in the medications there is not a lot of flexibility. It will only give you
certain options”
Increased workload and costs; delayed benefits
A common theme was the enormous amount of time required for data entry by physicians,
clearly far more than they had expected. Physicians had limited time during the work day to
implement EMR—the additional time was sometimes taken out of their personal lives.
Participants thought that EMRs would eventually lead to increased efficiency, but this had not
yet happened. There was a long perceived time lag between effort and reward, leading to
disappointment after almost two years into implementation.
70
“It is taking me longer when I am seeing them, I am staying later, I am working weekends, it
is like ok I like this why? And that is the most discouraging thing to me, this work load thing.
I didn’t mind the data entry; I just thought suck it up for a year, you know, you will be here
every night and every weekend. But now I am really not feeling very good about it.”
“The biggest frustration is the amount of work we had to do and in retrospect only if I was
starting a brand new practice with zero patients would I even consider doing EMR.”
“It is all front-end loaded. So, it is time, money and energy all up front, and then we are just
dribbling out the other end. But when faced with this ongoing thing it is hard for us to really
keep sight of why we were doing it.”
Participants felt that there were unexpected costs for the technology and for human resources
required to implement the innovation.
“I think it <the incentive funding> is a little amount compared to the overall costs and the
costs that we are now spending on IT and how we had to change the office and hire a new
staff person. So, these costs that we didn’t foresee are now part of what we have.”
The EMR appeared to have different effects for different staff positions:
“We are freeing up our front staff, but we are causing lots more work for our nurses because
we find the nursing function is more labour intensive; but the front staff love it.”
Technological barriers: lack of interoperability, lack of technical support and infrastructure failures
Several physicians mentioned the decrease in efficiency due to technological barriers such as
lack of system interoperability. All paper based materials coming from specialists, hospitals or
diagnostic imaging facilities need to be scanned in; in effect, the practice-based EMR functions
as an “electronic island”. Physicians also talked about the lack of ability to audit this paper-
based data, and about the effect of having to learn to operate in several different health care IT
environments. These barriers made it more difficult to adapt the EMR to office workflows.
71
“My secretary spends an hour or two every day scanning this stuff in and then I have to look
at what they scan.”
“So I can envision instead of using the fax that it emails and goes into the patient chart
directly. I can, down the road that is what they promised me, that will save me time, but it
will only save me time if it is in a retrievable form meaning I can see it and retrieve it. That
software is not here. So when things come in not only do I have to scan them but then I have
to write a summary of that and then I have to put it in various parts of the chart and all of
that is my time. I don’t know what you are up to but I am up to an hour and a half to 2 hours
after the last patient”
“The hospital should have the same systems so that when I go on the floor and try to learn
the system I can’t find half of the information so I am used to my system.”
“The analogy I keep using is it is like having a Bank card and you can only use it at your
branch and no where else. That is basically what we have.”
Several participants mentioned IT structural failures, lack of redundancy and lack of technical
support. Participants clearly had the impression that they were simply left to fend for
themselves, with little knowledge, time, or resources to manage a complex IT infrastructure.
Solving common IT problems was left to physicians, as offices initially lacked technical support
or non-physician personnel familiar with IT technology; there was no routine way to manage
problems, so that many issues escalated into larger problems. Dealing with IT issues had to be
done instead of seeing patients, and was learned “on the job”; participants found this to be
extremely stressful. There was no unified access point in case of problems with hardware,
software, or connectivity; participants often simply did not know who to contact and there was
no “disaster recovery plan” to deal with the inevitable equipment failures. Participants felt that
there was a lack of leadership needed to deal with the problems they were facing. Because of all
the failures, the system was perceived as being inherently faulty.
“We had huge connectivity issues so this gets into the mechanism. We had to institute our
own (connectivity) system, actually getting wiring through all of this, negotiate endlessly so it
really took up an awful lot of time and money”
72
“The system breaks down and you have to have all these recovery plans that are throwing us
all awry at a time when we are very stressed.”
“We need an office manager who could handle the printer going down, the scanner problems
that we have been having, the connectivity issues, then it would be ok; but one of us is always
running like a chicken with their head cut off crazily trying to put the finger in and nothing
ever happens.”
“I was broken into and my laptop was stolen the first few months. It was a great
consternation. I got a new one and just before Christmas I flipped the screen open and there
was a blue screen that says ‘contact your hardware vendor’. So I assumed <the EMR
company> covered us for hardware and software. So I phoned them and they sent me an
email of two places that I could take my laptop to be repaired.”
“Our monitors have gone down. It is not their responsibility and I had to buy a new monitor.
I just came in one day, and it was dead. Our printer also went.”
“We have bugs in some of our systems; we have printers that don’t work so we have become
increasingly dependent on something that is faulty.”
“We need to have a body that is really listening to us and can represent us, at the grassroot
level… All I see around us is continuing chaos.”
Lack of on-going training and education
Several participants described a lack of knowledge about basic computer operations and common
programs (IT skills). Some are unable to type, which can interfere with data entry into the
computer—yet they did not discuss taking a typing course.
“I have a huge problem because being older than everybody, I did not learn how to type.”
“I am not very sophisticated in terms of computers in general so for the newbie like myself
everything has been an adventure. So learning about not just our software but just how
<Microsoft> Office works or whatever application we are using. So we had to learn
everything and that slowed us down immeasurably.”
73
EMR training was offered prior to EMR start-up, but there were no formal sessions scheduled
later on. Participants recognized their need for on-going training in EMR use. They felt
frustrated, as they were often unable to even formulate the right question; they were truly out of
their “comfort zone“.
“I don’t even know what I could learn. I know there are buttons there that I am not using
efficiently so it would be nice if you could follow me around for 2 or 3 patients to see how I
am doing it and tell me probably how I can use it better”
“We should also have upgrades in terms of education. The education sessions were at a time
when too much was coming and now we do need to know how to use it.”
“I am still doing my own method that I learnt initially. There must be a faster way of
bypassing certain things.”
Facilitators and benefits
Participants talked about facilitators of EMR implementation and discussed their perceptions of
current and expected benefits of the system. We classified these as
• Availability of an EMR champion
• Increased efficiency for some practice processes
• Perception of time for the eventual pay-back of the investment
• Physicians’ perceptions of patient reaction
• Perceived improvements in quality
Availability of an EMR Champion
Physicians in one focus group (FHN2); mentioned the availability of a champion. The champion
provided support, helped solve some problems, and was perceived as facilitating and maintaining
enthusiasm for the transition.
“She (the EMR champion) makes sure that you understand the value of it and she is so
enthusiastic”
74
“She is very patient; she probably answered the same question five times.”
Increased efficiency for some practice processes
Participants found that some aspects of the EMR made them more efficient, once they had
learned the system. Prescription refills and consultation letters in particular were much quicker.
This occurred after an initial decrease in efficiency, once some data entry was completed.
Physicians felt that their administrative personnel were more efficient.
“I always put charts aside, particularly my physicals and did them later on and I don’t do
that. I don’t leave charts on my desk at the end of the day. They are done, they are done on
time and to me it is work it right then and there. Rather than remembering and writing it all
out, I do it right then. That part has been good.”
“I find that prescription renewals are great especially if you have someone on 10
medications and you have to start writing it out. It is a real pain, once it is in the system it is
kind of nice.”
“I found if I am doing a consult, it is easy to just click on medications in the consult letter or
past health or insurance forms. I just do a CPP and add it to the forms. I am not going back
and looking up things. It is all there.”
“Less paper shuffling when the person is in your office, it is compiled nicely and the
prescription printing clarity of physician names, patient’s name, the pharmacist can read
when we print off the scripts.”
“It saves huge amounts of time for the staff. They don’t have to pull them and refile them.
Prescriptions don’t have to be pulled; labs don’t have to be filed, so there is a lot of time
saved there for our staff.”
Perception of time for eventual pay-back of the investment
Participants felt that starting an EMR was becoming a necessity; however, they perceived that
there is an eventual benefit that decreases with increasing physician age. The transition was
viewed as extremely challenging; younger colleagues have more opportunity for the initial
investment in time, money and effort expended during the initial EMR implementation to
75
eventually pay dividends As well, younger colleagues are more familiar and comfortable with
the technology, decreasing the amount of initial investment needed; new physicians, in
particular, are faced with much less workload, as they do not have to input large amounts of data
from old charts.
“I don’t think there is any future for paper charts. Ten years from now it will not be
considered standard care. For anybody going into practice now who didn’t start with EMR
would be a total mess.”
“I have gone and spoken to different groups, and doctors over 45 have no interest. So those
with established practices who have been in practice 10 or 15 years and around age 45, not
always true, but the people that are under say 15-20 years of practice are those people that
are willing to invest the time. I was at a symposium and the 50+ said are just are you nuts
and why would I do that because they are not ready for us, it is not ready for Prime Time.”
Physicians’ perceptions of patient reaction
Physicians worried about their patients’ perceptions of the new technology. However, they felt
that patient reaction ranged from neutral to positive. Patients sometimes even encouraged their
physician.
“Actually it is not bad because I always thought that there wouldn’t be that much eye contact
but I really try and make an effort. I am not typing all the time so I don’t think they mind it.”
“I think patients are pleased. You know, oh finally you are in the modern age I see, good for
you.”
However, some physicians felt that the EMR interfered with the visits. Some of the difficulties
were related to data entry problems, such as being unable to type:
“It interferes between my relationship with my patients. I find that I want to look at them and
they want to look at me, they don’t want to see the back of my head or back and unfortunately
I cannot talk to them and make notes at the same time. I talk to them, I do everything and
then I walk out of the room and then I put my notes in.”
Some physicians were interested in giving patients greater access to their own charts:
76
“So it is far better that 2500 patients that I look after look after their own CPP than I try to
keep up to date and make sure it is current. That would be a huge benefit for all of us”
“Hospitals are working and testing some of the patient folders for chronic diseases like
diabetes and cancers. Where the patient can go and look at their own file”
However, physicians have found that there is a downside to some of these early attempts at
patient access to the chart. The coding system that is in common use for charting can lead to
misunderstandings.
“Lately, when I do referrals, I give the forms directly to the patients themselves. So, now
when I am printing that, I have to put (in the) past medical history and the problems are
sometimes very, like if you do a code for anxiety or depression and then the patient goes
home and starts reading it--and I get a call back, and they say ‘I don’t have this’, and I say
‘it is a whole category because you just have to whip them all together’, and I have had one
or two patients get very upset about it.”
Perceived improvements in quality
Physicians felt that the EMR implementation had improved the quality of their records: the chart
was better organized, and they were able to find data quickly. Legibility had improved as well.
Participants commented on the ability to generate reports, which was tied to their expectation
that the EMR would help them provide better care.
“It is nice to be able to find reports. If somebody comes in and says they had a mammogram
and I don’t remember I just look back and see the results. If they have seen a specialist it is
so much easier than trying to leaf through a chart.”
“The extraction is amazing. I can create lists to find out my hypertensive, if there has been a
drug recalls, just my elderly if I wanted to do something. I mean down the road I can really
see if I have got smokers then I can create that list and then use services to bring them in for
that or people with BMI’s of 30 and over if I am going to start.”
“I think the patients are benefiting; I am not sure I am, but patient care, I think, is
enhanced.”
77
Overall, participants expressed a lot of ambivalence about the EMR; while some of the promised
benefits were starting to be realized, there was certainly a feeling that the implementation was
much more difficult than anticipated.
“I want to go on record saying I hate the computer…I hate the tablet, which everybody else
seems to have. It makes mistakes like crazy and I spend more time correcting it. It was
wonderful if I needed a summation to send the patient to the hospital, fantastic retrieval
within a couple of minutes and I see the benefits. Once it is on the computer, once the
integration is loaded it is a remarkable tool. Hopefully once we have connectivity with the
hospital..”
“ His first sentence is I hate it and then too I love it (laugh)”
In conclusion, participants identified several barriers to EMR transition; amongst those, key
factor were the lack of technical support and system immaturity, and the lack of training (general
computer use as well as EMR specific training). Physicians were aware of but not well prepared
for the large time investment and additional funds needed for data transfer from paper to EMR
charts, leading to a mismatch between expectations and reality. Participants felt that age and IT
skills affected their ability to implement the EMR.
Some factors mitigated the difficulties, such as the presence of a champion. Physicians were still
hopeful about the benefits of EMR at this early stage of the transition, and often adopted the
strategic position that a certain number of years left to practice was needed to realize the
benefits. Some benefits, such as a well organized chart, positive patient reaction, or the ability to
extract some data were perceived to be present at this early stage.
78
Chapter 6 : Integrating theory, qualitative and quantitative results
We used mixed methods for this study of EMR implementation. Mixed methods studies
integrate different approaches, and seek to give a more complete assessment of the issues;168 this
may be particularly relevant to the implementation of health information technology,169 due to its
highly complex nature. In this chapter, we reflect on the theoretical concepts examined earlier
and integrate our qualitative and quantitative results.
In our qualitative findings, physicians perceived several factors as influencing EMR
implementation. They noted a low relative advantage during EMR implementation as
compared to the paper records previously used. This was described as a decrease in overall
efficiency, because of the substantial amount of time being spent on implementation. There were
technological barriers such as lack of interoperability and infrastructure failures interfering with
efficiency. There was also an unexpected increase in costs, and a lack of immediacy of reward
for the effort expended. Physicians felt that the relative advantage was greater for younger
physicians; perhaps younger age was a proxy for greater familiarity with computers, and
therefore greater efficiency. Younger physicians also have a longer time to benefit from possible
future efficiencies. However, in this study, the median year of graduation was 1978 for EMR
non-users, and was 1976 for users, indicating little differences between these two groups.
Physicians felt that relative advantage differed by staff role: it may have been greater for their
front office staff, but less so for the nurses and physicians.
There were, however, some perceived relative advantages to the EMR during implementation.
Physicians described some gains in efficiency after an initial decrease due to data entry,
specifically for prescription refills and generating consultation letters. As well, there was an
increase in efficiency for several administrative processes managed by the front staff. There
was a perception that relative advantage would improve over time, as more data were added and
the system became routinized.
79
There was limited compatibility, as implementers described a poor fit with most of their needs
and past experiences. For example, users described the need for ongoing and timely training
during implementation to address the multiple problems that arose; physicians felt that this was
not provided. A physician described a poor fit with his needs, as he could not efficiently enter
data due to an inability to type. Physicians also described limited past experiences and
familiarity with general computer use. However, there was some compatibility with values:
physicians felt that the quality of charting was improved: records were more legible and some
participants stated that they completed charts in a timelier manner. The chart was better
organized and patient data were easier to find. Physicians were able to generate some reports for
their practice; they felt that this would lead to improved quality of care. Physician perceptions of
patient reactions were mixed: while some physicians felt that patients approved of their use of
the new technology, others felt that the EMR interfered with their ability to interact with patients.
Observability was not mentioned during the focus groups. There was a high degree of
perceived complexity during implementation; the system was seen as difficult to learn and use.
Reasons mentioned for this were lack of software flexibility and difficulties with hardware
management. Initial expectations of usability were not met during this implementation.
The new system provided little in the way of a roadmap to implementation, and was likely a
radical departure from usual routine; physicians felt they did not have the knowledge, training or
assistance they needed to successfully reinvent the EMR or their practices. Implementation was
particularly challenging at early stages, during which time the organization restructures to adapt
to the EMR and simultaneously reinvents the EMR. There may not have been sufficient
organizational investment to ensure a smooth integration of the system into office routines.
There were some successes, as physicians described being able to routinely generate patient
summaries and retrieve information once sufficient data had been entered.
Physicians in one group described the presence of a champion; this was viewed as being a
significant facilitator. There was a perceived lack of leadership and support at the system level
to assist with implementation activities, described as difficulties with connectivity and a lack of
help with integration with other IT systems. At the practice organizational level, physicians did
not feel that they had sufficient capacity (or organizational slack) to enable them to learn and
test the new skills needed to effectively use the technology—in fact, they described being highly
80
stressed by the large additional time demands caused by data entry into the new EMR. This lack
of slack may have affected perceptions of complexity and compatibility as the EMR was being
implemented: if an innovation is initially viewed as being radical, it may be difficult to change
that perception without training and resources dedicated to implementation. For example, some
physicians mentioned that their familiarity with basic computer skills was lacking; however, they
did not appear to avail themselves of additional training—perhaps due to lack of time.
The fact that there continued to be some degree of perceived relative advantage and
compatibility in terms of improved quality of care, selected office efficiencies, future benefits
and some positive patient reactions to the EMR may help to explain why this group did not
discontinue EMR despite the many difficulties.
The experiences described in our qualitative results may provide some insights into our
quantitative results. We found that EMR implementation was not associated with an increase in
the provision of preventive services. This could be explained by the fact that there were multiple
perceived barriers to implementation. In other words, the simple presence of an EMR may be
insufficient. It is possible that effective implementation of the system may first be needed in
order to then produce a change in processes resulting in improved outcomes.
Our results show divergence in services between the two EMR-based physician groups using the
same software by the second year of implementation. There was a difference in processes:
physicians in FHN2 had sent reminder letters to patients overdue for a service, while physicians
in FHN1 had not. This was due to an internal decision to implement several EMR processes to
improve the provision of preventive services as detailed in the case study (Appendix A).
Consistent with theory, the change in processes appears to have occurred in a group with a
champion to drive the preventive project as well as some organizational “slack”: funds and
resources such as an administrator and data entry clerks were invested in this project.
Our qualitative and quantitative results reflect some of the theoretical propositions discussed
earlier. To summarize, EMR implementation in these small family practices was associated with
low relative advantage especially early in the implementation process and high complexity.
Views on compatibility were inconsistent. The EMR was perceived as being a relatively
inflexible innovation; technological barriers were prevalent. There was a lack of resources, time
or training to devote to implementation activities and address barriers, resulting in limited
81
change. The presence of a champion was perceived by colleagues to affect some implementation
processes; there was a perceived lack of leadership at a system level to address the lack of
interoperability. Due to these factors, practices had limited success in restructuring both the
EMR and their workflows to take advantage of the system’s potential.
Some of the theoretical factors discussed in chapter 2 were not addressed here; these include
perceptions, degree of control and influences of non-physician practice team members;
perceptions of attributes over time and correlation with specific stages of implementation (the
focus groups were conducted at a single point in time, approximately two years into
implementation); individual adopter categories and their perceptions and influences on
implementation. It is also possible that EMR implementation will be perceived as less radical
over time, as more primary care groups use these systems. Procedures, training and support for
implementation activities may become more routinized; technological barriers may decrease if
the systems mature and interconnectivity increases. Further studies of EMR implementation
should consider this wider context.
82
Chapter 7 : Discussion and Conclusions
Discussion
We found that the implementation of EMRs was not associated with an improvement in the
provision of the selected preventive services for the group of physicians we studied. There was
no difference in the change of service provision between physicians using EMR and those using
paper-based records. This mirrors other studies of EMR implementation and quality of care.23, 31,
170 A longitudinal study of diabetes care found no differences in A1c or LDL improvement
between practices using EMR (after two or four years of EMR implementation) and practices
using paper records.23 A systematic review identified no improvement in care when electronic
guideline implementation systems were used in ambulatory care as compared to paper-based
reminder systems.170 Another systematic review found that very few studies evaluated standard
EMRs in community-based care. The studies evaluating such software did not find any
consistent improvement in quality of care: there was a decrease in radiology services in one
study, which was not found in another study and there was no difference in care for depression.31
Poon found no relationship between the use or non-use of EMR and health care quality, but did
find a relationship between some quality measures (e.g. cancer prevention measures) and the
implementation of some EMR functions such as the use of an EMR-based problem list.171 The
lack of improvement may be due to the challenges encountered during EMR implementation.
Previous studies have shown limited use of more advanced EMR functions in individual
practices,25, 172, 173 and no improvement of care as additional experience with EMR accrues over
time.23, 25, 27
It may be very difficult and expensive for small practices using EMRs to implement the many
changes required to improve performance. For example, Halladay found substantial costs
associated with the reporting of performance data in primary care; these were related to data
entry, staff training and IT modification and maintenance.174 The highest costs were incurred by
small practices using EMR, as those physicians spent more than their counterparts practicing in
larger groups for assistance with EMR software modifications. Physicians in small practices also
needed more time for set-up and data gathering .174 Some conditions that may enable the
implementation of EMR-based quality improvement strategies in small practices may not be
83
widespread: physicians may not have time to modify their practice,175 there may be no local
EMR champion, EMR software systems with the flexibility to easily and accurately measure
service provision may not be available,176 and technical assistance to overcome software and
hardware problems may not be easily accessed.85
In this study, we categorized barriers to implementation as a lack of compatibility with
physicians’ needs and prior experiences, high system complexity, increased workload and
unexpected additional costs, delayed benefits, technological barriers and lack of ongoing training
and education. Facilitators were the availability of an EMR champion in one group, some
increases in efficiency, perceptions of eventual pay-back for the effort invested, some positive
patient reactions and perceptions of improvements in quality. These are broadly similar to
findings for small primary care practices described in the literature. Terry and colleagues used
qualitative methods to study small primary care offices implementing EMRs in south western
Ontario. Participants found that the time required for implementation was far greater than
expected; prior expectations of usability were not met; training was an important factor; and the
presence of a champion helped with implementation.96 A qualitative study of innovators and
early adopters of EMRs in small community practices in California177 found that initial costs
were higher than expected, with benefits not always being realized; physicians felt that the EMR
led to increased quality of care; the distribution of benefits was uneven with super-users
benefiting the most; and the presence of a champion was critical to implementation.177 Another
study178 found that several barriers to EMR implementation in community practices were
present: high initial costs, additional time requirements and immaturity of the technology,
difficulties with the ability to customize and adapt the EMR, inadequate interoperability with
external data sources and differing physician attitudes towards the EMR.178 A recent review of
studies on barriers to EMR implementation179 found that these could be broadly categorized as
concerns about costs, technical issues (including lack of interconnectivity, high complexity, and
lack of customizability), lack of time, psychological factors such as lack of belief in EMRs,
social factors such as a lack of support from colleagues, legal issues such as concerns over
privacy and security, differing organizational size and type (hospital vs community practice) and
difficulties with change management.179
The difficulties we found with EMR implementation led to problems with reinventing workflows
to take advantage of the new technology; these may be ultimately mitigated over time. We
84
found some increase in preventive services in the EMR cohort by the second year of
implementation, but this had not reached baseline levels. However, in a cross-sectional study,
Zhou et al found no association between the duration of EMR usage and performance,25
suggesting that EMR-based process improvements may not necessarily occur over time.
It is possible that performance improvement depends on careful implementation, reinvention and
integration of different aspects of the EMR, such as clinical decision support systems or point of
care alerts, into routine patient care and practice workflows. Physicians in this study perceived
that this integration was challenging. In a randomized controlled trial, Eccles found that a
computerized decision support system for asthma or angina embedded into the EMRs of fully
computerized primary care practices had no effect on care as measured by adherence to the
guidelines.22 A companion qualitative analysis found that the computerized system did not fit
well within the context of family practices; it was too difficult to use, it did not always present
relevant information, and there were too many alerts.180 It could be that integration into routine
care was challenging in that study, possibly leading to a lack of effectiveness of the intervention.
Baron described the implementation of a mammography recall program within an innovative,
fully computerized primary care group practice.181 The system was initially unable to properly
audit mammograms and to produce accurate lists of patients to be recalled; mammograms were
scanned in but were not recognized by the EMR. As Baron described, the change first required
awareness that this was an issue, then a decision to use new processes, then development and
implementation of a change in workflow to “tag” incoming mammograms so that patients could
be properly categorized as having or not having had a mammogram within the previous two
years.181 This process of “awareness to decision to implementation” has been described in the
literature.87 Although the authors do not describe the factors underlying the successful change
process, some of the theoretical factors discussed in chapter 2 (Figure 1) could be used to
complement and explain similar case studies in primary care.
In this study, physicians in the EMR cohort who did not use the EMR to record encounters had a
greater decrease in preventive services. As an explanation for this finding, it is possible that
those physicians may not have been aware of a point of care alert if the EMR was not used
during encounters. However, in a cross-sectional study, Keyhani and colleagues182 found no
association between implementation of various EMR components and functions (such as
85
electronic physician notes or reminder systems) and quality measures for the management of
chronic health conditions.182 A meta-analysis of point-of-care computerized reminders found
that these produced small effects (median 4.2% improvement).183 The authors commented on
the need to identify features that reliably predict improvement,183 but the process of
implementation and the ways in which providers use the EMR may be just as important as the
software. While we do not have data on what aspects of the EMR beyond recording encounters
were implemented by this group, it is possible to speculate that the receipt of electronic lab
results, scanning of documents or some data entry into health profiles may have occurred. This
would disperse data across paper records and EMR and may negatively affect the physician’s
ability to find needed information (such as the date of a last Pap smear) during clinical
encounters.
We found that preventive services changed at a similar rate in the first year of computerization
for the two groups of physicians implementing EMR. However, there was a difference in the
second year (2007), with FHN2 having a greater increase. By that year, FHN2 had implemented
an organized audit, recall and point-of-care reminder program, as detailed in our case study
(Appendix A) and the group was collectively mailing reminder letters. All physicians were
mailing letters, regardless of their stage of EMR implementation. There is a fee ($6.86 per
patient) for contacting patients who are overdue for a P4P preventive service. A recent
population-based study of Ontario primary care groups found that very few fee codes for this
contact service had been submitted and that the rate of submission was not increasing.184
Several of the processes in FHN2, such as chart audits and reminder letters, were implemented at
the group rather than the individual level. It is possible that collective actions may overcome
some of the challenges and costs that Halladay found at the small individual practice level.174
Our chart audits found a decrease in the proportion of Pap smears and mammograms in the EMR
cohort associated with early EMR implementation (2006), but there was no decline found in the
administrative data. Chart audits have traditionally been considered as the “gold standard” for
certain medical services, and have been used to validate administrative data.149, 151 The
administrative data we used to determine the rates of preventive services have not been validated
and cannot be used as an accurate estimate of the rates of preventive services. However, chart
audits were previously done on paper and EMR audits represent a new field of research with its
86
own challenges. EMR-based data may need to be validated. Good data may not always be
available or be readily accessible from the EMR. For example, Roth185 found that only a third
of the indicators needed for a quality assessment program could be easily extracted from EMRs,
and that there were difficulties associated with provider data entry habits and differences across
different EMR applications.185 The structure of the EMR is more complex than that of the paper
chart: physicians may not be entering data in consistent or expected locations, making it difficult
to extract.185 Physicians and auditors may have challenges in navigating the chart. Data from
external sources may be scanned in and may not be extractable electronically.181 Physicians may
continue to use both paper charts and EMR,35 scattering data across two different systems and
possibly increasing the amount of incomplete or duplicated data in audits. Research and quality
improvement projects using EMR data will need to consider the quality of data entered in the
EMR, as well as issues specific to the EMR application used.185
While there is uncertainty regarding administrative data, we studied the change in the proportion
of services provided, rather than the actual proportion provided. It is possible that administrative
data were less sensitive to variations in data quality and availability than EMR audits during the
transition. That is, there is no reason to suspect that the proportion of services varied from year
to year: for example, if 75% of influenza vaccinations were available in administrative data in
2006, we would expect that approximately 75% would be available in 2007. Our inclusion and
exclusion criteria for administrative data were held constant from year to year, as detailed in
Appendix G.
Administrative data for Pap smears were based on billing codes submitted by laboratories and
physician billing codes as detailed in Appendix G. Laboratory billings would not have been
affected by EMR implementation. It is possible that physicians use different workflows for
billing (example, day sheet lists used for billing) and for entering data in the chart. Physician
billings and chart entry may be affected in different ways by EMR implementation, although our
study does not provide information on this.
Administrative data for mammograms were based on radiology billing codes and data from the
Ontario Breast Screening Program, as shown in Appendix G. These were independent of EMR
implementation.
87
In our context, it is possible that administrative data suggest year over year changes in service
provision for Pap smears and mammograms while chart audits may suggest changes in the
availability of data documented in the EMR. Thus, it is possible that the decrease in Pap smear
and mammograms found in chart audits (but not administrative data) for the first year of EMR
implementation represent difficulties with EMR data entry rather than a true decline in service
provision.
The change in influenza vaccinations was similar in chart audits and in administrative data,
perhaps reflecting fewer problems with documentation. Documenting an influenza vaccination
does not require looking two years back for the presence of the service, as mammography or Pap
smears do; therefore, there may be a less complex workflow associated with recording this
service during the transition to EMR. Our chart audits showed a consistently higher provision of
influenza vaccinations than administrative data. Administrative data for influenza vaccines are
based on physician billings. They have limited sensitivity as they do not capture vaccinations
given in settings where the service is not billed such as pharmacies.186 Previous studies on
EMRs and documentation of influenza vaccinations have shown high specificity and lower
sensitivity, with most of the missing vaccines having been administered at sites other than where
the EMR record was kept.187, 188 Physicians in this study received P4P incentives for their rates of
influenza vaccinations (regardless of who provided the service), and may therefore have
recorded this service in the chart if a patient informed them that an influenza vaccination was
received elsewhere. This may have led to higher service provision levels in chart audits when
compared to administrative data. There were delays in vaccine delivery in 2006 and 2007, which
could account for the lower levels of vaccination found in both chart audits and administrative
data during the fall season of those two years.
The quality of information (accuracy, reliability, completeness) has been found to be associated
with empirical measures of success in implementing IT in the business literature.189 It is possible
that poorer information may make the system less usable and less useful, impacting
implementation efforts and decreasing the net benefits that could be obtained from the
techonology.190
Measuring performance depends on accurate documentation.2, 191, 192 Once accurate data have
been entered into the EMR, interventions that have been found to increase performance, such as
88
audits and feedback to clinicians,193, 194 point of care prompts for needed interventions,13, 79, 194
and reminder letters to patients80, 159 can then be implemented. Our quantitative results show a
lack of improvement in preventive service provision associated with early stages of EMR
implementation. It is possible that elements of those negative results were due to problems with
EMR data entry and data quality.
Limitations
This study was limited to a group of selected physicians in Toronto and as such, the findings may
not be generalizable to practices outside of this area, especially rural settings. However, all
physicians in this study were practicing in community-based settings, and many were in solo or
two physician offices—similar to the majority of family physicians in Ontario.47 19% of paper-
based physicians responded to the study invitation, and we do not know if their characteristics
differed from the non-responders.
The characteristics of physicians and patients in FHNs (Capitation) and FHGs (Enhanced fee-
for-service) in Ontario are shown in Appendix I (Table 18 and Table 19).132 The physicians in
the EMR cohort were members of a FHN, and those in the non-EMR cohort were members of a
FHG. Compared with their colleagues in major Ontario urban centres,132 physicians in our
cohorts had higher rates of some characteristics that may be associated with fewer preventive
services: being in practice for longer, having larger numbers of patients, and having patients
with higher morbidity and co-morbidity levels.133, 136, 138, 139, 144 They also had higher rates of
some characteristics that may be associated with more preventive services: fewer foreign
graduates and greater percentage of patients in the upper income stratum.75, 136, 141, 142 Glazier
found that FHG physicians looked after patients with greater morbidity and co-morbidity levels
than their FHN colleagues,132 similar to what was found in our study. Physicians in this study
were reasonably similar to their colleagues in Ontario urban centres.
The physicians in our EMR cohort initially provided preventive services to a very high
proportion of their eligible patients, and may therefore not be representative of the general
physician population with respect to these measures.75, 195 It is possible that physicians more
focused on quality may be more likely to self-select as earlier EMR adopters, since EMRs may
89
be viewed as a way to maintain or further increase quality. Physician attitude towards both
quality of care and EMRs could be an unmeasured confounder.
We studied a single EMR system; results may differ for physicians using other EMR systems.
This was an observational cohort study, and is therefore subject to both measured and
unmeasured confounders. In this study, we measured confounders that have been reported in the
literature to affect preventive services75, 133-144, 163 and used statistical adjustments to control for
some of these factors. In addition to chart audits, we also used administrative data to assess
differences between groups. However, these could not be linked at the patient level, and
therefore could not be used for adjustment. The groups mainly differed with regards to levels of
morbidity and co-morbidity. However, higher levels of morbidity and co-morbidity were found
in groups that had older patients and, therefore, may be related to patient age; we adjusted results
for patient age.
Physicians in our non-EMR cohort were introduced to incentives in 2007; pay for performance
incentives may increase the provision of preventive services59, 66, 184 and may have confounded
our results. However, the incentives payment was introduced as of April 2007 for targets
reached in the prior year, meaning that these physicians knew in fiscal 2006 that they would be
eligible for payments and may have changed their practices earlier in anticipation of the expected
payment. A recent Ontario population-based study184 found that physicians exposed to P4P had
greater rates of increases in Pap smears, mammograms and colorectal cancer screening when
compared to physicians not exposed to P4P. However, there were no significant differences
between the two groups for influenza vaccination or child vaccinations. For the majority of
physicians, the payment started in 2007, but the difference in the change in performance started
in 2006.184
In this study, the non-EMR physicians had higher rates of some personal and practice
characteristics possibly associated with less provision of preventive services (more male
physicians, more recent immigrants, more morbidities and co-morbidities)75, 134, 135, 142, 144 than
their EMR colleagues, along with some factors possibly associated with greater provision of
services (more CCFPs, fewer years in practice).133, 136, 137 We adjusted for physician gender,
CCFP status and years of practice, as well as patient age. We could not adjust for factors which
were not collected at the patient level, such as patient morbidities, co-morbidities or immigration
90
recency. However, greater morbidity and co-morbidity levels may have been related to higher
patient age, which we adjusted for. We used similar methods to adjust when comparing sub-
groups.
The group of physicians that had EMR, but did not use it by 18 months, had characteristics
possibly associated with fewer preventive services: there were fewer female physicians, fewer
CCFPs, and practices with more older patients with greater morbidity and co-morbidity levels.134,
135, 137, 141, 144 The non-user group also had some characteristics possibly associated with the
provision of more services, such as fewer years in practice and smaller practices.133, 138, 139 We
compared two groups of physicians using EMR. FHN1 physicians had some characteristics
possibly associated with more provision of preventive services (working in larger groups, more
CCFPs, and practices with younger patients with lower morbidity and co-morbidity levels).136,
137, 141, 144 They also had characteristics possibly associated with fewer preventive services, such
as more years in practice, fewer female physicians, larger size of practice, and fewer patients in
the upper income stratum.75, 133-135, 138, 139, 142, 143
We were limited to only two years of chart audits for the paper-based group; we used
administrative data to address this. We audited very small numbers of children’s medical
records in the EMR group due to the limited number of children in each practice, and were
unable to audit children’s vaccines in the parallel cohort. Administrative data on the children’s
vaccinations were not available. Due to the limitations in data collection methods, we used
different composite process scores for different comparisons. We did not have administrative
data to compare service provision between the two FHNs or between the EMR users and non
users.
There were differences between our two cohorts in terms of physician funding mechanisms.
Physicians may have self-selected on the basis of their preferences for either payment system and
these preferences may be associated with unmeasured differences in their attitudes towards
preventive care. However, a recent study using administrative data did not find any consistent
difference in the provision of preventive services between physicians in FHNs and FHGs.196 The
provision of Pap smears and mammograms did not change after joining either group; FOBT
performance improved more in FHNs than in FHGs.196 In our study, administrative data showed
little difference between the EMR group (FHN) and the non-EMR group (FHG) with respect to
91
influenza vaccinations and mammograms. The EMR group provided more Pap smears and
FOBTs. The difference in funding mechanism between the group using EMR and the group
using paper records may not have affected the change in the provision of preventive services.
A large number of Ontario physicians switched from reformed fee-for-service to capitated
payment methods during the years studied; we could not obtain longitudinal population level
administrative data comparing preventive services between FHGs and FHNs due to this
instability, although a recent study on P4P provides some of these data.184 Some physicians
changed funding group in the paper-based cohort; however, a reanalysis of administrative data
including only physicians in a FHG in each year did not show any differences. Changing to a
different group may not make an initial difference in service provision.196 Physicians in the
paper-based group were inconsistently exposed to P4P due to changes in groups, and we could
not adjust for these differences.
Delays in the delivery of influenza vaccines to family practices in 2006 and 2007 likely affected
our results. However, the effect was similar for all physicians studied.
Our qualitative data were limited to two focus groups held at one point in time during
implementation. To avoid leading questions, we did not ask directly about the impact of EMR
on preventive services. It would have been valuable to explore any changes in documentation of
these services and to collect data on participant use or non use of EMR during the focus groups.
However, we were limited due to the concurrent qualitative and quantitative design of this study.
Policy suggestions Government can influence behaviour through funding and/or regulations.197 Current subsidies in
Ontario appear to target the adoption of EMR systems by funding their purchase, rather than
their implementation into daily practice. Policies could address the disruption indicated in this
study during the initial stages of implementation, support the work of electronic data entry, and
promote the implementation of advanced EMR functions that have been shown to improve the
quality of care provided to patients.2, 21, 191, 193 For example, American physicians demonstrating
“meaningful use” of EMRs, which includes recording structured data and submitting information
on quality improvement measures and care coordination, are slated to receive financial
incentives as part of the American Recovery and Reinvestment Act.4
92
Improving practice IT infrastructure
Qualitative results from this study as well as others85, 96 178 point to weaknesses in IT
infrastructure (hardware and connectivity failures) and IT management, leading to difficulties
during implementation. Physicians perceived that they were responsible for overseeing and
managing these complex IT systems in their practices. Reallocating some adoption funding
towards IT support may be of benefit. Possible strategies could include allocating funds for
professional IT managers for groups of physicians, funding hardware management (ongoing
maintenance and upgrades of computers and peripherals, server maintenance), and improving
connectivity speed and stability.
Supporting EMR implementation
Policies could be targeted at increasing the amount of electronic data and improving data quality.
Policies could also support interventions that improve practice workflows.
Findings in the literature15, 39, 41 as well as from the qualitative results of this study, point out that
the lack of system-level interconnectivity impedes implementation by interfering with practice
workflows. Policies that could promote electronic connections between practices and other
members of the health care system could include: promoting the use of forms generated through
the EMR system; adopting electronic referrals, prescriptions and laboratory requisitions;
supporting interoperability projects; decreasing regulatory barriers to interoperability; promoting
common standards so that data generated by different organizations can be shared and collated
within the EMRs and between different systems;14, 198 and monitoring the amount of scanning
needed in practices (scanning may be a proxy for poor interoperability).
Physicians noted that the bulk of training occurred just prior to EMR implementation, with
limited on-going education. Possible strategies to address this could include accrediting and
funding on-going EMR training and mandating plans for ongoing, post implementation training
as part of the EMR certification process.
Physicians also valued peer support and EMR champions. Identifying and supporting local
champions may be worthwhile; for example, Canada Health Infoway’s Peer to Peer program
funds visits and support from experienced and proficient local EMR users.199
93
Physicians in this study noted the very large time demands required for data entry. Support for
data entry in the EMR could include incentives such as supplementary payments for patient
encounters that are charted electronically and financial incentives to encourage good data quality
such as consistent coding for health conditions.
Promoting the measurement and improvement of outcomes
Evidence from the literature23, 27, 200, 201 as well as from this study shows that advanced EMR-
based tools such as audits or patient reminders for overdue services may not be routinely used in
some practices. Without such tools, quality improvement efforts cannot be fully implemented.
Incentives and policies could target important quality outcomes, through initiatives such as
extending the number of P4P outcomes that can be measured, monitored and improved once
advanced EMR-based processes are implemented, and ensuring that incentives adequately
reward the work done.174 The Quality and Outcomes Framework (QOF) in the U.K. currently
comprises about 35% of family physicians’ incomes and requires reporting on a very large
number of outcomes.202 The program has been associated with consistent annual
improvements.59, 203 The QOF initiative strongly encourages the use of electronic systems and
tools to efficiently record data and achieve improvements in outcomes.202
As well, financial support could be targeted towards implementing registries (such as the Ontario
Diabetes registry)204 and including additional chronic disease registries. The generation and
maintenance of registries requires consistent, accurate and complete EMR data entry habits and
processes.
Practice suggestions Our results suggest that improvements in the provision of preventive services may not be
associated with the implementation of EMRs. A possible reason for this may be difficulties with
data quality in the EMR record.
Physicians planning to implement EMRs may wish to consider data entry issues from the outset.
Assistance from the EMR vendor and from experienced colleagues may be of value. Issues to
consider may include: consistency of data entry (coding when appropriate, entering data in
consistent areas, workflows related to data entry for scanned documents); completeness of data
94
entry (medications, avoiding non-electronic external sources if an electronic source is available);
and accuracy of data entry.
Practicing physicians may wish to consider some of the policy suggestions outlined earlier.
Hiring an IT professional and investing in good quality hardware may help with some of the
technical issues that were problematic. Systematically informing and redirecting patients away
from non-electronic external data providers such as non-electronic labs may be possible.
Scheduling regular, ongoing EMR training as well as training in ancillary IT skill such as
keyboarding (if needed) may be of value. Physicians may wish to hire qualified personnel to
assist with data entry to reduce the time demands they face with initial EMR implementation.
Several of these suggestions have costs in terms of money or time; physicians could periodically
review costs and benefits of various interventions.
In this study, a group of physicians used collective resources to manage preventive services and
this was associated with the implementation of reminder letters to patients who were overdue.
Physicians may consider associating with colleagues in order to leverage group resources for
selected EMR implementation activities.
Research suggestions In this study, there was no improvement in preventive care associated with EMR
implementation. We suggest that difficulties with implementing the EMR may be an underlying
factor for this. Theory-informed interventional studies could address some of the barriers and
facilitators to EMR implementation and could measure the effect of interventions on both
implementation of various aspects of the EMR and on the quality of care provided to patients.
For example, a study could add personnel specifically trained and tasked with furthering
implementation of various aspects of the EMR, therefore supporting reinvention and adding
slack. A recently funded study (the BETTER project) in Toronto and Calgary has randomly
allocated Practice Facilitators to help promote improved use of EMRs; the study will measure
chronic disease screening and prevention (current controlled trial number ISRCTN07170460).
Our results also suggest that problems with data quality in the EMR may be contributing to the
lack of improvement in services. Studies could help discover and quantify common difficulties
with data entry and could assess the feasibility, costs and impact of using different interventions
95
and incentives to correct problems. Research is also needed to validate data abstracted from
EMRs. It cannot be assumed that data obtained from EMR audits is identical to data from paper
charts.
Research identifying and targeting the stage of EMR implementation and the implementer
categories may be of value. Interventions targeting some quality improvement activities (such as
electronic audits and point of care alerts) may need to selectively enroll physicians who have
already entered structured data relevant to the intervention in the EMR. Given the limited
number of physicians reporting full use of EMR functions34, 172 and the high current proportion
of hybrid practices in the Canadian setting,32 these interventions may be targeting a minority of
physicians using EMR at the present time. Researchers may consider addressing the needs
physicians at different stages of implementation.
Interventions targeted at physicians at earlier stages of implementation may first need to address
EMR processes that may be problematic, such as workflow changes, data quality or IT support.
Interventions targeting physician performance during early EMR implementation may be less
effective. Interventional studies measuring patient outcomes using EMR during the transition
may need to address possible difficulties with chart documentation. Studies defining the
duration of the transition for various aspects of the EMR may be helpful.
Several non-users in our EMR cohort shared offices with colleagues who were actively
implementing the EMR; it is possible that the differing workflows within practices may have
exacerbated the implementation difficulties. The effect of having implementers at different
stages within the same setting could be explored further.
Our case study and exploratory analysis of the differences in preventive services between the two
FHNs indicates that some improvements may be possible through interventions at a group level.
The changes appeared to occur in the presence of an incomplete transition at the physician level
(four out of nine physicians in FHN2 were not entering encounters in the EMR by 18 months),
and may warrant further study. Some of the electronic tools used in our case study, such as
shared databases used by groups of physicians, make data entry and management at the group
level rather than at the individual physician level possible. Quality improvement initiatives and
research targeting larger primary care groups or organizations using a common EMR database
may represent a new avenue for knowledge translation efforts using EMR technology.
96
Finally, research in small community-based primary care practices using commercial off-the-
shelf EMRs is feasible. A large proportion of patient care occurs in these small practices,
improving generalizability. Studies pointing the way to improved EMR data quality, consistency
and quantity could impact the ability of clinicians to use their data for quality improvement
projects; as well, this could improve the availability and ease of data extraction from primary
care practices for research and surveillance projects.
Conclusions
In this group of practices, the implementation of EMRs was not associated with an increase in
the provision of the preventive services targeted by Ontario’s pay for performance incentive
when compared to the continued use of paper records. There was no improvement within the
group of physicians using EMR, and there may have been some difficulties with data entry for
these preventive services in the EMR during initial implementation. Physicians who had EMR
but were not using it to record encounters by 18 months may have had a greater degree of
decrease in preventive services than those who were using EMR for encounters. There appeared
to be a greater degree of increase in preventive services in the second year of EMR
implementation in one group of physicians using EMR; this group was mailing out reminder
letters to overdue patients.
Physicians generally felt that their EMR implementation was problematic. The EMR was
perceived as having a lack of relative advantage, high complexity and low compatibility.
Physicians had difficulties with adapting the system to their practices and reinventing their
workflows to take advantage of the EMR during implementation.
In conclusion, it should not be assumed that EMRs improve care. In this study, the first two
years of EMR implementation were not associated with any improvement in the provision of
preventive services.
97
References 1. Romanow R. Building on values: the future of health care in Canada. Ottawa; 2002.
2. Chassin MR, Galvin RW, and the National Roundtable on Health Care Quality. The Urgent Need to Improve Health Care Quality: Institute of Medicine National Roundtable on Health Care Quality. JAMA 1998;280:1000-5.
3. Audet AM, Doty MM, Shamasdin J, Schoenbaum SC. Measure, learn, and improve: physicians' involvement in quality improvement. Health Aff (Millwood) 2005;24:843-53.
4. Congress USA. "American Recovery and Reinvestment Act of 2009". In. Washington: United States Government Printing Office; 2009.
5. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, D.C.: Institute of Medicine; 2001.
6. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003;348:2635-45.
7. Rogers E. Diffusion of Innovations. 4th Edition ed. New York: The Free Press; 1995.
8. Hutchison B AJ, Lavis J. Primary Care In Canada: So Much Innovation, So Little Change. Health Aff 2001.
9. To Err Is Human: Building a Safer Health System. Washington, D.C.: The Institute of Medicine; 1999.
10. Bodenheimer T. Coordinating Care -- A Perilous Journey through the Health Care System. N Engl J Med 2008;358:1064-71.
11. van Walraven C, Taljaard M, Bell C, et al. Information exchange among physicians caring for the same patient in the community. CMAJ 2008;179:1013-8.
12. The e-Volution of health care: making a difference. 2007. (Accessed September 24, 2008, at http://www.infoway-inforoute.ca/Admin/Upload/Dev/Document/Infoway_Annual_Report_2007-2008_Eng.pdf?ireff=9.)
13. Garg AX, Adhikari NKJ, McDonald H, et al. Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes: A Systematic Review. JAMA 2005;293:1223-38.
14. The diffusion and value of healthcare information technology. RAND Corporation, 2005. (Accessed July 8, 2005, at http://www.rand.org/pubs/monographs/2005/RAND_MG272.pdf.)
15. Kilo CM. Transforming Care: Medical Practice Design And Information Technology. Health Aff 2005;24:1296-301.
16. EMR Toolkit. Health Canada, 2006. (Accessed November 7, 2008, at http://www.emrtoolkit.ca/view_section.php?section=7.)
17. Protti D, Bowden T, Johansen I. Adoption of information technology in primary care physician offices in New Zealand and Denmark, Part 3: Medical record environment comparisons. Inform Prim Care 2008;16:285-90.
98
18. Protti D, Bowden T, Johansen I. Adoption of information technology in primary care physician offices in New Zealand and Denmark, Part 4: Benefits comparisons. Inform Prim Care 2008;16:291-6.
19. The Health of Canadians – The Federal Role. The Standing Senate Committee on Social Affairs, Science and Technology, 2002. (Accessed November 4, 2005, at http://www.parl.gc.ca/37/2/parlbus/commbus/senate/com-e/SOCI-E/rep-e/repoct02vol6-e.htm.)
20. First Ministers' Accord on Health Care Renewal. 2003. (Accessed October 15, 2005, at http://www.hc-sc.gc.ca/hcs-sss/medi-assur/fptcollab/2003accord/index_e.html.)
21. Jha AK, Perlin JB, Kizer KW, Dudley RA. Effect of the transformation of the Veterans Affairs Health Care System on the quality of care. N Engl J Med 2003;348:2218-27.
22. Eccles M, McColl E, Steen N, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002;325:941.
23. O'Connor PJ, Crain AL, Rush WA, Sperl-Hillen JM, Gutenkauf JJ, Duncan JE. Impact of an electronic medical record on diabetes quality of care. Ann Fam Med 2005;3:300-6.
24. Linder JA, Ma J, Bates DW, Middleton B, Stafford RS. Electronic Health Record Use and the Quality of Ambulatory Care in the United States. Arch Intern Med 2007;167:1400-5.
25. Zhou L, Soran CS, Jenter CA, et al. The Relationship between Electronic Health Record Use and Quality of Care over Time. J Am Med Inform Assoc 2009;16:457-64.
26. Gordon JRS, Wahls T, Carlos RC, Pipinos II, Rosenthal GE, Cram P. Failure to Recognize Newly Identified Aortic Dilations in a Health Care System With an Advanced Electronic Medical Record. Annals of Internal Medicine 2009;151:21-7.
27. Crosson JC, Ohman-Strickland PA, Hahn KA, et al. Electronic medical records and diabetes quality of care: results from a sample of family medicine practices. Ann Fam Med 2007;5:209-15.
28. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. Jama 2005;293:1197-203.
29. Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected Increased Mortality After Implementation of a Commercially Sold Computerized Physician Order Entry System. Pediatrics 2005;116:1506-12.
30. Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980-97. Bmj 2001;322:279-82.
31. Chaudhry B, Wang J, Wu S, et al. Systematic Review: Impact of Health Information Technology on Quality, Efficiency, and Costs of Medical Care. Ann Intern Med 2006;144:742-52.
32. National Physician Survey: Information Technology in the main patient care setting. 2004. (Accessed April 29, 2005, at ww.cfpc.ca/nps/English/pdf/Physicians/Family_Physicians/Tables/National/Q23.pdf.)
33. Canada health infoway: IT in general practice: A 10-country comparison. 2007. (Accessed February 2, 2007, at http://www.cma.ca/index.cfm/ci_id/49047/la_id/1.htm.)
99
34. DesRoches CM, Campbell EG, Rao SR, et al. Electronic Health Records in Ambulatory Care -- A National Survey of Physicians. N Engl J Med 2008;359:50-60.
35. National Physician Survey. Canadian Medical Association, 2007. (Accessed March 30, 2008, at http://www.nationalphysiciansurvey.ca/nps/2007_Survey/2007results-e.asp.)
36. Valdes I, Kibbe DC, Tolleson G, Kunik ME, Petersen LA. Barriers to proliferation of electronic medical records. Inform Prim Care 2004;12:3-9.
37. Loomis GA, Ries JS, Saywell RM, Jr., Thakker NR. If electronic medical records are so great, why aren't family physicians using them? J Fam Pract 2002;51:636-41.
38. Audet AM, Doty MM, Peugh J, Shamasdin J, Zapert K, Schoenbaum S. Information technologies: when will they make it into physicians' black bags? MedGenMed 2004;6:2.
39. Health Information Technology: Can HIT Lower Costs and Improve Quality? RAND Corporation, 2005. (Accessed November 20, 2005, at http://www.rand.org/publications/RB/RB9136/.)
40. Have paper records passed their expiry date? CMAJ 2005;173:725.
41. Baron RJ, Fabens EL, Schiffman M, Wolf E. Electronic Health Records: Just around the Corner? Or over the Cliff? Ann Intern Med 2005;143:222-6.
42. Miller RH, West C, Brown TM, Sim I, Ganchoff C. The Value Of Electronic Health Records In Solo Or Small Group Practices. Health Aff 2005;24:1127-37.
43. Request for proposals ("RFP") for ePhysician Project, clinical management systems (appication service provider model) RFP no. 2002-01 (amended). Ministry of Health and Long Term Care, Ontario, 2002. (Accessed September 28th, 2008, at http://hdl.handle.net/1873/5352.)
44. Physician IT program: services. Ontario Medical Association, 2005. (Accessed November 27, 2005, at http://www.ontariomd.com/en/services/physicianIT.jsp.)
45. Physician IT Program: Put Physician IT to work. For your practice, your patients and you. Government of Ontario, 2005. (Accessed November 27, 2005, at http://www.health.gov.on.ca/english/providers/program/physician_it/physicianit_mn.html.)
46. Manuel DG, Maaten S, Thiruchelvam D, Jaakkimainen L, Upshur R. Primary Care in the health care system. In: Jaakkimainen L UR, Klein-Geltink JE, Leong A, Maaten S, Schultz SE, Wang L, ed. Primary Care in Ontario: ICES Atlas. Toronto: Institure for Clinical Evaluative Sciences; 2006.
47. Schultz S.E TJ, Guttmann A, Jaakkimainen L. Characteristics of Primary Care Practice. In: Jaakkimainen L UR, Klein-Geltink JE, Leong A, Maaten S, Schultz SE, Wang L, ed. Primary Care in Ontario: ICES Atlas. Toronto: Institute for Clinical Evaluative Sciences; 2006.
48. Maioni A. From Cinderalla to Belle of the Ball: the politics of primary care reform in Canada. In: R W, ed. Implementing primary care reform: barriers and facilitators. Kingston: Queen's University; 2004.
49. Information kit: your Family Health Network. In. Toronto: Ministry of Health and Long Term care, Queen's printer for Ontario; 2002.
50. Deadline for physicians to apply for CMS funding has passed. OntarioMD, 2008. (Accessed December 15, 2008, at https://www.ontariomd.ca/imageserver/OMDContent/tsp/physITDeadline.html.)
100
51. Memorandum of agreement between the Ontario Medical Association and the Ontario Ministry of Health and Long Term Care. In. Toronto: Government of Ontario; 2003.
52. Keshavjee K, Troyan S, Holbrook AM, VanderMolen D. Measuring the success of electronic medical record implementation using electronic and survey data. Proc AMIA Symp 2001:309-13.
53. Greenhalgh T, Glenn R, Bate P, Macfarlane F, Kyriakidou O. Diffusion of innovations in health services organizations. Malden: Blackwell publishing Ltd; 2005.
54. The Standish Group: Chaos. West Yarmouth, Massachussetts; 1995.
55. US Preventive Services Task Force. Agency for Health Care Research and Quality. (Accessed July 8, 2007, at http://www.ahrq.gov/clinic/uspstfix.htm.)
56. Canadian Task Force on Preventive Health Care. (Accessed July 8, 2007, at http://www.ctfphc.org/.)
57. Dubey V, Glazier R. Preventive care checklist form. Evidence-based tool to improve preventive health care during complete health assessment of adults. Can Fam Physician 2006;52:48-55.
58. Barnsley J, Berta W, Cockerill R, MacPhail J, Vayda E. Identifying performance indicators for family practice: assessing levels of consensus. Can Fam Physician 2005;51:700-1.
59. Doran T, Fullwood C, Gravelle H, et al. Pay-for-performance programs in family practices in the United Kingdom. N Engl J Med 2006;355:375-84.
60. Becker MH. Factors affecting diffusion of innovations among health professionals. Am J Public Health Nations Health 1970;60:294-304.
61. Greiver M, Kasperski J, Barnsley J, Rachlis V. Measuring Quality Improvements in Preventive Care Services in the First Two Family Health Networks in the Greater Toronto Area
In. Toronto: Ontario College of Family Physicians; 2006.
62. Prochaska JO, DiClemente C. Stages and processes of self-change of smoking: toward an integrative model of change. J Consult Clin Psychol 1983;51:390-95.
63. Epstein AM. Pay for performance at the tipping point. N Engl J Med 2007;356:515-7.
64. Fisher ES. Paying for Performance -- Risks and Recommendations. N Engl J Med 2006;355:1845-7.
65. Rosenthal MB, Landon BE, Normand S-LT, Frank RG, Epstein AM. Pay for Performance in Commercial HMOs. N Engl J Med 2006;355:1895-902.
66. Rosenthal MB, Frank RG, Li Z, Epstein AM. Early Experience With Pay-for-Performance: From Concept to Practice. JAMA 2005;294:1788-93.
67. Lindenauer PK, Remus D, Roman S, et al. Public reporting and pay for performance in hospital quality improvement. N Engl J Med 2007;356:486-96.
68. Petersen LA, Woodard LD, Urech T, Daw C, Sookanan S. Does Pay-for-Performance Improve the Quality of Health Care? Ann Intern Med 2006;145:265-72.
69. Epstein AM, Lee TH, Hamel MB. Paying Physicians for High-Quality Care. N Engl J Med 2004;350:406-10.
101
70. Casalino LP, Elster A, Eisenberg A, Lewis E, Montgomery J, Ramos D. Will Pay-For-Performance And Quality Reporting Affect Health Care Disparities? Health Aff 2007;26:w405-14.
71. Family health network agreement: General blended payment template. Ministry of Health and Long Term Care of Ontario and the Ontario Family Health Network Corporation; 2002.
72. Memorandum of agreement between the Ontario Medical Association and the Ontario Ministry of Health and Long Term Care. In. Toronto: Government of Ontario; 2004.
73. Campbell MJ. Statistics at square two. London: BMJ Publishing Group; 2001.
74. Swanson JG, Kaczorowski J. Mammography rates for 20 community-based family practices in Ontario: a full practice audit. Can J Public Health 2007;98:374-8.
75. Snider J, Beauvais J, Levy I, Villeneuve P, Pennock J. Trends in mammography and Pap smear utilization in Canada. Chronic Dis Can 1996;17:108-17.
76. Cancer care ontario: Cancer system quality index. 2006. (Accessed July 8, 2007, at http://www.cancercare.on.ca/qualityindex2006/.)
77. Guttmann A, Manuel D, Dick PT, To T, Lam K, Stukel TA. Volume Matters: Physician Practice Characteristics and Immunization Coverage Among Young Children Insured Through a Universal Health Plan. Pediatrics 2006;117:595-602.
78. Have paper records passed their expiry date?
10.1503/cmaj.051229. CMAJ 2005;173:725-.
79. Balas EA, Weingarten S, Garb CT, Blumenthal D, Boren SA, Brown GD. Improving Preventive Care by Prompting Physicians. Arch Intern Med 2000;160:301-8.
80. Szilagyi PG, Bordley C, Vann JC, et al. Effect of patient reminder/recall interventions on immunization rates: A review. Jama 2000;284:1820-7.
81. Kitzinger J. Qualitative Research: Introducing focus groups. Bmj 1995;311:299.
82. Grol RP, Bosch MC, Hulscher ME, Eccles MP, Wensing M. Planning and studying improvement in patient care: the use of theoretical perspectives. Milbank Q 2007;85:93-138.
83. Grimshaw J, McAuley LM, Bero LA, et al. Systematic reviews of the effectiveness of quality improvement strategies and programmes. Qual Saf Health Care 2003;12:298-303.
84. In: Oxford English Dictionary, 2nd edition. Oxford: Oxford University Press; 1997.
85. Greenhalgh T, Stramer K, Bratan T, Byrne E, Mohammad Y, Russell J. Introduction of shared electronic records: multi-site case study using diffusion of innovation theory. Bmj 2008;337:a1786.
86. Berwick DM. Disseminating Innovations in Health Care. Jama 2003;289:1969-75.
87. Pathman DE, Konrad TR, Freed GL, Freeman VA, Koch GG. The awareness-to-adherence model of the steps to clinical guideline compliance. The case of pediatric vaccine recommendations. Med Care 1996;34:873-89.
88. England I, Stewart D, Walker S. Information technology adoption in health care: when organisations and technology collide. Aust Health Rev 2000;23:176-85.
102
89. Meyer A, Goes J. Organisational assimilation of innovations: a multi-level contextual analysis. Academy of Management Review 1988;31:897-923.
90. Dirksen CD, Ament AJ, Go PM. Diffusion of six surgical endoscopic procedures in the Netherlands. Stimulating and restraining factors. Health Policy 1996;37:91-104.
91. Lemieux-Charles L, Barnsley J. An innovation diffusion perspective on knowledge and evidence in health care. In: Lemieux-Charles L, Champagne F, eds. Using Knowledge and evidence in health care: Multidisciplinary perspectives. Toronto: University of Toronto Press; 2004.
92. Deveugele M, Derese A, van den Brink-Muinen A, Bensing J, De Maeseneer J. Consultation length in general practice: cross sectional study in six European countries. BMJ 2002;325:472-.
93. Wilson A, Childs S. The relationship between consultation length, process and outcomes in general practice: a systematic review. Br J Gen Pract 2002;52:1012-20.
94. Lussier MT, Richard C. Doctor-patient communication. Time to talk. Can Fam Physician 2006;52:1401-2.
95. Hogan SO, Kissam SM. Measuring Meaningful Use. Health Aff;29:601-6.
96. Terry AL, Thorpe CF, Giles G, et al. Implementing electronic health records: Key factors in primary care. Can Fam Physician 2008;54:730-6.
97. Simon SR, Kaushal R, Cleary PD, et al. Correlates of Electronic Health Record Adoption in Office Practices: A Statewide Survey. J Am Med Inform Assoc 2007;14:110-7.
98. Foy R, MacLennan G, Grimshaw J, Penney G, Campbell M, Grol R. Attributes of clinical recommendations that influence change in practice following audit and feedback. J Clin Epidemiol 2002;55:717-22.
99. Kaplan B, Harris-Salamone KD. Health IT Success and Failure: Recommendations from Literature and an AMIA Workshop. J Am Med Inform Assoc 2009;16:291-9.
100. Rahimi B, Timpka T, Vimarlund V, Uppugunduri S, Svensson M. Organization-wide adoption of computerized provider order entry systems: a study based on diffusion of innovations theory. BMC Med Inform Decis Mak 2009;9:52.
101. Denis JL, Hebert Y, Langley A, Lozeau D, Trottier LH. Explaining diffusion patterns for complex health care innovations. Health Care Manage Rev 2002;27:60-73.
102. Champagne F, Denis JL, Pineault R, Contandriopoulos AP. Structural and political models of analysis of the introduction of an innovation in organizations: the case of the change in the method of payment of physicians in long-term care hospitals. Health Serv Manage Res 1991;4:94-111.
103. Grilli R, Lomas J. Evaluating the message: the relationship between compliance rate and the subject of a practice guideline. Med Care 1994;32:202-13.
104. Edsall R.L. AKG. The 2009 EHR user satisfaction survey. In: Family Practice Management. Leawood, Kansas: American Association of Family Physicians; 2009:10-6.
105. Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform 2007;76 Suppl 1:S21-7.
103
106. Dewar RD, Dutton JE. The Adoption of Radical and Incremental Innovations: An Empirical Analysis. MANAGEMENT SCIENCE 1986;32:1422-33.
107. Fuller RM, Dennis AR. Does Fit Matter? The Impact of Task-Technology Fit and Appropriation on Team Performance in Repeated Tasks. INFORMATION SYSTEMS RESEARCH 2009;20:2-17.
108. Gustafson DH, Sainfort F, Eichler M, Adams L, Bisognano M, Steudel H. Developing and testing a model to predict outcomes of organizational change. Health Serv Res 2003;38:751-76.
109. Gladwin J, Dixon RA, Wilson TD. Implementing a new health management information system in Uganda. Health Policy Plan 2003;18:214-24.
110. Timmons S. How Does Professional Culture Influence the Success or Failure of IT Implementation in Health Services? In: Ashburner L, ed. Organisational Behaviour and Organisational Studies in Health Care: Reflections on the Future. Basingstoke: Palgrave; 2001.
111. Lapointe L, Rivard S. Getting physicians to accept new information technology: insights from case studies. Cmaj 2006;174:1573-8.
112. Naveh E, Meilich O, Marcus A. The effects of administrative innovation implementation on performance: an organizational learning approach. Strategic Organization 2006;4:275-302.
113. Aarts J, Doorewaard H, Berg M. Understanding Implementation: The Case of a Computerized Physician Order Entry System in a Large Dutch University Medical Center. J Am Med Inform Assoc 2004;11:207-16.
114. Tyre MJ, Orlikowski WJ. Windows of Opportunity: Temporal Patterns of Technological Adaptation in Organizations. ORGANIZATION SCIENCE 1994;5:98-118.
115. Doumit G, Gattellari M, Grimshaw J, O'Brien MA. Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2007:CD000125.
116. Coleman JS. Foundations for a theory of collective decisions. AJS 1966;71:615-27.
117. Schon D. Champions for radical new inventions. Harvard Business Review 1963;41:77-86.
118. Anderson JG, Jay SJ. Computers and clinical judgment: the role of physician networks. Soc Sci Med 1985;20:969-79.
119. Valois RF, Hoyle TB. Formative evaluation results from the Mariner Project: a coordinated school health pilot program. J Sch Health 2000;70:95-103.
120. Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care 1998;7:149-58.
121. Armstrong CP, Sambamurthy V. Information Technology Assimilation in Firms: The Influence of Senior Leadership and IT Infrastructures. INFORMATION SYSTEMS RESEARCH 1999;10:304-27.
122. Scott JT, Rundall TG, Vogt TM, Hsu J. Kaiser Permanente's experience of implementing an electronic medical record: a qualitative study. Bmj 2005;331:1313-6.
104
123. Wager KA, Lee FW, White AW, Ward DM, Ornstein SM. Impact of an electronic medical record system on community-based primary care practices. J Am Board Fam Pract 2000;13:338-48.
124. Damanpour F. Organisational innovations: a meta-analysis of effects of determinants and moderators. Academy of Management Journal 1991;34:555-90.
125. Zaltman G, Duncan R, Holbek J. Innovations and organizations. New York: John Wiley and Sons; 1973.
126. Damanpour F. Organizational Complexity and Innovation: Developing and Testing Multiple Contingency Models. MANAGEMENT SCIENCE 1996;42:693-716.
127. Damanpour F. Organizational Size and Innovation. Organization Studies 1992;13:375-402.
128. Dijkstra RF, Braspenning JC, Uiters E, van Ballegooie E, Grol RT. Perceived barriers to the implementation of diabetes guidelines in hospitals in The Netherlands. Neth J Med 2000;56:80-5.
129. McCormick LK, Steckler AB, McLeroy KR. Diffusion of innovations in schools: a study of adoption and implementation of school-based tobacco prevention curricula. Am J Health Promot 1995;9:210-9.
130. Cooper RB, Zmud RW. Information technology implementation research: a technological diffusion approach. Manage Sci 1990;36:123-39.
131. Reid CM, Ryan P, Nelson M, et al. General practitioner participation in the second Australian National Blood Pressure Study (ANBP2). Clin Exp Pharmacol Physiol 2001;28:663-7.
132. Glazier RH, Klein-Geltink J, Kopp A, Sibley LM. Capitation and enhanced fee-for-service models for primary care reform: a population-based evaluation. CMAJ 2009;180:E72-81.
133. Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med 2005;142:260-73.
134. Franks P, Clancy CM. Physician gender bias in clinical decisionmaking: screening for cancer in primary care. Med Care 1993;31:213-8.
135. Lurie N, Slater J, McGovern P, Ekstrum J, Quam L, Margolis K. Preventive Care for Women -- Does the Sex of the Physician Matter? N Engl J Med 1993;329:478-82.
136. Thind A, Feightner J, Stewart M, Thorpe C, Burt A. Who delivers preventive care as recommended?: Analysis of physician and practice characteristics. Can Fam Physician 2008;54:1574-15755.
137. Norton PG, Dunn EV, Soberman L. What factors affect quality of care? Using the Peer Assessment Program in Ontario family practices. Can Fam Physician 1997;43:1739-44.
138. Zyzanski SJ, Stange KC, Langa D, Flocke SA. Trade-offs in high-volume primary care practice. J Fam Pract 1998;46:397-402.
139. Parkerton PH, Wagner EH, Smith DG, Straley HL. Effect of part-time practice on patient outcomes. J Gen Intern Med 2003;18:717-24.
105
140. McIsaac WJ, Fuller-Thomson E, Talbot Y. Does having regular care by a family physician improve preventive care? Can Fam Physician 2001;47:70-6.
141. Finkelstein M. Preventive screening. What factors influence testing? Can Fam Physician 2002;48:1494-501.
142. Lofters A, Glazier RH, Agha MM, Creatore MI, Moineddin R. Inadequacy of cervical cancer screening among urban recent immigrants: a population-based study of physician and laboratory claims in Toronto, Canada. Prev Med 2007;44:536-42.
143. Glazier RH, Creatore MI, Gozdyra P, et al. Geographic methods for understanding and responding to disparities in mammography use in Toronto, Canada. J Gen Intern Med 2004;19:952-61.
144. Walter LC, Lindquist K, Nugent S, et al. Impact of Age and Comorbidity on Colorectal Cancer Screening Among Older Veterans. Annals of Internal Medicine 2009;150:465-73.
145. Shortell SM. Continuity of medical care: conceptualization and measurement. Med Care 1976;14:377-91.
146. The Johns Hopkins University ACG Case-Mix Adjustment System. In. Baltimore, Md: Johns Hopkins University School of Hygiene and Public Health; 1997.
147. Sibley LM, Moineddin R, Agha MM, Glazier RH. Risk adjustment using administrative data-based and survey-derived methods for explaining physician utilization. Med Care;48:175-82.
148. The Johns Hopkins University ACG Case-Mix System. Johns Hopkins University. (Accessed November 22, 2009, at http://www.acg.jhsph.edu/html/HowDoACGsWork.htm.)
149. Hux JE, Ivis F, Flintoft V, Bica A. Diabetes in Ontario: Determination of prevalence and incidence using a validated administrative data algorithm. Diabetes Care 2002;25:512-6.
150. Ko DT, Alter DA, Austin PC, et al. Life expectancy after an index hospitalization for patients with heart failure: a population-based study. Am Heart J 2008;155:324-31.
151. Tu K, Campbell NR, Chen ZL, Cauch-Dudek KJ, McAlister FA. Accuracy of administrative databases in identifying patients with hypertension. Open Med 2007;1:e18-26.
152. Jackevicius CA, Li P, Tu JV. Prevalence, predictors, and outcomes of primary nonadherence after acute myocardial infarction. Circulation 2008;117:1028-36.
153. Gershon AS, Wang C, Guan J, Vasilevska-Ristovska J, Cicutto L, To T. Identifying patients with physician-diagnosed asthma in health administrative databases. Can Respir J 2009;16:183-8.
154. Gershon AS, Wang C, Guan J, Vasilevska-Ristovska J, Cicutto L, To T. Identifying individuals with physcian diagnosed COPD in health administrative databases. COPD 2009;6:388-94.
155. Steele LS, Glazier RH, Lin E, Evans M. Using administrative data to measure ambulatory mental health service provision in primary care. Med Care 2004;42:960-5.
156. Creswell JW. Research design: qualitative, quantitative, and mixed methods approaches. 3rd ed. ed. Thousand Oaks, California: Sage Publications Inc.; 2009.
106
157. Landrum MB, Bronskill SE, Normand S-LT. Analytic Methods for Constructing Cross-Sectional Profiles of Health Care Providers. Health Services and Outcomes Research Methodology 2000;1:23-47.
158. Gann P, Melville SK, Luckmann R. Characteristics of Primary Care Office Systems as Predictors of Mammography Utilization. Ann Intern Med 1993;118:893-8.
159. Stone EG, Morton SC, Hulscher ME, et al. Interventions that increase use of adult immunization and cancer screening services: a meta-analysis. Ann Intern Med 2002;136:641-51.
160. Epi Info. In. 3.32 ed. Atlanta: Centers for Disease Control and Prevention; 2005.
161. Donner A, Klar N. Design and analysis of cluster randomization trials in health research. London: Oxford University Press; 2000.
162. Ukoumunne OC, Gulliford MC, Chinn S, Sterne JA, Burney PG, Donner A. Methods in health service research. Evaluation of health interventions at area and organisation level. Bmj 1999;319:376-9.
163. Asch SM, Kerr EA, Keesey J, et al. Who is at greatest risk for receiving poor-quality health care? N Engl J Med 2006;354:1147-56.
164. Kitzinger J. Qualitative Research: Introducing focus groups. Bmj 1995;311:299-302.
165. Glaser B, Strauss A. The discovery of grounded theory: Strategies for qualitative research. New York: Aldine de Gruyter; 1967.
166. Pope C, Ziebland S, Mays N. Qualitative research in health care. Analysing qualitative data. BMJ 2000;320:114-6.
167. Strauss A, Corbin J. Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: Sage; 1990.
168. Adami MF, Kiger A. The use of triangulation for completeness purposes. Nurse Res 2005;12:19-29.
169. Ammenwerth E, Iller C, Mansmann U. Can evaluation studies benefit from triangulation? A case study. Int J Med Inform 2003;70:237-48.
170. Heselmans A, Van de Velde S, Donceel P, Aertgeerts B, Ramaekers D. Effectiveness of electronic guideline-based implementation systems in ambulatory care settings - a systematic review. Implement Sci 2009;4:82.
171. Poon EG, Wright A, Simon SR, et al. Relationship between use of electronic health record features and health care quality: results of a statewide survey. Med Care;48:203-9.
172. Simon SR, Soran CS, Kaushal R, et al. Physicians' Use of Key Functions in Electronic Health Records from 2005 to 2007: A Statewide Survey. J Am Med Inform Assoc 2009;16:465-70.
173. Schellhase KG, Koepsell TD, Norris TE. Providers' Reactions to an Automated Health Maintenance Reminder System Incorporated into the Patient's Electronic Medical Record. J Am Board Fam Pract 2003;16:312-7.
174. Halladay JR, Stearns SC, Wroth T, et al. Cost to Primary Care Practices of Responding to Payer Requests for Quality and Performance Data. Ann Fam Med 2009;7:495-503.
107
175. Yarnall KS, Pollak KI, Ostbye T, Krause KM, Michener JL. Primary care: is there enough time for prevention? Am J Public Health 2003;93:635-41.
176. Bates DW, Bitton A. The Future Of Health Information Technology In The Patient-Centered Medical Home. Health Aff;29:614-21.
177. Electronic medical records: lessons from small physician practices. California Health Care Foundation, 2003. (Accessed September 1, 2010, at http://www.chcf.org/publications/2003/10/electronic-medical-records-lessons-from-small-physician-practices.)
178. Miller RH, Sim I. Physicians' Use Of Electronic Medical Records: Barriers And Solutions. Health Aff 2004;23:116-26.
179. Boonstra A, Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Services Research;10:231.
180. Rousseau N, McColl E, Newton J, Grimshaw J, Eccles M. Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. Bmj 2003;326:314.
181. Baron RJ. Quality improvement with an electronic health record: achievable, but not automatic. Ann Intern Med 2007;147:549-52.
182. Keyhani S, Hebert PL, Ross JS, Federman A, Zhu CW, Siu AL. Electronic health record components and the quality of care. Med Care 2008;46:1267-72.
183. Shojania KG, Jennings A, Mayhew A, Ramsay C, Eccles M, Grimshaw J. Effect of point-of-care computer reminders on physician behaviour: a systematic review. CMAJ;182:E216-25.
184. Li J, Hurley J, DeCicca P, Buckley G. Physician response to pay-for-performance - evidence from a natural experiment Toronto: York University; 2010.
185. Roth CP, Lim YW, Pevnick JM, Asch SM, McGlynn EA. The challenge of measuring quality of care from the electronic health record. Am J Med Qual 2009;24:385-94.
186. Kwong JC, Manuel DG. Using OHIP physician billing claims to ascertain individual influenza vaccination status. Vaccine 2007;25:1270-4.
187. Sy LS, Liu IL, Solano Z, et al. Accuracy of influenza vaccination status in a computer-based immunization tracking system of a managed care organization. Vaccine;28:5254-9.
188. Greene SK, Shi P, Dutta-Linn MM, et al. Accuracy of data on influenza vaccination status at four Vaccine Safety Datalink sites. Am J Prev Med 2009;37:552-5.
189. DeLone WH, McLean ER. Information Systems Success: The Quest for the Dependent Variable. INFORMATION SYSTEMS RESEARCH 1992;3:60-95.
190. Delone WH, McLean ER. The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. J Manage Inf Syst 2003;19:9-30.
191. Newcomer LN. Perspective: Physician, Measure Thyself. Health Aff 1998;17:32-5.
192. Casalino L, Gillies RR, Shortell SM, et al. External incentives, information technology, and organized processes to improve health care quality for patients with chronic diseases. Jama 2003;289:434-41.
108
193. Perlin JB, Kolodner RM, Roswell RH. The Veterans Health Administration: quality, value, accountability, and information as transforming strategies for patient-centered care. Am J Manag Care 2004;10:828-36.
194. Craig TJ, Perlin JB, Fleming BB. Self-Reported Performance Improvement Strategies of Highly Successful Veterans Health Administration Facilities. American Journal of Medical Quality 2007;22:438-44.
195. Hutchison B, Woodward CA, Norman GR, Abelson J, Brown JA. Provision of preventive care to unannounced standardized patients. Cmaj 1998;158:185-93.
196. Jaakkimainen RL, Barnsley J, Glazier R, Klein-Geltink J, Kopp A, Sibley LM. Performance Measures Amongst Primary Care Groups In Ontario, Canada In: North American Primary Care Research Group. Montreal, Quebec; 2009.
197. Federal-Provincial relations and health care: reconstructing the partnership. Queen's University, 2002. (Accessed November 5, 2005, at http://www.hc-sc.gc.ca/english/pdf/romanow/pdfs/Summary_Fiscal_E.pdf.)
198. Infoway Standards Collaboration. Canada Health Infoway, 2005. (Accessed November 27, 2005, at http://www.infoway-inforoute.ca/ehr/standards_overview.php?lang=en.)
199. Nagle L. Infoway's EHR user engagement strategy. Nursing Leadership 2007;20:31-3.
200. Griffin S, Kinmonth AL. Diabetes care: the effectiveness of systems for routine surveillance for people with diabetes. Cochrane Database Syst Rev 2000:CD000541.
201. Welch WP, Bazarko D, Ritten K, Burgess Y, Harmon R, Sandy LG. Electronic Health Records in Four Community Physician Practices: Impact on Quality and Cost of Care. J Am Med Inform Assoc 2007;14:320-8.
202. Quality and Outcomes Framework guidance for GMS contract 2009/10. London; 2009.
203. Campbell SM, Roland MO, Middleton E, Reeves D. Improvements in quality of clinical care in English general practice 1998-2003: longitudinal observational study. BMJ 2005;331:1121-.
204. Diabetes testing report. Government of Ontario, 2010. (Accessed May 27, 2010, at http://www.health.gov.on.ca/en/ms/diabetes/en/about_diabetes_care_rep.html.)
205. Improvement: Testing and Implementing Change. Ontario Health Quality Council, 2007. (Accessed January 11, 2009, at http://www.ohqc.ca/en/change_2.php.)
109
Appendices
Appendix A
Case study of a Quality Improvement project using EMR
Organization of preventive care in FHN2
A physician (the principal investigator for this study) took the lead in organizing the preventive
services. She mapped and tried several possible processes using tools available in EMR
software, using repeated Plan-Do-Study-Act cycles.205 She refined and tested these within her
own practice in the first year until she was satisfied, and then spread what she had learned to her
FHN.
An important implementation factor was the availability of funding. The FHN maintains a
common account to receive Ministry payments associated with the provision of after hours care,
so funds are available for projects if approved by the group. Another factor was resources
(administrative assistance): the government funds a part-time FHN administrator. A staff
member was hired to manage the preventive services project.
Group processes and cohesiveness were also present. Physicians participating in a FHN can
function as a group (even if they do not co-locate), and the implementation of a common, EMR-
based preventive services process was discussed and agreed upon by the group, through FHN
meetings and emails. The FHN has a Lead and an Associate Lead physician; both supported and
approved funding for this project.
Finally, the structure of the EMR was a factor. All physicians in this group use a common EMR
database; this has involved difficulties in terms of speed of access to the EMR (the database is
accessed remotely from each practice) and increased complexity. However, the common
database also facilitates the implementation of group projects; it would have been more difficult
to conduct this if data was in several different servers, requiring separate log-ins for each
practice.
110
Description of processes
The initial step in the process was to “clean up” the group’s EMR database; only rostered
patients are eligible for incentives, and the EMR tracks services according to roster status.
Demographic data had been transferred from older software applications to the new EMR in
2006; the older programs often did not have a field to record roster status. Data on rostering was
rarely entered within practices once the EMR was started, and therefore the recorded roster status
in the EMR was incomplete and inaccurate. Students were hired as data entry clerks to generate
accurate EMR rosters, by entering roster status in the correct field using the paper roster lists sent
by the Ministry of Health to each provider.
Once this was done, the data entry clerks were then trained to use the paper list of preventive
services sent by the Ministry of health, and to transfer the data from those lists to the appropriate
field of the EMR (Figure 14). They added the fact that the service was done, along with the date
the service was provided. We then hired medical students to audit both paper and EMR charts
for all remaining services, and they systematically entered the date of the last service in the
correct field in the EMR. The physician lead for this project periodically audited the data for
accuracy.
111
Figure 14: Field for adding the presence of a preventive test
Once the data entry was finished, the group had a list of overdue patients. In order to enable a
mailing to those patients, templates for reminder letters for each service were sent to all
physicians for approval and corrections. After approval, the templates were uploaded to each
physician’s EMR application; the system then used the list of overdue patients to generate
personalized letters on individual physician’s letterhead (Figure 15). Letters were printed and
mailed by the preventive services coordinator.
112
Figure 15: List of patients used to generate reminder letters
After the initial mailing, physicians and their staff were reminded that it was their responsibility
to keep the roster and preventive services EMR lists up to date. The preventive services
coordinator sent out regular emails to that effect, and conducted a training session for all FHN
front staff. She also gave practices the opportunity to fax over the monthly roster list updates
sent by the Ministry of Health so that the EMR database could be maintained.
Physicians were regularly reminded by email to record the fact that a preventive service was
provided in the correct area of the EMR. All physicians agreed to have their EMR set up so that
the area containing the reminders was visible at each encounter. In some practices, the front staff
took the lead in ensuring that data were entered.
The Preventive Coordinator routinely audited the charts of overdue patients every three months,
as a backup to staff and physician data entry, and to ensure that no data were missing. After the
audit, she mailed letters to overdue patients, to a maximum of two letters per patient. During the
summer, we hired additional staff to call overdue patients who had already received two letters.
113
This process is still ongoing; overall service provision for 2008 (excluding FOBT) was 87%;
services had improved from the previous year for every physician. The EMR Company enabled
FOBT tracking in 2008, and the same process has now been implemented for this service,
starting in the summer of 2009.
At the end of each fiscal year (after March 31st), all physicians receive a report detailing their
performance as well as the group’s overall performance; the report also includes the previous
results at the individual and group level. Physicians can use their individual results to bill for the
incentives, or they can have the FHN administrator bill this on their behalf.
This initiative had funding, agreement and participation from all those involved, leadership to
implement and drive the project, a champion, training and reminders to use EMR tools
consistently, an administrator responsible for overall project management, and data entry clerks
to relieve physician and practice staff from repetitive data entry duties. This made the project
successful and sustainable. EMR can provide tools for Quality Improvement projects; however,
change management tools are needed for implementation. The presence of EMR is a necessary
but not sufficient condition for larger scale Quality Improvement projects.
114
Appendix B
Literature search strategy Database: Ovid MEDLINE(R) <1996 to February Week 2 2010>
Search Strategy:
--------------------------------------------------------------------------------
1 emr.mp. (1417)
2 exp Medical Records Systems, Computerized/ or emr.mp. (14741)
3 electronic health records.mp. or exp Electronic Health Records/ or exp Medical Informatics/
(176542)
4 exp Electronic Health Records/og, td, ut [Organization & Administration, Trends,
Utilization] (34)
5 2 or 4 (14741)
6 limit 5 to (english language and yr="2008 -Current") (2967)
7 implement*.mp. (107729)
8 6 and 7 (565)
9 purpose.mp. (390856)
10 6 and 9 (211)
11 benefit*.mp. and 6 (209)
12 innovations.mp. (4881)
13 theor*.mp. and 6 (110)
14 12 and 6 (18)
15 innovation*.mp. (28853)
16 13 and 15 (17)
17 emr.mp. (1417)
18 exp Medical Records Systems, Computerized/ or emr.mp. (14741)
19 exp "Diffusion of Innovation"/ or innovation* theory.mp. (9139)
20 18 and 19 (1090)
21 limit 20 to english language (1085)
22 innovation* theory.mp. (86)
115
23 18 and 22 (5)
24 from 23 keep 1-5 (5)
116
Appendix C
Letter of invitation and survey of AHC physicians Effect of electronic medical records (EMRs) on the provision of preventive services in a
Pay-for-Performance environment
Michelle Greiver, Jan Barnsley, Val Rachlis, MD, Jan Kasperski, Bart Harvey, Rick Glazier,
Rahim Moineddin
University of Toronto (Health Policy and Management Evaluation; Department of Family and
Community Medicine; Department of Public Health Sciences); North York General Hospital
(Department of Family and Community Medicine); Ontario College of Family Physician; ICES
Dear Colleague:
Our government has recently introduced incentives for preventive services; these are payments
for reaching targets for mammograms, Pap smears, influenza vaccinations and children’s
vaccinations. As well, several physicians in our area have started to use Electronic Medical
Records (EMRs). At the present time, we do not know to what degree the incentives are making
a difference to our quality care, and we do not know the effect of EMRs on preventive services.
We are conducting a study on the effect of financial incentives and Electronic Medical
Records (EMRs) on preventive services in our community. We are doing this study because
it is important to determine the extent to which the incentives and EMRs influence the provision
of preventive services, and the best way to do this is to study our own practices. We are asking
you to answer the questions below to indicate your interest in this study and to help us determine
whether you are eligible to participate in the study.
117
Effect of electronic medical records (EMRs) on the provision of preventive services in a
Pay-for-Performance environment
Your name:
Size of your practice (approximate number of active patients in your practice):
Number of days you work in your office per week:
Do you have or are you planning to implement an Electronic Medical Record (EMR) in your
practice in the next year?
Would you be interested in participating in this study?
Please fax the questionnaire to 416-xxx-xxxx, or return in the postage-paid envelope.
We thank you for considering this study!
118
Appendix D
Baseline survey of participants Effect of electronic medical records (EMRs) on the provision of preventive services in a
Pay-for-Performance environment
Please fax back to 416-xxx-xxxx
Yourself:
Name:
Year of graduation:
Gender: Male Female
CCFP member? Yes No
Your practice
Number of family physicians in your office:
Are there any other health professionals in your office (eg, specialist, physiotherapist,
chiropractor etc)? Yes No
If yes, which ones?
Do you employ a practice nurse (RN or RNA)? Yes No
Number of hours you normally work per week (excluding on-call coverage) providing direct
patient care (e.g. office visits, hospital rounds, telephone advice) and indirect patient care (e.g.
reviewing laboratory results, telephone prescriptions):
119
Approximately what percentage of your patients (directly under your care) are in each of the
following age groups?
Children (0-12 years) %
Adolescents (13-18 years) %
Adults (19-64 years) %
Seniors (65+ years) %
Total 100 %
How many patients are on your roster?
Your preventive tracking system
How are you planning to track, or are currently tracking preventive services in your practice?
Services you provide
Are you accepting new patients into your practice?
No Yes Yes, with some restrictions _______
(please specify type of restrictions) :
120
Do you provide prenatal/antenatal care? Yes No
Do you provide intrapartum care (deliveries)? Yes No
Do you provide palliative care? Yes No
Are you a member of the Freeman Centre for Palliative Care? Yes No
Do you provide in-hospital care to your patients? Yes No
Do you do housecalls for your patients? Yes No
121
Appendix E
Data recording form
Effect of electronic medical records (EMRs) on the provision of preventive services in a
Pay-for-Performance environment
Mammogram recording form: Non-EMR cohort, 2007 (year end March 31st 08)
Patient number:
Date recorded (DD-MM-YYYY):
Date of birth (DD-MM-YYY):
Age (years):
Inclusion criteria:
1. Is this woman age 50 to 70? Y / N
2. Is she a patient in this practice for two years or more? Y / N
3. Does she have a history of breast cancer? Y / N
122
If YES to first TWO and NO to THIRD, patient is Eligible, please record data
Is patient Eligible? Y / N
Data:
Physician number:
Mammogram done in past 30 months (report in chart, or note that it was done by another
provider written in chart): Y / N
Mammogram report: Y / N
OR Note that it was done through another provider: Y / N
If more than 24 months since last mammogram, was patient contacted to remind her to have a
mammogram (letter/phone call documented in chart): Y / N
Letter: Y / N
Phone call: Y / N
123
Appendix F
Example of electronic audit: Pap smears
Patient DOB Sex Serviced
Not Serviced - Declined
Not Serviced - No Response Letter
3831343 Dec 7 1943 F Dec 15 2005 0
3830296 Oct 10 1963 F Nov 13 2004 0
3831822 Apr 18 1938 F 1 0
3831799 May 6 1947 F 1
3831820 May 1 1971 F Sep 26 2006 1
3831013 Aug 24 1937 F Mar 17 2006 0
3830756 Aug 31 1967 F May 8 2006 0
3830260 Oct 6 1962 F Sep 5 2006 0
3830792 Feb 20 1970 F Jul 1 2005 0
3831723 Sep 9 1969 F 1 2
124
Appendix G
Inclusion and exclusion criteria for administrative cohorts
Pap smears
Inclusion: Women 35-69 by January 1st of each year of interest rostered to physicians in the
EMR cohort (N=18) and non-EMR cohort (N=9) on March 31, in each of the years of interest
(2005-2008 – fiscal 2004-2007). For each woman identified as of March 31st 2005, 2006, 2007
and 2008, their pap smear claims traced back for the past 30 months. Any woman who had at
least one of the following tests will be deemed to have been screened:
• OHIP claims G365A, G394A, E430A • OHIP claims L812, L716, L733
Exclusion: 1. Previous diagnosis of cervical cancer (ever) • ICD-9 180.0, 180.1, 180.8, 180.9 2. Women with hysterectomy (ever) • OHIP claims S810, S757, S758, S759 3. Died before December 31st 2007
Mammograms
Inclusion: Women 50-69 by January 1st of each year of interest; women rostered to physicians
in the EMR cohort (N=18) and non-EMR cohort (N=9) on March 31, in each of the years of
interest (2005-2008 – fiscal 2004-2007). For each woman identified as of January 1st 2004,
2005, 2006 and 2007, their mammography claims traced back for the past 30 months. Any
woman who had at least one of the following tests will be deemed to have been screened:
• Client screened (SCREENED, from Ontario Breast Screening Program) – where equal to 2 (mammogram only) or 3 (yes, both physical breast exam and mammogram), OR
• OHIP radiology claim X185 Exclusion:
1. Breast cancer diagnosed ever • ICD-9 code: 174
125
• Data sources: a)CIHI, Same Day Surgery (use discharge dates) or b)Ontario Cancer Registry (Ices Key Number linked)
2. Died before December 31st 2007
Influenza vaccinations
Inclusion: Persons age 65 or more by January 1st of each year of interest; persons rostered to
physicians in the EMR cohort (N=18) and non-EMR cohort (N=9) on March 31, in each of the
years of interest (2005-2008 – fiscal 2004-2007). Any person with at least one code for influenza
vaccination in the Fall (October 1st to December 31st) of each year of interest (2004, 2005,
2006, 2007) was deemed to be vaccinated:
• OHIP claims G590, G591, G538, G539 Exclusion:
• Died before December 31st 2007
Fecal occult blood testing Inclusion: Persons age 50-74 by January 1st of each year of interest; persons rostered to
physicians in the EMR cohort (N=18) and non-EMR cohort (N=9) on March 31, in each of the
years of interest (2005-2008 – fiscal 2004-2007). For each person identified as of March 31st
2005, 2006, 2007 and 2008, their fecal occult blood test claims traced back for the past 30
months. Any person who had at least one of the following tests will be deemed to have been
screened:
• L181
Exclusion:
1. Cases diagnosed with any colorectal cancer between January 1 2000 and December 31st
2007 (using Ontario Cancer Registry, ICES Key Number linked).
• ICD-9 codes: 153.0 to 153.4, 153.6 to 154.1
2. Cases diagnosed with any severe inflammatory bowel disease between January 1 2000
and December 31st 2007 (using CIHI, Same Day Surgery and Discharge Abstract
Database (use discharge dates))
• ICD-9 codes: 556, 556.0 to 556.9 and 555, 555.0 to 555.9
126
• ICD-10 codes: K50, K50.0, K50.1, K50.9, K50.9, K51, K51.0-K51.9
3. Cases who have undergone a colonoscopy between January 1 1999 and Dec 31st 2004
(2004 cohort); between Jan 1 2000 and Dec 31 2005 (2005 cohort); between Jan 1 2001
and Dec 31 2006 (2006 cohort); between Jan 1 2002 and Dec 31 2007 (2007 cohort)
(according to OHIP)
• OHIP claims Z555 plus one of E740 or E741 or E747 or E705 on the same day
4. Died before December 31, 2007
127
Appendix H
Focus Group Questions
EMR study Focus Groups
February 5 and 7, 2008
1. Are you currently using only EMR system or a combination of both EMR and paper
charts?
2. What motivated you to implement the EMR system?
3. What have been the most positive aspects of implementation? Please explain what has
worked for you? Why?
4. What have been the biggest negatives? Would you return to paper charts?
5. What have been the biggest barriers to implementation?
6. What have been the patients’ reactions to the introduction of the EMR?
7. Would you recommend EMR implementation to colleagues?
8. What would you tell your colleagues who are considering the introduction of an EMR?
9. What supports are needed to make the transition to an EMR easier?
10. What else would you like to tell us?
128
Appendix I
Characteristics of physicians and patients in FHNs and FHGs in Ontario Table 18: Characteristics of physicians in FHNs (capitation) and FHGs (enhanced fee-for-
service) in Ontario1
1 Glazier RH, Klein-Geltink J, Kopp A, Sibley LM. Capitation and enhanced fee-for-service models for primary care reform: a population-based evaluation. CMAJ 2009;180:E72-81. © 2009 Canadian Medical Association. Copied under licence from Access Copyright. Further reproduction prohibited.
129
Table 19: Characteristics of patients in FHNs and FHGs in Ontario2
2Glazier RH, Klein-Geltink J, Kopp A, Sibley LM. Capitation and enhanced fee-for-service models for primary care reform: a population-based evaluation. Cmaj 2009;180:E72-81. © 2009 Canadian Medical Association. Copied under licence from Access Copyright. Further reproduction prohibited.
130
Appendix J
Research Ethics Board Approval
131
Appendix K
Physician Focus Group Consent Form
I understand that I am agreeing to participate as a member of a focus group of physicians organized for a study on “The Effect of Electronic Medical Records (EMRs) on the Provision of Preventive Services in a Pay-for-Performance environment.” The focus group will discuss experiences with the Electronic Medical Record that was introduced into the family health team in 2006.
I understand that, with my permission, the focus group will be audio-taped and that the audio-tapes will be transcribed verbatim. The information from the focus group will be used for the sole purpose of research and will not be used for any other purpose either during or after the completion of this study. The focus group will be conducted by an experienced facilitator and will last approximately one hour. The risks to the study participants are considered to be minimal. The greatest risk to participants is the disclosure of information provided during the focus group in a manner in which the participant can be identified. All information obtained in the focus group will be kept strictly confidential by the research team and the following steps will be taken to maintain the anonymity of the participants. During the focus group, only first names will be used. During transcription, identification codes will be used in the place of names and practice settings so participant names and their practice setting will not be identified. Only research team members will be able to decipher these codes. Information from the focus group might be included in reports, publications or presentations; however, individual participants will not be identified at any time.
During the project, the master code sheet (linking names to codes), the audiotapes, and the transcriptions will be stored in a locked filing cabinet in the Co-Principal Investigator’s office at the University of Toronto. All audiotapes will be destroyed upon completion of the data analysis. The master code sheet and the transcriptions will be destroyed seven years after the completion of the study. While focus group participants will be instructed that the content of the focus group discussion is confidential, the research team cannot guarantee that this confidentiality will be maintained by all participants. However, only factual questions about experience with the EMR will be asked; none of the questions will be of a personal nature. Further, you are under no obligation to participate in the focus group and may refuse to discuss any topic or withdraw as a participant at any time without penalty or loss of benefits to which you are otherwise entitled. If a participant withdraws from the study, all material provided to the research team previously will be destroyed.
There is no compensation for participation in the focus group.
I have been provided the opportunity to ask questions. For any additional questions or concerns about the study, I can contact the study investigators, Dr. M. Greiver at 416 222-3011 or Dr. J. Barnsley, at 416 978-1782. All conversations will be confidential. Any concerns about my rights as a research subject may be directed to Ms Jill Parsons, University of Toronto Research Ethics Board, 416 946-3273 or [email protected]).
I have read the above description of the focus group, I understand what my participation will involve and that I will be given a copy of this signed consent for my records.
Name (print) Witness Name (print)
Signature Date Signature Date
132
Copyright Acknowledgements Table 18 Glazier RH, Klein-Geltink J, Kopp A, Sibley LM. Capitation and enhanced fee-for-service models for primary care reform: a population-based evaluation. CMAJ 2009;180:E72-81. © 2009 Canadian Medical Association. Copied under licence from Access Copyright. Further reproduction prohibited.
Table 19. Glazier RH, Klein-Geltink J, Kopp A, Sibley LM. Capitation and enhanced fee-for-service models for primary care reform: a population-based evaluation. CMAJ 2009;180:E72-81. © 2009 Canadian Medical Association. Copied under licence from Access Copyright. Further reproduction prohibited.