i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 480–489

journa l homepage: www. int l .e lsev ierhea l th .com/ journa ls / i jmi

Evaluating the success of an emergency response medicalinformation system

Stacie Pettera,∗, Ann Fruhlingb

a Information Systems & Quantitative Analysis, University of Nebraska at Omaha, The Peter Kiewit Institute, 1110 S 67th Street, PKI173B, Omaha, NE 68182-0392, United Statesb School of Interdisciplinary Informatics, University of Nebraska at Omaha, The Peter Kiewit Institute, Omaha, NE, United States

a r t i c l e i n f o

Article history:

Received 29 July 2010

Received in revised form

14 December 2010

Accepted 17 March 2011

Keywords:

Clinical laboratory information

systems

Emergencies

Program evaluation

a b s t r a c t

Objective: STATPackTM is an information system used to aid in the diagnosis of pathogens

in hospitals and state public health laboratories. STATPackTM is used as a communication

and telemedicine diagnosis tool during emergencies. This paper explores the success of this

emergency response medical information system (ERMIS) using a well-known framework

of information systems success developed by DeLone and McLean.

Method: Using an online survey, the entire population of STATPackTM users evaluated the suc-

cess of the information system by considering system quality, information quality, system

use, intention to use, user satisfaction, individual impact, and organizational impact.

Results: The results indicate that the overall quality of this ERMIS (i.e., system quality, infor-

mation quality, and service quality) has a positive impact on both user satisfaction and

intention to use the system. However, given the nature of ERMIS, overall quality does not

necessarily predict use of the system. Moreover, the user’s satisfaction with the informa-

tion system positively affected the intention to use the system. User satisfaction, intention

to use, and system use had a positive influence on the system’s impact on the individual.

Finally, the organizational impacts of the system were positively influenced by use of the

system and the system’s individual impact on the user.

Conclusions: The results of the study demonstrate how to evaluate the success of an ERMIS

as well as introduce potential changes in how one applies the DeLone and McLean success

model in an emergency response medical information system context.

Within the information systems literature, DeLone and

1. Introduction

By 2012, it is expected that spending on information tech-nology in healthcare will reach $10.8 billion annually [1],and pandemics, such as the H1N1 influenza virus, haveonly pushed healthcare technology spending higher [2]. After

implementing a medical information system, it is importantto measure and evaluate if the investment in the healthcaretechnology was worthwhile. This study describes the evalua-

∗ Corresponding author. Tel.: +1 402 554 2077; fax: +1 402 554 3284.E-mail address: spetter@mail.unomaha.edu (S. Petter).

1386-5056/$ – see front matter © 2011 Elsevier Ireland Ltd. All rights resdoi:10.1016/j.ijmedinf.2011.03.010

© 2011 Elsevier Ireland Ltd. All rights reserved.

tion of an emergency response medical information systems(ERMIS) named STATPackTM. This study not only evaluates ifSTATPackTM is indeed successful, but also ascertains if theevaluation of the success of an ERMIS (non-routine usage) isthe same as evaluating the success of other types of medicalinformation systems (routine usage).

McLean [3,4] have suggested that success can be measuredusing up to eight interdependent variables: system quality,information quality, service quality, system use, intention

erved.

a l i n

tinthili

2i

Aisttgasa

stpcciEswsadcmTIua

2i

ThtlbmS(waawtls

i n t e r n a t i o n a l j o u r n a l o f m e d i c

o use, user satisfaction, individual impact, organizationalmpact. This model of IS success suggests that IS success isot a unidimensional concept, but rather a concept with mul-iple dimensions and interrelationships. The IS success modelas been examined in several different contexts [5], includ-

ng healthcare [6–9]. However, this model may have someimitations in the context of emergency response medicalnformation systems mainly due to their non-routine usage.

. Emergency response medicalnformation systems

n emergency response medical information system (ERMIS)s a specialized group decision support system that includes aystematically organized group communication system wherehe protocols and communication structure are provided, buthere is little content about a particular crisis except in inte-rated electronic databases [10]. The system may also includen alert and notification process. ERMIS provide the neces-ary information for decision makers to determine a course ofction in the event of a man-made or natural disaster.

Public health emergency response medical informationystems, a type of ERMIS, are an important component ofhe national information infrastructure for bioterrorism pre-aredness. In general, ERMIS systems often have unique andhallenging system requirements. Public health ERMIS mustomply with the Health Insurance Portability and Accountabil-ty Act (HIPAA) in terms of compliancy, accuracy, and privacy.RMIS often require real-time immediate responses in a highlyecure environment. They must be easy to learn and useith information that does not overload the user especially

ince these systems are only used during emergencies whichre unpredictable and unexpected. These systems requireynamic interaction of data, multi-level statuses and notifi-ations, and real-time up-to-date information. The systemsust be accurate, reliable and process at peak performance.

he environment during an emergency is chaotic and volatile.n many situations, the process of responding to a crisis isnpredictable since almost everything in a crisis situation isn exception to the norm.

.1. STATPackTM emergency response medicalnformation system

hreats of bioterrorism and high-profile disease outbreaksave accelerated the efforts of public health laboratorieso establish better communication networks with clinicalaboratories. The intent of the STATPackTM Project, whichegan September 2002, was to address critical health com-unication and biosecurity needs in Nebraska [11]. The

ecure Telecommunications Application Terminal PackageSTATPackTM) system is a secure, patient-privacy compliant,eb-based network system that supports video telemedicinend connectivity among clinical health laboratories. The over-rching goal of this public health emergency response system

as to establish an electronic infrastructure, largely using web

echnology, to allow secure communication among state pub-ic health hub and spoke laboratory networks in emergencyituations.

f o r m a t i c s 8 0 ( 2 0 1 1 ) 480–489 481

Specifically, the STATPackTM concept involves takingmacroscopic (gross) as well as microscopic digital images ofculture samples and sending them electronically for con-sultation with experts at state public health laboratories.STATPackTM enables microbiology laboratories around thestate (and now the region) to send pictures of suspiciousorganisms to the state public health laboratory, instead ofthe samples themselves, thus lessening the risk of spread-ing potentially deadly select agents or infectious diseases. Thesystem includes an alert system that is bi-directional and hasvarious levels of priorities (emergency, urgent, and routine).

To date, 55 STATPackTM systems have been placed in keyclinical hospital laboratory locations throughout Nebraska,Kansas and Oklahoma. STATPackTM systems are also locatedin numerous food, water, environmental and veterinary sci-ence diagnostic testing laboratories. Now that the STATPackTM

systems are fully functional and widely deployed, it is impor-tant to measure the success of the STATPackTM system andits ability to meet the needs of first responders as intendedduring emergencies.

2.2. Research model and hypotheses

When any information system is implemented, it is done soto provide some value to the organization, such as improvepatient safety, lower costs, or comply with regulations. By eval-uating the information system after it is implemented, theorganization is able to determine if the system is achievingthose goals as well as identify how to improve the systemfurther or to learn from prior mistakes [12].

The IS success models proposed by DeLone and McLean[3,4] was used as the framework to evaluate the success ofSTATPackTM, a specific ERMIS. In their model of IS success,they state that the quality of the information system pos-itively affects other variables of IS success. Specifically thetypes of quality include the technical quality of the system(System Quality), the quality of the output provided by theinformation system (Information Quality), and the quality ofthe support provided to the users for the information system(Service Quality). These different types of information systemsquality are posited to positively influence the satisfaction thatusers have with the information system. Users that believethat the ERMIS has higher levels of System Quality, Informa-tion Quality, and Service Quality will also have higher levels ofsatisfaction with the ERMIS.

H1 (:). The overall quality of the information system positivelyimpacts User Satisfaction.

H1a (:). System Quality is positively associated with User Sat-isfaction.

H1b (:). Information Quality is positively associated with UserSatisfaction

H1c (:). Service Quality is positively associated with User Sat-

isfaction.

In an update to their original IS success model, DeLone andMcLean suggested that the different types of information sys-

i c a l

482 i n t e r n a t i o n a l j o u r n a l o f m e d

tem quality will influence how likely users intend to use thesystem in the future. Given that ERMIS are not systems thatare used on a routine basis, a user’s intention to use the sys-tem should an emergency arise is necessary to measure whenevaluating the success of the system. Therefore, the followinghypotheses are proposed.

H2 (:). The overall quality of the information system positivelyimpacts Intention to Use.

H2a (:). System Quality is positively associated with Intentionto Use.

H2b (:). Information Quality is positively associated withIntention to Use.

H2c (:). Service Quality is positively associated with Intentionto Use.

A common measure of information systems success iswhether or not the system is actually used. However, forERMIS, these are systems that are only used during anemergency. Regardless if users perceive the information sys-tem to be of high quality, users have no reason to usethe ERMIS unless an emergency arises. Therefore, for thehypotheses related to the quality of an information sys-tem and use, the null hypothesis is used to suggest thatthere is no relationship between System Quality, InformationQuality, and Service Quality and use of the information sys-tem.

H3 (:). The overall quality of the information system is notrelated to Use.

H3a (:). System Quality is not related to Use.

H3b (:). Information Quality is not related to Use.

H3c (:). Service Quality is not related to Use.

Not only should the user’s perception of the quality of theERMIS affect their intention to use the system, but also howsatisfied users are with the information system.

H4 (:). User Satisfaction is positively associated with Intentionto Use.

The reason that information systems are implemented is toprovide some type of benefit to the users, the organization, thecommunity, or some other stakeholder group [4]. For ERMIS,the satisfaction that users have with system should value thesystem more highly and perceive a stronger benefit from usingthe information system.

H5 (:). User Satisfaction is positively associated with Individ-ual Impact.

Furthermore, users that are more likely to use the systemin the emergency are also more likely to believe that the ERMIShas a stronger benefit to the user.

i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 480–489

H6 (:). Intention to Use is positively associated IndividualImpact.

Although H3 posits that the use of an ERMIS is not affectedby the perceived quality of the information system, it is pos-sible that those users that have experience using the ERMISare more likely to see the impact (or value) of the informationsystem for both the user and the organization.

H7 (:). Use is positively associated with impact of the system.

H7a (:). Use of the information system positively affects Indi-vidual Impact

H7b (:). Use of the information system positively affects Orga-nizational Impact.

Finally, the higher the value that users believe the ERMIShas on their work will influence the perceptions that usershave about the value that the ERMIS provides to the organiza-tion.

H8 (:). Individual Impact is positively associated with Organi-zational Impact.

Fig. 1 shows the proposed research model based on theDeLone and McLean IS success model that is customized forthe ERMIS context.

3. Method

3.1. Sample

The population of STATPackTM users is growing, but is stillrelatively small. Approximately 150 people have been trainedand are using the system across three Midwestern states inthe USA. At the time of the survey, there were 52 clinicalmicrobiology laboratories and public health laboratories usingSTATPackTM.

To administer the survey, the microbiology supervisor foreach STATPackTM system was contacted via email to explainthe purpose of the survey and to determine how the supervi-sor wanted the survey distributed. Since medical technologistswould be completing the survey at work, it was important tohave the supervisor understand the value and need for theassessment as well as the supervisor’s approval and endorse-ment. In the email, each laboratory was given the optionto receive either a paper-based or online version of the sur-vey.

3.2. Instrument

To measure the eight variables of IS success, one of theauthors thoroughly examined the large body of literaturerelated to IS success for existing measurement items [13–22].

This list was reviewed by the second author, who ensured thatthe measures for each thoroughly measure each constructin the ERMIS context. When necessary, new items specificto the ERMIS context, such as items to measure Use, were

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 480–489 483

Fig. 1 – IS success model for emergency response medical information systems. Note: Dotted lines represent the nullh cts).

dbb[

augi(csati8ttuati

twimrrttqdttp

Bps

the Spearman correlation coefficients among the constructsconsisted of both high and low values.1 Furthermore, most ofthe correlations between the variables were significant (seeTable 2).

ypothesis (i.e., no posited relationship between the constru

eveloped. Since frequency of use is correlated with the num-er of emergencies, Use was measured in a richer formaty considering the functionality and depth of system use23].

A small pool of ERMIS users and first-line support staff forn ERMIS examined the survey for clarity, redundancy, andnderstandability. After reviewing the feedback from this pilotroup of five individuals, the authors examined each surveytem to determine if the survey item (a) was relevant to ERMIS;b) was redundant with any other items; and (c) would affectontent validity if removed. Before finalizing the survey, theurvey was examined by two other individuals (a researchernd ERMIS user) to ensure the survey was not redundant, buthorough. Using these criteria, the final survey consisted of 47tems (9 items for System Quality; 6 for Information Quality;

for Service Quality; 2 for User Satisfaction; 3 for Intentiono Use; 10 for Use; 5 for Individual Impact; 4 for Organiza-ional Impact) plus some additional questions related to theser’s demographics and their exposure to STATPackTM. Userslso had the opportunity to answer several open-ended ques-ions about STATPackTM. The final survey could be completedn 15 min or less.

A 7-point Likert scale was used for all quantitative ques-ions. In addition, an eighth option of “Unable to Evaluate”as also included. Given that ERMIS are rarely used, some

ndividuals may feel that they are unable to answer one orore questions about the system, which creates a risk that

espondents may not answer one or more questions. A nullesponse in an evaluation questionnaire could suggest thathe person was unable to evaluate the ERMIS for that par-icular item or it could mean that the person overlooked theuestion. By including an “Unable to Evaluate” option, usersecide whether or not they are able to evaluate the success ofhe ERMIS on an item-by-item basis. This increases confidencehat the responses provided in the evaluation reflect the user’serceptions accurately.

Prior to administering the survey, an Institutional Review

oard examined the questions and the survey administrationrotocol. All institutional procedures, including informed con-ent, were followed for data collection.

3.3. Data analysis

Given the small population of STATPackTM users, structuralequation modeling (SEM) data analysis techniques would notbe feasible due to the expected small sample size [8]. Summa-tive scales were used for each of the constructs. Each of theitems used for a specific construct were averaged to obtain asingle value for each construct. Then, using this average score,each hypothesis was tested using simple regression in SPSS v.17.0.

4. Research findings

Of the approximately 150 STATPackTM users that were sur-veyed, 64 useable responses were received via online survey,facsimile, and mail, yielding a response rate of 42.7%. Ofthose that answered the demographic questions, 72% of therespondents were female, which is consistent with the userpopulation of STATPackTM. Only 9% of respondents werefrom state public health laboratories; the vast majority ofSTATPackTM users and respondents were at other laboratorieswithin three Midwestern states. 61% of the respondents ratedthemselves as a moderate or expert STATPackTM user. Table 1provides a summary of the demographic characteristics of theusers.

Given the use of a summative scale and that the items mea-suring the constructs are formative, rather than reflective, itis not necessary to assess the reliability of the items withinthe scales [24]. Consistent with other studies examining thesuccess of medical information systems [8], the means foreach construct were above the scale midpoint (i.e., 3.5) and

1 Because some of the constructs were nonnormal, withproblems with both skewness and kurtosis, a Spearmancorrelation matrix is reported.

484 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 480–489

Table 1 – Demographic characteristics of the sample population.

Age n % Organizational tenure n %

Under 25 0 0% ≤1 year 3 5%26–55 9 14% 2–5 years 10 16%36–55 13 20% 6–10 years 8 13%46–55 21 33% 11–15 years 5 8%56–65 16 25% 16–20 years 11 17%Over 65 2 3% >20 years 23 36%No response 3 5% No response 4 6%

Gender n % Organizational type n %

Male 14 22% Public health laboratory 6 9%Female 46 72% Other 55 86%No response 4 6% No response 3 5%

Organizational role n % STATPack experience n %

Manager 23 36% Novice 21 33%Technologist 21 33% Intermediate 34 53%Organizational support 8 13% Advanced 5 8%Scientist 10 16% No response 4 6%No response 2 3%

Table 2 – Spearman correlation coefficients for IS success variables.

IS success variable Mean Std Dev Spearman’s correlation

1 2 3 4 5 6 7 8

1. System Quality 5.618 0.751 1.002. Information Quality 5.783 0.731 0.692* 1.003. Service Quality 6.197 0.830 0.531* 0.459* 1.004. User Satisfaction 6.008 0.845 0.687* 0.706* 0.406* 1.005. Intention to Use 6.265 0.711 0.493* 0.506* 0.359* 0.565* 1.006. System Use 4.311 1.231 0.471* 0.227 0.135 0.429* 0.338* 1.007. Individual Impact 5.286 0.967 0.678* 0.598* 0.441* 0.767* 0.549* 0.484* 1.00

* 0.69 * * * * * *

8. Organizational Impact 5.519 1.013 0.727

∗ p < 0.01.

4.1. Information systems quality

As predicted, System Quality, Information Quality, and ServiceQuality had a direct, significant relationship with both UserSatisfaction (H1) and Intention to Use (H2). Consistently, Sys-tem Quality had a higher beta coefficient and a larger R2 thanboth Information Quality and Service Quality.

To examine H3, which was the null hypothesis that SystemQuality does not predict System Use, only one of the threerelationships was rejected. As predicted, the null hypothe-sis could not be rejected for Information Quality or ServiceQuality; however, the null hypothesis was rejected betweenSystem Quality and System Use in that System Quality sig-nificantly predicted System Use. To ensure the sample sizewas large enough to reject the null hypothesis for H3, powercalculations were performed. A meta-analysis on IS success[25] revealed that the relationship between Information Qual-ity and Use has a large effect size (0.49) and the relationshipbetween Service Quality and Use had a small effect size (0.09)and these effect sizes were used to determine if the samplesize is sufficient. For H3b (Information Quality to System Use),

given that a large relationship was expected, a sample size of18 is all that is necessary to reject the null hypothesis assum-ing a power level of 0.8 and an alpha of 0.05. For H3c (ServiceQuality to System Use), since its effect size is smaller, a sample

0 0.495 0.782 0.524 0.424 0.841 1.00

size of 88 is necessary to reject the null hypothesis assuminga power level of 0.8 and an alpha of 0.05 or a sample of 68assuming a power of 0.8 and an alpha of 0.1. Therefore, onlyH3b is fully supported.

4.2. Intention to use and impacts

As expected users that were more satisfied with the ERMISstated that they were more likely to use the system in future,supporting H4.

Several variables were predicted to positively influenceIndividual Impact, or the perceived value of the ERMIS on theuser. User Satisfaction (H5), Intention to Use (H6), and SystemUse (H7a) all significantly predicted Individual Impact.

The final dependent variable in the model is OrganizationalImpact, the effect of the ERMIS on the organization. Both theamount of system use (H7b) and user’s individual impact (H8)positively affect how the user views the ERMIS’ impact on theorganization. Table 3 summarizes the results of the study.

5. Discussion

Given the widespread use of DeLone and McLean IS successmodel to evaluate different types of information systems [5],

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 480–489 485

Table 3 – Regression results.

Hypotheses Beta R2 Results

System Quality 0.727*** 0.528 SupportedH1 Information Quality Positively impacts User satisfaction 0.714*** 0.509 Supported†

Service Quality 0.432** 0.184 Supported†

System Quality 0.509*** 0.259 SupportedH2 Information Quality Positively impacts Intention to use 0.411** 0.169 Supported

Service Quality 0.364** 0.133 SupportedSystem Quality 0.355** 0.126 Not supported

H3 Information Quality Not related to System use 0.140 0.020 SupportedService Quality 0.062 0.004 Supported

H4 User Satisfaction Positively impacts Intention to use 0.654*** 0.440 Supported†

H5 User Satisfaction Positively impacts Individual impact 0.773*** 0.598 SupportedH6 Intention to Use Positively impacts Individual impact 0.588** 0.334 Supported

Positively impacts Individual Impact 0.460*** 0.212 SupportedH7 System Use Positively impacts Organizational impact 0.329** 0.108 SupportedH8 Individual Impact Positively impacts Organizational impact 0.827*** 0.684 Supported

∗∗ p < 0.01.∗∗∗ p < 0.001.

† The residuals of these regression results were nonnormal due to a lack of normality in the User Satisfaction construct. Therefore, the UserSatisfaction construct for these regression tests was transformed by squaring the average value of the User Satisfaction scores. The resultingregression resulted in residuals with a normalized pattern. The regression coefficients for the transformed User Satisfaction construct is

iWistDam

TsbotEs

sqsstmoadtllvsFtpt

similar to the results with no data transformation.

t was expected that it would be useful in the ERMIS context.e have confirmed the value of the framework in evaluat-

ng ERMIS by illustrating that the DeLone and McLean’s ISuccess model can be adapted to emergency response sys-ems that are used on a non-routine basis. As expected, theeLone and McLean model was a useful framework to evalu-te a specialized medical information system with some slightodifications.The empirical results demonstrate support for the model.

he overall quality of the information system, as measured byystem quality, information quality, and service quality, influenceoth user satisfaction and intention to use. The individual impactf the system is affected by user satisfaction and intentiono use. An individual’s perceived organizational impact of theRMIS was affected by the level of individual impact of theystem.

In this study, system quality was a strong predictor of useratisfaction and intention to use. While all three measures ofuality were significant predictors of these downstream ISuccess variable, it appears that system quality has a sub-tantial influence on whether or not users are satisfied withhe system and likely to use the system in the future. The

easures of system quality in this study focused on aspectsf the system such as ease of use, accessibility, and avail-bility. Because an ERMIS is sporadically used and users areependent on the system during an emergency, an ERMIShat is difficult to use or unavailable could generate a highevel of dissatisfaction with the system. Without a certainevel of system quality, the information quality and ser-ice quality becomes irrelevant. Information quality was also atrong predictor of both user satisfaction and intention to use.or this particular information system, the users are cap-

uring images to send to experts to aid in the diagnosis ofathogens. Without information quality, the value of the sys-em to the users is diminished. In this particular context, the

users are using the system to help sick patients that needa diagnosis quickly or to ascertain the potential threat ofbioterrorism. In the medical domain, users are often awareof the need for precision and accuracy in their work andare likely more attune to issues in quality. Therefore, thesystem quality and information quality are paramount for IS suc-cess.

The interesting findings in this study relate to the IS suc-cess variable of system use. The original DeLone and McLeanmodel [3] states that quality also influences the use of an infor-mation system; however, this is not relevant in the context ofan ERMIS. Use is dictated by the number of emergencies, a fac-tor outside of the control of the user, and it is not a reasonablemeasure of IS success. In this context, the quality generallydoes not affect the depth of use of the information system.The only quality variable found to significantly impact systemuse is system quality. An ERMIS that is perceived negatively interms of system quality may be disregarded during an emer-gency. Users may believe that the system would only inhibittheir ability to address the situation; therefore, system qual-ity is necessary to ensure that users actually use the systemshould a need arise. Although measures of quality were nothypothesized to influence system use, it was believed thatsystem use would influence both individual impact and organi-zational impact. In their original model, DeLone and McLean[3] stated that system use would have a direct effect on indi-vidual impact, but did not suggest any relationship betweenuse and organizational impact. In their update to their model,DeLone and McLean [4], suggested that system use influencesnet benefits, which could be at the individual or organizationallevel. This study confirms that users that have had more expe-rience with using the ERMIS are more likely to see the value of

the system as measured by both individual and organizationalimpact. Users that may have only been exposed to the systemfor training purposes or have had less opportunity to directly

i c a l

486 i n t e r n a t i o n a l j o u r n a l o f m e d

interact with the system may struggle to find the value of thesystem until an emergency arises and system use is requiredto address the problem. Therefore, it is necessary during train-ing to clearly explain the impact of the information systemto the user, the organization, and the patient when trainingthe users. By sharing success stories from others that haveused the ERMIS during an emergency and the benefits of thesystem, it can encourage those users that have not used thesystem during an emergency to value the information sys-tem.

Although this study offers some interesting insights tounderstanding success in ERMIS, there are limitations toconsider. First, the population of users for this particular infor-mation system is quite small. Although the population isgrowing, there were only 150 users of the information sys-tem at the time of the evaluation of STATPackTM; therefore,even though the response rate was over 40% of the entire pop-ulation of users, the actual number of responses used in thequantitative analysis is relatively small (64 usable responses).Based on informal interactions with the STATPackTM popu-lation, there is no reason to believe there is any systematicresponse bias. In addition, the data revealed that users with awide variety of experience with the system and across lab-oratories and states completed the survey. It was expectedthat individuals in the population may not respond giventhe nature of their work and the limited free time availablefor the clinical staff. Any findings that yielded nonsignifi-cant results, power calculations were performed to identifyif the lack of results was due to a small effect size or asmall sample size and these issues were reported. In somecases, there is sufficient power to make an assessment aboutthe relationship between constructs; however, some relation-ships may have smaller effect sizes and a larger samplesize is necessary to more fully evaluate some of the rela-tionships of IS success in an ERMIS context. An additionallimitation of the data collected in this research is the non-normality of some of the data. This can be common in studiesof IS success because values tend to be skewed high or lowdepending on the quality of the information system understudy. In the analysis, care was taken to ensure that properstatistical techniques were used and no assumptions wereviolated in the statistical tests used. Another limitation ofthis research is that only a single ERMIS was evaluated. ThisERMIS is a very specific clinical information system, and moreinsight can be gained on assessing the success of other ERMIS.Hopefully this research will generate interest in evaluatingthe differences in evaluating IS success for ERMIS given itsdifferences from other types of healthcare information sys-tems.

This study demonstrates the need to consider the contextof the medical information system when using frameworks,such as the DeLone and McLean IS success model, to eval-uate the system. Some IS success variables become morerelevant, such as system quality, and other variables have dif-ferent implications, such as system use, in an ERMIS context.The primary outcome of this study is not only to demon-

strate the value of using models of IS success to evaluateERMIS, but also to consider the need to adapt these modelsbased on the context of the system under study. Stakehold-ers interested in understanding the success of ERMIS systems

i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 480–489

should consider using this adaptation of the DeLone andMcLean IS success model to evaluate other ERMIS. In thisevaluation of the IS success model for a specific application,STATPackTM, certain stakeholder implications emerged. Forexample, in an ERMIS, the quality of the system does havea downstream impact on the intention to use the system dur-ing an emergency and user satisfaction. It is important fordevelopers and support staff for ERMIS to keep needs of theusers in mind when designing the system and informationrequirements. For the organization to realize benefits from theinformation system, users that see the value of the ERMIS insupporting their individual tasks will also find the value inthe ERMIS for the organization. Managers can help develophigher user perceptions of individual impact by developinga system that users like to use. Those that have used thesystem are more likely to find value and given that use isoften dependent on an emergency occurring, it is importantto provide examples and scenarios to demonstrate the valueof the system to users. This should be performed before anemergency happens so users are prepared and remember touse the system during a time of crisis. Additional researchshould continue to test this adapted IS success model forother types of ERMIS to better understand the different vari-ables that influence the success of these types of informationsystems.

Author contributions

The first author is the primary author of the study andserved as the expert on information system success. Thesecond author conceived of the study, participated in thequestionnaire development and administration, and servedas the expert on emergency response medical informationsystems.

Conflicts of interests

One of the authors is the PI for the STATPackTM devel-opment team. This author has a staff that supportsSTATPackTM in the laboratories, which helped in contact-ing the laboratories for their participation. The other authoron this paper had no relationship to STATPackTM and waslisted as the primary contact for respondents on the sur-vey.

Acknowledgements

The authors would like to acknowledge Steven Hinrichs, M.D.,Stokes-Shackleford Professor of Pathology and Chair Depart-ment of Pathology/Microbiology and Anthony R. Sambol, MA,SM(NRM), SV(ASCP), Assistant Professor, Dept. of Pathology& Microbiology Assistant Director, Nebraska Public Health

Laboratory, for their ongoing enthusiasm and support dur-ing the research project. We would also like to thank SandraVlasnik for her assistance in coordinating the survey pro-cess.

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n

Summary pointsWhat was known before the study?

• Large amount of spending on medical information sys-tems, but rarely is the success of these informationsystems formally evaluated.

• Emergency response medical information systems area special type of medical information system thatis used during high stress situations and on a non-routine basis.

• DeLone and McLean developed a model of IS successthat has been adopted in a variety of contexts, includ-ing medical information systems.

What has the study added to the body of knowledge?

• The DeLone and McLean IS success model needs to bealtered for the context of emergency medical informa-tion systems because System Use is non-routine.

• Intention to Use the information system becomes animportant measure of IS success for emergency medi-cal information systems.

• The results suggest that System Quality, InformationQuality, and Service Quality affect both Intention toUse and User Satisfaction. Intention to Use, User Sat-isfaction, and System Use each directly affect theIndividual Impact of the system. Both System Use andIndividual Impact affect the perceived Organizationalimpact of the system.

• The study found that it is important to consider thecontext of the medical information system when eval-uating its success. Future research should continue toexamine how System Use is affected and impacts othervariables in the IS success model in an emergencyresponse medical information system context.

A

ppendix A. Survey items

Construct Items Measure

System Quality SysQ1 The STATPackTM system iseasy to use

SysQ2 I am knowledgeable on how touse the STATPackTM system

SysQ3 The STATPackTM system hasall of the features that I need

for remote public healthmicrobiology laboratoryconsultations and interactions

f o r m a t i c s 8 0 ( 2 0 1 1 ) 480–489 487

SysQ4 The STATPackTM systemprovides me appropriateinformation about camera,microscope and networkavailability

SysQ5 The STATPackTM system isavailable when I need it

SysQ6 The STATPackTM softwarealways does what I expect it todo

SysQ7 The STATPackTM softwareperforms quickly enough tocommands

SysQ8 The STATPackTM systemmakes information readilyaccessible to me

SysQ9 In terms of overall systemquality, I would rate theSTATPackTM system highly

InformationQuality

InfQ1 Information from STATPackTM

is in a form that is readilyusable

InfQ2 The information presented bySTATPackTM is easy tounderstand

InfQ3 The resolution of the imagescaptured and stored inSTATPackTM meets mystandards

InfQ4 Information available fromSTATPackTM is important

InfQ5 The images captured andstored in STATPackTM

accurately reflect the actualsample

InfQ6 In general, STATPackTM

provides me with high-qualityinformation (text and images)

Service Quality SrvQ1 The technical support staff forthe STATPackTM system isconsistently courteous withusers

SrvQ2 The technical support staff forthe STATPackTM system hasthe knowledge to do their jobwell

SrvQ3 When users have a problem,the technical support staff forthe STATPackTM system showsa sincere interest in solving it

SrvQ4 The technical support staff forthe STATPackTM system isdependable

SrvQ5 The technical support staff forthe STATPackTM system gives

prompt service to users

SrvQ6 The technical support staff forthe STATPackTM system isavailable when I need them

i c a l

r

informatics system for Intensive Care Unit Research, qualityof care improvement, and daily patient care, Journal of the

488 i n t e r n a t i o n a l j o u r n a l o f m e d

SrvQ7 The technical support staff forthe STATPackTM systemunderstands the specificneeds of their users

SrvQ8 Overall, I would rate thetechnical support staff for theSTATPackTM system highly interms of their ability toprovide quality service

UserSatisfaction

USat1 Overall, I am satisfied withSTATPackTM

USat2 I like having the STATPackTM

system available

System Use Use1 I often use the STATPackTM

capability to capturemicroscopic images forconsultation

Use2 I often use the STATPackTM

capability to capture grossimages for consultation

Use3 I often use the STATPackTM

capability to store microscopicand gross images locally

Use4 I often use the STATPackTM

capability to review imagesfrom the Image History

Use5 I often use the STATPackTM

Electronic Textbook capabilityUse6 When the STATPackTM system

is used for consultation, I amthe person in the lab whousually does this

Use7 When were you firstintroduced to/trained onSTATPackTM (i.e., how longhave you been usingSTATPackTM)?

Use8 How many times have youused the STATPackTM system(considering both the timeswhen you were trained on thesystem and for otherpurposes)?

Use9 How many times have youused STATPackTM for purposesother than being trained?

Use10 How often do you believe yourorganization uses theSTATPackTM system?(1 = weekly, 2 = monthly,3 = quarterly, 4 = yearly,5 = never)

Intention toUse

IUse1 I am likely to use theSTATPackTM system in anemergency

IUse2 I intend to use theSTATPackTM system in thefuture

i n f o r m a t i c s 8 0 ( 2 0 1 1 ) 480–489

IUse3 Should a situation arise, I planto use the STATPackTM system

IndividualImpact

IndI1 The STATPackTM systemenhances my effectiveness asa medical technologist

IndI2 I find the STATPackTM systemuseful for my job

IndI3 Using the STATPackTM systemimproves my decisions

IndI4 Using the STATPackTM systemgives me confidence toaccomplish my job

IndI5 In general, STATPackTM is apositive impact on my work

OrganizationalImpact

OrgI1 The STATPackTM systemstreamlines consultation andother work processes

OrgI2 Using the STATPackTM systemimproves our organization’scare to patients

OrgI3 The STATPackTM system hasresulted in overall qualityimprovement forconsultations

OrgI4 Overall, the STATPackTM

system provides value to ourorganization

e f e r e n c e s

[1] C. Dixon, State & Local Health Care IT Market, 2007–2012, in:INPUT Industry Report. Series State & Local Health Care ITMarket, 2007–2012, 2008.

[2] K. Mulholland, Will This Little Piggy Take HIT To Market? in:INPUT Blog on Government Marketplace. Series Will ThisLittle Piggy Take HIT To Market? 2009.

[3] W.H. DeLone, E.R. McLean, Information systems success: thequest for the dependent variable, Information SystemsResearch 3 (1) (1992) 60–95.

[4] W.H. DeLone, E.R. McLean, The DeLone and McLean modelof information systems success: a ten-year update, Journalof Management Information Systems 19 (4) (2003) 9–30.

[5] S. Petter, W.H. DeLone, E.R. McLean, Measuring informationsystems success: models, dimensions, measures, andinterrelationships, European Journal of Information Systems17 (3) (2008) 236–263.

[6] G. Pare, L. Lepanto, D. Aubry, et al., Toward amultidimensional assessment of picture archiving andcommunication system success, International Journal ofTechnology Assessment in Health Care 21 (4) (2005)1–9.

[7] J.F. Golob Jr., A.M.A. Fadlalla, J.A. Kan, et al., Validation ofsurgical intensive care—infection registry: a medical

American College of Surgeons 207 (2) (2008)164–173.

a l i n

i n t e r n a t i o n a l j o u r n a l o f m e d i c

[8] W.-Y Jen, C.-C. Chao, Measuring mobile patient safetyinformation system success: an empirical study,International Journal of Medical Informatics 77 (2008)689–697.

[9] C. Sicotte, G. Pare, K. Kra Bini, et al., Virtual organization ofhospital medical imaging: a user satisfaction survey, Journalof Digital Imaging (2009).

[10] M. Turoff, B. Van de Walle, Preface to the special issue onemergency preparedness and response informationsystems, Journal of Information Technology Theory andApplication 6 (3) (2004) iii–V.

[11] A. Fruhling, Information Systems, Health Care XIII,Examining the critical requirements, design approaches andevaluation methods for a public health emergency responsesystem, Communications of the Association for InformationSystems 18 (2006), 20.

[12] M.M. Yusof, A. Papazafeiropoulou, R.J. Paul, et al.,Investigating evaluation frameworks for health informationsystems, International Journal of Medical Informatics 77(2008) 377–385.

[13] R.O. Briggs, B.A. Reinig, G.-J. deVreede, The yield shift theoryof satisfaction and its application to the IS/IT domain,Journal of the Association for Information Systems 9 (5)(2008) 267–293.

[14] C.-J. Chang, J. King, Measuring the performance ofinformation systems: a functional scorecard, Journal ofManagement Information Systems 22 (1) (2005) 85–115.

[15] F.D. Davis, Perceived usefulness, perceived ease of use, anduser acceptance of information technology, MIS Quarterly 13

(3) (1989) 318–346.

[16] W.J. Doll, X.D. Deng, T.S. Raghunathan, et al., The meaningand measurement of user satisfaction: a multigroupinvariance analysis of the end-user computing satisfaction

f o r m a t i c s 8 0 ( 2 0 1 1 ) 480–489 489

instrument, Journal of Management Information Systems 21(1) (2004) 227–262.

[17] G.G. Gable, T. Chan, D. Sedera, Re-conceptualizinginformation system success: the IS-impact measurementmodel, Journal of the Association for Information Systems 9(7) (2008) 377–408.

[18] B. Ives, M. Olson, J.J. Baroudi, The measurement of userinformation satisfaction, Communications of the ACM 26(10) (1983) 785–793.

[19] J.J. Jiang, G. Klein, C.L. Carr, Measuring information systemservice quality: SERVQUAL from the other side, MISQuarterly 26 (2) (2002) 145–166.

[20] P. Seddon, S.-K. Yip, An empirical evaluation of userinformation satisfaction (UIS) measures for use with generalledger accounting software, Journal of Information Systems6 (1) (1992) 75–98.

[21] W. Sedera, G.G. Gable, A factor and structural equationanalysis of the enterprise systems success measurementmodel, in: International Conference on InformationSystems, Association for Information Systems, Washington,DC, 2004.

[22] B.H. Wixom, P.A. Todd, A theoretical integration of usersatisfaction and technology acceptance, InformationSystems Research 16 (1) (2005) 85–102.

[23] A. Burton-Jones, D.W. Straub, Reconceptualizing systemusage: an approach and empirical test, Information SystemsResearch MIS Quarterly 17 (3) (2006) 228–246.

[24] S. Petter, A. Rai, D.W. Straub, Specifying formative constructsin information systems research, MIS Quarterly 31 (2007), 4.

[25] S. Petter, E.R. McLean, A meta-analytic assessment of theDeLone and McLean IS success model: an examination of ISsuccess at the individual level, Information & Management46 (3) (2009) 159–166.