Reference checks: A tool for system selection A u t h o r : V i c k y B r a d l e y , R N , M S , L e x i n g t o n , K e n t u c k y

p u r c h a s i n g a c o m p u t e r i z e d cl inical d o c u m e n t a t i o n s y s t e m for your e m e r g e n c y d e p a r t m e n t is a major

inves tmen t . C h e c k i n g vendor r e fe rences is a key s t r a t e g y to faci l i ta te s y s t e m select ion. Deve lop ing a check l i s t of ques t i ons to ask co l l eagues w h o have im- p l e m e n t e d t h e s e s y s t e m s will help you ob ta in infor- m a t i o n a b o u t t he vendor, sof tware functionali ty, hard- ware , a n d imp lemen ta t i on . Your co l l eagues g a i n e d va luable e x p e r i e n c e w h e n t h e y ins ta l led thei r sys tem. The p u r p o s e of pe r fo rming re fe rence checks is to learn from o thers ' expe r i ences , as well as to va l ida te w h a t vendor s have p ro fe s sed a b o u t thei r p roduc t s .

W h o t o c a l l

Ask vendo r s to supp ly a l i s t ing of all thei r c l ients w i th a br ief d e s c r i p t i o n a b o u t w h a t t h e s e e m e r g e n c y de- p a r t m e n t s have instal led. Some vendors will not pro- v ide the i r en t i re c l ient list. If not, a sk the vendor for c l ients of s imilar ins t i tu t ions such as:

• Cl ients who have the vers ion or modu le s t ha t you p lan to p u r c h a s e

• Cl ients w h o s e e m e r g e n c y d e p a r t m e n t is s imilar in s ize a n d p r o g r a m s (e.g., 36,000 vis i t s a year, universi ty , level I t r a u m a center, t r ea t s adul t s and chi ldren)

• Cl ients who have s a m e in te r faces (e.g., ADT, order en t ry / resu l t s , bi l l ing sys tem)

• Cl ients who have i m p l e m e n t e d s y s t e m wi th in las t yea r

• Cl ients who vendo r be l i eves have fully imple- m e n t e d thei r s y s t e m A n o t h e r w a y to ob ta in r e fe rences is to ask for

help on a l is tserve. To ob ta in informat ion on h o w to p a r t i c i p a t e in a m a n a g e r ' s l is tserve, v is i t the ENA w e b s i te h t t p : / / w w w . e n a . o r g or s e n d an E-mai l to list- serv@ena.org . Leave the sub j ec t blank. Type in mes - sage: Subsc r ibe m a n a g e r s (don ' t t y p e a n y t h i n g else).

Vicki Bradley is Director of Nursing Inforrnatics, University of Kentucky Hospital, Lexington, Kentucky. J Emerg Nurs 1998;24:175-7. Copyright © 1998 by the Emergency Nurses Association. 0099-1767/98 $5.00 + 0 18/62/88794

The (hospital) emergency department is planning to purchase a computerized documentat ion system. We are currently evaluating vendors X, Y, and Z and are seeking input from any emergency department that has implemented these systems. We would like to set up an appointment to interview you about sys- tem functionality, hardware, support, and implemen- tation. If you would be wil l ing to participate in our survey, p lease send your name, hospital, and tele- phone number to (name, number). The interview will take approximately 45 minutes.

Lis tserve request for information

Send the m e s s a g e , and wa i t for a c k n o w l e d g m e n t via E-mai l and further d i rec t ions . A s a m p l e l i s t serve mes - s a g e is i l lus t ra ted in F igure 1. This r e que s t is p r imar- ily to f ind con tac t s to in t e rv iew later by phone . You could also ask your spec i f ic ques t i ons in a l i s t serve m e s s a g e .

D e v e l o p i n g y o u r i n t e r v i e w g u i d e

While you are d e t e r m i n i n g w h o m to call, deve lop an in t e rv iew guide . Sample ques t ions are i nc luded in F igure 2. The p u r p o s e of the in t e rv iew is to va l ida te w h a t you have b e e n told by the vendor, to ob ta in

informat ion tha t will help g u i d e your i m p l e m e n t a t i o n s t ra tegy, and to ident i fy po ten t i a l s i t es tha t you would like to visit. Have your pro jec t c o m m i t t e e r ev iew this l ist and revise i t to reflect your pro jec t objec t ives , spe - cific vendor funct ions , and ques t ions ra i sed b y the team. Invar iably s o m e o n e on your t eam, or wor se ye t hosp i ta l admin i s t r a t ion , will have h e a r d a rumor a b o u t a vendor such as, "I h e a r d tha t hosp i ta l X h a d w o r k e r ' s c o m p e n s a t i o n in jur ies w h e n t h e y u s e d h a n d h e l d s for d a t a entry." Be sure to follow-up on t h e s e leads.

I have found it eas ie r to list all the ques t ions on one p a g e and t hen c h e c k off e ach ques t ion as it is answered . I t r ied c r ea t i ng a form wi th b lanks after

~ e . ~ , . ~ . ~ > : ~ , ~ = = ~ r , ~ - ~ . ~ . , x ~ , , : ~ : > ~ : : ~ = = ~ . = = , , : , ~ , ~ = , ~ - ~ : e ~ : : ~ ~ ~ • ~ , = , , : •z ~ - c ¸, :%> ,~

April 1998 1 7 5

JOURNAL OF EMERGENCY NURSING/Bradley

1. Genera l • A s k r e f e r e n c e to d e s c r i b e t h e m s e l v e s (e.g.,

vo lume , a d u l t / p e d i a t r i c s , spec i a l p r o g r a m s :

t r a u m a cen te r , c h e s t p a i n cen te r , f as t track).

• W h i c h v e n d o r s d id t h e y e v a l u a t e in the i r se-

l e c t i on p r o c e s s ?

• W h a t w e r e t h e ma jo r r e a s o n s t h e y s e l e c t e d

th is v e n d o r ?

• On a sca le of I to 5, h o w w o u l d t h e y r a t e the i r

s y s t e m ?

2. S y s t e m func t ion a l i t y • W h a t v e r s i o n h a v e t h e y i m p l e m e n t e d ( some

s y s t e m s h a v e r e l e a s e d n e w v e r s i o n s or h a v e

d i f f e ren t l eve l s )?

• W h a t f u n c t i o n s h a v e t h e y i m p l e m e n t e d :

t r a c k i n g board , t r i age , o rde r entry, h i s to ry

a n d phys ica l s , a s s e s s m e n t s , p r o g r e s s no t e s ,

I&O, f l o w s h e e t s , m e d i c a t i o n s , IVs, d i s c h a r g e

i n s t ruc t i ons , que r i e s , r e p o r t s ?

• W h o u s e s t h e s y s t e m : nu r se s , t e c h n i c i a n s ,

phys i c i ans , r e s i d e n t s , resp i ra tory , c le rks?

• H o w w o u l d u s e r s d e s c r i b e t h e s y s t e m (e.g.,

u s e r f r iendly)?

• R e s p o n s e t ime: f a s t (bl ink s p e e d ) or s l o w ?

• If t h e y cou ld i m p r o v e t h e c u r r e n t s y s t e m ,

w h a t w o u l d t h e y s u g g e s t ?

• Are t h e r e a n y f u n c t i o n s / m o d u l e s t h a t t h e y

p u r c h a s e d b u t a re no t u s i n g ?

• W h a t k i n d s of r e p o r t s d o e s t h e s y s t e m

p r o v i d e ?

• A r e y o u ab le to w r i t e your o w n r e p o r t s ?

• D o e s t h e p r o d u c t fo l low spec i f i c a t i ons r e c o m -

m e n d e d in D a t a E l e m e n t s for E m e r g e n c y De-

p a r t m e n t S y s t e m s (DEEDS)? 2

3. I m p l e m e n t a t i o n • W h e n did t h e y s i g n c o n t r a c t ?

• W h e n did t h e y go l ive?

• H o w did t h e y roll ou t t h e p r o d u c t ?

• Did t h e y run pa ra l l e l s y s t e m s ? For h o w long?

• H o w did t h e y t r a in all t h e s ta f f for ini t ial

i m p l e m e n t a t i o n ? L e n g t h ?

• Did t h e y u s e s u p e r t r a i n e r s ? H o w l o n g to t r a in

s u p e r t r a i n e r ?

• H o w do t h e y t r a in n e w s ta f f n o w ?

• W h a t role d id t h e v e n d o r p l a y in i m p l e m e n -

t a t i on?

• W e r e m a n u a l s s u p p l i e d by t h e v e n d o r ?

• If t h e y h a d to do it all ove r aga in , w h a t w o u l d

t h e y do d i f fe ren t ly?

Figure 2 Reference interview guide. 1

4. Technical • W h a t k ind of h a r d w a r e did t h e y p u r c h a s e :

PCs, x - t e rmina l s , mon i to r s , f i l e se rve r s , lap-

tops , h a n d h o l d s , p r in te r s , car ts , a n d so on?

• W h e r e are t e rmina l s , moni to r s , a n d p r in t e r s

l o c a t e d ?

• H o w are d a t a e n t e r e d : ba r code , l igh t pen ,

m o u s e , t o u c h s c r e e n ?

• Did t h e y c o n s i d e r r a d i o f r e q u e n c y ?

• Do t h e y h a v e any rou t i ne d o w n t i m e ?

• A n y u n s c h e d u l e d d o w n t i m e ?

• W h a t i n t e r f a c e s do t h e y h a v e in p l a c e a n d

a re a n y s c h e d u l e d for t h e fu tu r e?

• A n y i s s u e s w i t h h a r d w a r e / n e t w o r k ?

• H o w m a n y u p g r a d e s h a v e t h e y h a d ?

• W h e n w a s t h e las t u p g r a d e ?

• C a n s c r e e n s b e c o n f i g u r e d by s y s t e m admin -

i s t r a to r or only by v e n d o r ?

5. Secur i ty • H o w is s e c u r i t y s e t up?

• W h o m a i n t a i n s ?

• Is t h e r e an aud i t t ra i l of t r a n s a c t i o n s ?

• A n y i s s u e s ?

6. Support • H o w w o u l d t h e y d e s c r i b e v e n d o r s u p p o r t ? Is

i t 24 x 7?

• W h a t k ind of s u p p o r t do t h e y o b t a i n f rom

the i r o w n IS, % of FTE?

• W h o is r e s p o n s i b l e for t h e s y s t e m in t h e

e m e r g e n c y d e p a r t m e n t , % of FTE?

• W h o is r e s p o n s i b l e for t r a in ing?

• W h o w r i t e s r e p o r t s ?

7. Contract • A n y s u g g e s t i o n s for w h a t s h o u l d be i n c l u d e d

in c o n t r a c t ?

• A n y u n e x p e c t e d c o s t s ?

8. C o s t / b e n e f i t a n a l y s i s • W h a t w e r e t h e ma jo r benef i t s : i m p r o v e d docu-

m en t a t i on , d e c r e a s e d l e n g t h of stay, i m p r o v e d

c o m m u n i c a t i o n b e t w e e n staff, i m p r o v e d s taff

morale , i m p r o v e d p a t i e n t care, a n d so for th?

• A n y i m p a c t on s t a f f ing?

• W e r e t h e y ab le to d o c u m e n t a n y cos t s a v i n g s ?

• A n y u n e x p e c t e d s e q u e l a e ?

9. Addi t iona l contac t s • Techn ica l

• P h y s i c i a n

• N u r s e m a n a g e r

176 Volume 24, Number 2

Bradley/JOU1LNAL OF EMERGENCY NURSING

e a c h ques t ion , b u t invar iab ly the i n t e r v i e w e e would

a n s w e r a ques t i on on p a g e 5 w h e n I w a s on p a g e 2

and I wou ld lose t ime t ry ing to f ind w h e r e to wr i te the answer . After you m a k e the first call or two, you will f ind a m e t h o d t ha t works for you.

Making the cal ls Discuss w i th your pro jec t t e a m a p lan for m a k i n g the calls. You can a s s i g n one pe r son to do all t he refer- ence checks , w h i c h p r o m o t e s s t anda rd iza t ion , or you can have the pro jec t t e a m d iv ide t he calls (e.g., e ach m e m b e r m a k e s two or t h r ee calls). This he lps each m e m b e r g e t a be t t e r "feel" for t he i s sues and chal-

l enges t h e y will face wi th th is project . A c o m b i n e d a p p r o a c h is to have one or two m e m b e r s in t e rv iew all the r e fe rences a n d to ask for a t e chn i ca l reference , nu r se manage r , a n d p h y s i c i a n contac t . Provide t h e s e c o n t a c t s to t e a m m e m b e r s so t h e y can call thei r col- l eagues . Develop an a b b r e v i a t e d in t e rv i ew gu ide for e ach of t h e s e t e a m m e m b e r s to follow. Each t e a m m e m b e r should s u b m i t his or her f ind ings to a centra l pe r son for summar iza t ion .

The p u r p o s e of p e r f o r m i n 9 r e f e r e n c e c h e c k s is to l earn f rom o thers ' e x p e r i e n c e s , a s w e l l as to v a l i d a t e w h a t v e n d o r s h a v e p r o f e s s e d abou'~ the ir p r o d u c t s .

From the list of s i t es p rov ided b y the vendor and r e s p o n s e to your l i s t serve ques t ion , p lan to call 7 to 10 different e m e r g e n c y d e p a r t m e n t s . Call t he con t a c t

pe r sons p rov ided to se t up an a p p o i n t m e n t to inter- v i ew t h e m for app rox ima te ly 45 minu tes . Rarely have I b e e n ab le to call s o m e o n e the first t ime and c o n d u c t the in terview. It f requent ly t akes a se r ies of "phone tag" m e s s a g e s b u t i t is wor th the effort. Offer to fax or E-marl your ques t ions in advance . O b t a i n i n g this in format ion is no t a qu ick process ; allow 1 to 2 m o n t h s to c o m p l e t e t h e s e in te rv iews .

S u m m a r i z e y o u r f ind ings After all r e fe rences have b e e n checked , s u m m a r i z e your f ind ings for p resen ta t ion . Inc lude a l i s t ing of hosp i t a l s con t ac t ed , a s u m m a r y for e a c h major ca te - gory, and a list of o u t s t a n d i n g ques t i ons or i ssues . This in format ion can b e u s e d to:

• De te rmine w h a t all the vendors have in common,

w h a t none of t h e m do, and w h e r e there a re major d i f fe rences

• Ident i fy i s sues tha t n e e d to b e a d d r e s s e d by ven- dor (e.g., you told us t ha t s y s t e m d o e s n ' t have any downt ime , bu t all the re fe rences r epor t ed a n igh t ly d o w n t i m e r a ng ing from a few m i n u t e s to m a x i m u m of 45 minu tes , p l ea se explain)

• A s s i s t you in d e t e r m i n i n g a s i te t ha t you would like to vis i t

• Provide s u g g e s t i o n s for i m p l e m e n t a t i o n • Provide s u g g e s t i o n s for a cos t /bene f i t ana lys is to

suppor t p u r c h a s e of th is s y s t e m

You car~ s a v e ~:he Oso]ec~c :earn d iv ide t h e cal l s (e.g.~ e a c h m e m b e r m a k e s t w o o~ !i:hree calls) . This n~..~os~ ~i ' each" m e m b e r g e t a b e t t e r °'feel" for t h e i s s u e s a n d c h a l l e n g e s 'they wi l l f ace w i t h this project°

Obta in ing t e l ephone re fe rences from your col- l e agues who have i m p l e m e n t e d s y s t e m s tha t you are cons ide r ing for p u r c h a s e is a va luable exercise . It will inf luence your dec i s ion and p rov ide ins igh t s into the i m p a c t tha t i m p l e m e n t a t i o n of such a s y s t e m will have on your e m e r g e n c y d e p a r t m e n t .

References

1. Hannah K J, Ball ME Edwards MJA. Introduction to nurs- ing informatics. New York: Springer-Verlag; 1994. p. 189-206. 2. National Center for Injury Prevention and Control. Data elements for emergency department systems, release 1.0 (DEEDS). Atlanta, GA: Centers for Disease Control and Prevention, 1997. Available at: http://www.cdc.gov/ncipc/ pub-res/deedspage.htm.

Contributions to this column may be sent to ~/gcday Bradley, RN, MS, 4017 Palomar Blvd., Lexington, KY 40513, phone (606) 257-2350 (E-mail address: bradley@pop.uky.edu) orMaa"se;"~a Zimmerma~g .~; ~Z4, 1414 AshlandAve., St. Paul, MN 55104; (612) 347-8796 (E-mail address.. marsha.zimmerman@co.bennepin, ran. us).

~ , - ~ ~ z ~ : ~ : . ~ = ~ , ~ . ~ ~-~T~:~ : c : ~ . ~ . ~ : x ~ ~ ~ : ~ , - ~ L = ~ e ~ , - , ~ : , ~ . : ~ w ~ : , ~ = : ~ - ~ , ~ = : ~ v : ~ : ~ - : : ~ p ~ , ~ ~ . ~ . ~ ~ r : , : ~ : ~ : : ~ ~ , i : : : : : n ~ . ~ r n ~ , , ~ : : , ~ n : : ¸ r ~ ~, ~ : ~ v , ~ > : ~ -~:~ ~ i~z,, - ~

April 1998 177