Surgical education

Learning curves and impact of previous operative experience onperformance on a virtual reality simulator to test laparoscopic

surgical skills

Teodor P. Grantcharov, M.D.a,b,c,*, Linda Bardram, M.D., D.Sc.b,Peter Funch-Jensen, M.D., D.Sc.a, Jacob Rosenberg, M.D., D.Sc.b

aDepartment of Surgical Gastroenterology L, Aarhus University, Kommunehospitalet, Aarhus, DenmarkbDepartment of Surgical Gastroenterology 435, University of Copenhagen, Hvidovre Hospital, Hvidovre, Denmark

cDepartment of Surgical Gastroenterology D 16, Copenhagen University, Glostrup Hospital, Ndr. Ringvej 29-67, DK-2600 Glostrup, Denmark

Manuscript received March 5, 2002; revised manuscript August 12, 2002

Abstract

Background: The study was carried out to analyze the learning rate for laparoscopic skills on a virtual reality training system and toestablish whether the simulator was able to differentiate between surgeons with different laparoscopic experience.Methods: Forty-one surgeons were divided into three groups according to their experience in laparoscopic surgery: masters (group 1,performed more than 100 cholecystectomies), intermediates (group 2, between 15 and 80 cholecystectomies), and beginners (group 3, fewerthan 10 cholecystectomies) were included in the study. The participants were tested on the Minimally Invasive Surgical Trainer–VirtualReality (MIST-VR) 10 consecutive times within a 1-month period. Assessment of laparoscopic skills included time, errors, and economyof hand movement, measured by the simulator.Results: The learning curves regarding time reached plateau after the second repetition for group 1, the fifth repetition for group 2, and theseventh repetition for group 3 (Friedman’s testsP �0.05). Experienced surgeons did not improve their error or economy of movement scores(Friedman’s tests,P �0.2) indicating the absence of a learning curve for these parameters. Group 2 error scores reached plateau after thefirst repetition, and group 3 after the fifth repetition. Group 2 improved their economy of movement score up to the third repetition and group3 up to the sixth repetition (Friedman’s tests,P �0.05). Experienced surgeons (group 1) demonstrated best performance parameters,followed by group 2 and group 3 (Mann-Whitney testP �0.05).Conclusions: Different learning curves existed for surgeons with different laparoscopic background. The familiarization rate on thesimulator was proportional to the operative experience of the surgeons. Experienced surgeons demonstrated best laparoscopic performanceon the simulator, followed by those with intermediate experience and the beginners. These differences indicate that the scoring system ofMIST-VR is sensitive and specific to measuring skills relevant for laparoscopic surgery. © 2003 Excerpta Medica Inc. All rights reserved.

Keywords: Surgical performance; Laparoscopic surgery; Virtual reality; Training; Assessment; Learning curve

Minimally invasive surgery has developed enormously dur-ing the past decade. The method is today the golden stan-dard for cholecystectomy and antireflux surgery and is be-ginning to demonstrate advantages compared withconventional open methods in a number of other proce-dures. However, a major limitation for laparoscopic surgeryis training. There is consensus that the educational activitiesin minimally invasive surgery should be intensified andassessment of surgeons’ skills introduced in order to ensure

good quality of treatment. Virtual reality simulators aregaining territory as means of training and objective assess-ment of psychomotor performance [1–3]. These systemsallow repeated practice of standardized tasks and provideunbiased and objective measurements of laparoscopic per-formance, but their wide application in the surgical trainingprograms is not yet generally accepted.

The Minimally Invasive Surgical Trainer–Virtual Reality([MIST-VR] Mentice Medical Simulation, Gothenburg,Sweden) is a simulator that has been in use for several years,but the familiarization rate on the system has not beeninvestigated. This study aimed to analyze the learning curve

* Corresponding author. Tel.:�45-2826-0934; fax:�45-4673-4616.E-mail address: ttgrant@dadlnet.dk

The American Journal of Surgery 185 (2003) 146–149

0002-9610/03/$ – see front matter © 2003 Excerpta Medica Inc. All rights reserved.doi:10.1016/S0002-9610(02)01213-8

patterns for surgeons with different laparoscopic experienceand to provide evidence on the construct validity of thecomputer system by comparing the performance scores ofthe three groups of participants.

Subjects and methods

Setting

The study was carried out in two gastroenterologicalsurgical units of teaching hospitals. None of the participantsin the study had had previous contact with the MIST-VR.

Procedures

Forty-one surgeons (30 male) were tested on theMIST-VR 10 consecutive times within a 1-month period.The participants were divided into three groups according totheir experience in laparoscopic surgery: masters (group 1,8 subjects who had performed more than 100 cholecystec-tomies); intermediates (group 2, 8 subjects who had per-formed between 15 and 80 cholecystectomies); and begin-ners (group 3, 25 subjects who had performed fewer than 10cholecystectomies).

Assessment system

Laparoscopic skills were objectively measured by per-forming the tasks on the MIST-VR system. The system isbased on a PC, linked to a jig containing two laparoscopicinstruments and a diathermy pedal. Movement of the instru-ments is translated as a real-time graphical display.MIST-VR has six tasks of progressive difficulty, based onabstract graphics, which simulate the technique of manipu-lations during laparoscopic cholecystectomy. All tasks be-gin with bilateral movements to touch a virtual sphere withthe virtual instruments’ tips. For task one the trainee isrequired to grasp a virtual sphere and place it in a virtualbox. As with all tasks this is repeated two times for eachhand. In the second task the virtual sphere is grasped,transferred between instruments, and then placed in the box.Task three consists of grasping alternately the segments ofa virtual pipe. Task four requires the trainee to grasp thevirtual sphere, touch it with the tip of the other instrument,withdraw and reinsert this instrument, and once more touchthe sphere. In the fifth task, once the virtual sphere has beengrasped, three plates appear on the surface of the sphere, 90degrees apart; these are then touched by the other instru-ment and, using the pedal, removed using virtual diathermy.Task six combines the actions of tasks four and five with theaim of diathermying the plates while holding the sphere inthe virtual box. Errors, economy of movement for each hand(actual path length/ideal path length), and time are regis-tered. For the analysis, a total economy of motion score was

calculated by summing the scores for each hand into asingle value.

Statistical analysis

Data analysis was performed using the scores from tasksix, which includes elements from most of the other tasks, isof highest complexity and requires the highest level ofconcentration and coordination. Previous studies haveshown that performance scores on this task correlate bestwith surgical skills measured during an in-vivo laparoscopicprocedure [3]. Nonparametric analysis (Friedman’s test)was carried out in order to examine the difference betweenthe performance score values from each attempt and thevalues from the consecutive attempts. The difference inperformance scores between the three groups of surgeonswas analyzed by use of the Kruskal-Wallis and Mann-Whitney tests. Values are given as median (range) if notstated otherwise.

Results

The learning curve patterns for the three groups of sur-geons can be seen on Fig. 1. The curves regarding timereached plateau after the second repetition for group 1, thefifth repetition for group 2, and the seventh repetition forgroup 3 (Friedman’s tests, P �0.05). Experienced surgeonsdid not improve their error- or economy of movementscores (Friedman’s tests, P �0.2) indicating the absence ofa learning curve for these parameters. Group 2 error scoresreached plateau after the first repetition, and group 3, afterthe fifth repetition. Group 2 improved their economy ofmovement score up to the third repetition and group 3 up tothe sixth repetition (Friedman’s tests, P �0.05).

Significant differences in the performance scores of sur-geons with different operative experience were observed(Kruskal-Wallis tests, P �0.05). Experienced surgeons(group 1) demonstrated best performance in all parameters,followed by group 2 and group 3 (Mann-Whitney test, P�0.05). The values are presented in Fig. 2.

Comments

The present study provides data on the rate of acquisitionof laparoscopic psychomotor skills in a virtual environmentas well as demonstrates the construct validity of the MISTsystem. Clinical experience has shown that there is a sig-nificant learning curve for each surgeon and for each newlaparoscopic procedure, such as fundoplication, cholecys-tectomy, and appendectomy [4]. This learning phase in-cludes 10 to 30 patients and results in longer operating roomtime, higher complication rates, and higher conversion ratesto open laparotomy [5], all contributing to higher costs. Forsurgeons and surgical residents laparoscopic training with a

147T.P. Grantcharov et al. / The American Journal of Surgery 185 (2003) 146–149

VR simulator would be more efficient than training onpatients and may replace some of the learning curve typicalof new laparoscopic procedures.

Previous work has attempted to identify the familiariza-tion rate on the MIST-VR comparing the curves for sur-geons against nonsurgeons based on six repetitions of eachtask [6]. The study showed a significant improvement in theperformance scores up to the third repetition in all tasks anddid not demonstrate difference in the familiarization curvesbetween the two groups. Our study compared the learningcurves for surgeons of three experience levels who per-

Fig. 1. Learning curves for masters (filled triangles), intermediates (filledsquares), and beginners (filled circles). (a) Time to complete the task(sec), (b) error scores, and (c) number of unnecessary movements. Fordetails of statistical analyzes, see text.

Fig. 2. Comparison of the performance scores between the three groups.Horizontal bands indicate medians, boxes indicate 25th and 75th percen-tiles, and whisker lines indicate highest and lowest values. (a) Time tocomplete the task (sec), (b) error scores, and (c) number of unnecessarymovements. Levels of experience: group 1 (performed more than 100cholecystectomies), group 2 (15 to 80 cholecystectomies), and group 3(fewer than 10 cholecystectomies).

148 T.P. Grantcharov et al. / The American Journal of Surgery 185 (2003) 146–149

formed 10 repetitions of the MIST-VR tasks. We foundsignificant differences in the familiarization curves on thesimulator among the three groups. Experienced surgeonshad a very rapid learning curve regarding time and made nosignificant improvements in their error and economy ofmotion scores. This absence of initial familiarization rateindicates both the high quality of the human-computer in-terface and the relevance of the performance parametersmeasured by the simulator. Surgeons with intermediate ex-perience made quick improvements in their performancescores, while the beginners required more time and repeti-tions in order to reach their maximum score. These resultssuggest that experienced surgeons will not benefit fromtraining on this simulator while surgeons with moderateexperience as well as beginners could probably gain signif-icant improvement of their psychomotor skills by training ina virtual environment. However, we cannot state at thispoint whether improvements of surgical skills on a com-puter simulator can be transferred into real surgery withimproved clinical outcome.

Another important finding of this study is the demonstra-tion of the construct validity of the MIST-VR scoring sys-tem. It has already been shown that MIST-VR can distin-guish between surgeons and nonsurgeons [7]. However, thegroups were very disparate in their surgical capabilities anddemonstrating difference in performance values betweenthese groups does not confidently prove the construct va-lidity of the simulator (as other factors than surgical skillscould influence the performance on the simulator). Ourstudy compared three groups of surgeons with differentlevels of experience, and in this way provides strong evi-dence that the computer system measures technical skills.Experienced surgeons demonstrated best laparoscopic per-formance on the simulator, followed by those with interme-diate experience and the beginners. Further, group 1 showedless intersubject variability, ie, they were all performingwell. These differences indicate that the scoring system ofMIST-VR is reliable, sensitive, and specific to measuringskills relevant for laparoscopic surgery.

Significant difference in performance scores between thethree groups was present in the beginning of MIST-VRtraining (attempt 1), but not at the end (attempt 10) indicat-ing that the basic manual skills for the performance oflaparoscopic surgery can be acquired after 10 repetitions ofthe six MIST-VR tasks. This finding can have impact on theconstruction of the training programs in minimally invasivesurgery for residents (for example by requiring the perfor-mance of minimum 10 repetitions of all virtual tasks on the

MIST-VR, before starting supervised procedures on pa-tients).

Previous studies have demonstrated the validity of theMIST-VR scoring system by showing good correlation be-tween performance scores in a live animal operation andscores registered by the simulator [3]. The present studyprovided additional evidence that MIST-VR can preciselydifferentiate among three groups of surgeons with differentlevels of experience. Further, the simulator system showedexcellent interface with the tested surgeons and had theability to teach skills in a virtual environment. This objec-tive and valid assessment provided by MIST-VR has impli-cations for the future evaluation of manual dexterity andhand-eye coordination in the context of laparoscopic skillsfor surgical trainees. Analysis of the participants’ learningcurves allows us to determine how many examination ses-sions are necessary to evaluate an individual’s motor skillsreliably and should be considered when designing surgicalassessment programs in the future.

Acknowledgments

This study was supported by Sygekassernes Helsefond,Copenhagen, Denmark.

References

[1] Jordan JA, Gallagher AG, McGuigan J, et al. A comparison betweenrandomly alternating imaging, normal laparoscopic imaging, and vir-tual reality training in laparoscopic psychomotor skill acquisition.Am J Surg 2000;180:208–11.

[2] Prystowsky JB, Regehr G, Rogers DA, et al. A virtual reality modulefor intravenous catheter placement. Am J Surg 1999;177:171–5.

[3] Grantcharov TP, Rosenberg J, Pahle E, Funch-Jensen PM. Virtualreality computer simulation—an objective method for evaluation oflaparoscopic surgical skills. Surg Endosc 2001;15:242–4.

[4] Watson DI, Baigrie RJ, Jamieson GG. A learning curve for laparo-scopic fundoplication. Definable, avoidable, or a waste of time? AnnSurg 1996;224:98–203.

[5] MacFadyen BV, Vecchio R, Ricardo AE, Mathis CR. Bile duct injuryafter laparoscopic cholecystectomy. The United States Experience.Surg Endosc 1998;12:315–21.

[6] Chaudhry A, Sutton C, Wood J, et al. Learning rate for laparoscopicsurgical skills on MIST-VR, a virtual reality simulator: quality ofhuman-computer interface. Ann R Coll Surg Engl 1999;81:281–6.

[7] Taffinder N, Sutton C, Fishwick RJ, et al. Validation of virtual realityto teach and assess psychomotor skills in laparoscopic surgery: resultsfrom randomized controlled studies using the MIST-VR laparoscopicsimulator. Stud Health Technol Inform 1998;50:124–30.

149T.P. Grantcharov et al. / The American Journal of Surgery 185 (2003) 146–149