biomedical paper - ucla | bionics labbionics.seas.ucla.edu/publications/jp_07.pdf · 2008. 9....

Biomedical Paper

Task Decomposition of Laparoscopic Surgery forObjective Evaluation of Surgical Residents’ Learning

Curve Using Hidden Markov ModelJacob Rosen, Ph.D., Massimiliano Solazzo, M.D., Blake Hannaford, Ph.D., and

Mika Sinanan, M.D., Ph.D.,Departments of Electrical Engineering (J.R., B.H.) and Surgery (M.So., M.Si.),

University of Washington, Seattle, Washington

ABSTRACT Objective: Evaluation of the laparoscopic surgical skills of surgical residents is usuallya subjective process carried out in the operating room by senior surgeons. The two hypotheses of thecurrent study were: (1) haptic information and tool/tissue interactions (types and transitions) per-formed in laparoscopic surgery are skill-dependent, and (2) statistical models (Hidden MarkovModels—HMMs) incorporating these data are capable of objectively evaluating laparoscopic surgicalskills.

Materials and Methods: Eight subjects (six residents—two first-year (R1), two third-year(R3), and two fifth-year (R5)—and two expert laparoscopic surgeons) performed laparoscopic cho-lecystectomy on pigs using an instrumented grasper equipped with force/torque (F/T) sensors at thehand/tool interface, and F/T data was synchronized with video of the operative maneuvers. Fourteentypes of tool/tissue (T/T) interactions, each associated with unique F/T signatures, were definedfrom frame-by-frame video analysis. HMMs for each subject and step of the operation were com-pared to evaluate the statistical distance between expert surgeons and residents with different skilllevels.

Results: The statistical distances between HMMs representing expert surgeons and resi-dents were significantly different (� < 0.05). Major differences occurred in: (1) F/T magnitudes; (2)type of T/T interactions and transitions between them; and (3) time intervals for each T/T inter-action and overall completion time. The greatest difference in performance was between R1 (juniortrainee) and R3 (midlevel trainee). Smaller changes were seen as expertise increased beyond the R3 level.

Conclusion: HMMs incorporating haptic and visual information provide an objective tool forevaluating surgical skills. Objective evidence for a “learning curve” suggests that surgical residentsacquire a major portion of their laparoscopic skill between year 1 and year 3 of training. Comp AidSurg 7:49–61 (2002). ©2002 Wiley-Liss, Inc.

Key words: hidden Markov Model, laparoscopic surgery, surgical skill evaluation, minimally invasivesurgery (MIS)Key links: http://rcs.ee.washington.edu/brl/, http://depts.washington.edu/cves/

Received 22 September 2000; accepted 12 November 2001.

Address correspondence/reprint requests to: Dr. Jacob Rosen, Department of Electrical Engineering, Box 352500, Universityof Washington, Seattle, WA 98195; Telephone: 206-616-4936; Fax: 206-221-5264; E-mail: [email protected].

Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/igs.10026

Computer Aided Surgery 7:49–61 (2002)

©2002 Wiley-Liss, Inc.

INTRODUCTIONThe technical performance of surgery in general,and minimally invasive surgery (MIS) in particular,requires the application of forces and torques (F/T)on and between tissues to achieve specific goals.Parameters that determine the magnitude of the F/Tused include the nature of the tissue being manip-ulated, the goal of the manipulation, the type ofinstrument being used, and the skill of the sur-geons. One of the more difficult tasks in MISeducation is to teach the optimal application ofinstrument F/T necessary to conduct an operation.

Although the acquisition of laparoscopictechnical skill and the assessment of performanceare paramount for surgical resident training, thecurrent method of choice for evaluating surgicalskill is a subjective evaluation of performance inthe operating room or review of a videotape of thesurgery. In the last 5 years, use of surgical simula-tion for training and evaluating surgical skills hasbecome a subject of ongoing active research.

Simulators for evaluating surgical skills andtesting of dexterity in MIS can be roughly dividedinto three categories: (1) training boxes includingphysical objects or latex organ packs; (2) virtualreality (VR) simulators including graphical repre-sentation of virtual objects or virtual anatomy; and(3) VR simulators with a force-feedback device(haptic display) for simulating forces and torquesgenerated as a result of interaction between thevirtual objects or organs and the surgical tools.

One of the most used surgical simulators isthe laparoscopic trainer box covered by an opaquemembrane through which different trocars areplaced at different working angles. The trainee isrequired to complete several structured laparo-scopic tasks that are scored for both precision andspeed of performance. Several studies1–5 havefound the laparoscopic training box to be a valuableteaching tool for training and evaluation of basiclaparoscopic skills. Using the simulator as a basisfor evaluation, performance improved over thecourse of residency training and correlated withpostgraduate year.

A VR simulator for laparoscopic surgerymodels the movements needed to perform MIS andcan generate a score for various aspects of psy-chomotor skill. A typical example of such a simu-lator is the MIST-VR system. The MIST-VR usestwo laparoscopic instruments mounted on a framewith position sensors that provide instrumentmovement data that is translated into interactivereal-time graphics on a PC. Targets appear ran-

domly within the operating volume according tothe skill task, and can be grasped and manipulatedwith the instruments. Accuracy and errors duringperformance of the tasks and completion time arelogged. Studies performed using the MIST-VRsimulator6,7 concluded that it can objectively assessa number of desirable qualities in laparoscopic sur-gery, and can distinguish between experienced andnovice surgeons.

The use of VR models for teaching complexsurgical skills while simulating realistic human/tooland tool/tissue interaction has been a long-termgoal of numerous investigators.8–12 Although hap-tic devices that provide force feedback to the sur-gical tool while interacting with the virtual tissue/organ are commercially available (for review, seeref. 13), simulating a realistic force feedback basedon biomechanical models of soft tissue is still thesubject of active research.14 The complexity ofthese biomechanical models is due to the viscoelas-ticity and nonlinear characteristics of soft tissues.Moreover, the F/T data measured in vivo15–18 arecrucial for designing and evaluating haptic force-feedback telerobotic systems19–22 and VR sim-ulators.

The methodology developed in the currentstudy was based on Hidden Markov Models(HMMs). HMMs were extensively developed inthe area of speech recognition.23–26 Based on thetheory developed for speech recognition, HMMshave become useful statistical tools in the fields ofhuman operator modeling in general, and roboticsin particular. HMMs were applied for studyingteleoperation,27–29 human manipulation actions,30

human skills evaluation for the purpose of transfer-ring human skill to robots,31–33 and manufacturingapplications.34,35 Gesture recognition with HMMshas also received increasing attention from the re-habilitation technology community (see ref. 36 forreview). The models are also being applied to therecognition of facial expressions from video imag-es.37 Moreover, HMMs may well prove useful inmany other emerging applications beyond human–computer interfaces, such as DNA and proteinmodeling,38 fault diagnosis in nuclear powerplants,39 and detection of pulsar signals.40 Theseapplications suggest that HMMs have high poten-tial to provide better models of the human operatorin complex interactive tasks with machines.

The goal of this study was to define the learn-ing curve of MIS based on new quantitative knowl-edge of the F/T applied by surgeons on their in-

50 Rosen et al.: Task Decomposition of Laparoscopic Surgery

struments, and the types of tool/tissue interactionsused during the course of MIS surgery. This goalwas pursued through several steps: (1) developinginstrumented endoscopic tools that contain embed-ded sensors capable of measuring and recordingF/T information; (2) creating a database of F/Tsignals acquired during actual operating conditionson experimental animals; (3) performing a taskdecomposition in terms of tool/tissue interactionsin MIS based on video analysis; and (4) developingstatistical models (HMMs) for evaluating an objec-tive laparoscopic skill level.

MATERIALS AND METHODS

Subjects and ProtocolEight subjects [six general surgery residents—twofirst-year residents (R1), two third-year residents(R3), and two fifth-year residents (R5)—and twoexpert laparoscopic surgeons (ES)] each completedthe experimental protocol, which consisted of twophases. During the first phase, subjects watched a45-min video of the surgical procedure guided by asenior surgeon, to demonstrate the technique of theprocedure. This study was not intended to testknowledge of the procedure, but whether the sub-jects could technically perform it. In the secondphase, each subject performed a laparoscopic cho-lecystectomy on a pig using a standardized seven-step procedure. All surgical procedures and animalcare were reviewed and approved by the AnimalCare Committee of the University of Washingtonand the Animal Use Review Division of the USArmy Veterinary Corps.

Data from three steps of the laparoscopiccholecystectomy (positioning of the gallbladder,LC-1, exposure of the cystic duct, LC-2, and dis-section of the gallbladder, LC-3) were recorded.During these steps, the instrumented endoscopictool was used with an atraumatic grasper and acurved dissector (Fig. 1c).

Experimental System SetupDuring the laparoscopic procedures, data were ac-quired from two sources: (1) force/torque (F/T)data measured at the human/tool interface, and (2)visual information of the tool tip interacting withthe tissues. The two sources of information weresynchronized in time and recorded simultaneouslyfor off-line analysis.

In MIS, the action/reaction forces and torquesbeing applied at the interface between the sur-geon’s hand and the tool (tool/hand interface) arethe sum of forces and torques at three different

interfaces: (1) tool tip/tissue of internal organ, (2)port/abdominal wall, and (3) port/tool shaft, in ad-dition to the gravitational forces and inertial forcesand torque. Two sets of sensors measured the F/T atthe interface between the surgeon’s hand and theendoscopic grasper handle (Fig. 1a). The first sen-sor was a three-axis force/torque sensor (F/T Mini,ATI, Gamer, NC) that was mounted in the outertube (proximal end) of a standard reusable 10-mmendoscopic grasper (Storz). The sensor was capableof simultaneously measuring the three componentsof force (Fx, Fy, Fz) and three components of torque(Tx, Ty, Tz) in a Cartesian frame (Fig. 1b) with afrequency response spectrum of 250 Hz (3 dB).Due to the location of the F/T sensor, it measuredthe forces and torques applied at the hand/toolinterface, which were equal to the sum of all theforces and torques applied at the various interfacesmentioned previously. However, with the currentsetup, it was impossible to measure the contributionof each separate interface. The sensor orientationwas such that the X and Z axes formed a planeparallel to the contact surfaces of the tool’s internaljaws, and the Y and Z axes defined a plane perpen-dicular to that surface (Fig. 1b). A second forcesensor (FR1010, Futek, Irvine, CA) was mountedon the endoscopic grasper handle to permit themeasurement of grasping force (Fg) applied by thesurgeon’s fingers.

The grasper’s mechanism had a structuralfeature, like most of the commercially availableendoscopic graspers, enabling the surgeon tochange the orientation of the tool tip relative to thetissue position without changing the handle orien-tation. This was achieved by rotating the entireshaft of the grasper relative to the handle, includingthe tool tip and its rod inside the shaft, using a knoblocated at the proximal end of the shaft. The im-portance of this setup from the engineering per-spective was that the alignment of the tool-tip ori-gin/coordinate system relative to the F/T sensororigin/coordinate system attached to the outer tuberemained unchanged, because the outer tube andthe tool tip are linked mechanically.

The F/T data were integrated with the lapa-roscopic camera view of instrument activity (Fig.2). The seven channels of F/T data (Fx, Fy, Fz, Tx,Ty, Tz, Fg) were sampled at 30 Hz using a laptopcomputer with a PCMCIA 12-bit A/D card (DAQ-Card 1200, National Instruments, Austin, TX) (Fig2a). Preliminary measurements showed that 99% ofthe force/torque signals’ energy (PSD) was in-cluded in the 0–10-Hz frequency bandwidth. ALabView (National Instruments) application was

Rosen et al.: Task Decomposition of Laparoscopic Surgery 51

developed with a graphical user interface for ac-quiring and visualizing the F/T data in real timeduring an actual operation. The video signal fromthe endoscopic camera that monitored the grasper’stip interacting with the internal organs or tissueswas integrated with the F/T data using a videomixer in a picture-in-picture mode (PIP), allowingcorrelation of F/T data with instrument activ-ity. The integrated interface was recorded duringthe operation for off-line frame-by-frame analysis(Fig. 2b).

Data AnalysisTwo types of analysis were performed on the rawdata: (1) video analysis, encoding the tool-tip/tissueinteraction into states; and (2) Hidden MarkovModeling, for modeling and comparing the perfor-mance of surgeons at different levels of their train-ing (i.e., R1, R3, R5, or ES).

Video AnalysisThe video analysis was performed by two expertsurgeons encoding the video of each step of the

surgical procedure frame by frame (NTSC, 30frames per second). The encoding process used acodebook of 14 different discrete tool maneuvers inwhich the endoscopic tool was interacting with thetissue (Table 2). Each identified surgical tool/tissueinteraction had a unique F/T pattern. For example,in the laparoscopic cholecystectomy, isolation ofthe cystic duct and artery (LC-2) involves perform-ing repeated pushing and spreading (PS-SP—seeTable 1) maneuvers, which in turn require pushingforces, mainly along the Z axis (Fz), and spreadingforces (Fg) on the handle.

These 14 states can be grouped into threebroader types based on the number of movementsperformed simultaneously. Fundamental maneu-vers were defined as type I, and included the idlestate (moving the tool in space without touchingany structures within the insufflated abdomen). Theforces and torques used in the idle state mainlyrepresented the interaction of the port with theabdominal wall, in addition to gravitational andinertial forces. In the grasping and spreading states,compression and tension were applied to the tissue

Fig. 1. The instrumented endoscopic grasper. (a) The grasper with the three-axis force/torque sensor implemented on theouter tube and a force sensor located on the instrument handle. (b) The tool tip and XYZ frame aligned with the three-axisforce/torque sensor. (c) Tool tips used in the surgical procedure: (from left to right) atraumatic grasper, Babcock grasper, andcurved dissector. [Color figure can be viewed in the online issue, which is available at www.interscience.wiley.com.]


Fig. 2. Experimental setup. (a) Block diagram of the experimental setup integrating the force/torque data and the view from theendoscopic camera. (b) Real-time user interface of force/torque information synchronized with the endoscopic view of the procedureusing picture-in-picture mode. (Color figure can be viewed in the online issue, which is available at www.interscience.wiley.com)


by closing/opening the grasper handle. In the push-ing state, compression was applied to the tissue bymoving the tool along the Z axis. For sweeping, thetool was placed in one position while rotatingaround the X and Y axes (port frame). Type II andtype III states were defined as combinations of twoor three states defined by type I (Table 2).

Hidden Markov Model (HMM)During the second step of the data analysis, HiddenMarkov Models (HMMs) and the methodology forevaluating surgical skill in laparoscopic surgerywere developed. HMMs were selected for model-ing the surgical procedure because their genericarchitecture fitted very well with the nature of lapa-roscopic surgical task assessment. Moreover, theHMM mathematical formulation provided a verycompact form for statistically summarizing rela-tively complex tasks, such as individual steps of alaparoscopic surgery procedure.

The rationale for developing a methodologyfor objective evaluation of surgical skills based onHMMs and F/T measurements at the human/toolinterface came from our previous study41 showingthat F/T applied at that interface varied accordingto skill levels and the tasks being performed. HighF/T magnitudes defined by the absolute value of theforce-torque vectors

�F� � � �Fx2 � Fy

2 � Fz2 (A)

�T� � � �Tx2 � Ty

2 � Tz2 (B)

were applied by novice surgeons (NS) compared toexpert surgeons (ES), while performing tissue ma-

nipulation. This might be a result of insufficientdexterity on the part of the NS that might havepotential for tissue damage. However, low F/Tmagnitudes were applied by the NS, compared tothe ES, during tissue dissection, which might alsoindicate excessive caution in an effort to avoidirreversible tissue damage. As a result, the NS hadto perform more repetitions of the dissection move-ments to tear the tissue, substantially decreasing theefficiency of the MIS procedure.

Each laparoscopic surgical step could be de-composed into a series of finite states defined bythe way the surgeon interacts with the tissues (Ta-ble 1). The surgeon could move from one state tothe other or remain in the same state for a certainamount of time. Once the surgeon was interactingwith the tissue in a specific state, a certain F/Tsignature was applied by the surgeon through thesurgical tool to the tissue. These F/T signatures,each defined as an observation, were composed ofseven component vectors of data (Fx, Fy, Fz, Tx, Ty,Tz, Fg). The F/Ts were continuous streams of datadistributed normally, each state being defined byseven normal distributions functions chartered by amean and a standard deviation [Ni(�, �) i � 1. . .7)]. Combining the seven-element vector into ajoint multivariable distribution function f(O) wasdone using Equation (1):

f�O� �1

��2��N��1/ 2e��O��¥�1�O��/ 2 (1)

where O is the F/T observation vector, � is the

Table 1. Definition of Tool/Tissue Interactions and the Corresponding Directions of Forces andTorques Applied During MIS

Type State NameStateacronym

Force/TorqueFx Fy Fz Tx Ty Tz Fg

I Idle ID * * * * * * *Grasping GR �Spreading SP �Pushing PS �Sweeping SW � � � �

II Grasping–Pulling GR-PL � �Grasping–Pushing GR-PS � �Grasping–Sweeping GR-SW � � � � �Pushing–Spreading PS-SP � �Pushing–Sweeping PS-SW � � � � �Sweeping–Spreading SW-SP � � � � �

III Grasping–Pulling–Sweeping GR-PL-SW � � � � � �Grasping–Pushing–Sweeping GR-PS-SW � � � � � �Pushing–Sweeping–Spreading PS-SW-SP � � � � � �


mean vector, � is the covariance matrix, and N isthe observation vector size.

The state diagram (Fig. 3) describes the de-composed process of a typical laparoscopic surgi-cal procedure step. Circles in this diagram repre-sented states, and lines represented transitionsbetween states. The F/T data observation signalswere not included in Figure 3.

The HMM is termed “hidden” due to the factthat tool/tissue interactions—the states—are hid-den, and the only observed signals are the F/T data.Although the state could be decomposed manuallyusing a frame-by-frame video analysis, this is timeconsuming and unnecessary, because the data canalso be evaluated mathematically by the HMMonce its parameters are optimized.

From the mathematical perspective, four ele-ments should be defined in order to specify anHMM (�):23 (1) the number of states in the modelN; (2) the state transition probability distributionmatrix A; (3) the observation symbol probabilitydistribution matrix B; and (4) the initial state dis-tribution vector �. The HMM is then defined by thecompact notation (2)

� � � A, B, �� (2)

Given the HMM architecture, there are three basicproblems of interest:23

1. The evaluation problem: computing theprobability (P) of the observation sequence,given the model (�) and the observationsequence (O).

Given: �� A,B,��O � o1,o1, . . . ,oT

� (3)

Compute: P�O � ��.

2. Uncovering the hidden states: computingthe corresponding hidden state sequence(Q), given the observation sequence (O) andthe model (l).

Given: �� A,B,��O � o1,o2, . . . ,oT

� (4)

Compute: Q � q1, q2, . . . , qT

3. The training problem: adjusting the modelparameters (A, B, �) to maximize the prob-ability (P) of the observation sequence (O).

Given: � � �A, B, ��

Adjust: A, B, �

Maximize: {P�O � �� (5)

Using the given HMM architecture (Fig. 3),HMMs were developed for each surgeon perform-ing each step of the surgical procedure (eight HMMmodels, one for each surgeon performing one sur-gical procedure step). The skill level of each sub-ject (R1, R3, R5) was evaluated based on the sta-tistical distance between his/her HMMs and thoseof the the expert surgeons (ES). Given two HMMs�1 and �2, the statistical distances between them,D(�1, �2) and D(�2, �1), were defined by Equation(6):

D��1,�2� �1

TO2

log P�O2��1� � log P�O2��2��

(6)

D��2,�1� �1

TO1

log P�O1��1� � log P�O1��2��

D(�1, �2) is a measure of how well model �1

matches observations generated by model �2 rela-tive to how well model �2 matches observationsgenerated by itself, whereas TO1

and TO2stand for

the time duration of the observation vectors O1 andO2, respectively. Because D(�1, �2) and D(�2, �1)

Fig. 3. HMM architecture defined by a 14 fully connect-ed-state diagram (arrowheads of all lines connecting twostates were omitted to simplify the drawing).


are nonsymmetrical, the natural expression of thesymmetrical statistical distance version is definedby Equation (7):

Ds��1,�2� �D��1,�2� � D��2,�1�

2(7)

To scale the statistical distance between each of thetwo subjects defined by the index j � 1, 2, andassociated with three group levels defined by theindex i � 1, 3, 5 (R1, R3, R5) and the two subjectsof the expert surgeons defined by the index k � 1,2 [ES(1), ES(2)], for each surgical procedure, thestatistical distance between a certain subject j ingroup i and a certain subject k in the expert group[DS(�Ri, �ESj)] was normalized with respect to thedistance between the two expert subjects [DS(�ES1,�ES2)] [see Equation (8)].

For i � 1, 3, 5, j � 1, 2, and k � 1, 2,

D� S��Ri� j�,�ES�k�� D��Ri� j�, �ES�k��

DS��ES1,�ES2�(8)

The practical meaning of the normalized sta-tistical distance (D� s��Ri� j�, �ES�k��) is to define quan-titatively how far each subject is from being anexpert surgeon. Because each group included twosubjects, calculating all the combinations of thestatistical distances between subjects of the ESgroup and subjects of any of the Ri group providesfour values: n � 4 (D� S(�Ri(1), �ES(1)), D� S(�Ri(1),�ES(2)), D� S(�Ri(2), �ES(1)), D� S(�Ri(2), �ES(2))). Theaverage and the standard deviation of these fourvalues were calculated to provide a single statisticalmeasurement between two skill-level groups. Theaverage of the normalized statistical distance be-tween two groups was defined by Equation (9):

For i � 1, 3, 5, and n � 4,

D� S��Ri,�ES� �

�j�1,k�1

j�2,k�2

D� S��Ri� j�, �ES�k��

n(9)

RESULTSTypical raw data of forces and torques were plottedin a 3D space, showing the loads developed at thesensor location while the gallbladder fossae weredissected in 482 s by an expert surgeon duringlaparoscopic cholecystectomy (Fig. 4). The forcesand torques measured by the F/T ATI sensor can bedescribed as vectors with an origin at the center of

the sensor and the coordinate system aligned withthe tool coordinate system (Fig. 1b). These vectorsare constantly changing both their magnitudes andorientations as a result of the F/T applied by thesurgeon’s hand on the tool while interacting withthe tissues. The F/T vectors can be depicted asarrows attached to the origin, changing theirlengths and orientations as a function of time. Fig-ure 4 shows the trace of the tips of these vectors(arrows) as they change during MIS. The forcesand torques are represented by a 3D plot in additionto the three orthogonal planes. The ellipsoid definesa region including 95% of the F/T samples.

The forces along the Z axis (in/out of theport) were higher compared to the forces in the XYplane. On the other hand, torques developed byrotating the tool around the Z axis were extremelylow compared to the torques generated while rotat-ing the tool along the X and Y axes and sweepingthe tissue or performing lateral retraction. Similartrends in terms of the F/T magnitude ratios betweenthe X, Y, and Z axes were found in the datameasured in other steps of the MIS procedures.These raw data demonstrated the complexity of thesurgical task. Deeper understanding of this task isgained by decomposing it to its prime elements, asdemonstrated by the video analysis.

Frame-by-frame analysis of videotapes of thesurgical procedures incorporating the visual viewof the tool/tissue interaction and graphs of the F/Tat the tool/hand interface allowed definition of theprimary tool/tissue interactions in the MIS proce-dure, and of the direction of forces and torquesassociated with them (Table 1). Once these tool/tissue interaction archetypes were defined, eachstep of the surgical procedure could be manuallydecomposed into a list of tool/tissue interactions.This list was further transformed into a more com-pact diagram (as shown in Fig. 5) defining a typicaltool/tissue transition for a surgical procedure. Thetool/tissue transition diagram (state diagram) de-picted in Figure 5 represents the surgical step inwhich the gallbladder fossa was dissected.

The idle state is the only state connected to allthe others in the state transition diagram (Fig. 5).This state, in which no tool/tissue interaction wasperformed, was mainly used by both expert andnovice surgeons to move from one operative stateto the other. However, the expert surgeons used theidle state only as a transition state, while the nov-ices spent significant amounts of time in this stateplanning the next tool/tissue interaction. Anothermajor difference between surgeons from differentskill groups was related to the tool/tissue interac-


Fig. 4. Forces (a) and torques (b) measured at the human/tool interface while dissecting the gallbladder fossa duringlaparoscopic cholecystectomy. For the definitions of the X, Y, and Z directions, see Figure 1b. (Color figure can be viewedin the online issue, which is available at www.interscience.wiley.com)

tion and tool/tissue transitions used by these twogroups. Surgeons took different paths to reach thesame goal (Fig. 5). Each group utilized states andtransitions not used by the other group. Figure 5,for example, was essentially constructed from twoseparate models representing the expert surgeon’smodel (ES) (solid line and dashed line) and thenovice surgeon’s model (R1) (solid line and dottedline). For the purpose of evaluating surgical skillsusing the HMM, the model representing differentgroups must share the same architecture. This re-quirement led to generalized model architecture, asdescribed previously in Figure 3.

The final and most profound analysis in-cluded the HMM analysis. HMMs were developedfor each one of the eight subjects (R1, R3, R5, ES)that performed the three steps of the laparoscopiccholecystectomy (LC-1, LC-2, LC-3). The normal-ized statistical distances [Ds—see Equation (8)]between ES and R1, R3, and R5 (Fig. 6) wereplotted in Figure 7. The strength of this methodol-ogy is that it brings together different aspects of the

surgical procedure into a single number that prac-tically indicates how far the surgical performanceof the subject under study is from that of an expertperforming the same surgical task. In this way, Ds

provides an objective criterion for evaluating sur-gical performance in MIS.

The objective laparoscopic surgical skilllearning curve showed significant differences be-tween all skill levels (Fig. 7). The value of thenormalized statistical distance (Ds) between vari-ous skill-level groups (R1, R3, and R5) and the ESconverged exponentially to a value of 1 as the levelof expertise increased. However, the highest gradi-ent was between R1 and R3. This results indicatethat surgical residents acquire a major portion oftheir laparoscopic surgical capabilities between thefirst and third years of their residency training.Calculating the Ds values for LC-1 (not plotted inFig. 7) showed no significant difference betweenthe groups. The practical meaning of this result isthat LC-1 does not include sufficient haptic infor-mation to use this data for differentiating betweengroups with different skill levels. On the otherhand, LC-2 and LC-3 do provide such information,as shown in Figure 7.

Fig. 5. The state diagram based on a frame-by-framevideo analysis of the dissection of the gallbladder fossaduring laparoscopic cholecystectomy. Each circle representsa different state characterized by a tool/tissue interaction,and the arrows represent transitions between states. In somecases, the surgeon stays within the same state: this is de-picted by an arrow to the same state. (Dashed line: statesand transitions performed by ES; dotted line: states andtransitions performed by R1 representing a novice surgeon;solid line: states and transitions performed both by R1and ES).

Fig. 6. Schematic representation of the averaged statisti-cal distance Ds defining the statistical similarities betweenthe expert group and various groups of residents associatedwith different skill levels. Note that each group includedtwo subjects, so the statistical distance D� s between twoselected groups is the average of four distances.


DISCUSSIONMinimally invasive surgery is a complex task thatrequires a synthesis between visual and haptic in-formation. Analyzing MIS in terms of these twosources of information is a key step towards devel-oping objective criteria for training surgeons andevaluating the performance of VR simulators in-corporating haptic technology and master/slave ro-botic systems for telesurgery. In addition, using theF/T information in real time during the course oflearning to provide feedback to the surgical resi-dents may improve the learning curve, reduce soft-tissue injury, and increase efficiency during endo-scopic surgery.

The force/torque signatures and the HMMsare objective criteria for evaluating skills and per-formance in MIS. The results suggest that HMMsof surgical procedures allow objective quantifica-tion of skill based on the statistical distance be-tween HMMs representing surgical residents at dif-ferent levels of their training and HMMsrepresenting expert surgeons. Moreover, this meth-odology can be used to determine if the perfor-mance of a student matches his/her training level.

The advantage of using this method in terms of datareduction is that it condenses enormous amounts ofmultidimensional data into a single datum ex-pressed as the normalized statistical distance froman expert performance, or, in other words, the sta-tistical similarity between the subject under studyand an expert performer. The strength of the meth-odology using HMM for objective surgical skillassessment is that it is not limited to the in vivocondition as demonstrated in the current study. Itcan be extended to other modalities, such as surgi-cal simulators and robotic systems for telesurgery.The proposed methodology derives its power fromdecomposing the surgical task to its prime ele-ments: tool/tissue interactions. These elements areinherent in MIS no matter which modality is beingused.

Increasing the size of the database to includemore surgical procedures performed by more sur-geons could extend the approach outlined in thisstudy. This extension is essential for full validationof the proposed methodology. An intermediate steptoward achieving that goal is to determine howmany subjects at each level are required to show astatistically significant difference between differentskill levels. These requirements initiated a moreextensive experimental protocol, currently under-way, which includes five subjects in each skillgroup. This new database will provide a deeperinsight into the complex multidisciplinary issue ofobjective evaluation of surgical skills.

Another possible approach to further explo-ration of the proposed methodology that avoids invivo experiments on laboratory animals is to useVR simulators incorporating haptic devices. Thistransformation from an in vivo surgical setup to VRsimulators will be possible only when these simu-lators can realistically represent the surgical setupfrom both the graphic and haptic perspectives. Thisinformation, combined with other feedback data,may form the basis of teaching techniques for op-timizing tool usage in MIS. The novice surgeonscould practice their skills outside the operatingroom using realistic VR simulators until they haveachieved the desired level of competence, and com-pare themselves to norms established by experi-enced surgeons.

ACKNOWLEDGMENTS

This work was supported by the U.S. Army Med-ical Research and Material Command underDARPA Grant DMAD17-97-1-725, and by a majorgrant from U.S. Surgical, a division of Tyco, Inc.,

Fig. 7. The learning curve of surgical residence whileperforming laparoscopic cholecystectomy: normalized sta-tistical distance between surgical residents at differentstages of their training (R1, R3, R5) and expert surgeonswhile performing selected steps of a laparoscopic cholecys-tectomy (exposure of the cystic duct, LC-2, and dissectionof the gallbladder, LC-3). The vertical bars around datapoints depict the standard deviations from the average sta-tistical distances between the various groups.


to the University of Washington, Center for Video-endoscopic Surgery.

REFERENCES1. Rosser JC, Rosser LE, Savalgi RS. Objective evalua-

tion of a laparoscopic skill program for residents andsenior surgeons. Arch Surg 1998;133:657–661.

2. Rosser JC, Rosser LE, Savalgi RS. Skill acquisitionand assessment for laparoscopic surgery. Arch Surg1997;132:200–204.

3. Derossis AM, Fried GM, Abrahamowicz M, SigmanHH, Barkun JS, Meakins JL. Development of a modelfor training and evaluation of laparoscopic skills.Am J Surg 1998;175:482–487.

4. Derossis AM, Antoniuk M, Fried GM. Evaluation oflaparoscopic skills: a 2-year follow-up during resi-dency training. CJS 1999;42(4):293–296.

5. Melvin WS, Johnson JA, Ellison EC. Laparoscopicskill enhancement. Am J Surg 1996;172:377–379.

6. Gallagher AG, McClure N, McGuigan J, Crothers I,Browning J. Virtual reality training in laparoscopicsurgery: a preliminary assessment of minimally inva-sive surgical trainer virtual reality (MIST VR). Endos-copy 1999;31(4):310–313.

7. Chaudhry A, Sutton C, Wood J, Stone R, McCloy R.Learning rate for laparoscopic surgical skills on MISTVR, a virtual reality simulator: quality of human–computer interface. Ann R Coll Surg Engl 1999;81(4):281–286.

8. Satava R. Virtual reality surgical simulator. Surg En-dosc 1993;7:203–205.

9. Ota D, Loftin B, Saito T, Lea R, Keller J. Virtualreality in surgical education. Comput Biol Med 1995;25:127–137.

10. Noar M. Endoscopy simulation: a brave new world?Endoscopy 1991;23:147–149.

11. Rosen J. Advanced surgical technologies for plasticand reconstructive surgery. Otolaryngol Clin N Am1998;31:357–381.

12. Bicchi A, et al., A sensorized minimally invasivesurgery tool for detecting tissue elastic properties.Proceedings of the IEEE International Conference onRobotics and Automation, 1996. p 884–888.

13. Chen E, Marcus B. Force feedback for surgical sim-ulators. Proc IEEE 1998;86(3):524–530.

14. Berkley J, Oppenheimer P, Weghorst S, Berg D,Raugi G, Haynor D, Ganter M, Brooking C,Turkiyyah G. Creating fast finite element models frommedical images. In: Westwood JD, Hoffman HM,Mogel GT, Robb RA, Stredney D, editors: MedicineMeets Virtual Reality 2000. Envisioning Healing: In-teractive Technology and the Patient–Practitioner Di-alogue. Studies in Health Technology and InformaticsVol. 70. Amsterdam: IOS Press, 2000. p 26–32.

15. Morimoto A, Foral R, Kuhlman J, Zucker K, Curet M,Bocklage T, MacFarlane T, Kory L. Force sensor forlaparoscopic Babcock. In: Morgan KS, Hoffman HM,Stredney D, Weghorst SJ, editors. Medicine Meets

Virtual Reality 1997, Global Healthcare Grid Studiesin Health Technology and Informatics vol. 39. Am-sterdam: IOS Press, 1997. p 354–361.

16. MacFarlane M, Rosen J, Hannaford B, Pellegrini C,Sinanan M. Force feedback grasper helps restore thesense of touch in minimally invasive surgery. J Gas-trointestinal Surg 1999;3(3):278–285.

17. Rosen J, MacFarlane M, Richards C, Hannaford B,Sinanan M. Surgeon–tool force/torque signatures—evaluation of surgical skills in minimally invasivesurgery. In: Westwood JD, Hoffman HM, Robb RA,Stredney D, editors: Medicine Meets Virtual Reality.The Convergence of Physical and Informational Tech-nologies: Options for a New Era in Medicine. Studiesin Health Technology and Informatics Vol. 62. Am-sterdam: IOS Press, 1999. p 290–296.

18. Rosen J, Richards C, Hannaford B, Sinanan M. Hid-den Markov models of minimally invasive surgery. In:Westwood JD, Hoffman HM, Mogel GT, Robb RA,Stredney D, editors: Medicine Meets Virtual Reality2000. Envisioning Healing: Interactive Technologyand the Patient–Practitioner Dialogue. Studies inHealth Technology and Informatics Vol. 70. Amster-dam: IOS Press, 2000. p 279–285.

19. Hannaford B, Trujillo J, Sinanan M, Moreyra M,Rosen J, Brown J, Lueschke R, MacFarlane M. Com-puterized endoscopic surgical grasper. In: WestwoodJD, Hoffman HM, Stredney D, Weghorst SJ, editors:Medicine Meets Virtual Reality. Art, Science, Tech-nology: Healthcare (R)evolution. Studies in HealthTechnology and Informatics Vol. 50. Amsterdam: IOSPress, 1998. p 265–271.

20. Rosen J, Hannaford B, MacFarlane M, Sinanan M.Force controlled and teleoperated endoscopic grasperfor minimally invasive surgery—experimental perfor-mance evaluation. IEEE Trans Biomed Eng 1999;46(10):1212–1221.

21. Buess GF, et al. Robotics in allied technologies. ArchSurg 2000;135:229–235.

22. Howe RD, Matsuoka Y. Robotics for surgery. AnnuRev Biomed Eng 1999;1:211–240.

23. Rabiner LR. A tutorial on hidden Markov models andselected application in speech recognition. Proc IEEE1989;77(2):257–286.

24. Baker JK. The DRAGON system—an overview.IEEE Trans Acoustics Speech Signal Processing1975;23(1):24–29.

25. Levinson SE, Rabiner LR, Sondhi MM. An introduc-tion to the application of the theory of probabilisticfunctions of a Markov process to automatic speechrecognition. Bell Sys Tech J 1983;62(4):1035–1074.

26. Reddy R. Foundations and grand challenges of artifi-cial intelligence: 1988 AAAI presidential address. AIMagazine, Winter 1988, p 9–21.

27. Hannaford B, Lee L. Hidden Markov model analysisof force/torque information in telemanipulation. Pro-ceedings of the 1st International Symposium on Ex-perimental Robotics, Montreal, June 1989.


28. Hannaford B, Lee P. Hidden Markov model of forcetorque information in telemanipulation. Int J RoboticsRes 1991;10(5):528–539.

29. Hannaford B, Lee P. Multi-dimensional hiddenMarkov model of telemanipulation tasks with varyingoutcomes. Proceedings of the IEEE International Con-ference on Systems, Man and Cybernetics, Los Ange-les, CA, November 1990.

30. Pook P, Ballard DH. Recognizing teleoperated manip-ulations. Proceedings of IEEE Robotics and Automa-tion, Atlanta, GA, May 1993. Vol. 2. p 578–585.

31. Nechyba MC, Xu Y. Stochastic similarity for validat-ing human control strategy models. IEEE Trans Ro-botics Automation 1998;14(3):437–451.

32. Yang J, Xu YS, Chen CS. Human action learning viahidden Markov model. IEEE Transactions on Sys-tems, Man and Cybernetics, Part A (Systems andHumans) 1997;27(1):34–44.

33. Itabashi K, Hirana K, Suzuki T, Okuma S, Fujiwara F.Modeling and realization of the peg-in-hole task basedon hidden Markov model. Proceedings of the IEEEInternational Conference on Robotics and Automa-tion, Leuven, Belgium, May 1998. p 1142.

34. Hannaford B. Hidden Markov model analysis of man-ufacturing process information. Proceedings of IROS1991, Osaka, Japan, November 1991.

35. McCarragher BJ, Hovland G, Sikka P, Aigner P, Aus-

tin D. Hybrid dynamic modeling and control of con-strained manipulation systems. IEEE Robotics andAutomation Magazine 1997;4(2).

36. Wachsmuth I, Frohlich M, editors. Gesture and SignLanguage in Human–Computer Interaction. Interna-tional Gesture Workshop Proceedings. Berlin:Springer, 1998.

37. Lien JJ, Kanade T, Cohn JF, Ching CL. Automatedfacial expression recognition based on FACS actionunits. Proceedings of the Third IEEE InternationalConference on Automatic Face and Gesture Re-cognition, Nara, Japan, 14 –16 April 1998. p 390 –395.

38. Baldi P, Brunak S. Bioinformatics. New York: MITPress, 1998.

39. Kwon K. Application of HMM to accident diagnosis innuclear power plants. Proceedings of Topical Meeting onComputer-Based Human Support Systems. AmericanNuclear Society, Philadelphia, PA, June 1995.

40. Freed P. Detecting pulsars with hidden Markov mod-els. Third International Symposium on Signal Process-ing and Applications. Vol. 1. p 179–183.

41. Rosen J, Hannaford B, Richards C, Sinanan M.Markov modeling of minimally invasive surgerybased on tool/tissue interaction and force/torque sig-natures for evaluating surgical skills, IEEE TransBiomed Eng 2001;48(5) p. 579–591.


biomedical paper - ucla | bionics labbionics.seas.ucla.edu/publications/jp_07.pdf · 2008. 9....

Documents