Real-Time In-Situ Visual Feedback of Task Performance in Mixed Environments for Learning Joint Psychomotor-Cognitive Tasks

Aaron Kotranza* D. Scott Lind

† Carla M. Pugh

‡ Benjamin Lok

*

College of Engineering Dept. of Surgery Feinberg School of Medicine College of Engineering University of Florida Medical College of Georgia Northwestern University University of Florida

ABSTRACT

This paper proposes an approach to mixed environment training of manual tasks requiring concurrent use of psychomotor and cognitive skills. To train concurrent use of both skill sets, the learner is provided real-time generated, in-situ presented visual feedback of her performance. This feedback provides reinforcement and correction of psychomotor skills concurrently with guidance in developing cognitive models of the task.

The general approach is presented: 1) Sensors placed in the physical environment detect in real-time a learner’s manipulation of physical objects. 2) Sensor data is input to models of task performance which output quantitative measures of the learner’s performance. 3) Pre-defined rules are applied to transform the learner’s performance data into visual feedback presented in real-time and in-situ with the physical objects being manipulated.

With guidance from medical education experts, we have applied this approach to a mixed environment for learning clinical breast exams (CBEs). CBE belongs to a class of tasks that require learning multiple cognitive elements and task-specific psychomotor skills. Traditional approaches to learning CBEs and other joint psychomotor-cognitive tasks rely on extensive one-on-one training with an expert providing subjective feedback. By integrating real-time visual feedback of learners’ quantitatively measured CBE performance, a mixed environment for learning CBEs provides on-demand learning opportunities with more objective, detailed feedback than available with expert observation. The proposed approach applied to learning CBEs was informally evaluated by four expert medical educators and six novice medical students. This evaluation highlights that receiving real-time in-situ visual feedback of their performance provides students an advantage, over traditional approaches to learning CBEs, in developing correct psychomotor and cognitive skills. KEYWORDS: mixed reality, information visualization.

INDEX TERMS: J.3 [Computer Applications]: Life and Medical Sciences—Health

1 INTRODUCTION

Mixed environments (MEs) are proposed to be well suited to training both psychomotor and cognitive components of real-world tasks [6]. MEs augment manipulation of physical objects with in-situ presented visual information to assist learners in becoming skilled in the psychomotor components of the task while simultaneously developing the cognitive models needed to perform the task in the real-world (i.e. without the virtual information). For example, a ME for learning a physical assembly task integrates graphical assembly instructions with the

physical hardware to be assembled [6][28]. However, evaluations of the efficacy of these MEs primarily

emphasize teaching and evaluating cognitive skills, with teaching of psychomotor skills a secondary goal (explored in Section 2).

To teach and evaluate both psychomotor and cognitive competencies, we propose leveraging real-time sensing of a learner’s manipulation of physical objects, in order to provide real-time generated in-situ visual feedback of the learner’s task performance. This feedback provides reinforcement and correction of a learner’s psychomotor actions while concurrently guiding development of cognitive models of the task.

This paper presents 1) a proposed method for providing real-time in-situ visualizations of a learner’s task performance in mixed learning environments, and 2) its application to a ME for learning clinical breast examinations (Figure 1).

Figure 1. A clinician palpates a physical breast to perform CBE of a

virtual human patient. He receives feedback of his palpation

pressure presented on an HMD or large screen display (inset).

The clinical breast exam (CBE) belongs to a class of joint psychomotor-cognitive tasks requiring concurrent learning of multiple cognitive components and task-specific (i.e. new to the learner) psychomotor skills. The CBE consists of three main concurrently occurring components, which can be classified as psychomotor or cognitive according to Bloom and Simpson’s taxonomies [5][26]: • Psychomotor: 1) Palpation of the breast in circular motions

with a specific motion at three distinct levels of palpation pressure; learning the correct pressure as a guided response (trial and error) until it becomes an overt response (accurate, skillful performance).

• Cognitive: 2) Recall of a procedural pattern, the pattern-of-search, in which the breast should be palpated; and 3) the search task for unseen breast masses – recognizing which areas of the breast remain to be palpated in order to ensure

*{akotranz, lok}@cise.ufl.edu; †dlind@mcg.edu; ‡drpugh@northwestern.edu

125

IEEE International Symposium on Mixed and Augmented Reality 2009Science and Technology Proceedings19 -22 October, Orlando, Florida, USA978-1-4244-5389-4/09/$25.00 ©2009 IEEE

complete coverage of the breast, and interpreting whether tissue feels like normal tissue or an abnormality.

Competence in CBE is required of all healthcare professionals, as CBE is an essential part of screening for the early detection of breast cancer [16]. However, current approaches to learning of CBE are hampered by a high degree of learner anxiety [20], a lack of standardized, detailed feedback, and limited opportunities for practice [24].

Current approaches to learning CBE: Accepted approaches to teaching and learning CBE include lectures covering technique, practice with silicone breast models, and evaluation through expert observation of a learner’s CBE of a human patient [24]. Using these approaches, medical students often struggle to reach competency in the psychomotor and cognitive components of the CBE. Learners struggle balancing the simultaneous psychomotor and cognitive tasks with the task of communicating with the patient. The result is a high degree of learner anxiety [20] and lack of confidence [16]. Medical educators have identified a need for increased number of practice opportunities and the incorporation of standardized, immediate, objective, and detailed feedback [24].

Proposed approach to learning CBE: Our approach to addressing the difficulties in learning CBE is the creation of a ME that incorporates real-time sensing of the learner’s physical actions in order to provide real-time in-situ visual feedback of the learner’s task performance.

This ME is an on-demand learning experience providing learners with guidance in the psychomotor and cognitive tasks of the CBE through feedback unavailable in the traditional approach of expert observation. Our approach measures learner performance objectively and quantitatively, affording feedback more fine-grained than can be provided by an expert observer. For example, without the ME’s sensing of the learner’s actions, it is not possible to ascertain whether the learner uses correct pressure [21]. The expert observer can not tell the learner how much palpation pressure to use, instead providing coarse level assistance such as press until you feel the chest wall. In our approach, real-time-captured sensor data affords presentation of real-time visual feedback indicating the correctness of the learner’s palpation pressure measured to within a gram of force (Figure 2 c-f). In this paper we will use the term fine-grained to refer to this detailed feedback based on quantitatively measured performance.

To assist learners with the psychomotor and cognitive tasks of the CBE, two visualizations have been developed:

1) The touch map (Figure 2) provides color-coded reinforcement and correction of the learner’s use of correct pressure (psychomotor) and guidance in recognizing when the entire breast has been palpated (cognitive).

2) The pattern-of-search map (Figure 3) guides the learner’s search strategy (cognitive) to follow that of an expert.

These visualizations are viewed together (Figure 4) to assist concurrent learning of the psychomotor and cognitive components of CBE.

Face validity of this feedback to provide reinforcement, correction, and guidance in psychomotor and cognitive components of the CBE has been established through informal evaluation by expert clinicians, educators, and medical students. Expert and student feedback highlights the advantages of the ME with real-time feedback of learner performance over traditional approaches to learning CBEs. For the class of joint psychomotor-cognitive tasks including CBEs, the presented approach may enhance the applicability of MEs to training joint psychomotor-cognitive tasks.

Figure 2. The touch map assists (a,b) the cognitive task of

recognizing which breast tissue remains to be palpated, (c-e)

reinforces use of correct pressure, and (f) indicates need for

correction when too much pressure is used.

Figure 3. The pattern-of-search map assists the learner in the

cognitive search task by providing the learner with an expert

search pattern to follow and feedback on the learner’s

deviation from this pattern. A learner successfully following a

vertical strip pattern (a, b), and not using a specific pattern (c).

Figure 4. The touch map and pattern-of-search map and

combination of the two for the same exam (a,b,c), and

progression of the combined visualizations (d,e).

(b)

(c)

(c) – low pressure

(d) – medium (e) – high (f) – too-high

(d) (e)

(a)

(b)

(a) (b) (c)

(a)

126

2 TRAINING PSYCHOMOTOR AND COGNITIVE COMPONENTS OF

MANUAL TASKS

Across fields such as mechanical, manufacturing, and industrial engineering, and medicine, training of manual tasks which require learning both psychomotor (i.e. coordinating muscle movements) and cognitive (e.g. spatial layout, search) components is often accomplished by pairing the learner with an expert (“learner-expert” model). The expert observes the learner performing the task and provides the learner with feedback [6][9][14]. Though often the standard, drawbacks of the “learner-expert” model include restricting a student’s practice to times the student can be observed (conversely, placing demands on an expert’s time) and the ability of experts to provide only inherently subjective feedback of varying granularity (e.g. a clinician can provide a medical student with more detailed feedback in a laparoscopic procedure than in a prostate exam, because the laparoscope allows the expert to view the student’s actions within the patient’s body; this is not the case for a prostate exam). MEs have been proposed to provide additional learning opportunities. In these MEs, the role of the expert is assumed by visual information overlaid on the physical objects being manipulated.

2.1 Benefits of MEs for Learning Joint Psychomotor-Cognitive Tasks

MEs have the potential to benefit learning joint psychomotor-

cognitive tasks:

(1) MEs provide physical manipulation simultaneous with

guidance for learning psychomotor and cognitive tasks [6].

(2) MEs may incorporate sensors (e.g. tracking) to provide

automated, objective, quantitative evaluations of learner

performance.

(3) This sensor data can be processed to provide constructive,

real-time feedback of performance (including feedback which is

unavailable in a purely physical environment). Constructive

feedback of learner performance is known to improve learning of

cognitive, psychomotor, and joint psychomotor-cognitive tasks in

physical environments [4][19].

(4) MEs can integrate this automated learner evaluation and

feedback in an on-demand learning environment. On-demand

learning environments improve learner performance by increasing

the frequency of learning opportunities and combining

information from multiple data sources [15].

2.2 Challenges in MEs for Learning

Although MEs have many potential benefits for learning joint psychomotor-cognitive tasks, there are substantial challenges to realizing these benefits. The challenges to ME developers are:

1) Overcome accuracy and latency limitations of tracking in order to accurately sense the learner’s psychomotor actions in real-time. Real-time sensing of learner actions is required to evaluate psychomotor task performance, and as more fine-motor skills are involved, the sensing approach required becomes more complex. In these cases, using only fiducial-based or glove-based tracking becomes less feasible due to the need to track complex hand and finger movement without encumbering this movement.

2) In real-time, extract quantitative measures of learner task performance from this noisy and possibly high-dimensional sensor data.

3) Provide real-time feedback of this performance data in a form meaningful to the learner [23].

From a search of MR and VR literature (Section 2.3), common approaches avoid the pitfalls represented by these challenges, but do not fully realize all potential benefits, by:

1) Focusing on only the cognitive components of the task [6][8][9][22][28]. These MEs provide feedback to guide cognitive tasks by presenting a priori acquired information, e.g. spatial relationships among task elements. Without providing feedback of psychomotor components, real-time sensing of learner actions is not required. However, intuitively, this reduces the ME’s effectiveness in training tasks requiring psychomotor skills not already well developed in the learner.

2) Having human experts process captured performance data into post-experiential feedback [10][14][18][25][29]. In this approach, real-time sensing of learner actions and evaluation of learner psychomotor performance is not required. However, reliance on offline expert analysis of learner performance lengthens the learning process and restricts the ability of the ME to provide on-demand learning opportunities.

2.3 Previous Work in Training Cognitive and Psychomotor Tasks in MEs

MEs have successfully been applied to train joint psychomotor-cognitive tasks in which the psychomotor components are previously well developed in the learner, such as assembly tasks. MEs for learning assemblies provide guidance in the form of static reference diagrams of the assembly [6] or dynamic step-by-step instructions [28]. Although these cognitive aids reduce learners’ errors and cognitive load, it has not been demonstrated that the psychomotor skills required for assembly tasks, e.g. reaching, grasping, aiming, and inserting [6] are learned within the ME or are well developed in learners prior to training with the ME. Similar approaches to presenting cognitive aids for spatial tasks have been applied in MEs for learning printer [8] and aircraft [9] maintenance. MEs also aided developing cognitive models of gas flow within anesthesia machines [22]. Similarly to MEs for learning assembly, the psychomotor skills used were simple, e.g. pressing buttons to alter gas flow, and were not a focus of the ME.

Learning of psychomotor skills in VEs and MEs has been demonstrated for two classes of tasks: tasks in which the learner uses only his or her body to interact with a virtual environment (e.g. tai chi [18], tennis [29]), and tasks which focus on the interaction between a few physical objects (manipulated by the user) and a virtual environment (e.g. rehabilitation [10], use of forceps in birthing [25], laparoscopy [14]). These systems train psychomotor tasks using the paradigm of mimicking a virtual expert (presented as pre-recorded video or a 3D reconstruction). Learner performance data is captured by tracking a small number of passive props [10], instrumented interface devices [14], or is generated offline by expert review. Incorporation of real-time feedback in these systems is uncommon and typically replicates feedback available in the physical learning environments for these tasks, e.g. patient bleeding in laparoscopic surgery [14].

In order to provide real-time visual feedback of performance, the feedback must be a function of the learner’s performance measured in real-time. This contrasts with the approach of augmenting the environment with visual elements computed from a priori measured information, e.g. as used in labeling [11] and “x-ray” vision for navigation in outdoor environments [2], and assembly and maintenance learning environments [6][8][28]. Among approaches at generating visual feedback “on the fly” (not necessarily real-time) are augmenting a driver’s view with color coded real-time-captured range data in a ME for evaluating driver awareness systems [30], integration of real-time captured ultrasound imagery for guiding biopsies [1], and the creation of visually presented Markov models for offline evaluation of learners in a ME anesthesia simulator [22].

127

Our approach of providing real-time feedback of learner performance for training psychomotor and cognitive components of manual tasks in MEs leverages the four benefits of MEs presented in Section 2.1, while addressing the challenges of Section 2.2. With respect to prior work providing feedback for learning manual tasks in MEs, the novelty of our approach is to provide real-time feedback of task performance, that is unavailable in purely physical learning environments and that reinforces and corrects psychomotor skills simultaneously with guiding development of cognitive models of the task.

3 PROPOSED APPROACH

We propose an approach to simultaneous learning of psychomotor and cognitive components of manual tasks in on-demand mixed learning environments. Real-time generated, in-situ presented, visual feedback provides learners with reinforcement and correction of psychomotor skills along with guidance in building cognitive models of the task.

Reinforcement and correction of psychomotor skills requires real-time sensing of the learner’s actions in order to indicate to the learner when his or her actions are correct and when they are incorrect. Learning cognitive task components requires visual guidance, e.g. temporal or spatial visualization of correct performance of the task, and calculation and visualization of the learner’s performance relative to correct performance.

In the proposed approach, 1) Many sensors placed on or in physical objects manipulated

by the learner capture data describing the learner’s actions in real-time. The type of sensors used is customized to the physical properties measured to evaluate task performance, e.g. in the ME for learning CBE, force sensors are used to measure palpation of a physical breast model. Flexibility in this sensing approach is gained by allowing sensors to be placed in an ad-hoc fashion. Limited fiducial-based tracking (e.g. 2D tracking with a single camera) of the learner’s hand position augments this sensing approach.

2) Data from these sensors is fed into offline created models of expert performance. These models are created automatically from a set of sensor data captured during an expert’s performance of the task in the ME. Expert performance can be considered “ideal” or “correct” performance, comparison to which allows for quantitative evaluation of the learner’s performance. Given a learner’s real-time sensor data, the models output quantitative, real-time measures of the learner’s performance relative to an expert.

3) Pre-defined rules describing the appearance of the visual feedback (e.g. color, shape of visual elements) are applied to the learner’s quantitative performance data, creating real-time visual feedback of the learner’s performance that is presented in-situ with the physical objects being manipulated.

We apply this approach to a ME for learning clinical breast exams (CBEs). Learners receive feedback providing reinforcement and correction of their palpation pressure as well as cognitive guidance in following a prescribed pattern-of-search and palpating the entire breast. This feedback takes the form of two visualizations, a touch map which visualizes use of correct palpation pressure and the area of the breast palpated (Figure 2), and a pattern-of-search map which visualizes use of correct pattern-of-search (Figure 3).

The touch map provides reinforcement and correction of the psychomotor task of palpating with correct pressure. Reinforcement is provided by seeing the palpation position colored green, indicating correct pressure – associating the

learner’s muscle movements with the knowledge that correct pressure was applied. Correction is provided from seeing the palpation position colored red, indicating inappropriately hard pressure. The touch map also guides the cognitive task of palpating all breast tissue, by coloring the breast tissue which has been palpated.

The pattern-of-search map aids in the cognitive task of recalling the correct pattern-of-search by presenting the user with an expert pattern overlaid on the breast, and visually encoding the learner’s deviation of this pattern. Guidance in building cognitive models of the task is provided from both visualizations, i.e. to find breast masses I should continue to use the amount of pressure that turns the breast green, along the pattern-of-search matching the expert’s, until I have covered the entire breast.

Section 4 describes the CBE and a ME developed for learning CBEs, and in Section 5 the proposed approach applied to training CBEs is presented in detail.

4 A MIXED ENVIRONMENT FOR LEARNING CBE

We have previously developed a mixed environment for

learning clinical breast examination [13]. This “mixed reality

human (MRH) patient” (Figure 5) integrates physical interface

elements (including a physical breast model) with a virtual human

simulating a patient with a breast mass. The physical interface

provides accurate haptic feedback and the virtual human provides

the context of patient interaction which is necessary for

developing the combination of interpersonal and physical exam

skills required for success performing CBEs [12].

This work aims to enhance the educational benefit of the MRH

by incorporating real-time visual feedback of the learner’s

performance. We hypothesize that a ME which provides a learner

with real-time feedback of the learner’s objectively and

quantitatively measured palpation pressure, coverage of the

breast, and pattern-of-search will accelerate and improve learning

of these essential clinical skills.

Three main components of a CBE which may benefit from the

proposed feedback approach are the extent, pressure, and pattern

of the examination. The extent of the examination refers to

palpating all breast tissue to ensure complete “coverage” of the

breast. Recognizing and recalling which areas have been palpated

and which remain to be palpated is a cognitive task. The

palpation pressure refers to the psychomotor task of palpating at

three levels of pressure: “low,” “medium,” and “high,” without

palpating with “too-hard” pressure which would cause the patient

pain. The pattern-of-search refers to a using systematic method

(e.g. vertical strip or “lawnmower pattern”) of ensuring that all

breast tissue is palpated. Following a prescribed pattern-of-search

aids the health care provider in the cognitive task of searching for

abnormalities, producing more effective exams [24]. Recalling

and following a specific pattern is a cognitive task.

Prior approaches to providing automated feedback of learner

performance in CBE are limited to the computer-assisted physical

learning environment of Pugh et al. Indication of which quadrants

of the breast were palpated and palpation pressure data are

presented as a series of graphs and checklists, which aid experts in

providing evaluation and feedback of the student’s CBE, but is

not presented in-situ in a form easily interpreted by novice

learners [21].

By incorporating real-time in-situ visual feedback of the

correctness of a learner’s palpation pressure, pattern-of-search,

and coverage, the ME of the MRH patient will provide medical

students with more objective and finer-grained feedback than

expert observation, more opportunities for practice, and a shorter

learning cycle (i.e. CBE-evaluation-feedback-repeat). Practice of

128

CBEs with the MRH is expected to result in long term

improvement of students’ CBEs of human patients.

Figure 5. The MRH system design. The color and IR cameras for

augmentation and expert calibration are circled. Sensors

placed in the physical breast model detect palpation of the

breast.

5 REAL-TIME IN-SITU VISUALIZATIONS OF TASK-PERFORMANCE APPLIED TO CBES

We apply the approach presented in Section 3 to provide

feedback of the clinical breast exam learner’s use of correct

palpation pressure, coverage, and use of correct pattern-of-search.

In an offline step, a model of expert palpation pressure and

pattern-of-search is created by capturing an expert’s CBE of the

MRH. During a learner’s CBEs, force sensors placed in the

physical interface of the MRH capture the learner’s palpations in

real-time. This sensor data is converted into quantitative

measures of the learner’s performance, relative to the expert.

These measures are then converted into visual feedback of the

learner’s performance, presented in real-time, in-situ with the

MRH patient.

5.1 Sensing User Manipulation of Physical Objects

To provide feedback of the learner’s palpation pressure, coverage, and pattern-of-search, a sensing approach is required which provides the pressure and position of each palpation of the physical breast.

To capture the palpation pressure, a set of force sensors are placed beneath the physical breast model (Figure 5). Force sensors are placed within the environment, rather than in, e.g., a glove worn by the learner, in order to accurately simulate the conditions of the CBE (in which feeling skin texture is important) and to maintain the previously demonstrated acceptability of learning CBE in a mixed environment [13].

No requirements are placed on the positions of the sensors (i.e. no regular pattern is required). Two versions of the ME have been developed, one with 64 sensors installed at Medical College of Georgia, and another with 24 sensors used for development of the visual feedback (the latter created the images of Figures 2-4, 7). Each force sensor reports a scalar value representing a voltage that varies linearly with the amount of force applied. The force sensors are sampled at 35 Hz with a measured maximum delay of ~90 ms between applying force and receiving the set of 64 sensor values. There is an additional maximum 20 ms of delay until corresponding visual feedback is displayed. This delay is

acceptable as CBE learners are taught that each palpation motion should have duration of 1-2 seconds.

The palpation position is found by tracking a 0.25 cm radius infrared-reflective marker placed on the middle fingernail of the hand used in palpation. The middle fingernail approximates the centroid of the index, middle, and ring finger-pads used to palpate the breast. The palpation position is determined in the image space of the camera providing the video augmentation (Figure 5), as the visual feedback is rendered into this video stream.

The palpation position and pressure data from these sensors

describes the learner’s CBE performance. To transform this data

into quantitative measures of the learner’s performance relative to

an expert, this sensor data is input into models of expert

performance.

5.2 Deriving Quantitative Measures of Task-Performance

Real-time, quantitative measures of the learner’s use of correct

pressure and pattern-of-search are computed from the real-time

sensor data. As each set of sensor data is acquired, it is input to a

model of expert task performance. This model is calculated in an

offline step by processing sensor data captured during an expert’s

CBE of the MRH. The outputs of the model are continuous

values describing the learner’s current level of palpation pressure

and deviation from the correct (expert’s) pattern-of-search.

5.2.1 Capturing Expert CBE Performance

Models of expert palpation pressure and pattern-of-search are

created by capturing an expert’s CBE of the MRH. In this expert

calibration exam, the expert wears an infrared (IR) reflective

marker covering the fingernails of the expert’s index, middle, and

ring fingers, the three fingers used in palpation. The infrared-

seeing camera (Figure 5) segments the pixels belonging to the

infrared markers at 30 Hz. This set of points is transformed into

the coordinate frame of the color-seeing camera (which provides

the video stream augmenting the MRH), resulting in the set of

image-space points Xi. The centroid of Xi is also recorded as the

palpation position, pi, for time step i. As each new set of force

sensor values Vi (values with respect to the “untouched” baseline

values of each sensor) arrives at 35 Hz, the tuple (pi, Xi, Vi) is

recorded. The area of the IR marker representing the area of the

expert’s fingertips is also calculated and the radius r of a circle

approximating this area is recorded. The recorded data is

processed offline to create models of correct pressure and correct

pattern-of-search.

5.2.2 Model of Correct Pressure

Using correct palpation pressure consists of progressing

through three distinct levels of pressure: low (superficial),

medium, and high (pressing to the chest wall); pressure that would

cause pain (“too-high”) should not be used.

The model of correct pressure consists of determining the range

of sensor values corresponding to the low, medium, high, and too-

high pressure levels at each possible palpation position. Because

of the high dimensionality of the sensor data (64 real values, one

from each sensor) and the large size of the space of possible

palpation positions (order of 105 in the 640x480 image), we

instead model the pressure ranges at each sensor. During the

learner’s exam, the pressure level is calculated at each sensor, and

then the pressure level at the palpation position is calculated as a

weighted average of the pressure levels at the sensors.

Modeling pressure ranges at each sensor avoids the

computational expense of working with high dimensional sensor

data. At each sensor, a pressure level can be modeled in one

129

dimension – if pressure levels were modeled at each palpation

position, one dimension per sensor would be required.

Modeling pressure ranges at each sensor: The low, medium,

and high pressure ranges are naturally present in the sensor data of

the expert’s exam. Calculating these ranges is an unsupervised

learning problem which can be solved using clustering. For each

sensor, initial estimates of the clusters Ck corresponding to low,

medium, and high pressure ranges are calculated using k-means

(k=3) applied to the non-zero values reported by the sensor.

}minarg|,{2

kxXC kxkkk =−= µµ

With k∈{low, medium, high}, Xk the set of image-space

points belonging to cluster Ck, and µk the mean of Xk.

(1)

The estimated clusters for sensor s serve as initial values for an

expectation-maximization algorithm fitting a Gaussian Mixture

Model (GMM) to the sensor’s non-zero data. The resulting GMM

contains three one-dimensional Gaussian distributions

corresponding to the naturally present low, medium, and high

pressure ranges. The too-high pressure range is then modeled as

an additional one-dimensional Gaussian component, which is a

shifted (by an experimentally chosen +2.5 std. deviations)

duplicate of the high pressure component. Each distribution takes

the form ),|(2

, kkskGMM vN σµ , k∈{low, medium, high, too-high}.

Modeling the effect of palpation position on each sensor’s

value: Given the palpation pressure ranges at each sensor, we

wish to calculate the palpation pressure at the learner’s palpation

position. This requires knowledge of how palpating at each

position on the breast affects the value of each sensor. Intuitively,

the value reported by sensor s increases as the palpation position

approaches the position of sensor s. This relation between

palpation position and the pressure at sensor s is modeled as a 2D

Gaussian centered at the position of sensor s in image-space.

The mean of the 2D Gaussian distribution ),|( ,2,2,2 ssimgs xN Σµvv

of sensor s is calculated as the weighted mean of expert palpation

positions corresponding to non-zero values of sensor s. The

values of sensor s serve as the weights in this calculation.

To reduce the impact of noise in the sensor data, this

calculation includes only those palpation positions corresponding

to values of sensor s that are one std. deviation greater than the

mean value reported by sensor s during the expert exam. Letting

Xs,VALID be the set of these positions and Vs,VALID be the set of

corresponding values of sensor s, the mean is calculated by Eq. 2.

This adaptive thresholding heuristic calculates the sensor’s

position in image-space to within the radius of the sensing

element, resulting in ~5 pixels (<0.5 cm) of error (to reduce this

error, a smaller sensing element could be used). The covariance

of the 2D Gaussian is calculated as the weighted covariance, again

with the sensor’s values as weights. Thresholding is not used in

the covariance calculation, to preserve the notion of the

covariance as the “area” of palpation positions in which the sensor

reports non-zero pressure.

),(),(, ,,,2 VALIDsVALIDss VXvxv

vx∈

∗=∑∑ v

vvµ (2)

Calculating level of pressure at the learner’s palpation position: The model of correct pressure can be described completely by the four-component 1D GMM and the 2D Gaussian at each sensor. At each simulation timestep during the learner’s exam, the model is evaluated at the current set of sensor values V, and the model returns a continuous value in the range [1 = low, 4 = too-high] representing the learner’s palpation pressure level.

Given the set of sensor values V and the learner’s palpation position ximg reported by 2D infrared tracking, the learner’s palpation pressure level is calculated using Eq. 3 and Eq. 4.

For each sensor s with non-zero value vs, calculate the

pressure level ls at sensor s as:

∑

∑ ∗

=

k

kkskGMM

k

kkskGMM

svN

vNk

l),|(

),|(

2

,

2

,

σµ

σµ

with k∈{1 = low, 2 = medium, 3 = high, 4 = too-high}.

(3)

The pressure level xl v at imgx

v is calculated as a weighted

average of pressure levels at each sensor:

∑

∑

Σ

Σ∗

=

s

ssimgs

s

ssimgss

xxN

xNl

l),|(

),|(

,2,2,2

,2,2,2

µ

µ

vv

vv

v

(4)

Figure 6. Informal correctness of the model is demonstrated by

showing that the output of the model fits the expected

progression of pressure levels. Shown here is data for a

single sensor. This behavior is consistent across all sensors

and repeatable across multiple calibration data sets.

The correctness of this model is evaluated informally by

demonstrating that it produces, for the range of all sensor values,

output consistent with 4 distinct levels of pressure. It is expected

that as the user reaches either of the low, medium, and high levels

of pressure, there is a range of sensor values for which the

pressure remains in the same level. The values of these ranges

vary with the thickness of the breast tissue and can not be known

a priori – these ranges are discovered by fitting of the GMM to

the expert calibration data. Also, as the user transitions between

levels, the output of the model should be approximately linear, as

the value returned by a force sensor scales linearly with the force

applied. The model reproduces this expected behavior (Figure 6).

We have explored other methods for modeling correct palpation

pressure, including the naïve approach of finding a nearest

neighbor in the expert’s sensor data to the learner’s sensor data,

but the high-dimensionality of the sensor data makes these

approaches too computationally expensive to evaluate in real-

time. In contrast, the presented model is computationally

inexpensive (evaluation of five Gaussian distributions at each

active sensor with typically less than five sensors active at once),

allowing the learner’s use of correct pressure to be evaluated in

real-time. The model is presented in detail as a roadmap for

130

application designers to create real-time-evaluable models of

expert performance from high-dimensional data.

5.2.3 Model of Correct Pattern-of-Search

A model of correct pattern-of-search takes as input a recent

subset of the learner’s palpation positions, and outputs a

quantitative measure of the learner’s deviation from expert

pattern-of-search. Modeling correct pattern-of-search consists of

recovering the expert’s pattern from the palpation position data of

the expert calibration exam, and creating a real-time evaluable

model to calculate the learner’s deviation from this expert pattern.

Recovering expert pattern-of-search: The set of palpation

positions },...,{ 1 nppPvv

= captured in the expert calibration exam

contains clusters corresponding to each distinct palpation. This is

shown for a vertical strip pattern in Figure 7a. The centroids of

these clusters are calculated by processing P in temporal order and

creating a new cluster when the distance between the current

cluster’s centroid and the next position pi is greater than the radius

r of a circle representing the area the expert’s fingertips cover in

each palpation. Resulting centroids are shown in Figure 7b.

Because the noise present in the IR tracking of the expert’s

palpation positions influences the centroid calculation, the

centroids are then filtered in temporal order by convolving with

the neighborhood (¼, ½, ¼). The final expert path is created by

constructing a Catmull-Rom spline with the filtered centroids as

control points (Figure 7c). The Catmull-Rom spline was chosen

because it passes through all control points. Direction indicators

are added when rendering the expert path (Figure 7e). The spline

reconstruction of the expert pattern is stored as a sequence of line

segments, S, which will be used to evaluate the learner’s pattern –

also represented a sequence of line segments L between the

learner’s successive distinct palpation positions.

Figure 7. Modeling pattern-of-search: Expert palpation position

data (a) is clustered (centroids in b), filtered, and interpolated

(c), resulting in the vertical strip pattern with direction arrows

added (e). In (f), the output of this process applied to a spiral

pattern-of-search. In (d), the scalar field used to calculate

learner deviation from the pattern of (e) is shown.

Evaluating learner pattern-of-search: Evaluating learner

pattern-of-search differs from prior approaches to evaluating

learner trajectories relative to expert trajectories (e.g. [25]). In

CBE, the learner’s goal is to use the same pattern-of-search as the

expert, but it is not necessary to follow the expert’s trajectory

exactly – for example, learner’s should not be penalized for

following the pattern in reverse temporal order or for a small

translation between learner and expert patterns. Thus, Euclidean

distance is a poor metric of learner deviation from the expert

pattern. Deviation from the expert pattern is instead taken to be

the angle between matched segments of learner and expert

patterns.

We experimented with a naïve approach to matching learner

and expert patterns: the two nearest (in Euclidean distance) expert

segments to the current learner segment were found, and the

deviation calculated as the average of the angles between the

learner segment and the two expert segments. However, this

approach exhibited poor performance in portions of the pattern

with high curvature and penalizes small translational offsets

between learner and expert. An approach which avoids these

problems is to not explicitly define a matching between learner

and expert segments, but instead to create from the expert pattern

a vector field which serves as a look-up-table.

Our approach is to place radial basis functions of the form

)exp()(2

2

iimgimgi mxrxfvvv

−−= − at the midpoints of the line segments

of the expert’s pattern, where mi is the midpoint of segment si and

r is the radius of the circle representing the area of each palpation.

Each radial basis is associated with the normalized vector ŝi of the

line segment at which it is placed. The vector field value at ximg is

calculated by Eqn. 5. Instead of storing the vector field, a scalar

field is stored to simplify computation during the learner’s exam.

The scalar field contains the absolute value of the dot product of

v(ximg) with the reference vector (0,1). The absolute value causes

forward and reverse traversal of the pattern to be equivalent. This

scalar field is visualized in Figure 7d.

{ } ∑∑ ∗=i

imgi

i

iimgiimg xfsxfxv )(ˆ)()(vvvv

(5)

To calculate the deviation of the current line segment of the learner’s pattern, the scalar field values s(x1), s(x2) at the endpoints of the segment are retrieved and the dot product d between the learner’s current line segment li and the reference vector calculated. The learner’s deviation from the expert pattern is then calculated as the average of the differences |d-s(x1)| and |d-s(x2)|.

The quantitative measures of user performance output by the

models of correct pressure and pattern-of-search are converted to

visual feedback by applying pre-defined rules governing the

appearance of the visual elements of the feedback.

5.3 Rule-based Generation of Real-Time Visual Feedback

To provide visual feedback of the learner’s performance, pre-

defined rules governing color and shape of the visual feedback are

applied to the measures of learner performance. These rules

define two visualizations, the touch map and the pattern-of-search

map.

5.3.1 Touch Map

The touch map provides visual feedback of the learner’s use of

correct pressure and coverage of the breast. The touch map

applies two rules to present this information visually: the

coverage is encoded as visual element shape, a circle, and the

pressure is encoded as the color of the visual elements, a

multicolored scale with distinct colors at each of the four pressure

levels.

Because each palpation consists of applying pressure using a

circular motion, the shape of each visual element of the touch map

is a circle. The radius of the circle is calculated during the expert

calibration exam to provide a circle of approximately the area the

expert’s middle finger covers during the palpation motion. This

underestimation of the area felt by the three middle fingers in each

palpation motivates the learner to overlap successive palpations –

part of the desired technique [24]. Each palpation of the breast

results in one of these circular elements. The union of these

(a) (b) (c)

(e) (f)

(d)

131

circles represents the area of the breast tissue palpated, guiding

the learner in palpating the entire breast.

The level of pressure (low, medium, high, too-high) the learner

uses is represented by the color of this circle. A multicolored

scale with a distinct color at each pressure level was chosen, as

multicolored scales are preferred for identification tasks (i.e.

identifying areas of the breast which have not been palpated with

low, medium and high pressure) [7]. The colors chosen for each

pressure level are influenced by prior information visualization

literature and discussion with medical educators. The ability of

the color scale to convey use of correct pressure is evaluated as

part of the informal evaluation presented in Section 6. As a blue-

green-yellow scale best encodes continuous data [17], these colors

are chosen for the low, medium, and high pressure levels (low =

blue, medium = yellow, high = green). The order of green and

yellow were swapped so that green’s connotation with good

would match good to reaching the high pressure level. Red was

chosen for the too-high pressure level, as red connotes stop.

Given the continuous pressure level value lx outputted by the

model of correct pressure, the color of the visual element is

calculated by linearly interpolating between the colors at the

neighboring pressure levels floor(lx) and floor(lx)+1.

5.3.2 Pattern-of-Search Map

The pattern-of-search map provides visual feedback of the

learner’s following and deviation from the expert’s pattern-of-

search.

The rules of the pattern-of-search map are to encode the

progression of the search pattern as a series of arrows, and to

encode the deviation of the student’s pattern from the expert’s

pattern as a multicolored scale.

Patterns of search are typically presented in medical texts as a

series of arrows (e.g. [24]). Thus, the series of line segments

which reconstruct the learner’s pattern are visualized as a series of

arrows which point towards increasing time. The arrows are

rendered as vector graphics, with parameters of arrow tail and

head widths and arrow length.

The color of each arrow represents its deviation from the

expert’s pattern-of-search. We chose a three-colored scale of

green, yellow, and red, as with traffic lights (go, caution, stop).

Green encodes that the student deviates by less than 15 degrees

(low deviation range); yellow that the student deviates between 15

and 30 degrees (medium deviation range); and red encodes

deviation of greater than 30 degrees (high deviation range). As

with the color of the touch map elements, the color of an arrow is

calculated by linearly interpolating between the two neighboring

ranges of the learner’s deviation value.

5.3.3 Rendering the Visual Feedback

An image-based approach is taken to locating the visual

feedback in-situ with the physical breast model being

manipulated. The visual elements of the touch map and the

pattern-of-search map are rendered into the real-time video stream

of the learner’s hands and the physical breast, which augments the

MRH patient. Fragment shaders and render-to-texture are used to

simulate an accumulation buffer for each of the touch map and

pattern-of-search map. As the touch map accumulates visual

elements, existing color is overwritten only by color representing

higher pressure. The final color of each touch map element thus

represents the highest pressure used at that position. For the

pattern-of-search map, more recent elements occlude older

elements. For each frame of video, the latest touch map and

pattern-of-search map (in that order) are alpha blended with the

video. The video frame is then projected onto the mesh of the

MRH patient.

5.4 Design Choices

How many experts are needed to build a model of

performance?: We chose to base models of expert performance

on a single expert’s CBE. Using a single expert is standard in

MEs which teach psychomotor skills through mimicking a pre-

recorded expert performance [10][18][25]. However, the validity

of the models of expert performance may be increased by

incorporating data from multiple experts. The model of correct

palpation pressure is trivially extended by having multiple experts

perform calibration exams in succession and processing the

aggregate data. We expect a single expert to be adequate for

modeling the pattern-of-search as it is highly standardized [3],

however we will verify as more experts test our MRH system.

Visual feedback elements and the learner’s hands: The

elements of the visual feedback are rendered “on-top” of the video

stream of the learner’s hands and the physical breast model to

guarantee the visibility of the feedback. Feedback elements are

rendered with (learner adjustable) partial transparency to prevent

occlusion of the learner’s hands. However, this approach may

still make it difficult for learners to locate their fingertips in the

ME. We have experimented with adaptive-k Gaussian mixture

models [27] for segmenting the learner’s hands from the video

stream (to render the feedback beneath the fingertips), but have

not tested this with end-users because the computational

complexity dictates the need to first adopt a more distributed

computing approach.

Drawbacks of the approach: Because we take an image-

based approach to locating and rendering the visual feedback, a

new expert calibration exam is required if the cameras are moved

relative to the physical breast model. In practice, we have made

the relative arrangement of the physical breast model and the

cameras permanent in an installation of the MRH at the Medical

College of Georgia. Recalibration has not been required after

approximately 60 uses.

6 INFORMAL EXPERT AND LEARNER EVALUATION

Expert clinicians and 2nd-year medical students provided

informal feedback concerning the efficacy of the real-time, in-situ

visual feedback to assist learning of the CBE.

After receiving lecture-based CBE instruction, six 2nd-year

medical students performed their first CBE using the MRH

integrating the touch map visualization (but not the pattern-of-

search map). All students reported that the touch map assisted

them in palpating the entire breast and in using correct pressure,

and felt that receiving this feedback was valuable in the learning

process, e.g.:

• “Being able to see which areas I had covered, helped me

to examine the entire breast.”

• “The map helped me realize how deep I was pressing, and

how large an area I covered.”

• “It was good being able to watch and see when I hadn't

gone far enough or when my next row was too far over.”

• “I thought the touch map was an excellent way to learn

how to do a breast exam. It was nice to feel how deep you

are supposed to palpate.”

• “I thought the most important feature was the pressure

sensing capability. [Before using the ME] we don't really

know how much force to apply and I think this would be

very useful if implemented into our curriculum prior to our

132

3rd year rotations [during which students are graded on

CBE].”

Students’ comments indicate that their psychomotor skill of

palpating with correct pressure was undeveloped before using the

ME, and that the touch map feedback assisted in developing this

psychomotor skill by allowing them to link the amount of force

applied with the level of pressure visualized. Students also

indicated that the touch map assisted them in the cognitive task of

palpating the entire breast, helping them keep track of the area

they had palpated and making sure adjacent palpations were

adequately close together.

All students also reported that the multicolor scale of the touch

map was easily interpreted, though one student suggested

alternately using the progression of colors of the visible light

spectrum.

Additionally, four expert clinicians who have each conducted

thousands of CBEs informally evaluated the touch map and

pattern-of-search map feedback, providing feedback concerning

how the real-time visual feedback could benefit students learning

the CBE.

• Expert 1: “One of the biggest complaints students have

when they finish medical school is that they don’t have

faculty or qualified people observe them conducting a

breast exam to the extent that they would like. …This is

information that they wouldn’t get otherwise. …This

technology allows [students] to get real-time feedback on

the quality of their exam and the ability to incorporate that

feedback into their learning and improve their

examination skills, so I think it could have a significant

impact on their ability to perform breast exams in the

future.” • Expert 2: “This seems good for making sure students

exam all portions of the breast. I have not seen much focus on pressure before. …I am sure it will help students with their own comfort level in the CBE which can be a hurdle for some students.”

• Expert 3: “Feedback is probably much better than what students normally receive. Particularly with the pressure feedback.”

• Expert 4: “Too often the breasts are provided only a cursory exam. The idea of placement, appropriate pressure and pattern of examination is quite useful as an educational tool. …[The expert expects that] having had experience on the mannequin will give them more confidence and understanding and will enable them to complete an exam on a living patient more efficiently.”

The expert clinicians identified that the touch map and pattern-

of-search map visual feedback provided students with feedback

on palpation pressure, pattern-of-search, and coverage of the

breast which students are not provided in purely physical learning

environments. The experts also remarked on the potential of the

real-time visual feedback to overcome barriers to learning CBEs,

such as student comfort (anxiety) and confidence.

Both expert and student feedback suggests that the

incorporation of visual feedback in the ME for learning CBE will

improve learning of the CBE in the ME – and that the visual

feedback provides the ME with an advantage, over traditional

CBE teaching approaches, in developing correct psychomotor and

cognitive skills.

7 CONCLUSIONS AND FUTURE WORK

In this paper we have presented an approach to providing real-

time in-situ visual feedback of a learner’s task performance in a

ME for learning joint psychomotor-cognitive tasks. This

approach leverages sensing customized to the task to calculate

quantitative measures of a learner’s performance and applies pre-

defined rules to transform these measures into visual feedback.

The presented approach has been applied to create two

visualizations of learner performance, the touch map and the

pattern-of-search map to assist novice medical students in

learning the clinical breast exam in a mixed environment. These

visualizations provide visual feedback of spatial, temporal, and

other measured physical data (e.g. pressure). Visual feedback of

these classes of data may benefit many other applications of MEs

for learning joint psychomotor-cognitive tasks, and can be

generated in real-time from a user’s interaction by applying the

general approach presented in this paper. We expect such

incorporation of real-time in-situ visual feedback of learner task

performance to improve learning of joint psychomotor-cognitive

tasks in MEs.

Future work will include exploring additional visual feedback

that can assist students in learning CBEs, e.g. detecting when the

student palpates a breast mass and providing visual feedback that

will assist the student in linking what he feels to information

about the mass – fixed, mobile, malignant, and benign. Work is

underway to apply the presented approach to MEs for training

additional medical exams, e.g. prostate and rectal exams.

To evaluate the impact of visual feedback on learning, we will

conduct a longitudinal study with medical students. This study

will seek to determine if learning the CBE using the MRH and

real-time visualization of task performance leads to improved

CBE performance on human patients, and whether visual

feedback is also able to reduce students’ cognitive load during the

exam, enabling them to also focus on improving their

communication with the patient. Based on expert evaluation of

the visual feedback, we expect the presented approach to improve

learning of the joint psychomotor-cognitive task of clinical breast

examination.

ACKNOWLEDGEMENTS

This work is supported by NSF Grant IIS-0803652, an

Association for Surgical Education CESERT Grant, and a

University of Florida Alumni Fellowship. Special thanks go to

Kyle Johnsen, Candelario Laserna, and evaluation participants.

Feedback and MRH technologies are patent pending.

REFERENCES

[1] M. Bajura, H. Fuchs, R. Ohbuchi, “Merging Virtual Objects with the

Real World: Seeing Ultrasound Imagery within the Patient,”

Computer Graphics, vol. 26, no 2, July 1992, p203-210.

[2] R. Bane and T. Hollerer, "Interactive tools for virtual x-ray vision in

mobile augmented reality," Proceedings of the 3rd IEEE and ACM

International Symposium on Mixed and Augmented Reality (ISMAR

2004), Nov. 2-5, 2004, pp. 231-239.

[3] M.B. Barton, R. Harris, and S.W. Fletcher, Does this patient have

breast cancer? the screening clinical breast examination: should it be

done? How? Journal of the American Medical Association, 1999. vol

282, 270-1280

[4] E.A. Bilodeau and I.M. Bilodeau. Motor skills learning. Annual

Reviews of Psychology, 1962, vol. 13, pp. 243–280.

[5] B.S. Bloom, Taxonomy of Educational Objectives, Handbook I: The

Cognitive Domain. New York: David McKay Co Inc, 1956.

[6] A.C. Boud, D.J. Haniff, C. Baber, and S.J. Steiner, "Virtual reality

and augmented reality as a training tool for assembly tasks,"

Proceedings of the 1999 IEEE International Conference on

Information Visualization, pp. 32-36.

[7] L.A. Breslow, T.J. Gregory, R.M. Ratwani, A perceptual process

approach to selecting color scales for complex visualizations,

133

Journal of Experimental Psychology: Applied, Mar. 2009, vol. 15,

no. 1, pp. 25-34.

[8] S. Feiner, B. MacIntyre, and D. Seligmann, Knowledge-based

augmented reality. Communications of the ACM, vol. 36, no. 7, July

1993, 52-62.

[9] T. Haritos, N.D. Macchiarella, "A mobile application of augmented

reality for aerospace maintenance training," Proceedings of the 24th

Digital Avionics Systems Conference, 2005. DASC 2005. Oct. 30 –

Nov. 5, 2005.

[10] M.K. Holden, Virtual environments for motor rehabilitation: review.

Cyberpsychology and Behavior. Jun. 2005, vol.8, no.3, pp.187-211.

[11] S. Julier, M. Lanzagorta, Y. Baillot, L. Rosenblum, S. Feiner, T.

Hollerer, and S. Sestito. Information Filtering for Mobile Augmented

Reality. In Proceedings of IEEE International Symposium on

Augmented Reality (ISAR 2000), pp. 3–11, October 2000.

[12] R. Kneebone, J. Kidd, D. Nestel, S. Asvall, P. Paraskeva, A. Darzi.

(2002). An innovative model for teaching and learning clinical

procedures. Medical Education, vol. 36, 628-634.

[13] A. Kotranza, B. Lok, A. Deladisma, C.M. Pugh, D.S. Lind, “Mixed

Reality Humans: Evaluating Behavior, Usability, and Acceptability,”

IEEE Transactions on Visualization and Computer Graphics, vol.

15, no. 3, May-June 2009, pp. 369-382.

[14] G. Lacey, D. Ryan, D. Cassidy, D. Young, "Mixed-Reality

Simulation of Minimally Invasive Surgeries," IEEE MultiMedia, vol.

14, no. 4, pp. 76-87, October-December, 2007.

[15] H.A. Latchman and S. M. Latchman, “Lectures on Demand in ALN–

Enhancing the Online Learning Experience” Journal of

Asynchronous Learning Networks, June 2001, vol. 5, no. 1, pp. 85-

98.

[16] A.K. Madan, et al., Education of medical students in clinical breast

examination during surgical clerkship. Am J Surg, 2002. 184(6): p.

637-40; discussion 641.

[17] D.H Merwin and C.D. Wickens. Comparison of eight color and gray

scales for displaying continuous data. In Proceedings of the Human

Factors Society, 1993, v. 2, 1330-1334.

[18] K. Patel, S. Jung, R. Diankov, R. Bajcsy, and J.N. Bailenson, The

effects of fully immersive virtual reality on the learning of physical

tasks, Proceedings of the 9th Annual International Workshop on

Presence, 2006, pp. 87-94.

[19] C. Pilgrim, et al., Improving clinical breast examination training in a

medical school: a randomized controlled trial. J Gen Intern Med,

1993. vol. 8, no. 12, pp. 685-688.

[20] C.M. Pugh and L.H. Salud, Fear of missing a lesion: use of

simulated breast models to decrease student anxiety when learning

clinical breast examinations. American Journal of Surgery, 2007.

vol. 193, no. 6, pp. 766-770.

[21] C.M. Pugh, Z.B Domont, L.H. Salud, and K.M Blossfield. A

Simulation Based Assessment of Clinical Breast Examination

Techniques: Do patient and clinician factors affect clinician

approach? American Journal of Surgery. 195(6):874-80, 2008.

[22] J. Quarles, S. Lampotang, I. Fischler, P. Fishwick, and B. Lok, “A

Mixed Reality System for Enabling Collocated After Action

Review" In Proceedings of the 7th IEEE and ACM International

Symposium on Mixed and Augmented Reality (ISMAR 2008), Sept.

15-18, Cambridge, UK, 107-116.

[23] A. Rizzo and G.J. Kim. A SWOT analysis of the field of virtual-

reality rehabilitation and therapy, Presence: Teleoperators and

Virtual Environments, 2005, vol. 14, no. 2, pp.119–146.

[24] D. Saslow et al., “Clinical Breast Examination: Practical

Recommendations for Optimizing Performance and Reporting,” CA:

Cancer J. Clinicians, vol. 54, pp. 327-344, 2004.

[25] T. Sielhorst, T. Blum, and N. Navab, "Synchronizing 3D movements

for quantitative comparison and simultaneous visualization of

actions," Proceedings of the 4th IEEE and ACM International

Symposium on Mixed and Augmented Reality (ISMAR 2005), Oct. 5-

8, 2005, pp. 38-47.

[26] E.J. Simpson, The Classification of Educational Objectives in the

Psychomotor Domain. Washington, DC: Gryphon House, 1972.

[27] R. Tan, H. Huo, J. Qian, and T. Fang, Traffic Video Segmentation

Using Adaptive-K Gaussian Mixture, Proceedings of the

International Workshop on Intelligent Computing in Pattern

Analysis/Synthesis (IWICPAS 2006), Xi’an, China, Aug. 26-27,

2006, pp. 125-134.

[28] A. Tang, C. Owen, F. Biocca, and W. Mou, Comparative

effectiveness of augmented reality in object assembly. In

Proceedings of the SIGCHI Conference on Human Factors in

Computing Systems (CHI 2003), Ft. Lauderdale, Florida, USA, April

05 - 10, 2003, pp. 73-80.

[29] E. Todorov, R. Shadmer, and E. Bizzi. Augmented feedback

presented in a virtual environment accelerates learning of a difficult

motor task. Journal of Motor Behavior, 1997, vol. 29, pp. 147–158.

[30] M. Tonnis, C. Sandor, G. Klinker, C. Lange, and H. Bubb,

“Experimental evaluation of an augmented reality visualization for

directing a car driver’s attention,” Proceedings of the Fourth IEEE

and ACM International Symposium on Mixed and Augmented

Reality (ISMAR 2005), pp. 56-59, 5-8 Oct. 2005.

134