abbreviations - kevincurran.org€¦ · web viewbsc (hons ) computer science ... through the...

Using Leap Motion and Gamification to

Facilitate and Encourage Rehabilitation in

those with Hand Injuries

Jamie Taylor

B00593273

[email protected]

BSc (Hons) Computer Science

School of Computing and Intelligent Systems

University of Ulster

Supervisor: Dr. Kevin Curran

December 2013

mailto:[email protected]

Declaration

I declare that this is all my own work and does not contain unreferenced

material copied from any other source. I have read the University’s policy on

plagiarism and understand the definition of plagiarism. If it is shown that

material has been plagiarised, or I have otherwise attempted to obtain an

unfair advantage for myself or others, I understand that I may face sanctions

in accordance with the policies and procedures of the University. A mark of

zero may be awarded and the reason for that mark will be recorded on my file.

- Jamie Taylor

of 130

Acknowledgements

I would like to extend my sincere thanks and gratitude to Dr Kevin Curran who

has been a tremendous help not only throughout this project, but throughout

all of my student career for which he has been a part.

I would also like to thank James Connolly for his vital contributions to the

research and development stages of the project.

of 130

Abstract

Injuries to the hand are more common than those of any other body region

and can have considerable financial, time-measured and psychological impact

on not only the victim but the community as a whole. Hand rehabilitation aims

to return people to their pre-injury roles and occupations and has proved

largely successful in doing so with the potential for technology to improve

these results further. However, most technology used in hand rehabilitation is

based on expensive and non-durable glove-based systems and issues with

accuracy are common among those which are not glove-based.

We propose an accurate, affordable and portable solution wherein we use the

Leap Motion as a tool for hand rehabilitation. User feedback will be given

primarily through an animated 3d hand model as the user performs

rehabilitative exercises. Exercise results will be recorded for later viewing by

patients and clinicians. The system will also include Gamification aspects,

techniques which (while proven to increase participation) have seen little to no

use in hand-rehabilitation systems.

In response to this need for new rehabilitation technologies, we have

developed a functional rehabilitation system using the Leap Motion. This

system is referred to as the Leap Motion Rehabilitation System or LMRS. The

LMRS delivers on the points described above and produces medically

relevant data accurate to within 100 milliseconds. The proof-of-concept that is

the LMRS coupled with the price and relative accuracy of the Leap Motion in

addition to its other unique qualities mean the LMRS represents the beginning

of a promising new avenue with regards to the use of technology in

rehabilitation.

of 130

ContentsAbbreviations.................................................................................................7

Table of Figures............................................................................................8

List of Tables.................................................................................................9

Table of Source Code Listings......................................................................9

Table of Graphs.............................................................................................9

1: Introduction.................................................................................................10

1.1 Existing Approaches..............................................................................10

1.2 Project Aim............................................................................................11

1.3 Chapter Overview..................................................................................11

2: Background and Related Work...................................................................12

2.1 Hand Injuries: Types, Occurrence and Impact......................................12

2.2 Treatment and Rehabilitation of Hand Injuries......................................14

2.3 The use of Glove-Based Technology in Hand Rehabilitation................16

2.4 The use of Non Glove-Based Technology in Hand Rehabilitation.........20

2.5 The Role and Potential of Gamification.................................................23

3: Requirements Analysis and Specification...................................................26

3.1 Problem Statement................................................................................26

3.2 Functional Requirements.......................................................................26

3.3 Non-Functional Requirements...............................................................27

3.4 Hardware Requirements........................................................................28

3.5 Software Requirements.........................................................................28

3.6 Development Methodology....................................................................28

4: Project Planning..........................................................................................30

4.1 Milestones and Deliverables..................................................................30

4.2 Project Plan...........................................................................................31

4.3 Time Management.................................................................................32

4.4 Risk Management..................................................................................34

5: Conclusion..................................................................................................36

6: Design.........................................................................................................37

6.1 System Component Overview...............................................................37

6.2 HCI Rules & Guidelines.........................................................................37

of 130

6.3 UI Mock-Ups..........................................................................................40

6.4 Diagrams (Use case and Activity Model)...............................................45

7: Implementation...........................................................................................48

7.1 Technical Overview...............................................................................48

7.2 XML Data Store.....................................................................................48

7.3 Patient Side...........................................................................................49

7.4 Clinician Side.........................................................................................65

8: Evaluation...................................................................................................73

8.1 Testing...................................................................................................73

8.2 Evaluation against Initial Requirements................................................77

8.3 Future Work and Enhancements...........................................................78

9: Conclusion..................................................................................................80

10: References...............................................................................................81

11.1: Appendix 1: Clinician User Scenario Test Case....................................85

11.2: Appendix 2: Patient User Scenario Test Case.......................................88

11.3: Appendix 3: Source Code......................................................................90

of 130

AbbreviationsVR – Virtual Reality

MMORPG – Massively Multiplayer Online Role-Playing Game

MOBA – Massively Online Battle Arena

IR – Infrared

WBS – Work Breakdown Structure

RAG – Red Amber Green

FPS – Frames per Second

IPP – Intel Performance Primitive

HCI – Human Computer Interaction

UI – User Interface

LMRS – Leap Motion Rehabilitation System

XML – Extensible Mark-up Language

WPF – Windows Presentation Foundation

of 130

Table of FiguresFigure 1 - Hand Therapy Session...................................................................16Figure 2 - CODA System................................................................................16Figure 3 - 5DT Data Glove Ultra.....................................................................17Figure 4 - GloveManager Software.................................................................17Figure 5 - HumanGlove..................................................................................18Figure 6 - Graphic Virtual Hand Software.......................................................18Figure 7 - Peregrine Gaming Glove................................................................19Figure 8 - GloveBox Configuration Software..................................................19Figure 9 - Rheumatoid Arthritis Hand.............................................................19Figure 10 - Rheumatoid Arthritis Hand...........................................................19Figure 11 - Kinect...........................................................................................21Figure 12 - Kinect in Hand Rehabilitation Setting...........................................21Figure 13 - Leap Motion..................................................................................22Figure 14 - Leap Motion Schematic View.......................................................22Figure 15 - Leap Motion Internals...................................................................23Figure 16 - Leap Motion Visualizer.................................................................23Figure 17 - Xbox 360 Achievements...............................................................24Figure 18 - Khan Academy Stat Tracking and Achievement System.............24Figure 19 - Gamified Application, Walking through a Forest..........................25Figure 20 - CONTRAST Serious Rehabilitation Game...................................25Figure 21 - Work Breakdown Structure...........................................................31Figure 22 - Gantt chart....................................................................................33Figure 23 - System Architecture.....................................................................37Figure 24 - Login Screen................................................................................41Figure 25 - Clinician Main Menu Screenshot..................................................41Figure 26 - Patient Main Menu Screenshot....................................................41Figure 27 - Clinician version...........................................................................42Figure 28 - Patient Version.............................................................................42Figure 29 - Register Patent.............................................................................43Figure 30 - Patient Screen..............................................................................43Figure 31 - Rehabilitation Exercise Screen.....................................................43Figure 32 - Patient User & Clinician User......................................................45Figure 33 - Patient User Interactions.............................................................46Figure 34 - Clinician User Interactions...........................................................47Figure 35 - LMRS Login Screen.....................................................................49Figure 36 - LMRS Perform Exercises Screen.................................................51Figure 37 - Metrics used to recognise user initiation of exercises..................54Figure 38 - Vectors used for angle calculation................................................57Figure 39 - Demonstration of XNA-powered 3d hand model..........................60Figure 40 - LMRS Exercise Results Screen...................................................60Figure 41 - LMRS Register New Patient Screen............................................68Figure 42 - LMRS Remove Patient Screen.....................................................70

of 130

List of TablesTable 1 - Non-Operative / Postoperative Hand Therapy Treatments..............15Table 2 - Time Usage during a Typical Day....................................................34Table 3 - Project Risks and Counter-Measures..............................................35Table 4 - Exercise Repetition Time Variations................................................76Table 5 - LMRS Compared against Functional Requirements.......................77Table 6 - LMRS Compared against Non-Functional Requirements................78

Table of Source Code Listings Listing 1 - LMRS Data Storage Format...........................................................48Listing 2 - Login Functionality.........................................................................50Listing 3 - LMRS Perform Exercises Form Start-up........................................52Listing 4 - Leap Listener Get() Addition..........................................................52Listing 5 - LMRS Exercise Calibration............................................................53Listing 6 - LMRS Exercise Instructions and timer start...................................54Listing 7 - LMRS Exercise 1 – Fist Clench core logic.....................................55Listing 8 - Exercise 2 – Wrist Flexion & Extension Key Metric Conditional.....56Listing 9 - Vector Calculation..........................................................................58Listing 10 - Finger Angle Calculation..............................................................59Listing 11 - Pitch & Yaw Calculation...............................................................59Listing 12 - LMRS Toggle Patient Combo-box................................................61Listing 13 - Load either Patient or Clinician Main Menu..................................62Listing 14 - Load Exercise Combo-box...........................................................62Listing 15 - Exercise Selection Changed........................................................63Listing 16 - Accept or Reject Call Depending on Tag.....................................64Listing 17 - Refresh/Reload Chart after Data Change....................................64Listing 18 - Chart Type Visibility Toggle..........................................................65Listing 19 - Line Chart XAML..........................................................................65Listing 20 - Create and Populate Patient Combo-box.....................................66Listing 21 - Updating Results Chart on Patient Changed event......................67Listing 22 - LMRS Register Patient.................................................................69Listing 23 - Automatic Username Generation.................................................69Listing 24 - Finding Patient to Remove...........................................................71Listing 25 - Removing Patient.........................................................................72

Table of GraphsGraph 1 - Exercise 1 - Fist Clench..................................................................74Graph 2 - Exercise 2 - Wrist Flexion & Extension...........................................75Graph 3 - Exercise 3 - Three Jaw Chuck Pinch..............................................75

of 130

1: IntroductionThe human hand is one of the most complex creations in existence and the

main enabler of our modern lifestyles. Given this intense and extensive use, it

should come as little surprise that injuries to the hand are more common than

those of any other body region (Trybus, et al., 2006). Injuries such as

Repetitive Stress Injuries (RSI’s), lacerations and crushing are just a few

common injuries to hand. Such injuries are treated through hand rehabilitation

(Amini, 2011). This includes measures such as splinting the hand and

prescribing rehabilitation exercises designed to strengthen the muscles in the

hand and prevent build-up of scar tissue which would otherwise affect joint

movement. Individuals who find themselves afflicted with these kinds of

injuries can experience great emotional and psychological since an injury to

our hands can threaten our independence and normality in a way few things

can. This process is not only time-consuming and costly for the person

injured; in the UK, over £100 million is spent every year treating these kinds of

injuries (Dias & Garcia-Elias, 2006). Current rehabilitation is largely analogue,

with no technological intervention, primarily due to cost. Data gloves, the most

common technological rehabilitation aid, can potentially cost thousands of

pounds (O'Donnell, 2010). There is a clear need for something accurate,

portable and affordable.

1.1 Existing ApproachesAt present, it is common for individuals with hand injuries to undergo

rehabilitation using no technical aids. Efforts to improve rehabilitation through

the use of technology have led to a number of systems being proposed, these

systems are most glove-based, with few alternatives. These glove-based

systems are (for the most part) prohibitively expensive (O'Donnell, 2010) and

the few alternatives such as Kinect (Bond, 2011) can suffer from portability

and accuracy issues. It should also be noted that none of these system take

advantage of gamification. Gamification is the use of game-like elements in

traditionally non-game like settings and has been proven to increase user

enjoyment and participation.

of 130

1.2 Project AimThis project aims to design and develop a software based system for hand

rehabilitation using the Leap Motion, a new, recently released motion-based

device which has yet to be investigated as a tool for hand rehabilitation. User

feedback comes primarily from an animated 3d hand model which will reflect

the users hand movements in real-time. The results from the exercises will

then be stored for later viewing by either the patient or a clinician.

Furthermore, the project proposes the addition of gamification elements to the

proposed system; this is done with the aim of better encouraging patients to

adhere to prescribed exercise programs. To achieve this, the project will

investigate current techniques and technologies used in the field of hand

rehabilitation to better inform the design of the proposed system.

1.3 Chapter OverviewThe remainder of this report is structured as follows: chapter two will provide a

literature overview, looking at hand injuries, treatments and the role of

rehabilitation, current rehabilitation systems using data gloves and other

technologies and finally: gamification. Chapter three will then describe the

requirements analysis for the proposed system, covering functional and non-

functional requirements in addition to essential hardware and software.

Chapter four will present project planning efforts, describing project

milestones, risk assessment and strategies for optimal time utilisation before

concluding with chapter five.

of 130

2: Background and Related Work

2.1 Hand Injuries: Types, Occurrence and ImpactHand injuries are among the most frequent injuries; accounting for between

6.6% and 28.6% of all injuries and 28% of musculoskeletal injuries with the

dominant hand being injured in 52.2% of cases. The most frequent place for

hand injuries to occur is not the work place as one might suspect. The most

frequent place for hand injuries to occur is in fact the home, accounting for

45.3% of all hand injuries, followed by the workplace at 19.7%, with young

male manual workers being most at-risk (Trybus, et al., 2006).

Common injuries to the hand, treatable with hand rehabilitation include acute

issues such as: fractures, lacerations, amputations, burns, surgical repairs of

tendons and nerves. This is in addition to more chronic and acquired

conditions such as: tendonitis, rheumatoid arthritis, osteoarthritis, RSI’s such

as carpal tunnel syndrome (Amini, 2011) and neurological issues (i.e. stroke)

(American Society for Surgery of the Hand, 2011). Of all the injuries listed

above, fractures of the hand make up for the majority of hand related injuries

seen and treated in hand surgery units, with nerve injuries having the most

prolonged and profound impact on the patient due to continuing disability

(Dias & Garcia-Elias, 2006).

In this report, we view the cost and impact of hand injuries as being three-fold,

where the impact can be divided into three main categories. These are

financial, time and psychological. We look first at the financial impact,

progressing to time-related impact and lastly psychological impact.

It is estimated that treatment for hand injuries costs the UK approximately

£100 million per year. However, this problem spans much farther than the UK;

the US for example, spends approximately $18 billion treating upper extremity

disorders and Germany spends approximately €2 billion treating severe

trauma with a ratio of 25 patients per 100,000 of the population (Dias &

Garcia-Elias, 2006). Looking at RSI as a more specific example, we see that

RSI alone is estimated to cost UK employers approximately £300 million per

of 130

year (Strategy One, 2008). This is again mirrored in other parts of the world,

with the US spending approximately $20 billion on RSI compensation each

year (Yassi, 1997). Of all the hand injuries described above, amputation is

deemed the most expensive, replantation of the hand or some part of the

hand can cost up to 1.6 times a patient’s annual salary. Nerve injuries are the

second most expensive injury to treat, costing between €51,238 and €31,186

(Holmberg, et al., 1996). Speaking in more general terms, (Trybus, et al.,

2006) calculate the mean cost of a hand injury to be $6126.76 or €4507.29.

When discussing the financial impact of hand injuries, it is interesting to note

the uneven distribution of direct to indirect cost. An example of a direct cost

would be that of a surgical procedure whereas examples of indirect cost would

include sick leave and outpatient travel. Direct cost was found to make up only

4% of the total expense whereas indirect costs made up the remaining 96%

(Trybus, et al., 2006).

The impact of hand injuries is not just measured in terms of financial cost to

employers through compensation or lost productivity; we can also use time

related metrics such as work days lost or treatment duration in days when

measuring the impact of injuries to the hand. Reports indicate that hand

injuries account for 27% of all work-related injuries requiring more than 1 day

of leave (G, 2003). Given that hand injuries are a world-wide concern, it is

realistic to suggest that hand injuries can result in millions of work days being

lost, as workers are forced to take leave in order to recover from their injuries.

RSI for example costs UK employers approximately 3.5 million working days

alone, with each affected person taking an average of 13 days off due to their

injury (Strategy One, 2008). Initial, hospital-based treatment of hand injuries

can last anywhere between 1 to 86 days with an average of 9.1 ± 9.3 days.

Total treatment duration, time in hospital and aftercare can last between 1 to

420 days with an average of 76.9 ± 67.8 days, meaning hand injuries often

take longer to treat than injuries to other regions of the body. It should also be

noted that the severity of the injury does, as one would expect, affect the

duration of treatment (Trybus, et al., 2006).

In addition to the financial and time-related impact observed above, hand

injuries can also have a severe psychological impact on those afflicted. It is

of 130

common for people to view themselves in relation to their occupational role,

rank and level of ability (Hasselkus, 2002). Injuries that then interfere with

one’s occupation or daily routine - such as those involving the hands - can

cause severe distress and a strong yearning for a return to normalcy

(Hasselkus, 2002). Of all the types of hand injuries described here, nerve

injuries have the most prolonged and profound psychological impact on the

patient, those suffering from a nerve injury in the hand are commonly left with

some form of persistent, residual disability that they are forced to contend with

for the remainder of their lives. The likelihood of persistent, residual disability

after hand injury spans from 1% to 100% with 13.6% of patients being affected

on average (Dias & Garcia-Elias, 2006). Psychological issues caused by hand

injuries and associated persistent, residual disability includes flashbacks,

Post-Traumatic Stress Disorder (PTSD) and concerns with personal

appearance (Sousa, et al., 2013).

2.2 Treatment and Rehabilitation of Hand InjuriesThis report is focused on the development of a hand rehabilitation system

using the Leap Motion; because of this we will be focusing on hand

rehabilitation and its use as a treatment for hand injuries to the exclusion of

other treatment measures such as surgical procedures.

Hand rehabilitation therapy is a form of occupational therapy (Amini, 2011)

and is depicted in figure 1. Hand rehabilitation/therapy is focused on “…

enabling the client to regain functional use of the traumatized arm and hand …

and return to their pre-injury occupations.” (Case-Smith, 2003). The treatment

offered by hand therapy can be divided into two main categories; these are

preventative, non-operative and post-operative. Using the information

presented in (American Society for Surgery of the Hand, 2011), a more

complete list of treatment options offered through hand therapy can be

compiled and is presented in Table 1.

of 130

Preventative, Non-operative, Conservative

Postoperative Rehabilitation

Management of acute or chronic pain Management of open or sutured

wounds

Desensitization following nerve injury

or trauma

Control of hypertrophic or

hypersensitive scars

Sensory re-education after nerve

injury

Reduction of swelling

Design and implementation of home exercise programs to increase motion, dexterity and/or strength

Fabrication of orthoses to protect

surgery or increase movement

Training in performance of daily life

skills through adapted methods and

equipment

Instruction in home exercise program

Splint fabrication for prevention or

correction of injury

Conditioning prior to returning to work

Table 1 - Non-Operative / Postoperative Hand Therapy Treatments

Of the treatments listed above, it is “design and implementation of home

exercise programs…” and “instruction in home exercise programs” that are of

particular relevance and interest to this project. (Lavanon, 2013) Points out

that such hand therapy exercises should be “motivating, repetitious,

interesting, challenging and graded”, (Amini, 2011) adds that these exercises

should incorporate “usual and customary occupation activities…”, this is

important, given that the aim of hand therapy as described above is to return

patients to their occupational and pre-injury roles. At present, it is common for

home exercise programs to be performed without the use of technological aids

or systems.

Hand therapy offers a high success rate as a treatment for hand injuries. Of

those studied and treated in (Case-Smith, 2003), 80% returned to work after

an 8 week course of treatment consisting of – on average – 13 hours of

treatment. These results are of particular relevance because during this time,

of 130

the occupational therapist was the patient’s sole provider of rehabilitation

services, showing that hand rehabilitation/therapy even in isolation can be

greatly successful and beneficial.

While hand therapy is already a successful form of treatment for hand injuries

(Case-Smith, 2003), there is evidence to suggest that this form of treatment

could be improved further through the use of technology (Lavanon, 2013).

Argues “…advanced technology can enrich treatment and help patients…”

looking in more detail, we see that technology can be applied to other areas of

hand injury treatment beyond rehabilitation. The CODA system seen in figure

2 for example, can be used as a diagnostic motion analysis tool. More

relevant to this project however, is the discussion of technology as a

rehabilitative tool, in particular, the use of everyday “off the shelf” technology

such as the Leap Motion. An example of such a system is described in

(Lavanon, 2013), where a VR system was constructed using the PlayStation

EyeToy, a common consumer device. The EyeToy based system was found

to be an effective and – more importantly – enjoyable way of exercising,

however the system fails to grade exercises. This is something we aim to

implement in the proposed system, even enhancing it further through the

introduction of gamification elements.

Figure 1 - Hand Therapy Session Figure 2 - CODA System

2.3 The use of Glove-Based Technology in Hand RehabilitationGlove-based technology, specifically data-glove technology, is arguably the

most common form of technological aid in treatment and managing of hand

injuries. Therefore we dedicate a section solely to it. Example applications

include motor assessment (Lautman, 2012) and as a tool for rehabilitative

of 130

exercises (O'Donnell, 2010). This high adoption rate is primarily a result of the

richness of the information provided by such systems (Dipietro, et al., 2008).

To define a glove-based system, we use the definition provided by (Dipietro,

et al., 2008), where a glove-based system is defined as “a system composed

of an array of sensors, electronics for data acquisition/processing, power

supply and a support for sensors that can be worn on the user’s hand.”. Such

gloves are typically made of Lycra onto which sensors are sewn, these

sensors then record data of the wearer’s hand movements, joint movement,

fingertip positioning and so forth. We now look at a few glove-based systems

that show promise in a hand rehabilitation environment.

5DT Data Glove Ultra

The 5DT Data Glove Ultra, shown in figure 3, developed by Fifth Dimension

Technologies (Fifth Dimension Technologies, 2011), is a data glove aimed

primarily at Motion Capture and Animation Professionals. The gloves has a

total of 14 sensors, uses proprietary optical-fibre flexors and supports 24=16

possible gestures (Dipietro, et al., 2008). The glove communicates with a

computer via USB cable or RS 232 serial port through an additional kit (sold

separately); another kit is available to allow for wireless operation via

Bluetooth (also sold separately), allowing 8 hours of use on a single battery at

a range of up to 20 meters. The glove itself has a base unit price of $995

(£608.79); this includes the glove and the ‘GloveManager’ proprietary

calibration software, shown in figure 4. The 5DT Data Glove Ultra is available

in left and right variants.

Figure 3 - 5DT Data Glove Ultra Figure 4 - GloveManager Software

of 130

Human Glove

The HumanGlove, shown in figure 5, developed by HumanWare

(HumanWare, 2010), is a glove-based system developed primarily for use in

medicine, rehabilitation, VR and Telerobotics. The glove uses Bluetooth

technology by default, emulating an RS 232 port in software and uses a total

of 22 hall of effect sensors to measure flexion/extension and

abduction/adduction (2 sensors per finger, 2 for the thumb and 2 for the wrist).

Like the 5DT described above, the HumanGlove uses proprietary software for

calibration, in this case, a package called “Graphical Virtual Hand” shown in

figure 6. Pricing information for the HumanGlove is not readily available.

Figure 5 - HumanGlove Figure 6 - Graphic Virtual Hand Software

Peregrine Gaming Glove

The Peregrine Gaming Glove, shown in figure 7, is a glove-based system

developed by Peregrine Canada (Peregrine, n.d.). The glove is designed for

use in games where the number of actions available to the player is vast,

games such as MMORPGs or MOBAs. The glove has 18 touchpads, 3

activator pads together with stainless steel conductive traces; allowing support

for 30 programmable actions configures using the proprietary GloveBox

software shown in figure 8. However, the glove cannot sense flexion/extension

or abduction/adduction of the fingers or thumb, instead, the glove detects the

thumb as it touches one of the 18 touchpads lining the fingers. The main

attraction of the Peregrine Gaming Glove from a hand rehabilitation standpoint

is the price; the glove has a unit price of $149.95 Canadian (£84.34) which

of 130

has allowed students to use the glove in numerous rehabilitation system

oriented projects (O'Donnell, 2010), (Lautman, 2012).

Figure 7 - Peregrine Gaming Glove Figure 8 - GloveBox Configuration Software

While glove-based systems offer a wealth of information to developers and

researchers, it should be noted that glove-based system suffer from a vast

number of flaws. Glove-based systems suffer from robustness and durability

issues due to the Lycra fabric, this lack of durability is exacerbated by the

price of these systems. Issues of portability when one is tethered to a

computer should also be considered (again, the extra cost for wireless options

exacerbates this) in addition to the need for constant calibration. The most

relevant draw-back of these system from a hand rehabilitation standpoint

however, is the simple fact that conditions such as rheumatoid arthritis, shown

in figures 9 and 10, can leave a patient unable to even wear the glove.

Figure 9 - Rheumatoid Arthritis Hand Figure 10 - Rheumatoid Arthritis Hand

of 130

2.4 The use of Non Glove-Based Technology in Hand RehabilitationThough glove-based systems have proven extremely popular and effective,

they do suffer from drawbacks as we have seen. Issues with pricing, durability

and simply being unable to wear the glove due to conditions such as

rheumatoid arthritis have generated great need, interest and opportunity for

non-glove-based systems.

OpenCV

The Open Source Computer Vision library or OpenCV is an open source

computer vision initiative. Started by Intel in the mid to late 90’s and released

to the public in 2000. The project has since been handed over to Willow

Garage and Itseez, ensuring a continuing release schedule.

OpenCV contains over 500 C/C++ based functions, allowing for a vast array of

computer vision based applications, including medical imaging, security and

robotics (Bradski & Kaehler, 2008). The library is compatible with a wide range

of commercially available camera equipment, the camera uses the position

and colour of a pixel to build up a matrix of numbers, this matrix is then

passed to the program. OpenCV has been shown to computationally

outperform other computer vision libraries such as LTI and VXL (Bradski &

Kaehler, 2008). Furthermore, OpenCV can benefit by as much as 20% from

IPP, if they are present in the host system. This makes OpenCV a powerful

and accessible library for computer vision. Such a resource would potentially

be a good supplement for glove-based systems; however, we do not plan to

use such a supplement technology in our Leap Motion-based system.

Kinect

The Kinect, shown in figure 10 is a gesture control device primarily aimed at

gaming applications for the Xbox 360 and later Windows based PCs.

However, since its initial release, engineers both professional and hobbyist

have used it in a wide array of applications ranging from robot guidance

(Ackerman, 2011) to hand rehabilitation shown in figure 11 (Bond, 2011). The

Kinect is made up of three main sensors, the first of which is an IR depth-

finding camera used to read input in the IR spectrum, the second is an IR

of 130

transmitter and the third is a standard RGB camera. Both the IR depth-finding

camera and the RGB camera run at a resolution of 640x480 with a frame-rate

of 30 frames per second.

The appeal of the Kinect with regards to hand rehabilitation lies in the fact that

it is relatively low priced compared to the glove based systems described

above, retailing for approximately £85. This can allow for a high adoption rate

among patients, furthermore, the Kinect is not bound by the issue of right VS

left handedness, the same unit can be used to train either hand, whereas with

glove-based systems, a second glove would have to be ordered. Lastly, the

Kinect is much more durable and robust, gloves wear out over time to the

point where they must be replaced, which is costly, and a Kinect by

comparison may never need to be replaced.

While an interesting device, the Kinect does suffer from an array of

drawbacks; for example, the device only supports a field of view of 57.8º.

However, the main drawback from a hand rehabilitation standpoint is

unquestionably its minimum range of 0.6m; there is however third party lenses

that try to reduce this with some success (Pc Mag, n.d.).

Figure 11 – Kinect Figure 12 - Kinect in Hand Rehabilitation Setting

Leap Motion

The Leap Motion, shown in figures 12, is a recently released (mass shipping

began July 2013) motion-based device for computer interaction developed by

of 130

Leap Motion Inc. (Leap Motion Inc, 2013) who claim the device offers

accuracy to within 0.01mm.

The device, as shown in figures13 and 14 is made up of 2 monochromatic IR

cameras (the grey dots in figure 13) and 3 infrared LEDs (the red dots in figure

13), giving the device a semi-spherical observational area with a distance of

approximately 1 meter. This observational area is smaller than that of the

Kinect, which is designed to monitor the entire body; however this allows the

Leap Motion to operate at a higher resolution and accuracy where accuracy is

defined as “the ability of a 3D sensor to determine a desired position in 3D

space” (Weichert, et al., 2013). The IR cameras can run at up to 300 frames

per second (as opposed to 30 with the Kinect) while the LEDs generate a 3D

pattern of dots made up of IR light (Anon., 2013). A study on the accuracy of

the Leap Motion found that while the claimed 0.01mm accuracy is not

achievable, a high precision accuracy of 0.7mm was (Weichert, et al., 2013)

achievable.

Figure 13 - Leap Motion Figure 14 - Leap Motion Schematic View

This makes the Leap far superior to the standard deviation of 1.5cm (15mm)

found in the Kinect. How the Leap Motion views the users hands can be seen

in figure 15, where the freely bundled “Leap Motion Visualizer” software is

demonstrated.

In addition to the technical improvements, the Leap Motion enjoys other

benefits over previous systems. Firstly, the Leap Motion is more affordable

than any other device discussed here - even the Kinect (£85) - the Leap

Motion is currently available for £65. The Leap Motion also benefits from its

of 130

small size, coming in at 0.5 inches in height, 1.2 inches in width and 3 inches

in depth with a weight of only 0.1 pounds (Leap Motion Inc, 2013), making it

more portable than any other device discussed here. Another advantage of

the Leap Motion (this one it shares with the Kinect) is durability; the Leap

Motion is not prone to wear and tear that eventually claims many a glove-

based system.

Figure 15 - Leap Motion Internals Figure 16 - Leap Motion Visualizer

It is clear that the Leap Motion is more accurate, more affordable and more

portable than anything that has come before it. Due to these advantages, we

have chosen Leap Motion as the means for delivering the system proposed in

this project.

2.5 The Role and Potential of GamificationGamification can be defined as “the use of game design elements in non-

game contexts” (Deterding, et al., 2011). Gamification is a fast growing

initiative, with the aim of increasing motivation and participation among users

of non-game applications and is expected to revolutionise all aspects of life in

the not too distant future (Chatfield, 2010), (The Pleasure Revolution: Why

Games Will Lead the Way, 2011).

One of the first examples of gamification been used in a popular commercial

product would be the achievement system used in the Microsoft Xbox360

console (Jakobsson, 2011). The achievement system allows users to

complete in-game challenges and accumulate “Gamerscore” as shown in

figure 16. Due to its success, the system has since been implemented in

numerous other platforms including the Sony PlayStation 3 and the popular

of 130

PC Steam network. An ideal example of game design elements being used in

a non-game context however would be the Khan Academy (Khan Academy,

2013). The Khan Academy is a non-profit organisation with the aim of

providing “a free world-class education for anyone anywhere”. The site allows

users to watch videos on a wide variety of educational topics, complete

exercises for which they can build up streaks, earn badges and a

Gamerscore-like collection of points in addition to an array of real-time stat

tracking tools as seen in figure 17.

Figure 17 - Xbox 360 Achievements Figure 18 - Khan Academy Stat Tracking and Achievement System

More relevant to this project however, is the use of gamification in a medical

and rehabilitation setting. (Gerling & Masuch, 2011) Explore the application of

gamification in augmenting the lives of frail elderly people who are no longer

able to participate in certain real-life activities due to age (such as a

recreational walk through a forest) shown in figure 18. They suggest that if we

are able to overcome challenges such as the lack of experience with digital

games and systems then elderly users can benefit not only cognitively,

physically thanks to increased participation in therapeutic activities, but also

socially from the experience, as gamified applications offer the opportunity for

friendly competition.

Gamification and ‘gamified applications’ such as serious games have been

proven to work in medical undertakings such as stroke rehabilitation (Burke, et

al., 2009). The authors look at the use of gamified applications in helping

those affected by strokes regain control of the affected limbs. Their results

of 130

show that gamified applications can be used to help solve a common issue

experienced by many stroke survivors undergoing therapy. The issue being

that the everyday actives assigned to them as part of their rehabilitative

therapy are boring and uninteresting. Couple this with the depression that is

common among stroke survivors and the result is low user enthusiasm, low

participation and poor results in terms of limb functionality regained through

therapy. The study proves that gamification can make activities these

engaging and stimulating, encouraging user participation and by extension,

leading to better results in terms of regained functionality.

Figure 19 - Gamified Application, Walking through a Forest

Figure 20 - CONTRAST Serious Rehabilitation Game

The study that is of most relevance to this project however is that conducted in

(Jacobs, et al., 2013), where the authors investigate the use of gamified

applications in arm-hand training for stroke survivors. Here, a proprietary

‘serious game’ (a form of gamified application) named CONTRAST, shown in

figure 19 was used wherein the user completes task-oriented exercises

involving the manipulation of everyday items. Results of the study show

increased user participation and by extension, improved arm-hand

functionality. They point out that gamified applications make rehabilitative

exercises “meaningful…”

However, research would suggest that gamification has yet to be used in a

hand-rehabilitation setting despite the fact that both hand rehabilitation and

gamification place emphasis on identifying the user/patients personal goals

“incorporating usual and customary occupational activities into treatment…”

(Amini, 2011), likewise, making the experience relevant to the user is also an

of 130

essential part of gamification “… it is important to catch the user’s personal

goals…” (Groh, 2012).

3: Requirements Analysis and Specification

3.1 Problem StatementPeople injure their hands in all manner of ways and as a result of which

normally have to go through a period of rehabilitation, this rehabilitation period

will typically include prescribed exercises aimed at restoring pre-injury

functionality. In addition to restoring pre-injury functionality, these exercises

are an important part of preventing the build-up of scar-like tissue that would

otherwise have a negative effect on the functionality of the patient’s hand. The

results of these exercises are typically timed or monitored to identify progress

and condition development.

This project aims to develop a system to guide the user through rehabilitative

exercises whilst tracking their progress. The novelty and contribution however

is that we are using the Leap Motion as our measuring device – our

technological aid or medium if you will. Though numerous glove-based

systems and other solutions such as the Kinect have been experimented with,

the Leap Motion has yet to be tested in this setting.

Like similar systems which have come before, the system will be able to

establish a database connection and store the results of exercises undertaken

by the user. These results can later be viewed by the patient or clinician in an

easy to interpret graph.

Furthermore, we aim to enhance the system by adding gamification elements

in an effort to further encourage sustained participation among patients.

Gamification has been proven to work in areas one would not normally

associate with games; areas such as education and indeed medical systems.

However, gamification has yet to be used in the area of hand rehabilitation.

of 130

3.2 Functional RequirementsIn this report, we view functional requirements as those which directly

describe the intended functionality and behaviour of the system – what it

should do. The functional requirements identified for this project are:

System should allow clinician to create an account for themselves and

patients (if the patient has not done so) and allow a patient to create an

account for themselves only

System should allow for a simple and straightforward login process

System should allow the user to choose an exercise to perform

System should track the user’s hand movements during the exercise

session

System should provide feedback, preferably graphical (such a 3d hand

animation) or at the very least text-based feedback

System should present user with their results and save them in a

database for easy retrieval by the patient or clinician

System should allow exercise progress and results to viewed by the

patient or clinician in chart form

System should shutdown gracefully and correctly, confirming user’s

wish to close and ensuring all important data is saved

System should be able to handle unexpected difficulties such as the

absence or removal of the Leap Motion or being unable to connect to

the data-base

System should run on Windows 7 or 8 in accordance with Leap Motion

minimum system requirements

3.3 Non-Functional RequirementsIn this report, we view non-functional requirements as those which are “not

directly concerned with the specific services delivered by the system…” or

“may define constraints on the system implementation…” (Sommerville,

2010). The non-functional requirements identified for this project are:

System should be robust and durable, the system may potentially see

daily use in clinics, not just in a patients home, so robustness and

durability are important properties

of 130

System should be easy for the user to operate, this is especially in the

case of patients whose injuries make interaction with computers difficult

System should provide guidance on how to perform exercises, this is

especially true in the case of home use where a clinician is not there to

provide guidance and assistance

System should capture accurate readings of the patient’s hand

movements, inaccurate readings lead to inaccurate results and

ultimately defeat the purpose of such systems

System should provide a meaningful use of gamification elements. As

described above, gamification is more than simply applying badges and

scores to a system, the gamification elements must encourage the

patient to pursue end goals which are important to them

3.4 Hardware RequirementsThe following hardware is considered essential to the project:

At least one PC must be available at any time for development, testing

and demonstration purposes. The PC in question must meet the

minimum system requirements for the Leap Motion:

o Windows 7 or 8, Mac OS X 10.6 Snow Leopard

o AMD Phenom II or Intel Core i3

o 2GB RAM

o USB 2.0 port

Leap Motion Device

3.5 Software RequirementsThe following software is considered essential to the project:

Windows 7 Operating System (any version)

Visual Studio 2010 (any version)

.NET 4.0

XNA Framework 4.0

Leap Motion Drivers

Leap Motion SDK (C# interfaces)

of 130

3.6 Development MethodologyIt is important in any project that the most suitable development methodology

be chosen. Failure to choose correctly can have potentially disastrous results

as different methodologies place priority on different aspects of the product

and its development process.

For this project, we have chosen the Rapid Application Development

methodology. This project is largely research-based and dynamic in nature;

system requirements and expected functionality may be subject to change as

development progresses, going against the traditional Waterfall model,

wherein development follows a linear progression from one stage to the next

and where concise development strategies and goals are expected for each

stage. In addition, this project will include constant user-input from the project

supervisor in addition to (hopefully) and others with relevant insight and

experience in the problem domain, making methods such as Spiral and

Prototyping unsuitable.

of 130

4: Project Planning

4.1 Milestones and DeliverablesIn this report, we define a milestone as a critical point in the projects

development and a deliverable as a tangible result of successfully met

milestones.

The milestones for this project are:

Completion of literature review

Completion of system specification

Completion of interim report

Completion of proto-type

Completion of application ‘final’ build

Completion of final report

The deliverables for this project are:

Interim report

Functioning proto-type

Functioning ‘final’ build

Final report

System discussion and demonstration

Again, it is critical that milestones be met and corrective action taken if this

proves not to be the case.

of 130

4.2 Project PlanWe now present a WBS in figure 20 wherein we explicitly list the tasks

required to complete the project, breaking those tasks down into smaller sub-

tasks where necessary. These tasks are listed in chronological order.

Figure 21 – Work Breakdown Structure

Task 1: Literature Review

This task is concerned with the collection and analysis of reference material.

Hand injuries, hand rehabilitation, technology used for hand rehabilitation and

gamification will be the main areas of focus leading up to the interim report.

This will grow to incorporate general software engineering theory when

designing and implementing the system.

Task 2: Interim Report

The interim report is concerned with presenting the problem the project aims

to solve. Background on hand injuries, hand rehabilitation and current

technological efforts will be covered in addition to project planning.

Task 3: System Design and Specification

Decisions related to the technologies used for development will be made here.

These include decisions about programming language and environment, as

of 130

System Discussion and Demonstration

Literature Review

Literature Search

Literature review

Interim ReportSystem Design

and Specification

Sytem Protoype

Implementation

Testing

Feedback

System 'Final' Build

Implementation

Testing

Feedback

Final Report

well as possible third party technologies. The system design and layout (class

listing, coding standards etc.) will also be decided here. The design and

specification will be documented as part of the final report.

Task 4: System Prototype

The first iteration of the system will be implemented, tested and feedback

obtained. Testing will be primarily aimed at ensuring the system functions

correctly and adheres to the specification. Feedback will be sought from the

project supervisor and other relevant individuals if possible.

Task 5: System ‘Final’ Build

The iteration to be used for the final discussion and demonstration will be

implemented. Testing will be aimed more at ensuring issues and feedback

received on the prototype has been correctly implemented. Feedback will

again be sought from the project supervisor and other relevant individuals if

possible to ensure that this is so.

Task 6: Final Report

The completion of the final report will be largely focused on discussing

whether or not the final system is fit for purpose and whether the project has

been successful.

4.3 Time ManagementWe now present a Gantt chart in figure 22, detailing the time assigned to each

task listed above. The term “float time” refers to a small number of days which

are left free to try and absorb sub-tasks that encounter difficulties or run over-

time.

of 130

Figure 22 - Gantt chart

of 130

In addition, steps have been taken to measure how time is spent during a typical day. The aim here is to identify ways in which each day can be more effectively utilised. Table 2 shows a typical day, how time is used and how it

can be used more effectively going forward.

Time Activity Effective-ness

Comments

8:40 – 9:00 Get ready for

university

80% Essential: Time used efficiently

9:00 – 9:15 Travel to

university

100% Time fully optimized

9:15 – 11:15 Class 100% Cannot be skipped

11:15 – 2:15 Lab session 70% Work could be completed faster

2:15 – 3:00 Relax 50% Meeting prep will be done prior.

3:00 – 4:00 Meet with

supervisor

80% Use meeting template below.

4:00 – 5:30 Lunch and MSc

course research

60% Serviceable/Essential: Could

spend a little less time on this

5:30 – 8:00 Project and non-

project

coursework

80% Other modules cannot be ignored.

Time-boxing will ensure optimal time

usage here.

8:00 – 9:00 Dinner 100% Essential: Need to eat

9:00 – 10:15 Project work 50% Efficiency lost due to time of day?

Move MSc research here instead?

Table 2 - Time Usage during a Typical Day

4.4 Risk ManagementFor this project, steps have been taken to identify the main risks; event-driven,

evolving in addition to technical and non-technical. In this report we measure

risks using the system described in (Turner, 1993) where risks are assessed

by multiplying the likelihood of the risk (where 1 = low, 2 = medium and 3 =

high) by the consequence of the risk (where 1 = very low, 2 = low, 3 =

medium, 4 = high and 5 = very high). We then classify these values using

RAG (where 1-5 = green, 6-10 = amber and >10 = red). We present identified

risks and counter-measures in table 2. Avoidance refers to reducing the

chances of a risks occurrence, whereas contingency refers to accepting that

of 130

the risk may occur and putting plans in place to deal with the consequences

should it occur.

Description Type Classification RA-G Trigger(s) Response/Alleviation

Crash Event Technical 3 N/A Contingency: keep

flash-drive and cloud

back-ups.

Illness Event Non-

Technical

3 N/A Avoidance: maintain good

personal hygiene.

Contingency: Incorporate float-

time into schedule.

Family issue Event Non-

Technical

3 N/A Contingency: Incorporate float-

time into schedule.

Hardware

failure

Event Technical 5 Recurring

system

problems.

Contingency: use

backups and setup

backup workstation.

Insufficient

technical

knowledge /

Technical

implementation

difficulties

Evolve Technical 8 Recurring

difficulties in

designing

and

implementin

g theory.

Avoidance: ensure

extensive literature

survey and research.

Contingency: incorporate float-

time into schedule.

Missing dead-

lines and

general falling

behind

Evolve Non-

Technical

10 Repeatedly

failing to

meet

personal

goals.

Avoidance: ensure

that workload and

scope are realistic.

Contingency: incorporate float-

time into schedule.

Table 3 – Project Risks and Counter-Measures

of 130

5: ConclusionBy now the report has identified a clear need for an accurate, affordable and

portable hand-rehabilitation system. Through the literature review in chapter 2

we have seen that hand injuries are in fact a world-wide issue and can have a

potentially devastating impact on individuals as well as those around them.

We reviewed current solutions both glove and non-glove-based and have

noted that these solutions suffer from issues such as affordability, accuracy

and durability.

We propose a new system aimed at overcoming the drawbacks described

above. By using a new device – the Leap Motion – and the incorporation of

gamification, we aim to develop a system that can improve system accuracy

and reliability in addition to user participation and enjoyment. The system is

intended to feature a real-time, 3D animated model of a human hand; this will

provide real-time feedback to the user. In addition, user results will be

recorded for later viewing by patients and clinicians; the system will also be

one of the first to incorporate aspects of gamification with the aim of improving

user enjoyment and by extension, continued user participation. A detailed

description of this system has been given in chapter 3, detailing the exact

requirements and expected functionality of such a system. This system

description is supported by chapter 4, wherein we discuss exactly how we aim

to use the resources at our disposal in order to give ourselves the best chance

of success.

Again, there is a clear need for an accurate, affordable and portable hand-

rehabilitation system. The system proposed in this report, through the coupling

of new technology – the Leap Motion - and the incorporation of effective yet

largely under-utilised theory – gamification -, offers the best opportunity for

realising such a system.

of 130

6: Design

6.1 System Component OverviewA visual layout of the system components and how they fit together is now

presented; the diagram includes all the major components of the system (the

patient/clinician machine, the Leap Motion and the user data). The patient’s

machine interacts with the Leap Motion controller and both the patient and the

clinician machines interact with the user data (exercise times, results etc…).

The most common approach to storing the user data would be in a database

of some description.

Figure 23 - System Architecture

6.2 HCI Rules & GuidelinesThe system we aim to develop must be suitable for use both by patients and

clinicians. In addition to this, both the patient and clinician variants of the

system will consist of multiple screens through which the user can navigate

and interact with. It is therefore essential that the system use a consistent and

easy to comprehend UI scheme throughout. Here we present Shneiderman’s

of 130

User Data

Patient Computer

Clinician Computer

Leap Motion

“8 Golden Rules of Interface Design” (Shneiderman B, 2005), these guidelines

will be used to inform the systems UI design.

Rule 1: Strive for consistency.

This rule permeates all aspects of the system. UI layout, colour

scheme, format and use of language should be consistent throughout.

Exceptions to consistency should be minimized.

The system being developed in this project will use the same UI layout,

colour-scheme and style of language throughout all its screens. An exception

might be calibrating the Leap Motion controller before the user starts a

session; in this case, calibration will be done once at the beginning of the

session, rather than at the beginning of each individual exercise to minimize

these exceptions to consistency.

Rule 2: Cater to universalizability.

A system should cater to users of diverse backgrounds (experience

with the system, technical literacy, age etc…).

The system being developed in this project will support this rule though the

use of succinct yet informative instructions (good for novice users), tool-tips

and the use of short-cuts (good for more experienced users).

Rule 3: Offer informative feedback.

Any and all user actions (from minor to the more infrequent and major)

should be met with a suitable response.

The system being developed will provide visual confirmation and feedback

when the user interacts with the application. This will range from controls

(such as buttons and drop-downs) responding as expected, to the system

using dialog screens to confirm the users intent for more major actions (like

quitting a session).

of 130

Rule 4: Design dialog to yield closure.

All sequences of actions in a system should have a beginning, middle

and an end with informative feedback at the end.

The system being developed will provide encouraging and informative at the

end of an exercise session, with a screen allowing the user to immediately

logout or to return to the main menu (thus providing closure).

Rule 5: Prevent errors.

The design of a system should make the probability of error miniscule

and should an error occur, offer supportive and informative feedback

allowing the user to easily correct the error.

The system being developed will employ techniques such as greying out non-

applicable options (e.g. trying to move to the next exercise before the current

exercise is finished), type-checking data entered into text-fields, prompting for

confirmation when a user-action may result in a loss of progress or data (e.g.

quitting an exercise before completion) and will use a supportive and

informative style of language for any and all error messages.

Rule 6: Permit easy reversal of actions.

As much as possible, actions should be reversible to promote

experimentation and familiarisation with the system whilst relieving any

anxiety.

The system being developed will allow for this by allowing the user to

recalibrate and continue a session should they remove their hand from the

Leap Motions field of view and by always allowing the user to return to the

main menu or the previous screen (if applicable).

Rule 7: Support internal locus of control.

Experienced users wish to feel like they are controlling and driving their

interaction with the system. In addition, surprising or tedious actions

build anxiety and dissatisfaction in the user.

of 130

The system being developed will adhere to this rule by requiring the user to be

the initiator, not the responder. Any and all actions must be initiated by the

user and can be stopped by the user at any time. The system will never force

anything upon the user.

Rule 8: Reduce short-term memory load.

There is a rule of thumb regarding human information processing in

short-term memory: “humans can remember seven plus or minus two

chunks of information”. To this end, displays should be kept simple,

multiple screens consolidated and sufficient training time allotted for

learning any codes, mnemonics or intricacies.

The system being developed will adhere to this rule by keeping the number of

individual screens to a minimum (a screen with date selection for results

giving way to a new screen with a graph could be reduced to one screen for

example). In addition, the system will use a consistent layout and theme so as

not to surprise or overload the user.

6.3 UI Mock-UpsWe now present UI mock-ups for the system, these mock-ups better describe

what an average user will see when they interact with the system. Where

possible, the UI has been designed to adhere to the rules described above.

of 130

Login

Figure 24 - Login Screen

In Figure 24 we have the login screen; this screen is re-used for both clinician

and patient users. Note the use of calming colours (the blue header) as

opposed to colours such as red seen in previous systems (which instil a sense

of anger and or panic) along with friendly language (the use of ‘please’ and

avoidance of jargon such as ‘credentials’).

Main Menu

Figure 25 - Clinician Main Menu Screenshot

Figure 26 - Patient Main Menu Screenshot

of 130

Figure 25 and Figure 26 show the main menu for a clinician user; this is where

usage paths between clinician and patient users start to diverge. A clinician

user has the ability to manage patient accounts (via register and remove)

whereas patient users have the option to perform exercises. Both users share

the ability to view results and logout.

View Exercise Results

Figure 27 - Clinician version Figure 28 - Patient Version

Figure 27 and Figure 28 demonstrate the view exercise results screen. Here

we see with the clinician version in Figure 27 and the patient version in Figure

28. Both versions will allows the user to view results by exercise type as well

as being able to specify a timeframe (via start and end dates) for the results.

The only variation is the ability of the clinician to view the results of numerous

patients, whereas a patient can only view their own results.

of 130

Manage Patients

Figure 29 - Register Patent Figure 30 - Patient Screen

Figure 29 shows the register patient screen and Figure 30 shows the remove

patient screens. These are exclusive to clinician users. The clinician user must

provide a certain amount of information about a new patient before they can

be registered. In the event of removing a patient, this is done by searching for

the patient’s username.

Performing Rehabilitative Exercises

Figure 31 - Rehabilitation Exercise Screen

Figure 31 shows the exercise screen which is exclusive to patient users. On

the left of the screen, a real-time 3D animated hand model will be used to

of 130

provide real-time feedback to the user. Information such as exercise

instructions, repetition count and time taken will be documented on the left of

the screen. Possibilities for continuing to the next exercise include a button (as

seen in the image), holding ones hand still or possibly leveraging the Leap

Motions built in swipe gesture. The back button will be used to return to the

main menu.

of 130

6.4 Diagrams (Use case and Activity Model)This section outlines various interactions possible between the system and

various types of user (both patient and clinician). We first present a use-case

diagram which is used to describe the system utilities available to each type of

user. We now supplement the use-case diagram above with the following

activity models; these are used to describe what a “typical session” with the UI

and functionalities described above may look like for a user with access to

those system utilities. First, a typical patient user is shown in Figure 32.

Figure 32 - Patient User & Clinician User

of 130

Figure 33 - Patient User Interactions

of 130

Lastly, we present the activity diagram for a typical clinician user:

Figure 34 - Clinician User Interactions

of 130

7: Implementation

7.1 Technical OverviewThe technologies used in the LMRS are: eXtensibe Mark-up Language (XML),

Windows Presentation Foundation (WPF), Leap Motion controller and

Microsoft XNA.

7.2 XML Data StoreThis initial version of the LMRS uses XML as its storage medium for user data

(both credentials and exercise related). XML was chosen due to a

combination of factors, these being mainly familiarity and time constraints. The

custom XML format used for the LMRS takes on the following form:

<?xml version="1.0" encoding="utf-8"?>

<LMRS_File> <clinicianCollection> <clinician username="c_curranK" password="password0"> <forename>Kevin</forename> <surname>Curran</surname> </clinician>  </clinicianCollection>

<patientCollection> <patient username="p_taylorJ" password="password1">

<forename>Jamie</forename> <surname>Taylor</surname> <addressLineOne>6 Knock Eden Close</addressLineOne> <addressLineTwo>Town</addressLineTwo> <addressLineThree>Co Antrim</addressLineThree> <postCode>BT53 6UE</postCode>

<activity name="Fist_Clench"> <session date="01/01/14"> <rep>1.23</rep> <rep>1.24</rep> <rep>1.55</rep> <rep>1.35</rep> <rep>1.37</rep> </session>  </activity>  </patient>  </patientCollection></LMRS_File>

Listing 1 - LMRS Data Storage Format

of 130

We first have the file element, followed by a collection element (holding either

a collection of patient or clinician elements). These patient and clinician

elements then hold information such as user name, password, actual name

and/or address and in the case of patients: exercise type and repetition times.

Meta-data (like session dates and usernames) is stored in attributes, with

“actual” data stored as child elements. The “p_” and “c_” sections of a

username allow us to quickly identify the user type and immediately jump to

the appropriate part of the file with which to work with.

7.3 Patient SideWe turn first to the “patient side” of the system; this is the system a typical

patient will interact with.

We will start with the login screen seen in Figure 35 - LMRS Login Screen. All

individual screens in the LMRS are made using WPF and share common

traits.

Figure 35 - LMRS Login Screen

The main functionality of this page can be seen in

Listing 2. After initial

sanity checks (such as ensuring both fields have text), we jump to the

appropriate part of the XML data store based on the username prefix

of 130

mentioned earlier; proceeding to the main menu if the credentials are found to

be correct and simply presenting an error and clear the fields if they are not.

private void btnLogin_Click(object sender, RoutedEventArgs e){ // Trivial sanity checks ommitted...

XDocument lmrsXml = XDocument.Load("LMRS_DataStore.xml");

if (txtUsername.Text.Contains("p_")) { // Find patient user XElement patientCollection = lmrsXml.Root.Element("patientCollection");

IEnumerable<XElement> patient = from el in patientCollection.Elements("patient") where (string)el.Attribute("username") == txtUsername.Text select el;

if (patient.Count() == 0) { MessageBox.Show("Please provide a valid username.", "Cannot Login", MessageBoxButton.OK, MessageBoxImage.Error);

txtUsername.Text = pwbxPassword.Password = ""; } else { XElement userOnRecord = patient.First<XElement>();

if (userOnRecord != null && pwbxPassword.Password.Equals( userOnRecord.Attribute("password").Value.ToString())) { UserInfoHelper.user = userOnRecord; UserInfoHelper.foreName = userOnRecord.Element("forename").Value; UserInfoHelper.userName = userOnRecord.Attribute("username").Value;

this.NavigationService.Navigate(new LMRS_MainMenu_Patient()); } else { MessageBox.Show("Please provide a valid password.", "Cannot Login", MessageBoxButton.OK, MessageBoxImage.Error);

pwbxPassword.Password = ""; } } } // btnLogin_Click()

Listing 2 – Login Functionality

of 130

The main menu screen will not be covered as it is a simple collection of calls

to the navigation system used to load the next screen depending on the user’s

selection of either performing exercises, viewing results or logging out.

We will describe the main focus of the patient side of the system and indeed

that of the entire LMRS: the perform exercise functionality. This part of the

system pulls together WPF, Leap Motion and XNA to deliver the user

experience seen in Figure 36.

Upon loading the page, we create a separate thread for the XNA-powered 3d

hand model and setup the Leap Motion, giving it its own background worker

so it does not block the main thread (which would cause the application to

hang).

private void Page_Loaded(object sender, RoutedEventArgs e){ // Set XNA hand running IntPtr handle = RenderPanel.Handle; xnaThread = new Thread(new ThreadStart(() => { game = new SkeletalAnimationSample(handle); game.Run(); } )); xnaThread.Start();

// Set Leap Motion running listener = new LeapListener(); controller = new Leap.Controller();

of 130

Figure 36 – LMRS Perform Exercises Screen

controller.AddListener(listener);

leapBackgroundWorker = new BackgroundWorker(); leapBackgroundWorker.WorkerSupportsCancellation = true; leapBackgroundWorker.DoWork += new DoWorkEventHandler(leapBackgroundWorker_DoWork); leapBackgroundWorker.RunWorkerAsync();}

Listing 3 – LMRS Perform Exercises Form Start-up

The Leap Listener class in

Listing 3 is a class in Leap API used to provide

listeners for key system events such as device connected, disconnected and

frame to name a few. The implementation used for the LMRS is simply a

standard listener with an added get method to help in retrieving the data from

the most recent frame (listener can be inherited and extended with custom

behaviours for each event).

// Additional listener methods omitted...

public override void OnFrame(Controller controller){ // Get the most recent frame from the device frame = controller.Frame();}

public Leap.Frame LeapFrame{ get { return frame; }}

Listing 4 – Leap Listener Get() Addition

The background workers “DoWork” method is where the core logic of the

perform exercise functionality resides. This method contains the code which

polls the Leap Motion device, processes this data against the exercise logic

and updates the 3D hand model.

We start with the calibration phase, this is in fact not to calibrate the Leap

Motion device itself, the calibration phase is actually required in order to

collect data needed to calculate key metrics which will then be used in the

exercise logic.

void leapBackgroundWorker_DoWork(object sender, DoWorkEventArgs e){ while (true) { Leap.Frame frameData = ((LeapListener)listener).LeapFrame;

of 130

if (frameData != null && frameData.Hands.Count > 0) { hand = frameData.Hands[0];

// Calibration #region Clibration if (!calibrated) { if (!calibratePromptGiven) { updateExerciseInstruction("Calibrating, please wait..."); calibratePromptGiven = true; }

if (!timer.IsRunning) { timer.Start(); }

// Collect hand pitch and vector magnitude readings, // large enough deltas in these parameters are used // to detect user movements and start the timer if (timer.Elapsed.Seconds < 5) { avgMag += (hand.Fingers.Frontmost.TipPosition - hand.PalmPosition).Magnitude; avgPitch += hand.Direction.Pitch; ++calls; } else { avgMag /= calls; avgPitch /= calls; calibrated = true; timer.Reset(); } } // if (!calibrated) #endregion

Listing 5 – LMRS Exercise Calibration

The calibration process in

Listing 5 will be repeated if the user takes their

hand away from or goes out of range of the Leap Motion.

These key metrics are described below and can also be seen in Figure 37:

The magnitude of the vector between the front-most fingertip and the

palm. A sizeable change (decrease) indicates that the front-most finger

is getting closer to the palm meaning the user has begun to clench their

hand into a fist. At this point we start the timer. Exercises 1 and 3 use

this logic (∆||v||>n?-> timer start).

of 130

The average pitch of the hand (rotation about the x-axis). Again, a

sizeable delta indicates the user has begun to perform the requested

action. Like with exercises 1 and 3 we again use this sizeable change

as an indicator to start the timer (∆ hand pitch ¿n? -> timer start).

Figure 37 – Metrics used to recognise user initiation of exercises

We now move onto the logic behind the rehabilitation exercises the user is

required to perform. We will look at the first exercise where the user is

required to go from holding their hand at rest, to forming a clenched fist before

finally returning their hand to a resting position, the code for this exercise can

be seen in Listing 6.

#region exercise1if (currentExercise == 1 && repsPerformed < repsRequired){ if (!exerciseDescGiven) { updateExerciseDescription("Exercise 1 of 3: Fist Clench. " + "\n1) Form a closed fist. \n2) Relax hand. \n\nThe timer will start automatically."); exerciseDescGiven = true; }

// Sufficient vector magnitude delta? Start timer if ((hand.Fingers.Frontmost.TipPosition - hand.PalmPosition).Magnitude < 0.9 * avgMag && !timer.IsRunning) { restartTimer(); }

Listing 6 – LMRS Exercise Instructions and timer start

of 130

Here we check the current exercise index and ensure that there are still

repetitions left for the user to perform. We then present the description/brief

for this exercise if we have not already done so. Lastly, we use the first of the

two metrics recorded during calibration to check for a sizable delta in the

vector stemming from the palm to the tip of the front-most finger, this means

the user has begun to move their hand and so we start the timer.

// Exercise part 0 - Form a fist if (hand.Fingers.Count == 5 && !clenchPromptGiven) { // Present time for this rep if (timer.IsRunning) { repTimes[repsPerformed] = timer.ElapsedMilliseconds;

updateRepInfo("This rep time: \nRep " + (repsPerformed + 1) + " of 5 : \n" + repTimes[repsPerformed] / 1000.0 + " seconds.");

// New best rep time? if (repTimes[repsPerformed] < bestRepTime) { bestRepTime = repTimes[repsPerformed]; updateBestRepInfo("Best rep time: \nRep " + (repsPerformed + 1) + " of 5: \n" + bestRepTime / 1000.0 + "seconds."); }

++repsPerformed; restartTimer(); }

// Give prompt if (repsPerformed < repsRequired) { updateExerciseInstruction("Clench your hand into a fist"); clenchPromptGiven = true; restPromptGiven = false; } }

// Exercise part 1 - Relax hand if (hand.Fingers.Count == 0 && !restPromptGiven) { updateExerciseInstruction("Place your hand at rest"); restPromptGiven = true; clenchPromptGiven = false; }} // if (currentExercise == 1 && repsPerformed < repsRequired)#endregion

Listing 7 – LMRS Exercise 1 – Fist Clench core logic

of 130

In Listing 7, we first look for 5 fingers; this indicates a hand at rest. When this

condition is met, we present the time for the current repetition and store the

time if it is a new personal best before prompting the user to continue to the

next rep by updating the exercise instruction field. The second condition we

check for is a finger count of zero (along with the necessary prompt having

been shown/not shown), a finger count of zero indicates a clenched fist. At

which point we update the exercise instruction field and toggle the prompt

displayed Booleans.

The second exercise (Wrist Flexion & Extension) uses similar core logic but

uses the second of the two metrics recorded during calibration (hand pitch) to

help decide when the timer should be started and instructions. This can be

seen in

Listing 8.

// Lower handif (hand.Direction.Pitch > avgPitch * 2.5 && repsPerformed < repsRequired && !lowerPromptGiven){

// ...}

// Raise handif (hand.Direction.Pitch < -2.5 * avgPitch && repsPerformed < repsRequired && !raisePromptGiven){

// ...}

Listing 8 – Exercise 2 – Wrist Flexion & Extension Key Metric Conditional

The third and final exercise (Three Jaw Chuck Pinch) uses the exact same

logic as the first exercise (Fist Clench) and will therefore not be covered. The

only notable difference between them is the number of fingers being checked

for at each stage of the exercise. Since the third exercise uses the thumb,

index finger and middle finger, we check for 3 fingers instead of 5 followed by

1 instead of 0.

After the data from the Leap Motion has been collected and used to progress

the exercises, the last remaining task for the current loop iteration is to update

the 3d hand model. The 3D hand model used in the LMRS is powered by

of 130

version 3.5 of XNA and uses data from the Leap Motion to animate the bones

of the model.

Before animating the hand however, we must first calculate two key vectors,

the angles between which will then be used to animate the hand model by

setting the joints equal to these angles. The first of these two vectors covers

the distance between the centre of the palm and the base of the finger in

question; the second of the two vectors covers the distance from the base of

the finger to the finger tip, these can be seen in Figure 38.

Figure 38 – Vectors used for angle calculation

However, before we can even calculate these two vectors, we must first

calculate the base of the finger itself as this data is not readily accessible via

the Leap Motion SDK, however, the Leap developers do provide a means to

calculate this in the SDK documentation (Leap Motion, 2013). These three

vectors (finger base, palm to base and base to tip) along with the resulting

angle data are calculated twice. Once for the thumb (using the left-most finger

member provided by the Leap Motion SDK) and once for the four remaining

fingers (all of which mimic the front-most finger member again provided by the

Leap Motion SDK). Due to the Leap Motions lack of skeletal tracking however,

it is currently near impossible to reliably identify individual fingers; left-most,

of 130

right-most and front-most are the only ones reliably accessible through the

SDK at present and even then are ambiguous (the left-most finger is the little

finger of your left hand and is, at the same time, the thumb of your right hand

for example). The code used to calculate these vectors can be seen in

Listing 9.

// ... } #endregion // Exercise 3/3

// Update hand model updateHandModel(frameData); } } // if (!calibrated) }} // leapBackgroundWorker_DoWork()

// Update the hand model based on information from the Leap Motionprivate void updateHandModel(Leap.Frame frame){ // Calculate necessary vectors Leap.Vector fingerBase = -frame.Hands[0].Fingers.Leftmost.Direction * frame.Hands[0].Fingers.Leftmost.Length;

fingerBase = fingerBase + frame.Hands[0].Fingers.Leftmost.TipPosition;

Leap.Vector palmCenterToBase = fingerBase - frame.Hands[0].PalmPosition;

Leap.Vector baseToTip = frame.Hands[0].Fingers.Leftmost.TipPosition - fingerBase;

Listing 9 – Vector Calculation

Once these vectors have been calculated, they are then used to calculate the

angles to which the joints in the 3d hand model will be set. For this, we borrow

the following quadratic equation from (Hillerbrand, et al., 2005):

qα , β=0.23+1.73d+1.5d2. Originally, this equation was used to define the

relationship between the bending angle of the outer-most and middle phalanx

(α ) and that of the middle and inner phalanx (β), with d denoting the distance

between the base joint and fingertip relative to the finger length. For the

LMRS, we use this quadratic for the 3d hand model; substituting the values

0.66 and 0.33 for α and β respectively for the index, middle, ring and little

of 130

finger, for the thumb, we only use the 0.33 value. In addition to this, we

multiply by the angle between the finger-base and the palm. This helps the

finger bend in a realistic fashion despite only having the angle between the

palm and finger base as our only accessible/calculable value. This can be

seen in

Listing 10 along with some basic clamping to prevent edge case

issues from occurring such as fingers bending the wrong way should the Leap

Motion lose sight/track of them, this code listing is for the index finger but the

same values/angles are applied to the other three (middle, ring and little).

// IndexfingerBase = -frame.Hands[0].Fingers.Frontmost.Direction * frame.Hands[0].Fingers.Frontmost.Length;

fingerBase = fingerBase + frame.Hands[0].Fingers.Frontmost.TipPosition;

palmCenterToBase = fingerBase - frame.Hands[0].PalmPosition;

baseToTip = frame.Hands[0].Fingers.Frontmost.TipPosition - fingerBase;

// Some basic clampingfloat fingerIndex3 = (-(3.0f - Leap.Vector.ZAxis.AngleTo(baseToTip))) > 0 ? 0 : (-(3.0f - Leap.Vector.ZAxis.AngleTo(baseToTip))); fingerIndex3 = fingerIndex3 <= -1.25f ? -1.25f : fingerIndex3;

float fingerIndex2 = (float)(((0.23 + 1.73 * 0.66 + 1.5 * (0.66 * 0.66))) * fingerIndex3);

float fingerIndex1 = (float)(((0.23 + 1.73 * 0.33 + 1.5 * (0.33 * 0.33))) * fingerIndex3);

Listing 10 – Finger Angle Calculation

We next calculate a pitch and yaw for the hand model (with pitch describing

rotation about the x-axis and yaw describing rotation about the y-axis). To do

this we can simply used the normalised direction property of the hand object in

the Leap SDK, the only modifications we make are to tone down the yaw as

leaving this value unaltered or too high was found to cause difficulties.

// Calulate pitch and yawfloat pitch = -frame.Hands[0].Direction.Normalized.Pitch;float yaw = -frame.Hands[0].Direction.Normalized.Yaw;

// Tone down yawyaw -= 0.5f;yaw /= 4;

of 130

// Update handgame.CameraDown(pitch);game.CameraLeft(yaw);

Listing 11 – Pitch & Yaw Calculation

The final result is shown in Figure 39.

Figure 39 – Demonstration of XNA-powered 3d hand model

We move now to the final part of the system with which a typical patient user

may interact, this being the viewing and graphing of results for that user, this

functionality can be seen in Figure 40.

of 130

Figure 40 – LMRS Exercise Results Screen

This is the clinician variation of the form being shown, however the main

difference between this and the patient variation is the patient combo-box

which is not visible for a patient user. We show the fully-featured clinician

version to avoid needless repetition. This mention of form variation depending

on user type would make the load and unload code a logical place to start; in

Listing 12 we see the code responsible for ensuring that the appropriate

features are made accessible depending on the user type.

public LMRS_ViewExerciseResults(string userType){ InitializeComponent();

// Set the window title WindowTitle = "Leap Motion Rehabilitation System – View Exercise Results";

// Load the XML file and populate the drop-downs lmrsXml = XDocument.Load("LMRS_TestFile.xml");

populatePatientComboBox(); populateExerciseComboBox();

this.userType = userType; if (userType.Equals("clinician")) { lblPatient.Visibility = Visibility.Visible; cmboPatient.Visibility = Visibility.Visible; } else

of 130

{ lblPatient.Visibility = Visibility.Hidden; cmboPatient.Visibility = Visibility.Hidden; }}

Listing 12 – LMRS Toggle Patient Combo-box

Note that we are using a custom constructor and decide whether or not to

make the patient selection functionality available depending on the user-type

argument passed to us. In

Listing 13, we see the relevant code from the

patient and clinician menus.

of 130

// LMRS_MainMenu_Patient.xaml.csprivate void btnViewExerciseResults_Click(object sender, RoutedEventArgs e){ this.NavigationService.Navigate(new LMRS_ViewExerciseResults("patient"));}

// LMRS_MainMenu_Clinician.xaml.csprivate void btnViewExerciseResults_Click(object sender, RoutedEventArgs e){ this.NavigationService.Navigate(new LMRS_ViewExerciseResults("clinician"));}

Listing 13 – Load either Patient or Clinician Main Menu

From here on we will assume a typical patient user. Any further code relevant

to clinician users will be covered when we look at the clinician side of the

system. The first thing to do for a patient user is load the exercise combo box;

this can be seen in

Listing 14.

// Load the exercise combo boxprivate void populateExerciseComboBox(){ // Now load the exercise combo box IEnumerable<XElement> exercises = (from el in selectedPatient.Elements("activity") select el); this.exercises.Add(exercises.ElementAt(0).Attribute("name").Value .ToString().Replace("_", " ")); this.exercises.Add(exercises.ElementAt(1).Attribute("name").Value .ToString().Replace("_", " ")); this.exercises.Add(exercises.ElementAt(2).Attribute("name").Value .ToString().Replace("_", " "));

cmboExercise.ItemsSource = this.exercises; cmboStartDate.ItemsSource = cmboEndDate.ItemsSource = dates;

chartDataList = new List<KeyValuePair<string, decimal>>();}

Listing 14 – Load Exercise Combo-box

We take each exercise from the XML data store, substitute the underscores in

the XML for plain spaces and add the item to the list. Lastly, we set the

combo-box’s data source to this enumerable. Next we look at the code for an

exercise changed event, for this we simply fetch the newly selected exercise

from the XML data store, load up the associated sessions and repetition times

of 130

and add the session dates to the date combo-box’s item source. This can be

seen.

private void cmboExercise_SelectionChanged(object sender, SelectionChangedEventArgs e){ // Load available sessions for this exercise, first find the // chosen exercise in the file String name = cmboExercise.SelectedItem.ToString().Replace( " ", "_");

IEnumerable<XElement> exercise = from el in selectedPatient.Elements("activity") where (string)el.Attribute("name") == name select el;

chosenExercise = exercise.First<XElement>();

// Now get the sessions and rep times sessions = from el in chosenExercise.Elements("session") select el;

repTimes = from el in sessions.Elements("rep") select el;

dates.Clear(); int i = 0; foreach (XElement exl in sessions) { dates.Add(exl.Attribute("date").Value); ++i; }

// Tag: arbitrary object value that can be used to store custom // information about this element. Use it here to avoid // triggering unwanted calls to refreshChart cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.Items.Refresh(); cmboEndDate.Items.Refresh(); cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.SelectedIndex = cmboEndDate.SelectedIndex = 0;

refreshChart();}

Listing 15 – Exercise Selection Changed

The use of the date combo-box’s tag property is to help avoid additional

unwanted to calls to refresh chart which were causing crashes during

development. The selection changed event for the start and end date combo-

box’s is the exact same, we check to see if the call should be accepted or

rejected, clamp the end date if necessary and finish with a call to refresh

chart.

private void cmboStartDate_SelectionChanged(object sender, SelectionChangedEventArgs e){ if (((string)cmboStartDate.Tag).Equals("ignoreCall"))

of 130

{ cmboStartDate.Tag = "acceptCall"; return; }

// End date cannot be before the start date if (cmboEndDate.SelectedIndex < cmboStartDate.SelectedIndex) { cmboEndDate.SelectedIndex = cmboStartDate.SelectedIndex; } refreshChart();}

Listing 16 – Accept or Reject Call Depending on Tag

The refresh chart method is responsible for updating the chart when either the

exercise or date selection (either start or end date) has been changed. After

an exercise change, it is responsible for fetching the session data (between

the start and end dates) linked with the newly chosen exercise. After a date

change event, it is responsible for loading the newly selected set of session

data.

// Changing the dates via the combo boxes modifies the dataset used // by the chart.private void refreshChart(){ chartDataList.Clear(); for (int i = 0; i < ((cmboEndDate.SelectedIndex - cmboStartDate.SelectedIndex) + 1) * 5; ++i) { chartDataList.Add(new KeyValuePair<string, decimal>(sessions.ElementAt(cmboStartDate.SelectedIndex + (i / 5)).Attribute("date").Value + ": Rep " + ((i % 5) + 1).ToString() + " ", Convert.ToDecimal(repTimes.ElementAt<XElement>(i).Value) )); }

chrtBarChart.DataContext = chrtLineChart.DataContext = chrtColumnChart.DataContext = null;

chrtBarChart.DataContext = chrtLineChart.DataContext = chrtColumnChart.DataContext = chartDataList;

// Update page title lblHeader.Content = "Results for " + exercises[cmboExercise.SelectedIndex] + " (" + cmboStartDate.SelectedValue + " - " + cmboEndDate.SelectedValue + ")";}

Listing 17 – Refresh/Reload Chart after Data Change

of 130

The three radio buttons on the form simply toggle between three types of

graph, these being bar, line and column. The code for each of the radio

button’s checked event handlers is the exact same, only toggling the visibility

of different graphs.

private void rdoBarChart_Checked(object sender, RoutedEventArgs e){ chrtBarChart.Visibility = Visibility.Visible; chrtLineChart.Visibility = Visibility.Collapsed; chrtColumnChart.Visibility = Visibility.Collapsed;}

Listing 18 – Chart Type Visibility Toggle

To allow for these different types of graph we use several graph items in the

xaml code for the page, the code for each is graph again the same, only the

series type and name differ. The line chart in

Listing 19 provides an example

of this.

<chartingToolkit:Chart Name="chrtLineChart" Margin="27.151,171.48,28.58,30.009" Grid.Column="1" Visibility="Collapsed">

<chartingToolkit:Chart.Axes> <chartingToolkit:LinearAxis Title="Time in Seconds" Orientation="Y" Interval="0.1"/> <chartingToolkit:CategoryAxis Title="Date and Rep #" Orientation="X"/> </chartingToolkit:Chart.Axes>

<!—- Series type --> <chartingToolkit:LineSeries Title="lineSeries DependentValuePath="Value" IndependentValuePath="Key" ItemsSource="{Binding}"/>

 <chartingToolkit:Chart.LegendStyle> <Style TargetType="datavis:Legend"> <Setter Property="Width" Value="0" /> </Style> </chartingToolkit:Chart.LegendStyle></chartingToolkit:Chart>

Listing 19 – Line Chart XAML

7.4 Clinician SideNow we have finished looked at the patient side of the LMRS, we turn to the

clinician side of the system.

of 130

We will start where we previously left off on the patient side: the viewing and

graphing of patient results. It was mentioned above that the results page

either displays or hides a combo-box of patients depending on the type of user

loading the form and some code to that affect was shown. We now look at the

code responsible for loading that combo-box.

private void populatePatientComboBox(){ // Now load the exercise combo box XElement patientCollection = lmrsXml.Root.Element("patientCollection");

IEnumerable<XElement> patients = (from el in patientCollection.Elements("patient") select el);

int i = 0; int j = 0; foreach (XElement el in patients) { this.patients.Add(patients.ElementAt(i).Element("forename"). Value + " " + patients.ElementAt(i).Element("surname").Value);

if (patients.ElementAt(i).Attribute("username").Value == UserInfoHelper.userName) { j = i; break; } ++i; }

cmboPatient.ItemsSource = this.patients; cmboPatient.Tag = "ignoreCall"; cmboPatient.SelectedIndex = 0;

selectedPatient = patients.ElementAt(j);}

Listing 20 – Create and Populate Patient Combo-box

In Listing 20 – Create and Populate Patient Combo-box, we add each patient

user found in the patient collection element (found in the XML data store). The

if statement is used when a patient user is loading the form. The condition

stops the loop if the patient currently being added to the combo-box is the

same patient user loading the form in which case we stop the loop and return

to the constructor where the combo-box (and corresponding label) is then set

to hidden, in the case of a clinician user however we continue loading in

patients from the XML data store.

of 130

The event handler for the patient combo-box’s selection changed event simply

finds the appropriate patient element and updates the graph.

private void cmboPatient_SelectionChanged(object sender, SelectionChangedEventArgs e){ if (((string)cmboPatient.Tag).Equals("ignoreCall")) { cmboPatient.Tag = "acceptCall"; return; }

// Fetch the newly selected patient XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patients = (from el in patientCollection.Elements("patient") select el);

int i = 0; foreach (XElement el in patients) { if(cmboPatient.SelectedItem.ToString(). Contains(patients.ElementAt(i).Element("forename").Value) && cmboPatient.SelectedItem.ToString(). Contains(patients.ElementAt(i).Element("surname").Value)) { selectedPatient = patients.ElementAt(i); break; } ++i; }

cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.Items.Refresh(); cmboEndDate.Items.Refresh(); cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.SelectedIndex = cmboEndDate.SelectedIndex = 0; cmboExercise.Tag = "ignoreCall"; cmboExercise.SelectedIndex = 0;

// Update the graph refreshChart();}

Listing 21 – Updating Results Chart on Patient Changed event

Another unique functionality available only to clinician users is the ability to register new patient users as seen in Figure 41.

of 130

Figure 41 – LMRS Register New Patient Screen

The main functionality of this page is held within the register button’s click

event seen in Listing 22. Here we ensure that all fields have been completed

and that the user does not already exist, if no match is found we create a new

user with the credentials being provided, if a match is found, we notify the user

and clear the text fields.

private void btnRegister_Click(object sender, RoutedEventArgs e){ // Necessary information provided? if (!textFieldsNotComplete()) { MessageBox.Show("Please ensure all text fields have been completed.", "Missing Information", MessageBoxButton.OK, MessageBoxImage.Error); } else { // Load the XML file XDocument lmrsXml = XDocument.Load("LMRS_TestFile.xml");

// Does the patient user exist already? String patientUsername = "p_" + txtSurname.Text.ToLower() + txtForename.Text.ToUpper()[0];

// Get patient collection node XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in patientCollection.Elements("patient") where (string)el.Attribute("username") == patientUsername select el;

if (patient.Count() != 0)

of 130

{ MessageBox.Show("A patient is already registered with those details.", "Patient Already Exists", MessageBoxButton.OK, MessageBoxImage.Error); } else { // Add new patient information to record XElement newPatient = new XElement("patient", new XAttribute("username", patientUsername), new XAttribute("password", txtPassword.Text), new XElement("forename", txtForename.Text), new XElement("surname", txtSurname.Text), new XElement("addressLineOne", txtAddress1.Text), new XElement("addressLineTwo", txtAddress2.Text), new XElement("addressLineThree", txtAddress3.Text), new XElement("postCode", txtPostcode.Text), new XElement("Fist_Clench", null), new XElement("Wrist_Flextion_Extension", null), new XElement("Three_Jaw_Chuck_Pinch", null));

// Save and close patientCollection.Add(newPatient); lmrsXml.Save("LMRS_TestFile.xml");

MessageBox.Show("Patient added successfully.", "Patient Added", MessageBoxButton.OK, MessageBoxImage.Information);

clearTextFields(); } }} // btnRegister_Click(object sender, RoutedEventArgs e)

Listing 22 – LMRS Register Patient

The methods to check for text field completion and to subsequently clear them

again are trivial assignment operations and are therefore omitted. Another

method worthy of mention is the text changed event handler for the surname

field, when this field is modified we automatically generate a username of the

format: “p_(surname all lower case)(forename initial in upper case)” i.e.

p_smithJ. This is done to ensure consistency across all usernames.

private void txtSurname_TextChanged(object sender, TextChangedEventArgs e){ if (txtForename.Text != "") { txtUsername.Text = "p_" + txtSurname.Text.ToLower() + txtForename.Text.ToUpper()[0]; }}

Listing 23 – Automatic Username Generation

of 130

The final functionality available only to clinicians (and the last uncovered

functionality of the LMRS as a whole) is the ability to remove registered

patient users, seen in Figure 42.

Like with the register button in the register new patient form, the main

functionality for the remove patient form is held in the click event handler for

the search button. When this button is pressed we check the XML data store

for a user matching the provided username, if one is found we enable the

remove button, otherwise we inform the user that no such user exists on

record and clear the search field.

of 130

Figure 42 – LMRS Remove Patient Screen

private void btnSearch_Click(object sender, RoutedEventArgs e){ lmrsXml = XDocument.Load("LMRS_TestFile.xml");

// Find patient user XElement patientCollection = lmrsXml.Root.Element("patientCollection");

IEnumerable<XElement> patient = from el in patientCollection.Elements("patient") where (string)el.Attribute("username") == txtUserToRemove.Text select el;

if (patient.Count() == 0) { MessageBox.Show("No user found with this username, please check the username and try again.", "User Not Found", MessageBoxButton.OK, MessageBoxImage.Error);

txtUserToRemove.Text = ""; } else { userOnRecord = patient.First<XElement>();

txtResults.Text = "User found: \n\n" + userOnRecord.Element("forename").Value + " " + userOnRecord.Element("surname").Value + "\n" + userOnRecord.Element("addressLineOne").Value + "\n" + userOnRecord.Element("addressLineTwo").Value + "\n" + userOnRecord.Element("addressLineThree").Value + "\n" + userOnRecord.Element("postCode").Value;

btnRemove.IsEnabled = true; }}

Listing 24 – Finding Patient to Remove

Finally, if a patient user has been found and the clinician user wishes to

remove that user we prompt for confirmation, removing the patient user only if

the clinician user wishes to proceed.

private void btnRemove_Click(object sender, RoutedEventArgs e){ if (MessageBox.Show("Removing this user will permanently delete all related exercise data, proceed?", "Confirm User Removal", MessageBoxButton.YesNo, MessageBoxImage.Question) == MessageBoxResult.Yes) { userOnRecord.RemoveAll(); lmrsXml.Save("LMRS_DataStore.xml");

MessageBox.Show("Patient removed successfully.", "Patient Removed", MessageBoxButton.OK, MessageBoxImage.Information); }

of 130

txtUserToRemove.Text = txtResults.Text = ""; btnRemove.IsEnabled = false;}

Listing 25 – Removing Patient

of 130

8: Evaluation

8.1 Testing

8.1.1 Unit and Regression TestingThroughout initial development, the most commonly used testing methods

were those of unit and regression testing. With unit testing used to test

individual functionalities and units of code. Unit testing for the LMRS took the

form of compiling and running the system to check that it is first: stable and

secondly: that it functions as expected/designed. Regression testing was used

in addition to unit testing to confirm correct and expected behaviour after

notable changes/refactoring. Regression testing for the LMRS took the form of

compiling and re-running the system, with the aim of testing certain

functionalities which had seen significant re-working, commonly as a result of

the prior mentioned unit testing. This was done before testing the system

against more formal, drafted test cases.

8.1.2 User Scenario Test CasesThe first formal means of testing the LMRS was through the use of

conventional test-cases. These test cases describe the typical usage patterns

of both a clinician and patient user. The tests include basic sanity testing (the

ability to detect and reject false credentials and other ‘junk data’) in addition to

testing the various components of the system (can exercises be performed

without issue, does the results graph show correctly etc…). The full test-cases

can be found in Appendix 1 and Appendix 2 respectively along with original

results and comments.

The issues uncovered by these test cases were largely XML related. For

example, both crashes on the results screen (for either patient or clinician)

were the result of empty XML elements. One due to potentially missing

session elements (if a user quits before completing an exercise for example)

and the other due to an error in the original remove patient code which would

remove all child elements of the patient but not the patient element itself. In

addition to this, testing revealed a lack of any suitable prompt for exiting while

of 130

an exercise session was in progress. A prompt was added in response to this

to keep the system in-line with its initial requirements, in this case, the

requirement of keeping the user informed at all times.

8.1.3 Testing Timing AccuracyOne of – if not the single most – important aspect of the LMRS is that it

provides highly accurate timings of exercise repetitions; if the LMRS cannot

provide accurate exercise repetition timings then any medical

relevance/usefulness of the timings and that of any other LMRS-generated

data is dramatically reduced.

To measure the accuracy of the LMRS in this respect, the system has been

compared against video references; actual video recording of the exercises

being performed. The timings recorded by the LMRS are then compared

against the timings taken from the video reference (acquired by measuring the

time taken in video for a repetition to be performed). Graphs 1, 2 and 3 show

the accuracy of the LRMS relative to the video reference for each of the three

exercises (Fist Clench, Wrist Flexion & Extension and Three Jaw Chuck

Pinch). This data has been collected by performing each exercise three times

(three LMRS sessions and three reference videos) and taking the average

time for each rep (1, 2, 3 etc…).

Graph 1 – Exercise 1 - Fist Clench

As we can see in Graph 1, deviations in the times recorded by the LMRS

compared to those in the reference video are minimal (often around 100

milliseconds). This trend is maintained in exercises 2 and 3, as seen in Graph

2 and Graph 3.

of 130

Rep 1 Rep 2 Rep 3 Rep 4 Rep 50

0.20.40.60.8

11.21.41.61.8

22.2

Exercise 2 - Wrist Felxion & Ex-tension

Graph 2 – Exercise 2 – Wrist Flexion & Extension

Rep 1 Rep 2 Rep 3 Rep 4 Rep 50

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Exercise 3 - Three Jaw Chuck Pinch

Graph 3 - Exercise 3 – Three Jaw Chuck Pinch

It is worth noting however that the one consistent area of variation between

the LMRS and the reference video is the first repetition of each exercise, this

would suggest that adjustments and/or refinements to the values used in the

timer related conditions (when to start/restart) may be in order for future

iterations.

of 130

In addition to the repetition timings, the deviation observed for each exercise

has also been calculated and are presented in Table 4. This is an important –

if not critical – metric for the LMRS and any rehabilitation system of this nature

and will be a key metric used to judge any future changes or additions made

to the LMRS (a more accurate version of the hand model cannot come at the

cost of a loss in repetition timing accuracy for example). As with the repetition

time data discussed above, this deviation data has been calculated by taking

the average deviation for each repetition across the three sessions and then

taking the average (the deviation value for rep 1 in exercise 1 for example is

the average rep 1 deviation observed for that exercise across the three

sessions).

Repetition # Exercise 1 – Fist Clench

Exercise 2 – Wrist Flexion & Extension

Exercise 3 – Three Jaw

Chuck Pinch1 -0.109 -0.116 -0.3222 0.107 -0.003 0.1493 -0.031 0.035 0.1244 -0.106 0.149 0.2675 -0.153 0.153 0.076

Average -0.0584 0.0436 0.0588

Table 4 – Exercise Repetition Time Variations

The results are encouraging. Despite the Leap Motion being an as yet new

and untested device, we see that the average inaccuracy (deviation) in the

timings measured by the LMRS is less than 100 milliseconds and rarely is the

100 millisecond barrier broken for any individual repetition.

of 130

8.2 Evaluation against Initial Requirements

8.2.1 Evaluation against Functional RequirementsThe LMRS will now be compared against its initial functional requirements.

Req # Description Requirement Met?1 System should allow clinician and

patients to create accounts for themselves

Partially: Clinicians are responsible for creating patient accounts

2 System should allow for a simple and straightforward login process

Met

3 System should allow the user to choose an exercise to perform

Not met: System progresses through exercises in set order

4 System should track the user’s hand movements during the exercise session

Met

5 System should provide feedback, preferably graphical (such a 3d hand animation) or at the very least text-based feedback

Met: LMRS supports real-time feedback through an animated model

6 System should present user with their results and save them in a database for easy retrieval by the patient or clinician

Partially met: Results can be presented to user but are stored via XML not SQL database

7 System should shutdown gracefully and correctly, confirming user’s wish to close and ensuring all important data is saved

Met

8 System should be able to handle unexpected difficulties such as the absence or removal of the Leap Motion or being unable to connect to the data-base

Met

9 System should run on Windows 7 or 8 in accordance with Leap Motion minimum system requirements

Partially Met: Unable to confirm Windows 8 due to dependencies induced by animated 3d model module

Table 5 – LMRS Compared against Functional Requirements

of 130

Overall, the LMRS meets the majority of the functional requirements set out in

the original specification. On the subject of Windows 8 compatibility

(requirement 9), this could not be confirmed due to XML dependencies

introduced by the 3d animated hand module. If the appropriate XML binaries

were present however it stands to reason that the LMRS would be able to run

without issue on Windows 8.

8.2.2 Evaluation against Non-Functional RequirementsThe LMRS will now be compared against its initial non-functional

requirements.

Req # Description Requirement Met?1 System should be robust and durable Met

2 System should be easy for the user to operate

Met

3 System should provide guidance on how to perform exercises

Met

4 System should capture accurate readings of the patient’s hand movements

Met

5 System should provide a meaningful use of gamification elements

Not Met

Table 6 – LMRS Compared against Non-Functional Requirements

Unfortunately, due to time constraints, gamification aspects of the LMRS could

not be implemented. However, the research for these elements of the system

has largely been done, making gamification an ideal starting point for further

enhancements.

8.3 Future Work and EnhancementsWhilst developing the LMRS, many a functionality had to be cut and/or

changed due to time constraints. Future work or systems in the vein of the

LMRS should consider these areas for potential enhancements.

The implementation of an SQL database. While the flexibility and

portability offered by XML – in addition to time constraints – have

resulted in a relatively powerful and easy to work with storage

of 130

mechanism, actual deployment of the LMRS would require an internet-

based storage mechanism.

In future, efforts should be made to improve the accuracy of the LMRS.

The Leap Motion SDK currently doesn’t offer skeletal tracking which

makes any work with the device inherently more difficult than with

traditional data gloves. Effort should be made to implement a means

by which individual fingers can be reliably identified; this will allow for

more accurate readings and the addition of more complex and intricate

exercises.

Tying into the point made above, unlike traditional data-gloves, the

Leap Motion easily allows for two handed exercises and potentially

exercises involving items such as cups or other everyday items

(through the SDK “point-able” object which allows for tracking of

additional items, not just fingers). Due to time constraints, these could

not be implemented. However, additional exercise types (two-handed

and those with items) represents a good opportunity to build on the

strengths of the Leap Motion.

of 130

9: ConclusionThis project saw us set out with the aim of researching and subsequently

developing a rehabilitation system for those with hand injuries and to do so

using the Leap Motion as our medium, rather than more traditional

technologies like data-gloves. The system was to allow a user to perform

rehabilitative exercises while receiving stimulating feedback via a real-time

animated model. The system was then required to store this data for later

viewing by either the patient or a clinician.

Upon completing the system, a more detailed set of functional and non-

functional requirements were drafted up to better inform the system design.

From these, mock-ups of the UI were created in addition to use-cases

describing a typical usage session. Development then began, with C#, WPF,

XAML and XNA as the technologies chosen to help with realising the system.

The end result is a system that bares striking resemblance to the system

described in those initial requirements and depicted in those initial UI mock-

ups.

After completing development, testing was then carried out to ensure the

functional correctness and accuracy of the system. Alongside traditional unit

and regression testing, the system has been subject to more formal test-cases

and finally to more specialised tests. These more ‘specialised’ tests took the

form of comparing the exercise repetition timings as recorded by the system to

those observed from a video recording of the same exercise. These tests

have proven that any differences between repetition timings as recorded by

the system are minimal (rarely above 100 milliseconds and below 100

milliseconds on average) when compared to those observed from a video

recording, suggesting the system holds much promise.

In closing, the LMRS can be deemed successful. We have been able to craft a

functional rehabilitation system using an entirely new medium – the Leap

Motion. The price and relative accuracy of this device in addition to its other

unique qualities mean the LMRS potentially represents the beginning of a

promising new avenue with regards to use of technology in rehabilitation.

of 130

10: ReferencesAckerman, E., 2011. Top 10 Robotic Kinect Hacks. [Online] Available at: http://spectrum.ieee.org/automaton/robotics/diy/top-10-robotic-kinect-hacks[Accessed 12 December 2013].

American Society for Surgery of the Hand, 2011. Hand and Arm Conditions. [Online] Available at: http://www.assh.org/Public/HandConditions/Pages/HandTherapy.aspx[Accessed 11 November 2013].

Amini, D., 2011. The Unique Role of Occupational Therapy in Rehabilitation of the Hand, s.l.: The American Occupational Therapy Association Inc.

Anon., 2013. Youtube - Leap Motion Structured Light Pattern. [Online] Available at: http://www.youtube.com/watch?v=UI5EBzU_QqM[Accessed 15 December 2013].

Bond, A., 2011. BSc Thesis: An Automated System for Reading Hand Measurements in Patients with Rheumatoid Arthritis, Londonderry: University of Ulster.

Bradski, G. & Kaehler, A., 2008. Learning OpenCV. s.l.:O'Reily.

B, S. & C, P., n.d. In: s.l.:s.n.

Burke, J. et al., 2009. Optimising Engagement for Stroke Rehabilitation using Serious Games. The Visual Computer, 25(12), pp. 1085-1099.

Case-Smith, J., 2003. Outcomes in Hand Rehabilitation Using Occupational Therapy Services. The American Journal of Occupational Therapy, Volume 57, pp. 499-506.

Chatfield, T., 2010. Why Gaming Will Dominate the Twnty-First Century. s.l.:Pegasus.

Deterding, S., Dixon, D., Khaled, R. & Nacke, L., 2011. From Game Design Elements to Gamefulness: Defining "Gamification". Mindtrek.

Dias, J. J. & Garcia-Elias, M., 2006. Hand Injury Costs. INJURY - International Journal of the Care of the Injured, Nov;37(11), pp. 1071-1077.

Dipietro, L., Sabatini, A. M. & Dario, P., 2008. A Survey of Glove-Based Systems and their Applications. IEE Transactions of Systems, Man, and Cybernetics, 38(4), pp. 461-482.

of 130

Fifth Dimension Technologies, 2011. 5DT Data Glove 5 Ultra Product Page. [Online] Available at: http://www.5dt.com/products/pdataglove5u.html[Accessed 6 December 2013].

Gerling, K. M. & Masuch, M., 2011. Exploring the Potential of Gamification Among Frail Elderly Persons. Vancouver, s.n.

G, R., 2003. Couts des Urgences Mains. Chir Main, Volume 22, pp. 258-63.

Groh, F., 2012. Research Trends in Media Informatics: Gamification: State of the Art Definition and Utilization. s.l., s.n., pp. 39-46.

Hasselkus, B., 2002. The Meaning of Everday Occupation. NJ: Slack Inc.

Hillerbrand, G., Bauer, M., Achatz, K. & Linker, G., 2005. Inverse Kinematic Infrared Optical Finger Tracking. [Online] Available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.62.9632&rep=rep1&type=pdf[Accessed 22 April 2014].

Holmberg, J., Lindgren, B. & Jutemark, R., 1996. Replantation-Revascularization and Primary Amputation in Major Hand Injuries. Resources Spent on Treatment and the Indirect Costs of Sick Leave in Sweden.. Journal of Hand Surgery, Volume 21, pp. 576-80.

HumanWare, 2010. Product Page. [Online] Available at: http://www.hmw.it/prodotti_i.html[Accessed 7 December 2013].

Jacobs, A. et al., 2013. CONTRAST: Gamficiation of Arm-Hand Training for Stroke Survivors. CHI EA '13 Extended Abstracts on Human Factors in Computing Systems, pp. 415-420.

Jakobsson, M., 2011. The Achievement Machine: Understanding Xbox 360 Achievements in Gaming Practices. [Online] Available at: http://gamestudies.org/1101/articles/jakobsson[Accessed 8 December 2013].

Khan Academy, 2013. Khan Academy. [Online] Available at: https://www.khanacademy.org/[Accessed 8 December 2013].

Lautman, R., 2012. Assessing Hand Movement i Arthritic Patients Using Wearable Glove Technology, Londonderry: s.n.

Lavanon, Y., 2013. The Advantages and Disadvantages of using High Technology in Hand Rehabilitation. Journal of Hand Therapy, Volume 26, pp.

of 130

179-183.

Leap Motion Inc, 2013. Leap Motion Product Page. [Online] Available at: https://www.leapmotion.com/product[Accessed 13 December 2013].

Leap Motion, 2013. SDK Docuementation - Tracking Hands, Fingers, and Tools. [Online] Available at: https://developer.leapmotion.com/documentation/python/devguide/Leap_Tracking.html[Accessed 22 April 2014].

O'Donnell, B., 2010. Hand Rehabilitation Using a Peregrine Gaming Data Glove, Londonderry: University of Ulster.

Pc Mag, n.d. Nyko Zoom for Kinect Review. [Online] Available at: http://xbox.about.com/od/projectnatal/fr/Nyko-Zoom-For-Kinect-Review-X360-Kinect.htm[Accessed 13 December 2013].

Peregrine, n.d. Peregrine Product Page. [Online] Available at: http://theperegrine.com/product/[Accessed 6 December 2013].

Shneiderman B, P. C., 2005. In: Designing the User Interface - Fourth Edition. s.l.:Pearson, pp. 92-93.

Sommerville, I., 2010. Software Engineering. 9 ed. s.l.:Addison Wesley.

Sousa, A. D. et al., 2013. Psychological Issues In Hand Trauma. ASEAN Journal of Psychiatry, 14(1).

Strategy One, 2008. Ergonomics and Repetitive Strain Injury, s.l.: Strategy One for Microsoft.

The Pleasure Revolution: Why Games Will Lead the Way. 2011. [Film] Directed by Jesse Schell. s.l.: s.n.

Trybus, M., Lorkowski, J., Brongel, L. & Hladki, W., 2006. Causes and Consequences of Hand Injuries. The American Journal of Surgery, July;129(1), pp. 52-57.

Turner, J. R., 1993. The Hanbook of Project-Based Management. s.l.:McGraw-Hill Book Co..

Weichert, F., Bachmann, D., Rudak, B. & Fisseler, D., 2013. Analysis of the Accuracy and Robustness of the Leap Motion Controller. Sensors, 13(5), pp. 6380-6393.

of 130

Willow Garage, 2011. Willow Garage Home Page. [Online] Available at: http://www.willowgarage.com/[Accessed 19 December 2013].

Yassi, A., 1997. Repetitve Strain Injuries. The Lancet, Volume 349, pp. 943-47.

of 130

11.1: Appendix 1: Clinician User Scenario Test CaseTest ID Description Steps Expected

OutcomeActual

OutcomePass/Fail

TestDate

Comment(s)

0 Login with junk credentials

Enter false username and/or password

System should show a failed login prompt

Failed login prompt shown

PASS 18/04/14

1 Login with valid credentials

Enter valid username and password

System should login successfully

Login successful

PASS 18/04/14

2 Able to view patient user exercise results

Select “View Patient Results” menu option. Toggle between patients, exercises and dates

System should allow graphs of various types, reflecting changes in patient, exercise and dates

Data is shown but crashes if no session data is available/also no guard against duplicate sessions

FAIL 18/04/14 Added checks for empty session data and attempts to make duplicate sessions under same data now ignored. PASS after retesting

3 Leaving fields in “Register New Patient” screen empty

Leave any field in the “Register New Patient” screen blank and click “Register”

System should show a prompt asking the user to fill in the empty field

Prompt shown PASS 18/04/14

4 Register a duplicate patient

Enter the details for an already registered

System should show a prompt informing the user that such


patient user in the “Register New Patient” screen and click “Register”

a patient is already registered

5 Able to register new patient user

Enter the details for a new patient user in the “Register New Patient” screen and click “Register”

System should show a prompt confirming successful addition

Prompt shown, new patient found in XML data store

PASS 18/04/14

6 Unable to remove non-existing patient user

Enter the username of a non-existent patient in the search field of the “Remove Existing Patient” screen and click “Search”

System should show a prompt requesting the user to enter a valid patient username


7 Able to remove existing patient user

Select “Register New Patient”, enter new patient credentials and click “Register”

System should show a prompt confirming successful addition

Prompt shown, patient removed from XML data store

PASS/FAIL 18/04/14 Empty remaining patient XML element causing results screen crash.

8 Log out Click “Log Out” System should show a prompt, click OK at


of 130

which point user should be returned to the login screen

of 130

11.2: Appendix 2: Patient User Scenario Test CaseTest ID Description Steps Expected

OutcomeActual

OutcomePass/Fail

TestDate

Comment(s)

0 Login with junk credentials

Enter false username and/or password

System should show a failed login prompt


1 Login with valid credentials

Enter valid username and password

System should login successfully

Login successful

PASS 19/04/14

2 Can perform exercises

From the main menu, select the “Perform Exercises” button

System should show “Perform Exercises” screen, exercises should progress logically, with instructions and timing updated

Able to perform exercises, instructions, timing and 3d hand model update and respond

PASS 19/04/14

3 Progress loss prompt shown when leaving while exercise in progress

While performing an exercise, click the “Back” button

System should show a prompt, informing of data loss and requiring user confirmation to continue

No prompt shown

FAIL 19/04/14 Prompt added

4 Can view exercise results

When exercises have

The “View Results” screen

Crash if any exercises have

FAIL 19/04/14 Added guard against empty

of 130

from exercise page

been completed, the “Continue” button should be enabled, click it

should be shown. The chart should respond to changes in exercise type, start/end date and chart type.

no data recorded against them

sessions

5 Can view exercise results from main menu

From the main menu, select the “View Results” option

The “View Results” screen should be shown. The chart should respond to changes in exercise type, start/end date and chart type.

Same as above FAIL 19/04/14

6 Log out Click “Log Out” System should show a prompt, click OK at which point user should be returned to the login screen


of 130

11.3: Appendix 3: Source Code

LMRS_LeapListener.cs

using System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;

using Leap;using System.Diagnostics;

// Derives from the default Listener provided by the Leap API,// provides callback implementations for key events.namespace LMRS_Main{ class LeapListener : Listener { private Leap.Frame frame = null;

public override void OnConnect(Controller controller) { Console.WriteLine("Leap Motion Connected"); // Register gesutres here... }

public override void OnDisconnect(Controller controller) { Console.WriteLine("Leap Motion Disconnected"); }

public override void OnInit(Controller controller) { Console.WriteLine("Initilising..."); }

public override void OnExit(Controller controller) { Console.WriteLine("Exiting..."); }

public override void OnFocusGained(Controller controller) { Console.WriteLine("Gained Focus"); }

public override void OnFocusLost(Controller controller) {

Console.WriteLine("Lost Focus"); }

public override void OnFrame(Controller controller) { // Get the most recent frame from the device frame = controller.Frame();

int STOP; if(frame != null) STOP = 1; }

public Leap.Frame LeapFrame { get { return frame; } } }}

of 130

LMRS_Login.xaml

<Page x:Class="LMRS_Main.LMRS_Login" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"

Title="LMRS_Login" Width="1366">

<Grid> <Grid.ColumnDefinitions> <ColumnDefinition Width="52.873*" /> <ColumnDefinition Width="1313.127*" /> </Grid.ColumnDefinitions> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Welcome to the Leap Motion Rehabilitation System" Height="119" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White" Grid.ColumnSpan="2" />

 <Label FontSize="32" HorizontalAlignment="Left" Margin="357.25,302.948,0,351.534" Name="lblUsername" Width="160" Grid.Column="1">Username:</Label> <TextBox FontSize="32" Height="50" Margin="0,305.806,440.132,351.534" Name="txtUsername" TabIndex="1" Grid.Column="1" HorizontalAlignment="Right" Width="349.981" />

<Label FontSize="32" Height="50" HorizontalAlignment="Left" Margin="371.54,0,0,257.22" Name="lblPasword" VerticalAlignment="Bottom" Width="150" Grid.Column="1">Password:</Label> <PasswordBox FontSize="32" Grid.Column="1" Height="50" Margin="523.014,0,440.132,257.22" Name="pwbxPassword" VerticalAlignment="Bottom" PasswordChar="*"/>

<Button Content="Login" Height="46" HorizontalAlignment="Right" Margin="0,661,41,0" Name="btnLogin" VerticalAlignment="Top" Width="150" Click="btnLogin_Click" Grid.Column="1" /> <Button Content="Exit" Height="46" HorizontalAlignment="Left" Margin="32,661,0,0" Name="btnExit" VerticalAlignment="Top" Width="150" Click="btnExit_Click" Grid.ColumnSpan="2" /> </Grid></Page>

LMRS_Login.xaml.cs

using System;using System.Collections.Generic;using System.Linq;using System.Text;

of 130

using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;using System.Xml.Linq;using System.Data;

namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_Login.xaml /// </summary> public partial class LMRS_Login : Page { public LMRS_Login() { InitializeComponent();

// Set the window title this.WindowTitle = "Leap Motion Rehabilitation System - Login"; }

private void btnLogin_Click(object sender, RoutedEventArgs e) { // Have credentials been provided? if (txtUsername.Text == "" || pwbxPassword.Password == "") { MessageBox.Show("Please provide both a username and password.", "Cannot Login", MessageBoxButton.OK, MessageBoxImage.Error);

txtUsername.Text = pwbxPassword.Password = ""; } // Is the username valid? else if (!txtUsername.Text.Contains("p_") && !txtUsername.Text.Contains("c_")) { MessageBox.Show("Please provide a valid username.", "Cannot Login", MessageBoxButton.OK, MessageBoxImage.Error);

txtUsername.Text = pwbxPassword.Password = ""; } else { XDocument lmrsXml = XDocument.Load("LMRS_TestFile.xml");

if (txtUsername.Text.Contains("p_")) { // Find patient user XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in

of 130

patientCollection.Elements("patient") where (string)el.Attribute("username") == txtUsername.Text select el;

if (patient.Count() == 0) { MessageBox.Show("Please provide a valid username.", "Cannot Login", MessageBoxButton.OK, MessageBoxImage.Error);

txtUsername.Text = pwbxPassword.Password = ""; } else { XElement userOnRecord = patient.First<XElement>();

if (userOnRecord != null && pwbxPassword.Password.Equals( userOnRecord.Attribute("password").Value. ToString())) { UserInfoHelper.user = userOnRecord; UserInfoHelper.foreName = userOnRecord.Element("forename"). Value; UserInfoHelper.userName = userOnRecord.Attribute("username"). Value;

this.NavigationService.Navigate( new LMRS_MainMenu_Patient()); } else { MessageBox.Show("Please provide a valid password.", "Cannot Login", MessageBoxButton.OK, MessageBoxImage.Error); pwbxPassword.Password = ""; } } } else { // Find clinician user XElement clinicianCollection = lmrsXml.Root.Element("clinicianCollection"); IEnumerable<XElement> clinician = from el in clinicianCollection.Elements("clinician") where (string)el.Attribute("username") == txtUsername.Text select el;

if (clinician.Count() == 0) { MessageBox.Show("Please provide a valid username.", "Cannot Login", MessageBoxButton.OK,

of 130

MessageBoxImage.Error); txtUsername.Text = pwbxPassword.Password = ""; } else { XElement userOnRecord = clinician.First<XElement>(); if (userOnRecord != null && pwbxPassword.Password.Equals(userOnRecord.Attribute("password").Value.ToString())) { UserInfoHelper.user = userOnRecord; UserInfoHelper.foreName = userOnRecord.Element("forename").Value; UserInfoHelper.userName = userOnRecord.Attribute("username").Value;

this.NavigationService.Navigate(new LMRS_MainMenu_Clinician()); } else { MessageBox.Show("Please provide a valid password.", "Cannot Login", MessageBoxButton.OK, MessageBoxImage.Error); pwbxPassword.Password = ""; } } } } } // btnLogin_Click()

private void btnExit_Click(object sender, RoutedEventArgs e) { (this.Parent as NavigationWindow).Close(); } }}

of 130

LMRS_MainMenu_Clinician.xaml

<Page x:Class="LMRS_Main.LMRS_MainMenu_Clinician" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"

Title="LMRS_MainMenu_Clinician" >

<Grid> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Welcome [CLINICIAN-NAME], Please Select an Option" Height="119" HorizontalAlignment="Center" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" Width="1342" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White"/>

<Button Content="View Patient Results" Height="85" HorizontalAlignment="Left" Margin="455,181,0,0" Name="btnViewPatientResults" VerticalAlignment="Top" Width="455" Click="btnViewPatientResults_Click"/> <Button Content="Register New Patient" Height="85" HorizontalAlignment="Left" Margin="455,328,0,0" Name="btnRegisterPatient" VerticalAlignment="Top" Width="455" Click="btnRegisterPatient_Click" /> <Button Content="Remove Existing Patient" Height="85" HorizontalAlignment="Left" Margin="455,474,0,0" Name="btnRemovePatient" VerticalAlignment="Top" Width="455" Click="btnRemovePatient_Click"/> <Button Content="Logout" Height="85" HorizontalAlignment="Left" Margin="455,620,0,0" Name="btnLogout" VerticalAlignment="Top" Width="455" Click="btnLogout_Click"/> </Grid></Page>

LMRS_MainMenu_Clinician.xaml.cs


of 130

namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_MainMenu_Clinician.xaml /// </summary> public partial class LMRS_MainMenu_Clinician : Page { public LMRS_MainMenu_Clinician() { InitializeComponent();

// Set the window title and header lblHeader.Content = "Welcome " + UserInfoHelper.foreName + ", Please Select an Option"; this.WindowTitle = "Leap Motion Rehabilitation System - Main Menu"; }

private void btnViewPatientResults_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_ViewExerciseResults("clinician")); }

private void btnRegisterPatient_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_RegisterNewPatient()); }

private void btnRemovePatient_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_RemovePatient()); }

private void btnLogout_Click(object sender, RoutedEventArgs e) { if (MessageBox.Show("Are you sure you wish to logout?", "Logout Confirmation", MessageBoxButton.YesNo, MessageBoxImage.Question) == MessageBoxResult.Yes) { this.NavigationService.Navigate(new LMRS_Login()); } } }}

of 130

LMRS_MainMenuPatient.xaml

<Page x:Class="LMRS_Main.LMRS_MainMenu_Patient" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"

Title="LMRS_MainMenu_Patient">

<Grid> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Welcome [PATIENT-NAME], Please Select an Option" Height="119" HorizontalAlignment="Center" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" Width="1342" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White"/>

<Button Content="Perform Exercises" Height="85" HorizontalAlignment="Left" Margin="455,227,0,0" Name="btnPerformExercises" VerticalAlignment="Top" Width="455" Click="btnPerformExercises_Click"/> <Button Content="View Results" Height="85" HorizontalAlignment="Left" Margin="455,414,0,0" Name="btnResults" VerticalAlignment="Top" Width="455" Click="btnViewExerciseResults_Click" /> <Button Content="Logout" Height="85" HorizontalAlignment="Left" Margin="455,601,0,0" Name="btnLogout" VerticalAlignment="Top" Width="455" Click="btnLogout_Click"/> </Grid></Page>

LMRS_MainMenu_Patient.xaml.cs


namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_MainMenu_Patient.xaml

of 130

/// </summary> public partial class LMRS_MainMenu_Patient : Page { public LMRS_MainMenu_Patient() { InitializeComponent();

// Set the window title and header lblHeader.Content = "Welcome " + UserInfoHelper.foreName + ", Please Select an Option"; this.WindowTitle = "Leap Motion Rehabilitation System - Main Menu"; }

private void btnPerformExercises_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_PerformExercise()); }

private void btnViewExerciseResults_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_ViewExerciseResults("patient")); }

private void btnLogout_Click(object sender, RoutedEventArgs e) { if (MessageBox.Show("Are you sure you wish to logout?", "Logout Confirmation", MessageBoxButton.YesNo, MessageBoxImage.Question) == MessageBoxResult.Yes) { this.NavigationService.Navigate(new LMRS_Login()); } } }}

of 130

LMRS_PerformExercise.xaml

<Page x:Class="LMRS_Main.LMRS_PerformExercise" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:wf="clr-namespace:System.Windows.Forms;assembly=System.Windows.Forms" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"

Title="LMRS_PerformExercise" Loaded="Page_Loaded" Unloaded="Page_Unloaded" Width="1366">

<Grid> <Grid.ColumnDefinitions> <ColumnDefinition Width="124.323*" /> <ColumnDefinition Width="51.444*" /> <ColumnDefinition Width="1190.233*" /> </Grid.ColumnDefinitions> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Perform Exercises" Height="119" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White" Grid.ColumnSpan="3" />

<Button Content="Back (Main Menu)" Height="46" HorizontalAlignment="Left" Margin="32,661,0,0" Name="btnExit" VerticalAlignment="Top" Width="150" Click="btnBack_Click" Grid.ColumnSpan="3" />

<StackPanel HorizontalAlignment="Left" Margin="32,158.619,0,65.734" Width="644" Grid.ColumnSpan="3"> <WindowsFormsHost Width="640" Height="480" Margin="2"> <wf:Panel x:Name="RenderPanel" BackColor="Black"/> </WindowsFormsHost> </StackPanel>

<TextBlock TextAlignment="Center" FontSize="32" Background="Gray" Height="42" Grid.Column="2" Margin="538.159,0,35.725,287.229" Name="txtExerciseInstructions" Text="Exercise Instructions..." VerticalAlignment="Bottom" /> <TextBlock TextAlignment="Center" FontSize="32" Background="LightGray" Margin="538.159,160.619,35.725,350.105" Name="txtExerciseDescription" Text="Exercise Description..." Grid.Column="2" /> <TextBlock TextAlignment="Center" FontSize="32" Foreground="Green" Background="LightGray" Height="164" Margin="538.159,0,356.074,98.601" Name="txtCurrentRepInfo" Text="Current Rep Info..." VerticalAlignment="Bottom" Grid.Column="2" /> <TextBlock TextAlignment="Center" FontSize="32" Background="LightGray" Height="164" HorizontalAlignment="Right" Margin="0,0,35.725,98.595" Name="txtBestRepInfo" Text="Best Rep Info..." VerticalAlignment="Bottom" Width="296" Grid.Column="2" />

of 130

<Button Height="46" Margin="0,0,35.725,31" Name="btnContinue" VerticalAlignment="Bottom" Grid.Column="2" HorizontalAlignment="Right" Width="150" IsEnabled="False" Click="btnContinue_Click">Continue</Button> </Grid></Page>

LMRS_PerformExercise.xaml.cs

using System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;using System.Threading;

using SkeletalAnimation;using Leap;using System.ComponentModel;using System.Diagnostics;using System.Xml.Linq;

namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_PerformExercise.xaml /// </summary> public partial class LMRS_PerformExercise : Page { // For animated hand private SkeletalAnimationSample game; private Thread xnaThread;

// For Leap Motion private Thread leapThread; private Leap.Controller controller; private Leap.Listener listener; private BackgroundWorker leapBackgroundWorker;

// For exercise progress tracking uint calls = 0; bool calibrated = false; bool calibratePromptGiven = false; int currentExercise = 1; float avgMag = 0.0f; float avgPitch = 0.0f; bool exerciseDescGiven = false; bool clenchPromptGiven = false; bool restPromptGiven = false; bool lowerPromptGiven = false;

of 130

bool raisePromptGiven = false; uint repsRequired = 5, repsPerformed = 0; long[] repTimes = new long[5]; long bestRepTime = 60000; Stopwatch timer = new Stopwatch();

Hand hand = null; bool exercisesFinished = false;

public LMRS_PerformExercise() { InitializeComponent();

// Set the window title this.WindowTitle = "Leap Motion Rehabilitation System - Perform Exercise"; }

private void btnBack_Click(object sender, RoutedEventArgs e) { if (!exercisesFinished) { if (MessageBox.Show("Are you sure you wish to quit, progress on your current exercise will be lost?", "Exercises Still in Progress", MessageBoxButton.OKCancel, MessageBoxImage.Question) == MessageBoxResult.OK) { this.NavigationService.Navigate(new LMRS_MainMenu_Patient()); } } else { this.NavigationService.Navigate(new LMRS_MainMenu_Patient()); } }

private void btnContinue_Click(object sender, RoutedEventArgs e) { NavigationService.Navigate(new LMRS_ViewExerciseResults("patient")); }

private void Page_Loaded(object sender, RoutedEventArgs e) { // Set XNA hand running IntPtr handle = RenderPanel.Handle; xnaThread = new Thread(new ThreadStart(() => { game = new SkeletalAnimationSample(handle); game.Run(); } )); xnaThread.Start();

// Set Leap Motion running listener = new LeapListener(); controller = new Leap.Controller(); controller.AddListener(listener);

of 130

leapBackgroundWorker = new BackgroundWorker(); leapBackgroundWorker.WorkerSupportsCancellation = true; leapBackgroundWorker.DoWork += new DoWorkEventHandler(leapBackgroundWorker_DoWork); leapBackgroundWorker.RunWorkerAsync(); }

private void Page_Unloaded(object sender, RoutedEventArgs e) { // Shutdown hand xnaThread.Abort();

// Shutdown Leap Motion controller.RemoveListener(listener); controller.Dispose(); controller = null; leapBackgroundWorker.CancelAsync(); }

void leapBackgroundWorker_DoWork(object sender, DoWorkEventArgs e) { while (true) { Leap.Frame frameData = ((LeapListener)listener).LeapFrame;

if (frameData != null && frameData.Hands.Count > 0) { hand = frameData.Hands[0];

// Calibration #region Clibration if (!calibrated) { if (!calibratePromptGiven) { updateExerciseInstruction("Calibrating, please wait..."); calibratePromptGiven = true; }

if (!timer.IsRunning) { timer.Start(); }

// Collect hand pitch and vector magnitude readings, // large enough deltas in these parameters are used to // detect user movements and start the timer if (timer.Elapsed.Seconds < 5) { avgMag += (hand.Fingers.Frontmost.TipPosition - hand.PalmPosition).Magnitude; avgPitch += hand.Direction.Pitch; ++calls; } else

of 130

{ avgMag /= calls; avgPitch /= calls; calibrated = true; timer.Reset(); } } #endregion else { // Move onto next exercise/move onto results? if (repsPerformed == repsRequired) { logExerciseResults();

++currentExercise; repsPerformed = 0; exerciseDescGiven = false; bestRepTime = 60000;

if (currentExercise <= 3) { updateExerciseInstruction("Next exercise in three seconds..."); restartTimer(); while (timer.Elapsed.Seconds <= 3) ; timer.Reset(); } else { updateExerciseInstruction("Exercises complete"); game.ResetCameraArcAndRoll(); Dispatcher.Invoke(new Action(() => { btnContinue.IsEnabled = true; })); exercisesFinished = true; } }

// Demo exercise 1 - Going from hand at rest to clenched fist #region ex1 if (currentExercise == 1 && repsPerformed < repsRequired) { // Give exercise brief if (!exerciseDescGiven) { updateExerciseDescription("Exercise 1 of 3: Fist Clench. " + "\n1) Form a closed fist. \n2) Relax hand. \n\nThe timer will start automatically"); exerciseDescGiven = true; }

// A sufficient delta in the vector between finger-tip and palm signals // the user has begun to moving their hand, start the timer

of 130

if ((hand.Fingers.Frontmost.TipPosition - hand.PalmPosition).Magnitude < 0.9 * avgMag && !timer.IsRunning) { restartTimer(); }


updateRepInfo("This rep time: \nRep " + (repsPerformed + 1) + " of 5 : \n" + repTimes[repsPerformed] / 1000.0 + " seconds.");



// Go if (repsPerformed < repsRequired) { updateExerciseInstruction("Clench your hand into a fist"); clenchPromptGiven = true; restPromptGiven = false; } }

// Exercise part 1 - Relax hand if (hand.Fingers.Count == 0 && !restPromptGiven) { updateExerciseInstruction("Place your hand at rest"); restPromptGiven = true; clenchPromptGiven = false; }

// Update xna hand model // updateHandModelExercise1(frameData); } #endregion

of 130

// Demo exercise 2 - Wrist flexion/extension #region ex2 if (currentExercise == 2 && repsPerformed < repsRequired) { // Start the exercise if (hand.Fingers.Count >= 1) { // Give exercise brief if (!exerciseDescGiven) { updateExerciseDescription("Exercise 2 of 3: Wrist Flexion/Extension. " + "\n1) Tilt your hand upward \n2) Tilt your hand downward \nThe timer will start automatically"); exerciseDescGiven = true;

updateExerciseInstruction("Raise your hand from the wrist"); }

// Lower hand if (hand.Direction.Pitch > avgPitch * 2.5 && repsPerformed < repsRequired && !lowerPromptGiven) { // Present time for this rep if (timer.IsRunning)// && !firstRep) { repTimes[repsPerformed] = timer.ElapsedMilliseconds; updateRepInfo("Rep " + (repsPerformed + 1) + " complete. \nTime: " + repTimes[repsPerformed] / 1000.0 + " seconds.");


++repsPerformed; } restartTimer(); //firstRep = false;

if (!lowerPromptGiven) { updateExerciseInstruction("Lower your hands from the wrist"); } lowerPromptGiven = true; raisePromptGiven = false; }

of 130

// Raise hand if (hand.Direction.Pitch < -2.5 * avgPitch && repsPerformed < repsRequired && !raisePromptGiven) { if (!raisePromptGiven) { updateExerciseInstruction("Raise your hands from the wrist"); } lowerPromptGiven = false; raisePromptGiven = true; } }

// Update xna hand //updateHandModelExercise2(frameData); } #endregion

// Demo exercise 3 - Three jaw chuck pinch #region ex3 if (currentExercise == 3 && repsPerformed < repsRequired) { // Give exercise brief if (!exerciseDescGiven) { updateExerciseDescription("Exercise 3 of 3: Three Jaw. " + "\n1) Form a closed fist. \n2) Relax hand. \n\nThe timer will start automatically"); exerciseDescGiven = true; game.ResetCameraArcAndRoll(); }

// A sufficient delta in the vector between finger-tip and palm signals // the user has begun to moving their hand, start the timer if ((hand.Fingers.Frontmost.TipPosition - hand.PalmPosition).Magnitude < 0.9 * avgMag && !timer.IsRunning) { restartTimer(); }


updateRepInfo("Rep " + (repsPerformed + 1) + " of 5 complete. \nTime: " + repTimes[repsPerformed] / 1000.0 + " seconds.");

// New best rep time?

of 130

if (repTimes[repsPerformed] < bestRepTime) { bestRepTime = repTimes[repsPerformed]; updateBestRepInfo("Best rep time: \nRep " + (repsPerformed + 1) + " of 5: \n" + bestRepTime / 1000.0 + "seconds."); }


// Go if (repsPerformed < repsRequired) { updateExerciseInstruction("Pinch your fingers and thumb together"); clenchPromptGiven = true; restPromptGiven = false; } }

// Exercise part 1 - Relax hand if (hand.Fingers.Count <= 1 && !restPromptGiven) { updateExerciseInstruction("Place your fingers and thumb at rest"); restPromptGiven = true; clenchPromptGiven = false; } } #endregion

// Update hand model updateHandModel(frameData); } } // if (!calibrated) else { // ... } } } // leapBackgroundWorker_DoWork()

// Update the hand model based on information from the Leap Motion private void updateHandModel(Leap.Frame frame) { // Calculate necessary vectors Leap.Vector fingerBase = -frame.Hands[0].Fingers.Leftmost.Direction * frame.Hands[0].Fingers.Leftmost.Length; fingerBase = fingerBase + frame.Hands[0].Fingers.Leftmost.TipPosition;

Leap.Vector palmCenterToBase = fingerBase - frame.Hands[0].PalmPosition; Leap.Vector baseToTip =

of 130

frame.Hands[0].Fingers.Leftmost.TipPosition - fingerBase;

// Calculate angles float thumbIndex2 = -(1.0f - Leap.Vector.ZAxis.AngleTo(palmCenterToBase)); // Some basic clamping thumbIndex2 = thumbIndex2 < -1.75f ? -1.75f : thumbIndex2;

// Set the angles for the fingers // Thumb game.Thumb2Radians = thumbIndex2; game.Thumb1Radians = (float)(((0.23 + 1.73 * 0.33 + 1.5 * (0.33 * 0.33))) * thumbIndex2);

// Index fingerBase = -frame.Hands[0].Fingers.Frontmost.Direction * frame.Hands[0].Fingers.Frontmost.Length; fingerBase = fingerBase + frame.Hands[0].Fingers.Frontmost.TipPosition; palmCenterToBase = fingerBase - frame.Hands[0].PalmPosition; baseToTip = frame.Hands[0].Fingers.Frontmost.TipPosition - fingerBase;

// Some basic clamping float fingerIndex3 = (-(3.0f - Leap.Vector.ZAxis.AngleTo(baseToTip))) > 0 ? 0 : (-(3.0f - Leap.Vector.ZAxis.AngleTo(baseToTip))); fingerIndex3 = fingerIndex3 <= -1.25f ? -1.25f : fingerIndex3;

float fingerIndex2 = (float)(((0.23 + 1.73 * 0.66 + 1.5 * (0.66 * 0.66)))) * fingerIndex3; float fingerIndex1 = (float)(((0.23 + 1.73 * 0.33 + 1.5 * (0.33 * 0.33)))) * fingerIndex3;

// Calulate pitch and yaw float pitch = -frame.Hands[0].Direction.Normalized.Pitch; float yaw = -frame.Hands[0].Direction.Normalized.Yaw;

// Tone down yaw yaw -= 0.5f; yaw /= 4;

// Update hand game.CameraDown(pitch); game.CameraLeft(yaw);

game.Index3Radians = fingerIndex3; game.Index2Radians = fingerIndex2; game.Index1Radians = fingerIndex1;

// Middle game.Middle3Radians = fingerIndex3; game.Middle2Radians = fingerIndex2; game.Middle1Radians = fingerIndex1;

// Ring if (currentExercise == 3) {

of 130

fingerIndex3 = -1.0f; fingerIndex2 = fingerIndex1 = (float)(((0.23 + 1.73 * 0.33 + 1.5 * (0.33 * 0.33))) * fingerIndex3); }

game.Ring3Radians = fingerIndex3; game.Ring2Radians = fingerIndex2; game.Ring1Radians = fingerIndex1;

// Pinky game.Little3Radians = fingerIndex3; game.Little2Radians = fingerIndex2; game.Little1Radians = fingerIndex1; }

private void logExerciseResults() { // Load the XML file XDocument lmrsXml = XDocument.Load("LMRS_TestFile.xml");

// Get patient collection node XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in patientCollection.Elements("patient") where (string)el.Attribute("username") == UserInfoHelper.userName select el;

string exerciseName = ""; if (currentExercise == 1) { exerciseName = "Fist_Clench"; } else if (currentExercise == 2) { exerciseName = "Wrist_Flexion_Extension"; } else { exerciseName = "Three_Jaw_Chuck_Pinch"; }

IEnumerable<XElement> exercise = from el in patient.Elements("activity") where el.Attribute("name").Value.Equals(exerciseName) select el; XElement e = exercise.First();

// Make sure this isn't a duplicate session (date already on record). IEnumerable<XElement> sessions = from el in e.Elements("session") where el.Attribute("date").Value.Equals(DateTime.Now.Date.ToString("dd/MM/yy")) select el;

if (sessions.Count() == 0)

of 130

{ XElement session = new XElement("session", new XAttribute("date", DateTime.Now.Date.ToString("dd/MM/yy")), new XElement("rep", repTimes[0] / 1000.0), new XElement("rep", repTimes[1] / 1000.0), new XElement("rep", repTimes[2] / 1000.0), new XElement("rep", repTimes[3] / 1000.0), new XElement("rep", repTimes[4] / 1000.0));

e.Add(session); lmrsXml.Save("LMRS_TestFile.xml"); } }

// Helper methods private void restartTimer() { timer.Reset(); timer.Start(); }

private void updateExerciseDescription(string newDescription) { Dispatcher.Invoke(new Action(() => { txtExerciseDescription.Text = newDescription; })); }

private void updateExerciseInstruction(string nextInstruction) { Dispatcher.Invoke(new Action(() => { txtExerciseInstructions.Text = nextInstruction; })); }

private void updateRepInfo(string repUpdate) { Dispatcher.Invoke(new Action(() => { txtCurrentRepInfo.Text = repUpdate; })); }

private void updateBestRepInfo(string newBestRep) { Dispatcher.Invoke(new Action(() => { txtBestRepInfo.Text = newBestRep; })); } }}

of 130

LMRS_RegisterNewPatient.xaml

<Page x:Class="LMRS_Main.LMRS_RegisterNewPatient" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"

Title="LMRS_RegisterNewPatient">

<Grid> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Please Enter Patient Details" Height="119" HorizontalAlignment="Center" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" Width="1342" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White"/>

<Button Content="Back (Main Menu)" Height="46" HorizontalAlignment="Left" Margin="32,661,0,0" Name="btnExit" VerticalAlignment="Top" Width="150" Click="btnBack_Click"/>

 <Label Height="50.00" HorizontalAlignment="Left" Margin="107.175,160.048,0,0" Name="lblForename" VerticalAlignment="Top" Width="150" FontSize="32">Forename:</Label> <TextBox Height="50.00" HorizontalAlignment="Left" Margin="295.803,160.048,0,0" Name="txtForename" VerticalAlignment="Top" Width="350" FontSize="32" />

<Label Height="50.00" HorizontalAlignment="Left" Margin="107.175,240.072,0,0" Name="lblSurname" VerticalAlignment="Top" Width="150" FontSize="32">Surname:</Label> <TextBox Height="50.00" HorizontalAlignment="Left" Margin="295.784,240.072,0,0" Name="txtSurname" VerticalAlignment="Top" Width="350" FontSize="32" TextChanged="txtSurname_TextChanged"/>

<Label Height="50.00" HorizontalAlignment="Left" Margin="107.175,320.096,0,337.244" Name="lblAddress1" Width="155" FontSize="32">Address 1:</Label> <TextBox Height="50" HorizontalAlignment="Left" Margin="295.784,322.096,0,335.244" Name="txtAddress1" Width="350" FontSize="32"/>

<Label Height="50.00" HorizontalAlignment="Left" Margin="107.175,0,0,255.791" Name="lblAddress2" VerticalAlignment="Bottom" Width="155" FontSize="32">Address 2:</Label> <TextBox Height="50.00" HorizontalAlignment="Left" Margin="295.803,0,0,255.791" Name="txtAddress2" VerticalAlignment="Bottom" Width="350" FontSize="32" />

<Label Height="50.50" HorizontalAlignment="Left" Margin="107.175,0,0,171.48" Name="lblAddress3"

of 130

VerticalAlignment="Bottom" Width="155" FontSize="32">Address 3:</Label> <TextBox Height="50.00" HorizontalAlignment="Left" Margin="295.803,0,0,171.48" Name="txtAddress3" VerticalAlignment="Bottom" Width="350" FontSize="32"/>

<Label Height="50.00" HorizontalAlignment="Left" Margin="107.175,0,0,90.027" Name="lblPostcode" VerticalAlignment="Bottom" Width="175" FontSize="32">Post Code:</Label> <TextBox Height="50.00" HorizontalAlignment="Right" Margin="0,0,720.216,90.027" Name="txtPostcode" VerticalAlignment="Bottom" Width="350" FontSize="32"/>

 <Label FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,160.048,467.283,0" Name="lblPassword" VerticalAlignment="Top" Width="175">Password:</Label> <TextBox FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,160.048,104.317,0" Name="txtPassword" VerticalAlignment="Top" Width="350" />

<Label FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,240.072,367.283,0" Name="lblConfirmPassword" VerticalAlignment="Top" Width="275">Confirm Password:</Label> <TextBox FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,240.072,104.317,0" Name="txtConfirmPassword" VerticalAlignment="Top" Width="255" />

<Label FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,0,467.283,88.027" Name="lblUsername" VerticalAlignment="Bottom" Width="175">Username:</Label> <TextBox FontSize="32" Height="50" HorizontalAlignment="Right" Margin="0,0,104.317,88.027" Name="txtUsername" VerticalAlignment="Bottom" Width="350" IsReadOnly="True"/>

<Button Height="46" HorizontalAlignment="Right" Margin="0,0,32,32" Name="btnRegister" VerticalAlignment="Bottom" Width="150" Click="btnRegister_Click">Register</Button> </Grid></Page>

LMRS_RegisterNewPatient.xaml.cs


of 130

using System.Xml.Linq;

namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_RegisterNewPatient.xaml /// </summary> public partial class LMRS_RegisterNewPatient : Page { public LMRS_RegisterNewPatient() { InitializeComponent();

// Set the window title this.WindowTitle = "Leap Motion Rehabilitation System - Register New Patient"; }

private void btnBack_Click(object sender, RoutedEventArgs e) { LMRS_MainMenu_Clinician mainMenu = new LMRS_MainMenu_Clinician(); this.NavigationService.Navigate(mainMenu); }

private void btnRegister_Click(object sender, RoutedEventArgs e) { // Necessary information provided? if (!textFieldsNotComplete()) { MessageBox.Show("Please ensure all text fields have been completed.", "Missing Information", MessageBoxButton.OK, MessageBoxImage.Error); } else { // Load the XML file XDocument lmrsXml = XDocument.Load("LMRS_TestFile.xml");

// Does the patient user exist already? String patientUsername = "p_" + txtSurname.Text.ToLower() + txtForename.Text.ToUpper()[0];

// Get patient collection node XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in patientCollection.Elements("patient") where (string)el.Attribute("username") == patientUsername select el;

if (patient.Count() != 0) { MessageBox.Show("A patient is already registered with those details.", "Patient Already Exists", MessageBoxButton.OK, MessageBoxImage.Error);

of 130

clearTextFields(); } else { // Add new patient information to record XElement newPatient = new XElement("patient", new XAttribute("username", patientUsername), new XAttribute("password", txtPassword.Text), new XElement("forename", txtForename.Text), new XElement("surname", txtSurname.Text), new XElement("addressLineOne", txtAddress1.Text), new XElement("addressLineTwo", txtAddress2.Text), new XElement("addressLineThree", txtAddress3.Text), new XElement("postCode", txtPostcode.Text), new XElement("activity", new XAttribute("name", "Fist_Clench")), new XElement("activity", new XAttribute("name", "Wrist_Flexion_Extension")), new XElement("activity", new XAttribute("name", "Three_Jaw_Chuck_Pinch")));

// Save and close patientCollection.Add(newPatient); lmrsXml.Save("LMRS_TestFile.xml");

MessageBox.Show("Patient added successfully.", "Patient Added", MessageBoxButton.OK, MessageBoxImage.Information);

clearTextFields(); } } }

private bool textFieldsNotComplete() { return txtForename.Text != "" && txtSurname.Text != "" && txtAddress1.Text != "" && txtAddress2.Text != "" && txtAddress3.Text != "" && txtPostcode.Text != "" && txtPassword.Text != "" && txtConfirmPassword.Text != ""; }

private void clearTextFields() { txtForename.Text = txtSurname.Text = txtAddress1.Text = txtAddress2.Text = txtAddress3.Text = txtPostcode.Text = txtPassword.Text = txtConfirmPassword.Text = txtUsername.Text = ""; }

private void txtSurname_TextChanged(object sender, TextChangedEventArgs e) { if (txtForename.Text != "") { txtUsername.Text = "p_" + txtSurname.Text.ToLower() + txtForename.Text.ToUpper()[0]; }

of 130

} }}

of 130

LMRS_RemovePatient.xaml

<Page x:Class="LMRS_Main.LMRS_RemovePatient" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"

Title="LMRS_RemovePatient">

<Grid> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background> <Label Content="Select a Patient to Remove" Height="119" HorizontalAlignment="Center" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" Width="1342" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White"/>

<Label Content="Search for a user i.e. p__smithJ (case sensitive):" FontSize="32" Height="50" Margin="350.105,224.353,315.809,0" Name="lblInstructions" VerticalAlignment="Top" /> <Label Content="User:" FontSize="32" HorizontalAlignment="Left" Margin="455.851,280.084,0,0" Name="label1" Width="80" Height="52.563" VerticalAlignment="Top" /> <TextBox FontSize="32" Margin="541.591,280.084,474.428,0" Name="txtUserToRemove" Height="52.563" VerticalAlignment="Top" TextChanged="txtUserToRemove_TextChanged" />

<Button IsEnabled="False" Content="Search" Height="46" Margin="0,0,32,32" Name="btnSearch" VerticalAlignment="Bottom" HorizontalAlignment="Right" Width="149.921" Click="btnSearch_Click"/> <TextBlock TextAlignment="Center" FontSize="16" Height="128" Margin="325.812,0,290.087,135.755" Name="txtResults" VerticalAlignment="Bottom" />

<Button Content="Back (Main Menu)" Height="46" HorizontalAlignment="Left" Margin="32,661,0,0" Name="btnExit" VerticalAlignment="Top" Width="150" Click="btnBack_Click"/> <Button Content="Remove" Height="46" HorizontalAlignment="Right" IsEnabled="False" Margin="0,0,32,110.033" Name="btnRemove" VerticalAlignment="Bottom" Width="149.921" Click="btnRemove_Click"/> </Grid></Page>

LMRS_RemovePatient.xaml.cs

using System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;

of 130

using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;using System.Xml.Linq;

namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_RemovePatient.xaml /// </summary> public partial class LMRS_RemovePatient : Page { // The xml file XDocument lmrsXml; // User to remove (if found on file) XElement userOnRecord;

public LMRS_RemovePatient() { InitializeComponent();

// Set the window title this.WindowTitle = "Leap Motion Rehabilitation System - Remove Existing Patient"; }

private void btnBack_Click(object sender, RoutedEventArgs e) { this.NavigationService.Navigate(new LMRS_MainMenu_Clinician()); }

private void txtUserToRemove_TextChanged(object sender, TextChangedEventArgs e) { if (txtUserToRemove.Text.Equals("")) { btnSearch.IsEnabled = false; } else { btnSearch.IsEnabled = true; } }

private void btnSearch_Click(object sender, RoutedEventArgs e) { lmrsXml = XDocument.Load("LMRS_TestFile.xml");

// Find patient user XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patient = from el in patientCollection.Elements("patient") where

of 130

(string)el.Attribute("username") == txtUserToRemove.Text select el;

if (patient.Count() == 0) { MessageBox.Show("No user found with this username, please check the username and try again.", "User Not Found", MessageBoxButton.OK, MessageBoxImage.Error); txtUserToRemove.Text = ""; } else { userOnRecord = patient.First<XElement>();

txtResults.Text = "User found: \n\n" + userOnRecord.Element("forename").Value + " " + userOnRecord.Element("surname").Value + "\n" + userOnRecord.Element("addressLineOne").Value + "\n" + userOnRecord.Element("addressLineTwo").Value + "\n" + userOnRecord.Element("addressLineThree").Value + userOnRecord.Element("postCode").Value;

btnRemove.IsEnabled = true; } }

private void btnRemove_Click(object sender, RoutedEventArgs e) { if (MessageBox.Show("Removing this user will permanently delete all related exercise data, proceed?", "Confirm User Removal", MessageBoxButton.YesNo, MessageBoxImage.Question) == MessageBoxResult.Yes) { userOnRecord.RemoveAll(); userOnRecord.Remove(); lmrsXml.Save("LMRS_TestFile.xml");

MessageBox.Show("Patient removed successfully.", "Patient Removed", MessageBoxButton.OK, MessageBoxImage.Information); } txtUserToRemove.Text = txtResults.Text = ""; btnRemove.IsEnabled = false; } }}

of 130

LMRS_ViewExerciseResults.xaml

<Page x:Class="LMRS_Main.LMRS_ViewExerciseResults" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:chartingprimitives="clr-namespace:System.Windows.Controls.DataVisualization.Charting.Primitives;assembly=System.Windows.Controls.DataVisualization.Toolkit"

xmlns:datavis="clr-namespace:System.Windows.Controls.DataVisualization;assembly=System.Windows.Controls.DataVisualization.Toolkit"

xmlns:charting="clr-namespace:System.Windows.Controls.DataVisualization.Charting;assembly=System.Windows.Controls.DataVisualization.Toolkit"

mc:Ignorable="d" d:DesignHeight="768" d:DesignWidth="1366"

Title="LMRS_ViewExerciseResults" xmlns:chartingToolkit="clr-namespace:System.Windows.Controls.DataVisualization.Charting;assembly=System.Windows.Controls.DataVisualization.Toolkit" Width="1366">

<Grid> <Grid.ColumnDefinitions> <ColumnDefinition Width="528.73*" /> <ColumnDefinition Width="837.27*" /> </Grid.ColumnDefinitions> <Grid.Background> <ImageBrush ImageSource="Background_1366x768.png" /> </Grid.Background>

<Label Content="View Exercise Results" Height="119" Margin="12,12,12,0" Name="lblHeader" VerticalAlignment="Top" HorizontalContentAlignment="Center" VerticalContentAlignment="Center" FontSize="32" Foreground="White" Grid.ColumnSpan="2" />

<Button Content="Back (Main Menu)" Height="46" HorizontalAlignment="Left" Margin="32,661,0,0" Name="btnExit" VerticalAlignment="Top" Width="150" Click="btnBack_Click"/>

 <Label Content="Exercise:" Height="28" HorizontalAlignment="Left" Margin="32,185,0,0" Name="lblExercise" VerticalAlignment="Top" Width="150" /> <ComboBox Height="23" HorizontalAlignment="Left" Margin="30.009,218.637,0,0" Name="cmboExercise" VerticalAlignment="Top" Width="150" SelectionChanged="cmboExercise_SelectionChanged" />

<Label Content="StartDate:" Height="28" Margin="202.918,185,175.812,0" Name="lblStartDate" VerticalAlignment="Top" /> <ComboBox Height="23" Margin="202.918,218.637,175.812,0"

of 130

Name="cmboStartDate" VerticalAlignment="Top" SelectionChanged="cmboStartDate_SelectionChanged"/>

<Label Content="EndDate:" Height="28" HorizontalAlignment="Right" Margin="0,185,1.474,0" Name="lblEndDate" VerticalAlignment="Top" Width="150"/> <ComboBox Height="23" HorizontalAlignment="Right" Margin="0,218.637,1.474,0" Name="cmboEndDate" VerticalAlignment="Top" Width="150" SelectionChanged="cmboEndDate_SelectionChanged"/>

 <chartingToolkit:Chart Name="chrtBarChart" Margin="27.151,171.48,28.58,30.009" Grid.Column="1"> <chartingToolkit:Chart.Axes> <chartingToolkit:CategoryAxis Title="Date and Rep #" Orientation="Y" /> <chartingToolkit:LinearAxis Title="Time in Seconds" Orientation="X" Interval="0.1"/> </chartingToolkit:Chart.Axes>

 <chartingToolkit:BarSeries Title="barSeries" DependentValuePath="Value" IndependentValuePath="Key" ItemsSource="{Binding}" />

 <chartingToolkit:Chart.LegendStyle> <Style TargetType="datavis:Legend"> <Setter Property="Width" Value="0" /> </Style> </chartingToolkit:Chart.LegendStyle> </chartingToolkit:Chart>

 <chartingToolkit:Chart Name="chrtLineChart" Margin="27.151,171.48,28.58,30.009" Grid.Column="1" Visibility="Collapsed"> <chartingToolkit:Chart.Axes> <chartingToolkit:LinearAxis Title="Time in Seconds" Orientation="Y" Interval="0.1"/> <chartingToolkit:CategoryAxis Title="Date and Rep #" Orientation="X"/> </chartingToolkit:Chart.Axes>

 <chartingToolkit:LineSeries Title="lineSeries" DependentValuePath="Value" IndependentValuePath="Key" ItemsSource="{Binding}"/>


 <chartingToolkit:Chart Name="chrtColumnChart" Margin="27.151,171.48,28.58,30.009" Grid.Column="1"

of 130

Visibility="Collapsed"> <chartingToolkit:Chart.Axes> <chartingToolkit:LinearAxis Title="Time in Seconds" Orientation="Y" Interval="0.1"/> <chartingToolkit:CategoryAxis Title="Date and Rep #" Orientation="X"/> </chartingToolkit:Chart.Axes>

 <chartingToolkit:ColumnSeries Title="columnSeries" DependentValuePath="Value" IndependentValuePath="Key" ItemsSource="{Binding}"/>


 <GroupBox Header="Chart Type" Margin="32,0,1.474,223.845" Name="grpChartTypes" Height="99.675" VerticalAlignment="Bottom"></GroupBox> <Grid Height="76.715" Margin="32,0,13.474,223.845" VerticalAlignment="Bottom"> <RadioButton HorizontalAlignment="Left" Margin="44.299,32.867,0,27.151" Name="rdoBarChart" Width="120" IsChecked="True" Checked="rdoBarChart_Checked">Bar Chart</RadioButton> <RadioButton Margin="148.616,33.564,214.35,27.151" Name="rdoLineChart" IsChecked="False" Checked="rdoLineChart_Checked">Line Chart</RadioButton> <RadioButton HorizontalAlignment="Right" Margin="0,33.564,100.03,27.151" Name="rdoColumnChart" IsChecked="False" Checked="rdoColumnChart_Checked" Width="120">Column Chart</RadioButton> </Grid>

 <Label Visibility="Hidden" Height="28" HorizontalAlignment="Left" Margin="30.009,274,0,0" Name="lblPatient" VerticalAlignment="Top" Width="150">Patient:</Label> <ComboBox Visibility="Hidden" Height="23" HorizontalAlignment="Left" Margin="28.018,308,0,0" Name="cmboPatient" VerticalAlignment="Top" Width="150" SelectionChanged="cmboPatient_SelectionChanged"/> </Grid></Page>

LMRS_ViewExerciseResults.xaml.cs

using System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Windows;

of 130

using System.Windows.Controls;using System.Windows.Data;using System.Windows.Documents;using System.Windows.Input;using System.Windows.Media;using System.Windows.Media.Imaging;using System.Windows.Navigation;using System.Windows.Shapes;using System.IO;using System.Xml.Linq;

namespace LMRS_Main{ /// <summary> /// Interaction logic for LMRS_ViewExerciseResults.xaml /// </summary> public partial class LMRS_ViewExerciseResults : Page { // The xml file storing user/session data XDocument lmrsXml;

// Store information about specific exercise XElement chosenExercise; IEnumerable<XElement> sessions; IEnumerable<XElement> repTimes;

// Used to populate combo boxes List<String> exercises = new List<String>(); List<String> dates = new List<String>(); // Used to populate the conditional patient combo box List<String> patients = new List<String>();

// Currently selected patient, used to filter date data XElement selectedPatient;

// Used to populate the chart List<KeyValuePair<string, decimal>> chartDataList;

// Used to toggle patient controls and ensure proper navigation when back is cliced string userType;

public LMRS_ViewExerciseResults(string userType) { InitializeComponent();

// Set the window title WindowTitle = "Leap Motion Rehabilitation System - View Exercise Results";

// Load the XML file and populate the drop-downs lmrsXml = XDocument.Load("LMRS_TestFile.xml");

populatePatientComboBox(); populateExerciseComboBox();

this.userType = userType; if (userType.Equals("clinician")) { lblPatient.Visibility = Visibility.Visible; cmboPatient.Visibility = Visibility.Visible;

of 130

} else { lblPatient.Visibility = Visibility.Hidden; cmboPatient.Visibility = Visibility.Hidden; } }

private void btnBack_Click(object sender, RoutedEventArgs e) { if (this.userType.Equals("clinician")) { this.NavigationService.Navigate(new LMRS_MainMenu_Clinician()); } else { this.NavigationService.Navigate(new LMRS_MainMenu_Patient()); } }

// Start things off with the XML parsing by loading the patient combo box private void populatePatientComboBox() { // Now load the exercise combo box XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patients = (from el in patientCollection.Elements("patient") select el);

int i = 0; int j = 0; foreach (XElement el in patients) { this.patients.Add(patients.ElementAt(i).Element("forename").Value + " " + patients.ElementAt(i).Element("surname").Value);

if (patients.ElementAt(i).Attribute("username").Value == UserInfoHelper.userName) { j = i; break; }

++i; }

cmboPatient.ItemsSource = this.patients; cmboPatient.Tag = "ignoreCall"; cmboPatient.SelectedIndex = 0;

selectedPatient = patients.ElementAt(j); }

private void cmboPatient_SelectionChanged(object sender, SelectionChangedEventArgs e) { if (((string)cmboPatient.Tag).Equals("ignoreCall"))

of 130

{ cmboPatient.Tag = "acceptCall"; return; }

// Now load the exercise combo box XElement patientCollection = lmrsXml.Root.Element("patientCollection"); IEnumerable<XElement> patients = (from el in patientCollection.Elements("patient") select el);

int i = 0; foreach (XElement el in patients) { if (cmboPatient.SelectedItem.ToString().Contains(patients.ElementAt(i).Element("forename").Value) && cmboPatient.SelectedItem.ToString().Contains(patients.ElementAt(i).Element("surname").Value)) { selectedPatient = patients.ElementAt(i); break; } ++i; }

cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.Items.Refresh(); cmboEndDate.Items.Refresh(); cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.SelectedIndex = cmboEndDate.SelectedIndex = 0; cmboExercise.Tag = "ignoreCall"; cmboExercise.SelectedIndex = 0;

refreshChart(); }

// Load the exercise combo box private void populateExerciseComboBox() { // Now load the exercise combo box IEnumerable<XElement> exercises = (from el in selectedPatient.Elements("activity") select el); this.exercises.Add(exercises.ElementAt(0).Attribute("name").Value.ToString().Replace("_", " ")); this.exercises.Add(exercises.ElementAt(1).Attribute("name").Value.ToString().Replace("_", " ")); this.exercises.Add(exercises.ElementAt(2).Attribute("name").Value.ToString().Replace("_", " "));

cmboExercise.ItemsSource = this.exercises; cmboStartDate.ItemsSource = cmboEndDate.ItemsSource =

of 130

dates; chartDataList = new List<KeyValuePair<string, decimal>>(); }

// Retrieves all sessions for the selected exercise private void cmboExercise_SelectionChanged(object sender, SelectionChangedEventArgs e) { // Load available sessions for this exercise, first find the chosen exercise in the file String name = cmboExercise.SelectedItem.ToString().Replace(" ", "_"); IEnumerable<XElement> exercise = from el in selectedPatient.Elements("activity") where (string)el.Attribute("name") == name select el;

chosenExercise = exercise.First<XElement>();

// Now get the sessions and rep times sessions = from el in chosenExercise.Elements("session") select el; repTimes = from el in sessions.Elements("rep") select el;

dates.Clear(); int i = 0; foreach (XElement exl in sessions) { dates.Add(exl.Attribute("date").Value); ++i; }

// Tag: arbitrary object value that can be used to store custom information about this element // Use it here to avoid triggering unwanted calls to refreshChart cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.Items.Refresh(); cmboEndDate.Items.Refresh(); cmboStartDate.Tag = cmboEndDate.Tag = "ignoreCall"; cmboStartDate.SelectedIndex = cmboEndDate.SelectedIndex = 0;

refreshChart(); }

private void cmboStartDate_SelectionChanged(object sender, SelectionChangedEventArgs e) { if (((string)cmboStartDate.Tag).Equals("ignoreCall")) { cmboStartDate.Tag = "acceptCall"; return; }

// End date cannot be before the start date if (cmboEndDate.SelectedIndex < cmboStartDate.SelectedIndex) { cmboEndDate.SelectedIndex =

of 130

cmboStartDate.SelectedIndex; } refreshChart(); }

private void cmboEndDate_SelectionChanged(object sender, SelectionChangedEventArgs e) { if (((string)cmboEndDate.Tag).Equals("ignoreCall")) { cmboEndDate.Tag = "acceptCall"; return; }

// End date cannot be before the start date if (cmboEndDate.SelectedIndex < cmboStartDate.SelectedIndex) { cmboEndDate.SelectedIndex = cmboStartDate.SelectedIndex; } refreshChart(); }

// Changing the dates via the combo boxes modifies the dataset used by the chart. private void refreshChart() { if (sessions.Count() != 0) { chartDataList.Clear();

int repOffset = (cmboStartDate.SelectedIndex * 5) < 0 ? 0 : (cmboStartDate.SelectedIndex * 5); for (int i = 0; i < ((cmboEndDate.SelectedIndex - cmboStartDate.SelectedIndex) + 1) * 5; ++i) { chartDataList.Add(new KeyValuePair<string, decimal>(sessions.ElementAt(cmboStartDate.SelectedIndex + (i / 5)).Attribute("date").Value + ": Rep " + ((i % 5) + 1).ToString() + " ", Convert.ToDecimal(repTimes.ElementAt<XElement>(repOffset + i).Value))); } chrtBarChart.DataContext = chrtLineChart.DataContext = chrtColumnChart.DataContext = null; chrtBarChart.DataContext = chrtLineChart.DataContext = chrtColumnChart.DataContext = chartDataList;

// Update page title lblHeader.Content = "Results for " + exercises[cmboExercise.SelectedIndex] + " (" + cmboStartDate.SelectedValue + " - " + cmboEndDate.SelectedValue + ")"; } else { chrtBarChart.DataContext = chrtLineChart.DataContext = chrtColumnChart.DataContext = null; } }

of 130

private void rdoBarChart_Checked(object sender, RoutedEventArgs e) { chrtBarChart.Visibility = Visibility.Visible; chrtLineChart.Visibility = chrtColumnChart.Visibility = Visibility.Collapsed; }

private void rdoLineChart_Checked(object sender, RoutedEventArgs e) { chrtLineChart.Visibility = Visibility.Visible; chrtBarChart.Visibility = chrtColumnChart.Visibility = Visibility.Collapsed; }

private void rdoColumnChart_Checked(object sender, RoutedEventArgs e) { chrtColumnChart.Visibility = Visibility.Visible; chrtBarChart.Visibility = chrtLineChart.Visibility = Visibility.Collapsed; } }}

of 130

UserInfoHelper.cs

using System;using System.Collections.Generic;using System.Linq;using System.Text;using System.Xml.Linq;using System.Threading;

namespace LMRS_Main{ static class UserInfoHelper { static public XElement user; static public string foreName; static public string userName; }}

of 130