Prince Sultan University College of Computer & Information Sciences
Department of Software Engineering
A Software Quality Model for Evaluating Medical Simulation Tools
Prepared By: Norah Naif Al-Romi
Under Supervision of Dr. Areej Al-Wabil
June 2015
Submitted in partial fulfilment of the requirements for the Degree of Master in Software Engineering at the Department of Software Engineering at the College of Computer and
Information Sciences
A Decision Support System for Evaluating
Medical Simulation Tools
By Norah Naif Al-Romi
This thesis was defended and approved on June 3, 2015
Supervisor: Dr. Areej Al-Wabil
Members of the Exam Committee
Dr. Areej Al-Wabil Chair
Dr. Nor Shahriza Abdulkarim Examiner
Dr. Iman Al-Momani Examiner
I
ACKNOWLEDGEMENT
First and foremost I want to express my highest gratitude to Allah almighty for providing
me guidance, ability, knowledge and patience in completing this thesis. I wish to thank my
supervisor, Dr. Areej AlWabil, Vice Dean of Academic Affairs at Prince Sultan University,
you have been a tremendous mentor for me. I would like to thank you for encouraging the
research and for allowing me to grow as a research scientist. Your advice on both research
as well as on my career have been priceless. I would also like to thank my committee
members, professor Nor Shahriza Abdulkarim, Dr. Iman AlMomani for serving as my
committee members even at hardship. I would especially like to thank the physicians,
quality managers and medical student for helping me collect data for my thesis. A special
thanks to my family. Words cannot express how grateful I am to my mother, father, and my
sister for all of the sacrifices that you’ve made on my behalf. I have to express my sincere
appreciation and gratitude for their support and understanding during my master journey.
At the end I would like express appreciation to my beloved husband Dr. Khaled AlRabiah
for his continuous support.
II
ABSTRACT This thesis describes a software quality model designed for assisting decision makers in
evaluating medical simulation systems. Decision-making in medical domains is an
increasingly complex task that involves a number of stakeholders, subspecialties and
technologies. Medical simulation tools create a lifelike situation for individuals to examine
the impact of decisions and changes in procedural activities in clinical and healthcare
contexts. They provide a relatively safe context for the patients and professionals as it
involves simulated human patients, scenarios of emergency response, and flow in
procedural activities (e.g. laparoscopic surgery). Evidence suggests that medical simulation
systems have the potential to improve the effectiveness, safety, and efficiency in health care
services. Moreover, it has been shown to consistently deliver significant value to the
organization, staff, or trainees in decision-making. Although medical simulation provided
ideal approaches for addressing healthcare issues, the number of successful software
implementation and development is inadequate since it is relatively small compared with
other established engineering disciplines, such as industrial and mechanical engineering.
Medical simulation issues have been highlighted in the literature, such as usability and
scalability. Considering a software quality model, which includes quality assurance factors,
specific to the medical simulation context is foreseen as an approach to augment current
practice in the design and development of such systems. This decision support system
works as an interactive tool and is designed to help in examining the effectiveness of
medical simulation tools, for specific contexts of use, before procurement for hospitals,
training centers, or education purposes. A systematic review of the literature on medical
simulation tools is conducted with a focus on quality assurance factors such as the ones
applied in the Mccall, Boehm, ISO 25022 and ISO 9126.
The contributions of this thesis are the conceptual design of the proposed decision support
system (DSS) and user acceptance testing of the developed system. The DSS is designed to
process the software quality metrics and conduct a comparative evaluation with other tools
in the domain and provides visualizations that assist in decision-making and quality
assessment. In addition, the software quality model that is designed for evaluating medical
simulation systems is introduced which aggregates the relevant quality assurance factors for
III
the context of medical simulation from previously developed software quality models.
Another contribution is the development of the medical simulation decisions support
system (DSS) which integrates the software quality model in a web-based system. A case
study is presented to examine the usability of the DSS in iterative user-centered design
cycles and in the user-acceptance testing of the system on a sample of medical simulation
tools.
IV
ABSTRACT IN ARABIC
ملخص االرسالة
في مجالل االمحاكاةة االطبيیة ووتطويیر أأنظمة تفاعليیة جوددةة االبرمجيیاتت االمستخدمة إإلى ددررااسة تقيیيیمهھھھذهه ااالططرووحة تهھدفف
ووذذلك لموااكبة االمستجدااتت في برمجيیاتت االمحاكاةة االطبيیة وو مجالل االرعايیة االصحيیة مساعدةة أأصحابب االقراارر فيمصممة ل
من حيیث آآليیاتت االتفاعل االمرئي وواالحسي وواالصوتي وو برمجيیاتت نظمةتقيیيیم أأنظمة االمحاكاةة باعتباررهھھھا من أأعقد ااألمعايیيیر
. ووااجهھاتت ااالستخداامم
نظراا يید ووذذلك لمعقدةة بشكل متزاا امم اا االمهھ لقراارر في االمجاالتت االطبيیة يیعد من االجهھاتت االمعنيیة باختيیارر ووااستخداامم لتعدددااتخاذذ اا
وواالمجاالتت االعلميیةاالتخصصاتت االدقيیقة متلباطط االرعايیة االصحيیة ووتعددد ووتقيیيیم جوددةة ااألنظمة ووااررتباطط االمنظومة االتشغيیليیة ب -
الفراادد . أأددووااتت االمحاكاةة االطبيیة تخلق حالة تالمس االوااقع وواالتي تهھدفف االي مساعدةة ااقنيیاتتاالتاالمستجدااتت في مجالل وواالبيینيیة
لدررااسة أأثر االقرااررااتت وواالتغيیرااتت في ااألنشطة ااإلجراائيیة االسريیريیة وواالرعايیة االصحيیة. هھھھذهه ااالجهھزةة آآمنهھ نسبيیا للمرضى
وواالمهھنيیيین ألنهھا تحاكي االمرضى من االبشرمن خاللل سيینارريیوهھھھاتت ااالستجابة لحاالتت االطوااررئئ٬، وواالتدفق في ااألنشطة
لدالئل إإلى أأنن أأنظمة االمحاكاةة االطبيیة لديیهھا االقدررةة على تحسيین االفعاليیة ااإلجراائيیة (مثل االجرااحة االتنظيیريیة). تشيیر اا
قدمم ذذلك٬، فقد تبيین أأنهھا ت االصحيیة. ووعالووةة على ة االرعايی فاءةة في خدماتت لك لسالمة وواا أأوو االمنشأةة كبيیرةة للمنظمة منفعةوواا
عمليیة صنع االقراارر. مجالل االرعايیة االصحيیة أأوو ااإلددااررةة االطبيیة وو ٬، وواالموظظفيین٬، أأوو االمتدرربيین في االطبيیة
اجحة لطبيیة االمقدمة تعد ن اةة اا ااجهھزةة االمحاك من اانن لرعايیة االصحيیة٬، ااال ااننفي مجالل االتدرريیب ووتنميیة االمهھاررااتت لعلى االرغم
إإلى أأنن آآليیاتت تقيیيیم االجوددةة لهھذهه ااألنظمة لم يیتوصل االباحثونن إإلى توحيیدهھھھا بما يیتوااكب مع االتطوررااتت ااألبحاثث االحديیثة تشيیر
مقاررنة مع غيیرهھھھا من االتخصصاتت ااالخرىى االمعمولل بهھا٬، مثل االهھندسة االصناعيیة وواالميیكانيیكيیة. ووقد تم تقنيفي االمجالل اال
ل االمحاكاةة االطبيیة مثفي مجالل ددررااسة االتحديیاتت االتي يیوااجهھهھا االمستخدميینلى عفي هھھھذهه االرسالة االعلميیة تسليیط االضوء
االبرمجيیاتت٬، يیتضمن عواامل ضمانن االجوددةة٬، خاصة الجهھزةة هھيیكل جوددةة تقدمم هھھھذهه االرسالة مقترحح لااالستخداامم. صعوبة
االمحاكاةة مصمم من مقرحاتت تقيیيیم جوددةة االبرمجيیاتت ااألساسي مع االتركيیز على االعواامل االمتخصصة بمجالل أأنظمة االطبيیة
يیستهھدفف تصميیم ووتقيیيیم نظامم لضمن منهھجيیة االدررااسة لهھذهه االرسالة االعلمي . تم تصميیم االهھيیكليیة االتفاعليیة االمحاكاةة االطبيیة
في ددررااسة فعاليیة أأددووااتت االمحاكاةة االطبيیة٬، لسيیاقاتت محدددةة االمختصيین ووااإلدداارريیيین لمساعدةة ااإلددااررةة االطبيیة أأوو االصحيیة
يیقومونن بتقيیيیم ااجهھزةة االمحاكاةة االطبيیهھ قبل ااعتماددهھھھا أأوو شراائهھا للمستشفيیاتت٬، وومرااكز االقرااررااالستخداامم حيیث اانن ااصحابب
االتدرريیب٬، أأوو ألغرااضض االتعليیم.
االبحث تمت مرااجعة ما تم ددررااستهھ مسبقا فيیما يیخص أأددووااتت االمحاكاةة االطبيیة مع االتركيیز على عواامل ضمانن في هھھھذاا
نظامم تقيیيیم ااجهھزةة . ووعالووةة على ذذلك٬، فإننا نقدمم االتصميیم االنظريي من ISOاالجوددةة مثل تلك االمطبقة في ماكولل٬، بوهھھھم٬، وو
االمقاررناتت مع أأددووااتت أأخرىى في االمجالل وويیقدمم االتصوررااتت االذيي يیعالج في ااالستداللل للتقيیيیم ووإإجرااء االمحاكاةة االطبيیة
V
االالززمة االتي تساعد في ااتخاذذ االقرااررااتت ووتقيیيیم االجوددةة. إإسهھاماتت هھھھذهه ااألططرووحة هھھھي عباررةة عن هھھھيیكلة جوددةة االبرمجيیاتت
. لتقيیيیم جوددةة أأجهھزةة االمحاكاةة االطبيیةاالمستخدمة لتقيیيیم أأنظمة االمحاكاةة االطبيیة. ووباإلضافة إإلى ذذلك٬، نظامم خاصص
VI
Table of Contents ACKNOWLEDGEMENT .......................................................................................................................... I
ABSTRACT .................................................................................................................................................. II
ABSTRACT IN ARABIC ........................................................................................................................ IV
LIST OF TABLES ................................................................................................................................. VIII
LIST OF FIGURES ................................................................................................................................... IX
LIST OF ABBREVIATIONS ................................................................................................................... X
CHAPTER ONE: INTRODUCTION ..................................................................................................... 1 1.1 MEDICAL SIMULATION TOOLS ................................................................................................................. 2 1.2 PROBLEM DEFINITION ................................................................................................................................. 4 1.3 RESEARCH SCOPE ......................................................................................................................................... 5 1.4 AIMS AND OBJECTIVES ............................................................................................................................... 6 1.5 RESEARCH QUESTIONS ............................................................................................................................... 6 1.6 METHODOLOGY ............................................................................................................................................ 7
1.6.1 Work Phase One: Data Collection .................................................................................................. 8 1.6.2 Work Phase Two. Model Designing ................................................................................................ 8 1.6.3 Work Phase Three. Model Developing .......................................................................................... 8
1.7 CONTRIBUTION .............................................................................................................................................. 9 1.8 THESIS STRUCTURE ..................................................................................................................................... 9
CHAPTER TWO: LITERATURE REVIEW .................................................................................... 10 2.1 MEDICAL SIMULATION ............................................................................................................................ 11
2.1.1 Medical Simulation Categorizations ............................................................................................ 12 2.1.2 Medical Simulation Challenges ..................................................................................................... 13 2.1.3 Human Factors in Medical Simulation ....................................................................................... 15 2.1.4 Medical Simulation Types ................................................................................................................ 17
2.2 DECISION SUPPORT SYSTEMS (DSSS) ................................................................................................ 18 2.2.1 Data-driven ............................................................................................................................................ 18 2.2.2 Knowledge-based ................................................................................................................................. 18 2.2.3 Document-driven ................................................................................................................................. 19 2.2.4 Communication-driven ...................................................................................................................... 19 2.2.5 Model-driven ......................................................................................................................................... 19
2.3 SOFTWARE QUALITY MODELS .............................................................................................................. 22 2.3.1 McCall’s Quality Model ................................................................................................................... 23 2.3.2 Boehm’s Quality Model ..................................................................................................................... 23 2.3.3 FURPS Quality Model ....................................................................................................................... 24 2.3.4 Dromey's Quality Model ................................................................................................................... 25 2.3.5 ISO 9126 Quality Model ................................................................................................................... 26 2.3.6 ISO/IEC 25000 SQuaRE ................................................................................................................... 27 2.3.7 Comparison between Software Quality Models and Quality Measures ......................... 29
CHAPTER THREE: SOFTWARE QUALITY MODEL FOR MEDICAL SIMULATION TOOLS ........................................................................................................................................................ 31
3.1 USER CENTERED DESIGN ............................................................................................................................... 32
VII
3.2 METRICS AND MEASUREMENTS IN THE QUALITY MODEL ................................................................. 37 3.2.1 Effectiveness Measures ...................................................................................................................... 37 3.2.2 Efficiency Measures ............................................................................................................................ 38 3.2.3 Satisfaction Measures ........................................................................................................................ 39 3.2.4 Freedom from Risk Measures ......................................................................................................... 40 3.2.5 Context Coverage Measures ........................................................................................................... 43
CHAPTER FOUR: MEDICAL SIMULATION DSS ....................................................................... 45 4.1 CONCEPTUAL DESIGN OF THE DSS .......................................................................................................... 46 4.2 PERSONAS AND SCENARIOS ......................................................................................................................... 47 4.3 SYSTEM VISUALIZATION ............................................................................................................................. 48
4.3.1 First Cycle .............................................................................................................................................. 48 4.3.2 Second Cycle ......................................................................................................................................... 49
4.4 CHAPTER SUMMARY ...................................................................................................................................... 50
CHAPTER FIVE: INTERACTIVE DECISION MAKING IN ASSESSING THE QUALITY OF MEDICAL SIMULATION TOOLS .............................................................................................. 51
5.1 SCENARIO ONE: EVALUATE A MEDICAL SIMULATION TOOL .................................................... 52 5.2 SCENARIO TWO: MEDICAL SIMULATION TOOLS REVIEW BASED ON THE QUALITY ASSURANCE FACTORS ............................................................................................................................................ 53 5.3 THE NEED FOR A SOFTWARE QUALITY MODEL FOR EVALUATING MEDICAL SIMULATION TOOLS 57 5.4 AIM AND SCOPE OF SOFTWARE QUALITY MODEL STUDY ........................................................... 57 5.5 EXPLORATORY STUDY ............................................................................................................................. 57
5.5.1 Participants ............................................................................................................................................ 57 5.5.2 Apparatus and Materials .................................................................................................................. 59 5.5.3 Test Material ......................................................................................................................................... 59 5.5.4 Task Scenarios ...................................................................................................................................... 59 5.5.5 Task List .................................................................................................................................................. 59
5.6 OBJECTIVE MEASURES OF SATISFACTION ........................................................................................ 60 5.7 SUBJECT MATTER EXPERTS REVIEWS ................................................................................................ 63 5.8 CHAPTER SUMMARY ................................................................................................................................ 64
CHAPTER SIX: CONCLUSION .......................................................................................................... 65 6.1 RESEARCH QUESTIONS ................................................................................................................................. 66 6.2 FUTURE WORK ................................................................................................................................................ 67 6.3 PUBLICATIONS ................................................................................................................................................. 67 6.4 REFERENCES .................................................................................................................................................... 68
APPENDIX A: CONSENT FORM FOR PARTICIPATION IN A RESEARCH STUDY ...... 74
APPENDIX B: SYSTEM USABILITY SCALE ................................................................................. 76
APPENDIX C: PERSONAS ................................................................................................................... 77
APPENDIX D: DSS DATA DICTIONARY ........................................................................................ 78
VIII
LIST OF TABLES
Table 1 Implementing DSSs and Examples from Previous Research ................................................... 20 Table 2 A Synthesis of DSS Categorizations in Medical Simulation ...................................................... 21 Table 3 Square Divisions ......................................................................................................................................... 28 Table 4 Quality Factors and Their Availability in Each Model ................................................................ 29 Table 5 UCD Participants ......................................................................................................................................... 34 Table 6 UCD Summarization and Results ......................................................................................................... 34 Table 7 Effectiveness measures ........................................................................................................................... 37 Table 8 Efficiency Measures .................................................................................................................................. 38 Table 9 Usefulness Measures ................................................................................................................................ 39 Table 10 Risk Mitigation Measures .................................................................................................................... 40 Table 11 Financial Measures ................................................................................................................................ 41 Table 12 Economic Measures ................................................................................................................................ 42 Table 13 Health And Safety ................................................................................................................................... 42 Table 14 Environmental Measures ..................................................................................................................... 43 Table 15 Context Completeness .......................................................................................................................... 43 Table 16 Flexibility Measures ............................................................................................................................... 44 Table 17 Participants Technical Experience ................................................................................................... 58 Table 18 Participants Demographic Information ......................................................................................... 58 Table 19 Task List ....................................................................................................................................................... 59 Table 20 Subject Matter Experts Review .......................................................................................................... 63
IX
LIST OF FIGURES
Figure 1 Comparison between the usage of medical simulation in Medical School Learning and the Hospital ..................................................................................................................................................................... 4 Figure 2 Research Design Methodologies ........................................................................................................... 7 Figure 3 Mapping Medical Simulation Categories ........................................................................................ 13 Figure 4 Surgical Simulations ............................................................................................................................... 17 Figure 5 Cosmetic and Plastic Surgery .............................................................................................................. 17 Figure 6 Dental Simulation .................................................................................................................................... 18 Figure 7 McCall Quality Model ............................................................................................................................. 23 Figure 8 Boehm Quality Model ............................................................................................................................. 24 Figure 9 FURPS Quality Model ............................................................................................................................. 25 Figure 10 Doremy's Quality Model ..................................................................................................................... 26 Figure 11 ISO 9126 Quality Model ..................................................................................................................... 27 Figure 12 ISO/IEC 25000 SQuaRE ...................................................................................................................... 28 Figure 13 User Centered Design ........................................................................................................................... 32 Figure 14 Medical Simulation Software Quality Models ........................................................................... 36 Figure 15 Conceptual Designs for DSS ............................................................................................................... 46 Figure 16 Persona 1 ................................................................................................................................................... 48 Figure 17 Persona 2 ................................................................................................................................................... 48 Figure 18 System Visualization First Cycle ...................................................................................................... 49 Figure 19 Visualization -‐ Iteration 2 ................................................................................................................... 50 Figure 20 Home Page ................................................................................................................................................ 52 Figure 21 Software Quality Model Embedded In the DSS ......................................................................... 53 Figure 22 Medical Simulation Tool Review ..................................................................................................... 54 Figure 23 Efficiency Measure ................................................................................................................................ 54 Figure 24 Medical Simulation Tool Measurement ........................................................................................ 55 Figure 25 Freedom Form Risk Measure ............................................................................................................ 55 Figure 26 View Results as a Chart ....................................................................................................................... 56 Figure 27 SUS Scores ................................................................................................................................................. 61 Figure 28 Relationship between Technical and Medical Simulation ................................................... 62 Figure 29 SUS Average Score for All Participants ......................................................................................... 63 Figure 30 Medical Simulation DSS ....................................................................................................................... 66
X
LIST OF ABBREVIATIONS
AAMC Association of American Medical
Colleges
BSC Balanced Score Card
CPI Collaborative, Participative and
Interactive
DSS Decision Support System
IV&V Independent Verification and
Validation
OR Operating Room
QA Quality Assurance
QME Quality Measure Elements
ROI Return On Investment
SDLC Software Development Life Cycle
SME Subject Matter Expert
SQuaRE Systems And Software Engineering
- Systems And Software Quality
Requirements And Evaluation
SUS System Usability Scale
TORS Transoral Robotic Surgery
UCD User Centred Design
1
Chapter One: Introduction
2
1.1 Medical Simulation Tools The proliferation of medical simulation has recently led to a surge in research investigating
complexity in data analysis and in the evaluation of tools for training and decision making
in different healthcare contexts [1]. Decision-making in medical domains is an increasingly
complex task that involves a number of stakeholders, sub-specialities and technologies.
Simulation is defined as the process of designing a model of a real system [2]. The model
often represents the operation or act of the selected system or process. Experiments are
often conducted using the model for the purpose either of evaluating various strategies for
the operation of the system over time or for understanding the system’s behaviour [2], [3].
Simulation can be classified in different ways depending on the interaction level of users
[4], the context of decision making, and characteristic of modeling [4]. For example,
simulation based on interaction levels of users includes active, passive, and cooperative;
whereas classification based on modeling techniques include three types [4]:
• Live Simulation: Involves real people and equipment performing an activity that
operates for real.
• Virtual Simulation: Involves real people that operate on simulated (computer
controlled) equipment in a virtual environment.
• Constructive Simulation: Involves all the elements including people, equipment
and environment are simulated.
In the past two decades, simulation and modeling have continued to become one of the
leading experimentation techniques, especially for problems associated with uncertainty
[5]. Simulation has been used in assisting decision making in many fields such as education
[6], management [7], and healthcare [8]. Nowadays, a rapidly growing interest has been
emerging in healthcare modeling and simulation in order to improve patient safety and
patient care [9].
Medical simulation is the utilization of technology related to education, training, and
management [9]. It creates a lifelike situation for individuals to practice decision-making
3
and procedural activities in a safe environment for the patients and professionals where it
involves simulated human patients, emergency response and simulated animation.
Evidence suggests that medical simulation improves the effectiveness, safety, and
efficiency in health care services [10][11]. Moreover, it consistently delivers significant
value to the organization, staff, or medical students involved in decision-making [11].
Medical simulation has been reported in the literature as categorizations in different ways.
For example, complexity of visualization, platforms, and contexts of use [12]. One way
which I followed, is categorized into:
● Clinical and Training Simulation: Is a training technique for physicians, medical
students, nurses, and other healthcare professionals, which is used to study, and
analyse the behaviour of diseases, including biological processes in human body
[3]. Example of this category include haptic device (e.g. robotic arm or endoscope
simulation) [13][14][15].
● Operational Simulation: Is a technique for modeling a process and is used for
capturing, and analysing health care operations, patient flow, service delivery, and
scheduling, healthcare business and optimization design [3]. Example of this
category includes the patient flow at the emergency department at a hospital [16].
● Managerial Simulation: Is a type of interactive training and feedback and is used
as a tool for managerial purposes, decision-making, strategic planning, and policy
implementation. Example of this category includes comprehensive management
planning for healthcare processes, staffing, equipment and buildings [17].
● Educational Simulation: Is training and educational techniques, where virtual and
physical objects are extensively used to help a learner explore, navigate or obtain
more information about the environment [3]. Example of this category include
haptic device (e.g. robotic arm or endoscope simulation) [14][15].
Brailsford (2007) classifies medical simulation models into three groups [18] which are;
Models of the human body that include biological processes, Models for modeling patient
flow in the clinic, ward, department, or hospital, and Models for strategies that are used for
strategic planning of the organization.
4
Notably, hospitals and medical schools increasingly relying on medical simulation systems
and tools in different specialties [1]. In
Figure 1 a survey was conducted by the Association of American Medical Colleges
(AAMC) to show uses of medical simulation.
Figure 1 Comparison Between The Usage Of Medical Simulation In Medical School Learning And The Hospital [19]
1.2 Problem Definition Although medical simulation provided ideal approaches for addressing healthcare issues,
the number of successful implementation and development is relatively small compared
with the manufacturing industry [18]. Modern hospitals and clinics are encountering high
levels of competition in both domestic and global markets. In addition, patients, physicians,
medical students and trainees are increasingly demanding for more quality in health care
services delivered with a reasonable cost. Notably, research in [20] has suggested that
medical simulation models needs to be structured in a way that it optimizes the safety and
quality of health care systems [20]. On the other hand, implementing qualified simulation
5
systems may increase the complexity of system and negatively influences the efficiency of
these systems. This consequently led to the emergence of software solutions that focused
more on simulation for basic science education and skill development when compared to
simulation for clinical training [21].
Issues prevalent in the literature that have been cited as challenges in modelling and
training in healthcare setting, which are relevant for medical simulation systems include:
● Complexity and multiple interactions associated with healthcare systems [22].
● High cost of simulation tools [4].
● Medical errors [23].
● Lack of reliable data and relevant tools [24].
● Usability problems and lack of a user-friendly interface [25].
The proposed solution for such issues is applying a software quality model for decision
support systems (DSSs) to evaluate the software quality of medical simulation tools based
on quality assurance factors (e.g. usability, accuracy, efficiency, performance, robustness,
and acceptance) [2]. Healthcare providers, physicians, medical students, trainees and other
healthcare professionals can use and benefit from such system especially in early
stages. Moreover, the selected tool went through a process for evaluation embedded in the
model and then, it will be linked with other tools in the domain and visualized in order to
assist in the decision making process. It is envisaged to help in investigating the
effectiveness of medical simulation tools before buying or approving them for hospitals,
training centres, or education purposes.
1.3 Research Scope This research investigates, develops, implements and tests a software quality model for a
DSS designed for evaluating medical simulation tools based on quality assurance factors.
The proposed software quality model assists decision makers in evaluating medical
simulation tools by embedding the quality factors into the system. Modeling for decision-
making development is an effective method that assists in investigating, testing or
understanding complex simulation tools. So, one of the main thrusts of this research is to
6
evaluate and improve these simulation tools especially in early stages. Developing such a
software quality model can be used as a pre-process, since it provides health care providers
the ability to evaluate the tool before approving it. Stakeholders were involved in the
iterative design of the DSS to assess the software quality model's appropriateness for the
medical simulation tools [26]. To measure and evaluate the proposed DSS, tests were
performed on a representative sample of medical simulation tools in the clinical and
training simulation domain (e.g. surgery, dermatology) and evaluated based on the model
that is embedded in the DSS.
1.4 Aims and Objectives • The main aim of this study is to gain an insight on medical simulations tools, their
challenges, and best practices to overcome these challenges, DSSs, and previous
software quality models.
• Propose a software quality model for the DSS in the context of medical simulation.
• Design and develop a DSS for evaluating medical simulation tools by applying
quality assurance factors and measures.
• Tool analysis on the medical simulation tools through a case study.
1.5 Research Questions How can software systems assist decision makers in assessing the quality of medical
simulation tools?
Subsidiary research questions:
1. Why do professionals need to evaluate medical simulation tools?
2. What are the DSSs used in medical domains?
3. What are the software development models used to design DSS in medical
simulation domains?
4. How can we test the usability, performance, robustness, and accessibility of the
medical simulation tools?
5. What are the widely accepted Software quality models used among Software
Engineers?
7
1.6 Methodology The research methodology was designed to adopt an experimental design strategy [27] and
the methodology used in collecting and analysing the data is a mixed approach (quantitative
and qualitative). The approach involved a User Centred Design (UCD) methodology. This
methodology is to ensure efficient, satisfying, and user-friendly experiences for the user
[28] because the product will be designed and developed based on the perspective of how it
will be understood and used by the stakeholder [29]. The flow of the research design
methodology is illustrated in Figure 2.
Figure 2 Research Design Methodologies
To elaborate more on the methodology lifecycle, each work phase is described in detail in
the following sections.
8
1.6.1 Work Phase One: Data Collection In this research, the data was collected through a mixed approach which combined
quantitative and qualitative data. Two methods were applied for data collection:
1. Literature Review: A systematic literature review of published work [30] in
medical simulation and software quality models computing has been applied to
help in understanding the research directions, best practices and collect the
required data for designing and developing an embedded medical simulation
software quality model in a decision support system (DSS).
2. User Centred Design (UCD): The ISO standard for the UCD methodology
has been applied in this research to ensure that the end users' needs and
limitation are taken into consideration [31]. Moreover, this participatory design
methodology facilitates the creation of efficient, satisfying, and user-friendly
systems because the product was designed and developed based on the
perspective of how it will be understood and used by the stakeholders [29].
This phase is described in details in chapter 2 and 3.
1.6.2 Work Phase Two. Designing the Model This phase involved a knowledge background that was gained after applying the
systematic literature review and user centred design methodology. This background
knowledge and insights into design considerations for simulation contexts in
general and medical domains in particular, guided the design and development of
the quality model for medical simulation software. This quality model adhered to
the International Organization for Standardization (ISO) standards. The ISO
standard selected to be followed is Systems and Software Engineering - Systems
and software Quality Requirements and Evaluation (SQuaRE) [32]. This phase is
described in details in chapter 3.
1.6.3 Work Phase Three. Developing the Model After designing the medical simulation software quality model, the DSS was
developed and the quality model was embedded in the system with the layers of
interfaces to provide intuitive interaction. The medical simulation software quality
9
model has been embedded in the system for the purpose of assisting decision
makers in evaluating medical simulation tools. The Software Development Life
Cycle SDLC guided this phase where requirements, analysis, design and
development adopted. This phase involved the creation of personas, to enhance
engagement and build empathy in the design process by gaining a clear
understanding of the DSS and it is targeted users [33]. Afterwards, three iterative
cycles of system visualization have been done. This phase has been described in
details in chapter 4.
1.7 Contribution In this thesis, the contribution of embedding a software quality model in a DSS for
evaluating medical simulation tools was described as it evolved in iterative cycles of
development. In addition, the design approach of UCD in developing the DSS and the
participatory design activities in assessing the software quality model with healthcare
decision makers can be emulated in similar contexts of software design and development.
1.8 Thesis Structure Chapter 2 is a review of literature related to medical simulation categorization, challenges,
human factors, and tools, in addition, to software quality models, and DSS. In Chapter 3,
we discuss and present Phase 2 that covers the medical simulation software quality model.
Chapter 4 “describes Phase 3, the DSS used for evaluating medical simulation tools. In
Chapter 5, a case study has been applied on the DSS including the software quality model.
We conclude the thesis in Chapter 6 by presenting how all the objectives and research
questions have been achieved during the work in this thesis, future work and publications.
10
Chapter Two: Literature Review
11
Quality models of medical simulation are an emerging field of research in software
engineering [34]. The design and development of DSSs often involves a good
understanding of the functional requirements of the systems in addition to the best practices
and standards in the domains relevant to the applied context [35]. However, the knowledge
that underlies the quality models of DSSs, either knowledge-based or model-based, is
unfortunately scattered throughout the literature. This chapter presents a systematic
literature review of published work in medical simulation and software quality models
computing in a way that helps software engineers in understanding the research directions
and best practices for designing and developing a DSS for medical simulation tools.
In this chapter, we start by presenting an overview on medical simulation and it is
challenges and DSSs. Following that, a review of software quality models and standards
examines the quality assurance factors used to assess DSS. We conclude with a comparison
between previous software quality models, and design recommendations for software
quality models in the context of medical simulation.
2.1 Medical Simulation Medical simulation is the utilization of technology related to education, training, and
management in medical contexts [10][20]. Simulation creates a lifelike situation for
individuals to practice decision-making and procedural activities in a safe environment for
the patients and professionals. For example, simulation can provide scenarios in which it
involves simulated human patients in clinical procedures or surgery [14]. Another
simulation example in medical contexts is in emergency response systems such as the
system in [36].
Evidence suggests that medical simulation improves the effectiveness, safety, and
efficiency in health care services [10] [11]. Moreover, it has been shown to consistently
deliver significant value to the organization, staff, or medical students in decision-making
[11][20].
12
2.1.1 Medical Simulation Categorizations Medical simulation has been reported in the literature as categorizations in different
ways. For example, complexity of visualization, platforms, contexts of use [12]. In
this section, we examine the different categorizations of medical simulation and
map the relevant categorizations across sub-domains to gain an insight into common
themes, trends and challenges in the simulation categories.
In [3], Barjis et al. categorized medical simulation tools into four categories as
clinical, operational, managerial, and educational.
• Clinical and Training Simulation: Is a training technique for physicians,
medical students, nurses, and other healthcare professionals, which is used
to study, and analyse the behaviour of diseases, including biological
processes in human body [3]. Example of this category include haptic device
(e.g. robotic arm or endoscope simulation) [13][14][15].
• Operational Simulation: Is a technique for modeling a process and is used
for capturing, and analysing health care operations, patient flow, service
delivery, and scheduling, healthcare business and optimization design [3].
Example of this category includes the patient flow at the emergency
department at a hospital [16].
• Managerial Simulation: Is a type of interactive training and feedback and
is used as a tool for managerial purposes, decision-making, strategic
planning, and policy implementation. Example of this category includes
comprehensive management planning for healthcare processes, staffing,
equipment and buildings [17].
• Educational Simulation: Is training and educational techniques, where
virtual and physical objects are extensively used to help a learner explore,
navigate or obtain more information about the environment [3]. Example of
this category include haptic device (e.g. robotic arm or endoscope
simulation) [14][15].
13
In [18], Brailsford (2007) classifies medical simulation models into three groups a
models of the human body that include biological processes, models for modeling
patient flow in the clinic, ward, department, or hospital, models for strategies that
are used for strategic planning of the organization.
In [37], Kathleen R. Rosen MD classified medical simulation into five types;
including Standardized patients, Human Patient Simulation, Virtual reality, Task
trainers, Software-based simulation. Researchers have varied in categorizing
medical simulation, where each author has a different name for the same category.
The following Figure 3 highlights the mapping.
Barjas [3] Brailsford[18] Kathleen R. Rosen MD [37]
Clinical Human body Standardized patients, Human patient Simulations
and Task trainer.
Operational Patient flow Virtual reality and Software-based.
Educational Human body Standardized patients, Human patients and Task
trainer.
Managerial Strategies Software-based simulation.
Figure 3 Mapping Medical Simulation Categories
2.1.2 Medical Simulation Challenges
Medical simulation is a multi-disciplinary and involves increasingly complex
problems and rapidly developing fields [3]. It is a theoretical and practical domain
that focuses on multi-methods, multi-paradigms, multi-modeling and multi-
disciplines [3][38]. One of the main challenges in medical simulation is the need for
medical simulators that truly apply multi-modeling to present visual, auditory,
haptic, and olfactory displays [39]. And based on that, the correlation of the
14
different displays is the challenge that lies here because there should be proper
relation between user perceives sensory cues and to user interactions [3][39].
Medical simulation values and benefits that are produced to improve clinical,
operational and management processes are clear and easy to perceive nowadays
[38]. And although medical simulation is now more known and prepared in the
healthcare industry, it is still a sophisticated and a highly technical tool for non-
technical user’s comprehension. This challenge triggers user resistance, which is
considered as barrier to a successful simulation implementation in healthcare. This
barrier exists in with the fact that detailed simulation such as medical simulation,
requires tremendous time and effort [38]. In addition, evidence has shown that
Human-Simulator Interfaces are often critical [25]. In order to avoid errors, poor
training, and to provide a complete solution for different cases in medical
simulation (e.g. applying and surgery on a patient, or training for healthcare
practices), a medical simulator designed interface should replicate that of the real
world and easily used [25]. Thus, the physician, medical student or any healthcare
practitioner should touch instruments that provide the same feel and look as their
real world counterparts. These tool’s requirements can be challenging, especially if
the simulator addresses open surgery where the degrees of freedom and the numbers
of instruments are significantly larger than in minimally invasive surgery. As a
conclusion for the previous challenge, user acceptance is an important matter in
healthcare simulation [28].
A simulation model can only provide good and accurate results based on the input
data, although the data collection is a challenge in healthcare simulation [39]. In
healthcare, often the medical simulation tool developers lack sufficient input data
for their simulation models, which leads to delivering rather approximate results
[38]. Data collection is a challenge due to not available useful formats for historic
data; data collection should take place over a long span of time; meeting with
healthcare professionals for gathering data collection and verification purpose is
also a hard task due to their busy schedules [3]. The input data need to be complete,
accurate and real. The entered data play a big role in assessing health professional in
15
decision making since they are considered as DSSs. In order to provide ideal data
collection it may require integrating simulation models with the organization
information systems to support the daily operation [38].
Validation and verification in medical simulation is a subject of extensive research
[40]. It is considered as a real devil because without applying verification and
validation methods, it would be risky, if not disastrous, to make any decisions or
forecasts based on the model outcomes [41]. To overcome this challenge,
innovative modeling approaches, model validation, especially for complex models
should be used [42]. And in order to enhance model verification an approach such
by applying CPI modeling, in which models are designed with the collaboration of
the users using the medical simulation tool and business owners [40]. Moreover,
validation quite differs from the verification process. A significant research
challenge may increase when developing a valid simulation model, designing valid
experiments based on the model, and carrying out a rigorous analysis of the
experiments [42].
The cost, is a major issue for medical modeling and simulation [40]. Although the
popular perception may be that the medical enterprise is well funded, the reality,
especially in medical education, is quite the opposite. The cost of simulators must
be significantly reduced if they are to become commonly available tools within the
medical school curriculum.
To summarize medical simulation challenges, evidence have shown that the cost is
major challenge. In addition, the validation and verification for medical simulation
tools is also considered as a challenge. Also, the complexity and data entered is a
challenge.
2.1.3 Human Factors in Medical Simulation Although medical simulation provided ideal approaches for addressing healthcare
issues, the number of successful implementation and development is relatively
small compared with the manufacturing industry [18]. Modern hospitals and clinics
are encountering high levels of competition in both domestic and global markets. In
addition, patients, physicians, medical students and trainees are increasingly
16
demanding for more quality in health care services delivered with a reasonable cost.
Notably, research in [20] has suggested that medical Simulation models needs to be
structured in a way that it optimizes the safety and quality of health care systems
[20]. On the other hand, implementing qualified simulation systems may increase
the complexity of system and negatively influences the efficiency of these systems.
This consequently led to the emergence of software solutions that focused more on
basic science education with simulation than less clinical training.
Human error plays a crucial role in the safety of medical simulation tools. Human
errors can frequently be traced back to deficiencies in the design of the human-
machine interface as been highlighted in the literature review [24]. If the system and
interface design was not designed with human capabilities and by considering the
limitations of the cognitive, perception and physical human factors, physician,
operators and healthcare providers are being placed in situations where the demands
imposed on them are unrealistic from a psychological point of view [43].
Subsequently, the result will be an inevitable error. The discipline of human factors,
or ergonomics, deals with the highlighted medical simulation issues and challenges
by designing interfaces that take into account human capabilities and limitations.
The lack of attention to human factors during the design phase seriously jeopardizes
the human safety. Following design principles related to medical simulation may
decrease human error and lead to a better understanding and using for the tools.
As an example of human factors design principles that can be adopted when
designing medical simulation tools include [43][44]:
• User should be provided with prompt feedback after each action.
• Make the functions of the various controls clear and obvious.
• Displayed messages should be easy to understand.
• Minimize the load on the users’ memory as much as possible.
• Increase efficiency by providing users with shortcuts.
• Performance and clinical evaluation.
• Background information of the medical simulation tool. Example of this
category includes intended purpose of the device, device components
17
general warnings conditions of device use, supplies and materials, and user
preparation.
2.1.4 Medical Simulation Types Medical simulation tools have been successful in the clinical, training and learning
context of healthcare. In this thesis, we are focusing on the clinical and training
simulation. Evidence has shown that medical simulation is increasingly being
adopted in surgical, dental and cosmetic usage contexts. Examples on such medical
simulation tools are illustrated in Figure 4, Figure 5, and Figure 6.
Figure 4 Surgical Simulations [45]
Figure 5 Cosmetic and Plastic Surgery [45]
18
Figure 6 Dental Simulation [45]
The simulation tools vary on the spectrum of complexity from simple limited scope
training on specific skills which implement direct-manipulation interfaces, to more
complex interactive procedure that require more multimodal interaction (e.g.
tangible interfaces and tactile feedback).
2.2 Decision Support Systems (DSSs) DSSs emerged in the 1970. It is often defined as a set of computer programs designed to
assist, and engage decision makers and supports individuals (e.g. stakeholders) in decision-
making and analysis as highlighted by [46] [47] [48].
The decision process refers to some techniques that include: gathering and exchanging
information; identifying scenarios, and producing a final choice among a set of options
[46][49]. A DSS can be classified as passive, or active and cooperative with regards to its
mediating role in decision [50][51]. Approaches applied for developing or categorizing
DSSs range from data levels to model-driven DSS categories [48][50][51].
2.2.1 Data-driven Data-driven DSSs are targeted by managers, staff and also product/service
suppliers. The data is used for querying a database or a data warehouse to find
answers for specific purposes. A Data-Driven DSS can provide the highest level of
functionality when applying On-line Analytical Processing. It is designed and
implemented via a mainframe system, client/server link, or via the web.
2.2.2 Knowledge-based Knowledge-based DSSs are systems with specialized problem-solving expertise.
Knowledge-based DSSs contain knowledge about a particular domain, the ability to
19
understand problems within that domain, and also can solve some of these problems.
It is designed and implemented via the web, or software running on stand-alone PCs.
2.2.3 Document-driven Document-driven DSSs uses processing technologies and computer storage to
provide document retrieval and analysis. Examples on this type of DSS are
databases that may include scanned documents, hypertext documents, images,
sounds and video. To access these kinds of documents a search engine is needed to
be associated with a document-driven DSS.
2.2.4 Communication-driven Communication-driven DSSs use network connection and communications
technologies to facilitate decision-relevant communication. This type of DSS assists
in conducting meetings, or for individual’s collaboration. Examples include tools
that use computer-based bulletin boards, video conferencing and groupware.
2.2.5 Model-driven Model-driven DSSs are systems that assist in analysing decisions. This type of DSS
emphasizes access to and manipulation model, optimization or financial. The data
and parameters used are limited in order to aid in decision-making. These DSS s can
be designed and implemented via software/hardware in stand-alone PCs,
client/server systems, or the web.
These DSS types also differ based on the type of communication network whether it is
LAN based or Web-based [52] as shown in Table 1.
20
Table 1 Implementing DSSs and Examples from Previous Research
Technology
DSS Type LAN Based Web-Based
Data-driven Thick client Thin client
Examples: [53] Examples: [54]
Knowledge - based Standalone PC Shared knowledge
Examples: [55] Examples:[56], [57],[58], [59]
Document-driven Limited to certain files such as Doc., .xls
Not limited to certain files such as HTML and search engines
Examples: [52] Examples: [52]
Communication-driven Narrow scope Global scope
Examples: [60] Examples: [61]
Model-driven Single user Multiple users
Examples: [62] Examples:[56], [63]
In the past 50 years, researchers have pointed of the rapidly growing computational
complexity in simulation which consequently lends itself to playing an increasingly
important role in decision-making in healthcare. In recent years, most of the DSSs are
considered as knowledge-base [63][64]. Evidence has suggested that medical DSSs have
been beneficial and considered as computational artefacts that assess various decision
making tasks in the medical domain (e.g. diagnosis, therapy planning, interacting with
patients) [58]. The authors I. Fatima, M. Fahim, D. Guan, Y.-K. Lee, And S. Lee, have
pointed out that in 1961 the first mathematical equation to aid in the diagnosis of congenital
heart disease has been proposed by Warner et al. and from this work many clinical DSSs
have been developed [64]. The most common Clinical Decision Support System (CDSS)
applications include alerts, diagnostic assistance, decision support in the prescription,
information retrieval, image recognition, and therapy critiquing and planning [65][66].
The authors I. Fatima, M. Fahim, D. Guan, Y.-K. Lee, And S. Lee, have designed and
developed a socially interactive clinical DSS where it can interact and automatically learn
21
new knowledge from users’ experience [64]. In addition, the proposed system supports user
high-level context recognition ability. Traditional CDSSs require physicians or patients to
manually enter patient information, which are used in decision-making. The proposed
CDSS is intelligent and generates patient-oriented decision when more related information
obtained.
A synthesis of DSS Medical simulation is listed in Table 2. To reflect on trends in the
design of DSS systems, this synthesis aggregates the DSS healthcare systems in a lens
looking at types which were reported in the literature and mapped to the DSS category. The
mapping in Table 2 refer to the DSS types as follows:
a. Data-driven b. Knowledge-driven
c. Document-driven d. Communication-driven
e. Model-driven
Table 2 A Synthesis of DSS Categorizations in Medical Simulation
Tools name
a b c d e
Dentist [45] x
CAE Healthcare AccuTouch [67]
x
The GI-Bronch Mentor [67]
x
Transoral Robotic Surgery (TORS) [68]
x
DentalNavi [45] x
Kidneys [45] x
Touch surgery [45] x x
Ear surgery [45] x
Heart surgery [45] x
22
Virtual dentist [45] x
Radiology 2.0: One Night in the ED [45] x
Radiology 2.0: Pregnant Appendicitis [45] x
Knee surgery [45] x
iURO Kidney [45] x x
Plastic surgery simulator [45] x
Virtual dentist [45] x
Medscape [45] x
2.3 Software Quality Models In software or anything else evaluating the quality means measuring value. A product with
a lower quality has less value than a product with a higher quality. Measuring and
evaluating the quality of software products is considered to be a fundamental task in the
different organizations [69][70]. Poor quality may lead to a diversity of issues such as task
failure, financial loss, permanent injury, or in critical systems it may lead to human life loss
[69]. Software quality has several definitions, for example, IEEE define it as the degree to
which a system, component, or procedure meets specified requirements. And in order to
meet the specified requirements software quality assurance factors should be applied due to
the variety and complexity of software which is increasing day after day [69]. To improve
the quality of software products and make them measurable, models containing quality
assuring factors have been developed and applied. These models are usually used to
support stakeholders in evaluating the quality of a software [70].
In the software engineering and systems engineering literature, several software quality
models have been repeatedly reported in different applied contexts, which are McCall’s
Quality Model, Boehm’s Quality Model, Dromey's Quality Model, FURPS Quality Model,
ISO 9126 Quality Model. Each model’s content is discussed briefly in the following
sections.
23
2.3.1 McCall’s Quality Model McCall Quality Model is a quality model presented by Jim Mccall in 1977 [71]. It is
one of the frequently cited predecessors of quality models in general and is
primarily aimed towards system development process and system developers
[71][72]. The McCall quality model mainly bridges the gap between users and
system developers by concentrating on a number of software quality factors that
reflect both the users’ vision and the developers’ priorities [72]. The McCall
quality model has three major characteristics for defining and identifying the quality
of a software product as shown in Figure 7.
Figure 7 McCall Quality Model [72]
2.3.2 Boehm’s Quality Model In 1978 Barry W. Boehm presented a quality model that addresses the
contemporary shortcomings of models by prediction. This quality model works by
quantitatively and automatically evaluating software’s quality by applying a set of
attributes and metrics [73]. Moreover, Boehm's model and McCall quality model
are similar, for example their quality model is structured in a hierarchical way as
24
shown in Figure 8. This model decomposes its characteristic into high-level
characteristic, intermediate level characteristic and primitive level characteristics.
Figure 8 Boehm Quality Model [72]
2.3.3 FURPS Quality Model Robert Grady and Hewlett Packard proposed a model called FURPS in 1987 [74].
The term FURPS is an abbreviation, which refers to Functionality, Usability,
Reliability, Performance and Supportability. FURPS quality model decomposes its
characteristics in two classifications of requirements that are functional requirement
and non-functional requirement [74] as shown in Figure 9. Functional requirement
define a specific behaviour where it has an input and an expected output, while non-
functional requirement that only include Usability, Reliability, Performance and
Super-operability and also called as URPS.
25
Figure 9 FURPS Quality Model [34]
2.3.4 Dromey's Quality Model R. Geoff Dromey presented a quality model in 1995. This model is considered as
one of the recent models that are similar to the McCall’s, Boehm’s and the FURPS
quality models [75][76]. Dromey quality model is a product based quality model
that states that the evaluation differs for each product which leads to a dynamic idea
of modeling is required [75]. Moreover, the model seeks to increase the
understanding due to the relationship between the attributes (characteristics) and
sub-attributes (sub- characteristics) of quality as shown in Figure 10.
26
Figure 10 Doremy's Quality Model [72]
2.3.5 ISO 9126 Quality Model ISO 9126 is an international standard for the evolution of software. The standard is
decomposed of four parts, which address the following subjects; Quality model
9126 Part-1, External metrics 9126 Part-2, Internal metrics 9126 Part-3, Quality in
use metrics 9126 Part-4 [72]. The ISO quality model is an extension to previous
quality models presented by McCall, Boehm, FURPS and Doremy [72]. This
international quality model evaluates the quality of a software product in terms of
internal and external quality assurance factors and their connection to characteristics
[72], [77]. ISO 9126 structure is decomposed of 2 levels. The highest level includes
the quality characteristics and the lowest level includes of the software quality
criteria (e.g. metrics [77]). The quality model has six characteristics consisting of
Functionality, Reliability, Usability, Efficiency, Maintainability and Portability as
shown in Figure 11. Moreover, these characteristics are further decomposed of 21
sub characteristics. The set of characteristics defined by the ISO 9126 quality model
are applicable to every kind of software [77].
27
Figure 11 ISO 9126 Quality Model [78]
2.3.6 ISO/IEC 25000 SQuaRE Although ISO/IEC 9126 have been considered as the best software quality model
since it is an extension and an enhanced version of previous software equality
models [79], evidence have found that it has major flaws [32]. An Example on such
flaws are ISO 9126 measurements is not consistent with the science and engineering
measurements so there is a real concern regarding validating the proposed measures
[79]. Therefore, ISO 9126 has been replaced with ISO/IEC 25000 SQuaRE due to
its extensive series of standards [72]. The SQuaRE is divided to five divisions as
shown Figure 12, which are the Quality Management division 2500n, Quality
Model Division 2501n, Quality Measurement Division 2502n, Quality
Requirements Division 2503n, and Quality Evaluation Division 2504n. The
SQuaRE division is elaborated more in Table 3[32].
28
Figure 12 ISO/IEC 25000 SQuaRE [32]
Table 3 Square Divisions
Type Description
ISO/IEC 2501n
Quality Model
Division
This division consists of detailed quality models for software
products, quality in use, data, and guidelines on how to use this
model.
ISO/IEC 2502n
Quality Measurement
Division
This division consists of software quality measurement model,
metrics, and guidelines on how to use this model.
ISO/IEC 2503n
Quality Requirements
Division
This division consists of software quality measurement model,
metrics, and guidelines on how to use this model.
SO/IEC 2504n Quality
Evaluation Division
This division consists of requirements, recommendations and
guidelines for evaluating a software product.
ISO/IEC 25050 25099
SQuaRE Extension
Division
This division consists of the Common Industry Formats for
usability reports and Requirements for quality of Commercial
Off-The-Shelf software.
29
2.3.7 Comparison between Software Quality Models and Quality
Measures Table 4 highlights the quality assurance factors and their availability in each
software quality model. Such comparison between the software quality models
assist in choosing the right model.
Table 4 Quality Factors and Their Availability in Each Model
Criteria Mccall Boehm Dormey ISO 9126
ISO/IEC 25010:2011
Correctness x x
Reliability x x x x x
Integrity x x
Usability x x x x x
Efficiency x x x x x
Maintainability x x x x x
Testability x
Interoperability x
Flexibility x x
Reusability x x x
Portability x x x x x
Clarity x
Modifiability x
Documentation x
Realistic x
Understandability x
Validity x
Functionality x x x
Generality
30
Economy
Compatibility x x
Security x x
31
Chapter Three: Software Quality Model
for Medical Simulation Tools
32
3.1 User Centered Design Since involving stakeholders provides a better understanding of evaluation and decision
making process a User Centered Design method has been applied in this research. A User
Centred Design (UCD) is a method in which the desires, requirements, and limitations of
end users of a product, process, or service are taken into consideration at each stage of the
design process [28].
Figure 13 User Centered Design
UCD has been applied by the semi-structured interview sessions conducted throughout the
project with professionals in both healthcare and software quality subject-matter domains in
different organizations. Semi-structured interviews are interviews with a small duration to
collect new ideas that are be brought up from the interviewee during the interview [80]. The
interviews conducted with these participants were designed as participatory design sessions
in which insights were elicited on specific aspects of the system. Sessions were carried out
33
in a semi-structured guided dialog mode, and ranged in duration from brief 20min sessions
to 50-60 sessions; overall, these sessions didn’t take more than one-hour conversations with
participants. We considered the DSS context; objectives; environment and goals. A set of
questions was asked to them in order to gain feedback and insights were integrated as
design considerations for the iterative cycles of developments of the DSS.
Semi-structured interviews were conducted as part of the UCD activities in which
participants from the group listed in Table 5 were presented with a set of enquiry-driven
questions. These questions include:
• What are the best techniques to evaluate a product?
• How is a product evaluated in your organization?
• What are the software quality models used in your organization?
• What is technical software quality model used in your company?
• Have you ever used a medical simulation system in the Operation Room (OR)
room? If yes, what are they?
• Are they easy to use? If not, what are the bottlenecks?
• How common is medical simulation used in your hospital?
• Have you been trained on these tools? If yes, for how long?
• Is there any DSS for medical simulation tools?
Answers were annotated and content analysis was conducted to elicit themes and trends in
the insights provided by the participants in this UCD activity. The information and
feedback from the participants were gathered as an aggregate set of design
recommendations, summarized in Table 6.
34
Table 5 UCD Participants
Participants Position Organization or
Company
Meeting day and
time
P1 Senior Project
Manager
PriceWaterhouseCooper 18-Feb-2015
P2 Quality Assurance
Manager
Saudi Skills Standards 25-Feb-2015
P3 Quality analyst Tamkeen Technologies 7-Mar-2015
P4 Dentist King Khalid University
hospital
9-Mar-2015
P5 Dental student Riyadh collage of
Dentistry and Pharmacy
15-Mar-2015
P6 Medical student King Saud University
P7 Dermatologist National Guard Hospital 2-Jan-2015 and 14-
Feb-2015
P8 Surgeon National Guard Hospital 1-April-2015
P9 Plastic surgeon National Guard Hospital 2-Mar-2015
Table 6 UCD Summarization and Results
UCD Results
• Most of the quality managers commonly follow standards in their organizations to
ensure product quality.
• Most of the software engineering domain participants that were interviewed follow
the ISO standards in their work.
• Organization commonly evaluates a product’s quality by following international
standards.
• The physicians using medical simulation tools are complaining from the tool’s
35
complexity, efficiency, usability, and high risk and considering them as a
bottleneck.
• The medical simulation tools are mostly abandoned due to their complexity and
difficulty to use.
• Using the medical simulation tools in a wrong way will affect the patients’ safety.
• Medical simulation tools are cost affective so they should be selected based on
measurements.
• It takes many days, effort and resources to be trained on medical simulation tools.
• Medical simulation tools are not commonly used in hospitals in the region, and in
Saudi Arabia in particular.
The literature in this research has highlighted medical simulation categories, challenges,
human factors and previous software quality models. After a long period of search in
software quality models and standards, the ISO and International Electro Technical
Commission (IEC) has been selected that form the specialized system for worldwide
standardization [32] in designing and developing the Software quality model. Findings
have led to creation of a proposed aggregate model for our DSS system; the Medical
Simulation Software Quality Model shown in Figure 14 that best matches and overcomes
the challenges found in the literature and by applying the UCD.
It is good practice to base the evaluation process on internationally agreed upon standards
which provide an authoritative source of reference. The selected standard is the Systems
and software engineering - Systems and software Quality Requirements and Evaluation
(SQuaRE) standard. It has been chosen due to its replacement of the ISO 9126. Moreover,
as highlighted in the literature the SQuaRE standard is an improved version of the ISO
9126 [81] that is an extension of previous work done by Mccall quality model which is
considered as the gold standard of software quality models [72]. Therefore, the presented
software quality model is a combination of previous software quality models with a focus
on criteria specific to medical simulation tools.
36
Figure 14 Medical Simulation Software Quality Models [32]
37
This International Standard defines quality in use measures for the characteristics listed in
ISO/IEC 25010, and is intended to be used together with ISO/IEC 25010. This International
Standard contains an explanation of how to apply software and computer system quality
measures, a basic set of quality measures for each characteristic, an example of how to
apply quality measures during the product life cycle. It includes as informative annexes a
quality in use evaluation process and a reporting format.
3.2 Metrics and Measurements in the Quality Model The models are comprised of key measurements for assessing the software product’s
quality. The measures were elicited form the referenced models and in this section, we
describe the key metrics for each measurement.
3.2.1 Effectiveness Measures Effectiveness measures ensure the completeness and accuracy with which users
achieve a specific requirement. And a QME is a quality measure element method.
An elaboration on this measure described in details in Table 7.
Table 7 Effectiveness measures [32]
Task name Description QME
Task completion What are the tasks completed correctly?
Z = A/B
A = number of tasks completed.
B = total number of tasks attempted.
Effectiveness task How much is the proportion of the required goal is achieved correctly?
{X = 1-∑Ai | X>0}
Ai= proportional value of each missing or incorrect performance in a task output.
Error frequency What is the error frequency made in a task by the user compared to a target value?
X= I/J
I= number of errors made by the user.
J= number of tasks.
38
3.2.2 Efficiency Measures Efficiency measures assess the accuracy and completeness in relation to resources
expansion with which users achieve a specific requirement. An elaboration on this
measure described in details in Table 8.
Table 8 Efficiency Measures [32]
Task name Description QME
Time How long does it take to complete a task.
W = Tt/At
Tt = target time.
At= Actual time.
Relative task time A comparison between a user completing a task and an expert.
X= J/I
I= normal user’s task time.
J= expert user’s task time.
Efficiency
A task compared with the target?
W = (Tt – At) / Tt
Tt = target time.
At= Actual time.
Task efficiency How efficient are the users? X = Te/ T
Te = task effectiveness.
T = task time.
Relative task
A comparison between the user efficiency and a target.
X= I/J
I= normal user’s task efficiency.
J= target task efficiency.
Economic productivity How cost-effective is the user? W = Te / C
Te = task effectiveness.
C = total cost of the task.
39
Productive proportion What proportion of the time are the user performing productive actions?
Z= Ta / Tb
T a = productive time
Tb = task time
Relative number of user actions
Is the minimum number of actions needed performed?
X=I/J
I= Number of actions performed by the user
J= Number of actions needed
3.2.3 Satisfaction Measures Satisfaction measures is the degree to which user is satisfied when using a software
product as shown in Table 9.
• Usefulness Measures Usefulness measures are the degree to which a user is satisfied with their
achievement of requirement, and result of using a software product. This measure is
related to the usability measures which have evolved over the years with their own
quality assurance factors specific to subjective and objective measures of user
experiences with interactive software products. An elaboration on this measure
described in details in Table 9.
Table 9 Usefulness Measures [32]
Task name Description QME
Discretionary usage
How much potential users using the system?
X = I/J
I= number of times that
specific software requirements
are used.
J = number of times intended to
be used.
Discretionary utilization of What is the average utilization X = ∑(Ai)/n
40
3.2.4 Freedom from Risk Measures Freedom from risk measure is the degree to which a software product mitigates a
potential risk to an environment, health, human life, and economic status.
• Risk Mitigation Measures Risk mitigation measure is the process of mitigating a potential risk to an
environment, health, human life, and economic status performed by a product
qualities (functional suitability, performance efficiency, compatibility, usability,
reliability, security, maintainability or portability). An elaboration on this measure
described in details in Table 10.
Table 10 Risk Mitigation Measures [32]
Task name Description QME
Risk mitigation
To what extent can product quality mitigate risk?
Z = A/B
A = Risk with high quality.
B = Risk with low quality.
functions
of functions?
Ai =Proportion of users using
function i.
B= number of function i.
Customer complaints
What proportion of complaints
submitted by customers?
Z = A/B
A = number of complaining
customers.
B = total number of customers.
41
• Financial Measures Financial measures are the process of assessing financial status related to economic
objectives, commercial property, efficient operation, and reputation. An elaboration
on this measure described in details in Table 11 and Table 12.
Table 11 Financial Measures [32]
Task name Description QME
Return Of Investment (ROI)
What is the return on investment?
Z = A/B
A = Benefits obtained.
B = Invested amount.
Time to achieve a return of investment
Is a return on investment achieved in an acceptable time?
Z = A/B
A = Time to reach ROI.
B = Acceptable time to reach ROI.
Relative business performance
How the business performance is compared to top class companies in the industry or in same business?
Z = A/B
A= IT investment amount or sales of the company.
B= IT investment amount or sales of target company
Balanced Score Card (BSC)
Do the benefits of IT investment evaluated using the Balanced Score Card meet objectives?
Z = A/B
A= BSC results.
B= BSC objectives.
Delivery time
Does delivery time and the length and number of late deliveries meet targets?
Z = A/B
A= Actual delivery times or late deliveries.
B= Target for delivery time or late deliveries.
Missing items
Do the number of missing items meet targets?
Z = A/B
A = Actual missing items
42
Table 12 Economic Measures [32]
• Health and Safety Measures Health and safety measures assess risky health and safety factors. An elaboration on
this measure described in details in Table 13.
Table 13 Health And Safety [32]
Task name Description QME
User health and safety frequency
Calculate the health problem among users of the software?
Z = A/B
A = number of users reporting health problems.
B = total number users.
User health and safety impact What is the safety of users X=N*T*S
B = Target missing items.
Task name Description QME
Revenue for each customer
Does the Revenue for each customer meet targets?
Z = A/B
A = Actual revenue from each customer.
B = Target revenue for customer.
Error with economic consequences
What is the frequency and size for human or system errors?
Z = A/B
A = Number of errors with economic consequences
B = Total number of usage situation.
Software corruption What is the frequency and size of software corruption?
Z = A/B
A= Number of occurrences of software corruption.
B= Total number of usage situations.
43
using the product? N = Number of affected people
T = Time
S = Degree of significance
Safety of people affected by use of the system
What is the incidence of hazard to people affected by use of the system?
X= I/J
I= number of people put at hazard
J = total number of people potentially affected by the system.
• Environmental Measures Environmental measures assess risky environmental factors. An elaboration on this
measure described in details in Table 14.
Table 14 Environmental Measures [32]
Task name Description QME
Environmental impact
What is the environmental impact of using software?
Z = A/B
A = environmental impact B = acceptable impact.
3.2.5 Context Coverage Measures Context coverage measures is a process ensuring the degree to which a product or
system can be used with effectiveness, efficiency, freedom from risk and
satisfaction in specified contexts of use. An elaboration on this measure described in
details in Table 15.
Table 15 Context Completeness [32]
Task name Description QME
Context complete
Proportion of the context of use a product can be used with acceptable usability?
Z = A/B
A = Number of contexts with unacceptable usability.
44
B = Total number of distinct contexts of description use.
• Flexibility Measures Flexibility measures is the process to ensure the degree to which a product or
system can be used with effectively, efficiently, free from risk and with satisfaction
in contexts and with flexibility. An elaboration on this measure described in details
in Table 16.
Table 16 Flexibility Measures [32]
Task name Description QME
Flexible context of use
How can the product be used with additional context of use?
Z = A/B
A = Number of additional contexts in to ensure usability
B = Total number of additional contexts in which the software might be used
Flexible design features
How much can the product adapt to meet the user needs?
X=I/J
I = Number of design features with compete flexibility
J = Total number of design features
45
Chapter Four: Medical Simulation DSS
46
4.1 Conceptual Design of the DSS The decision support system is designed to facilitate access to disparate information for a
collection of medical simulation tools. The quality assessment of software is a
comprehensive process of objective and subjective measures that need to be quantified to
aid in the sense-making process. To this end, the concept of the system was to build layers
of intuitive interaction on the aggregate software quality model. Minimalist approach was
considered in the interfaces so as to maintain the focus of building the knowledge base
when new medical simulation tools are introduced. In addition, to facilitate exploratory
approaches of trend analysis when querying the knowledgebase. Figure 15 depicts the high-
level concept diagram in which the users, tasks and contexts of use are envisioned. It is also
shown in the figure the DSS input and output.
Figure 15 Conceptual Designs for DSS
47
4.2 Personas and Scenarios
In requirements engineering [82], a technique called “persona” is often considered when
systems are designed for non-computational disciplines so as to accurately reflect on the
potential users of the system. Involving personas, which are essentially fictitious characters
to document technical, domain-knowledge and socio-economic profiles of target users, is as
an interaction design technique that assists in a software product development lifecycle
[33]. Personas are considered as design aids and are often elicited from the domain
knowledge or in participatory design sessions such as semi-structure interviews of UCD
activities, nevertheless they are considered fictional characters representing people who are
intended to use the software or have interest in it [83]. A persona’s description holds
information of fictional people, they have names, occupation, life stories and goals [83].
The purpose of applying this techniques is to enhance engagement and reality by gaining a
clear understanding of the software and it is targeted users [33] [83]. In addition, it will also
include a description of the usage patterns that a persona would have when using the
software [83].
A complementary artefact is wire-framing with low-fidelity prototypes so that they both
can be combined as communication tools in UCD activities with subject-matter experts to
elicit feedback in the iterative cycles of design and development.
In this thesis, I have applied this technique before starting the development phase to gain a
wider understanding of the DSS system and potential users in the healthcare profession.
The intended system persona in the Medical Simulation DSS is shown in Figure 16 and
Figure 17 to demonstrate the type of information communicated between the developer(s)
of the system and the stakeholders. Notably, other personas used in the design cycles of the
system are included in Appendix C.
48
Figure 16 Persona 1 representing an experienced medical simulation user
Figure 17 Persona 2 representing a user in a learning context in healthcare (medical student)
4.3 System Visualization The UCD activities for designing the DSS involved iteration in the Software Development
Life Cycle SDLC. Two iterative cycles were conducted to design the DSS for medical
simulation tools and the interaction evolved to reflect the design considerations and
recommendations provide by subject matter experts in both the healthcare and the QA
domains.
4.3.1 First Cycle The first cycle is introduced in Figure 18. A brief explanation of the system was
given to the subject matter experts along with the system’s functionality. The
49
system visualization has been presented to the subject matter experts in order to get
their feedback and insights on the system.
Figure 18 System Visualization First Cycle
4.3.2 Second Cycle After receiving the feedback from the experts about the system visualization for the
first cycle. Comments and feedback were applied in the second cycle, and the
changes in the design of the interface are depicted in Figure 19.
50
Figure 19 Visualization -‐ Iteration 2
4.4 Chapter Summary In this chapter, the concept of the system was describe as well as the low-fidelity
prototyping process for integrating the Software Quality Model knowledge base with the
interface in the DSS. The interface of the DSS was designed for populating the database as
well as querying the knowledge base. In the following chapter, we describe specific
contexts of usage in the envisioned healthcare simulation scenarios to reflect on the design
of the DSS as well as a pre-cursor step for independent verification and validation that was
sought in collaboration with subject matter experts in the UCD activities of the iterations of
this system's SDLC.
51
Chapter Five: Interactive Decision Making
in Assessing the Quality of Medical
Simulation Tools
52
This chapter will showcase the procedural flow in the DSS by applying two case studies on
the medical simulation software quality model embedded in the DSS. We presented two
scenarios to showcase the interaction within the DSS. Professionals in healthcare, one
physician and one dentist, were involved in the feedback elicitation sessions for this
workphase, and medical simulation tools for the context of training were selected to be
used and evaluated in the DSS.
5.1 Scenario One: Evaluate a Medical Simulation Tool The user will access the DSS to evaluate Dental Navi tool and will be able to compare it to
other tools in the same category. The home page contains two buttons, “Evaluate a medical
simulation tool” and “Medical simulation tools review” as shown in Figure 20.
Figure 20 Wireframe design of Home Page's interface
In our case, the user will press the “Evaluate a medical simulation tool” to evaluate the
Dental Navi tool. After that he was directed to the software quality model embedded in the
DSS as shown in Figure 21 Software Quality Model Embedded In the DSS. The user will fill
in the fields by entering each value for a certain measure and these measures (QME) are
explained in chapter 3 (e.g. Time to complete the task). The software quality model will
calculate all the measurements and will be saved in knowledge base. Following that, the
user is directed to the result page, which contain all the entered and calculated values. The
user was also able to compare the Dental Navi tool with other tools in the dental category to
assist him in decision making by choosing the most applicable tool.
53
Figure 21 Software Quality Model Embedded In the DSS
5.2 Scenario Two: Medical Simulation Tools Review Based On
the Quality Assurance Factors The user is a Professor at King Saud University in the Medical school and is searching for
medical simulation tools to be used for training and education purposes. The user accesses
the DSS to search for previous evaluated medical simulation tools in the fields needed. The
54
home page contains two buttons, “Evaluate a medical simulation tool” and “Medical
simulation tools review”. The user pressed the “Medical simulation review” button and was
directed to the review page shown in Figure 22. The review page contains different medical
simulation tools which can be compared with each other by pressing the table button shown
in Figure 23.
Figure 22 Medical Simulation Tool Review
After the user pressed the table button, he was directed to the page shown in
Figure 22 Medical Simulation Tool Review. If the user wants to review a tool based on a
certain measure which is very helpful in decision making, he/she can press the
measure/metric that he/she wants as depicted in Figure 23. Following that, and a page
containing the measurement will be displayed for further interaction in the DSS.
Figure 23 Efficiency Measure
55
Figure 22 22 shows all the evaluation values for each entered tool to assist in decision-
making. In addition, the DSS provides the user a functionality to view tools based on a
certain measure such as freedom from the risk measure shown in Figure 25.
Figure 24 Freedom From Risk Measure
The system provides another feature by allowing the user to view the results in a chart as
shown in Figure 26. After calculating and entering all the values for each measure they are
presented in a chart based on the entered values from the user. Such presentation gives the
user the ability to visualize the effectiveness of the system. As an example, Figure 26
shows the distribution of quality assurance factors that are relevant for the simulation.
56
Notably, this example shows a tool that scored relatively high on efficiency measures but
less on subjective satisfaction measures from actual users of the simulation tool.
Figure 25 View Results as a Chart
The usability-engineering techniques for independently verifying the system's utility from a
user’s point of view have been established in the literature [e.g. 84]. Usability techniques
assist in structuring the process of designing and developing a good user interface [85].
Usability testing was applied with subject matter experts and practitioners to elicit their
insights on the high-fidelity design of the design. Qualitative and quantities approaches
were combined in these UCD activities. To augment the UCD activities applied in the early
research stages for requirements discovery and engineering, a cognitive walkthrough
usability technique was applied in the validation process to conduct the IV&V independent
verification and validation to ensure that system was developed right and that the right DSS
was developed from the perspective of professionals in the healthcare context [84]. IV&V
independent verification and validation is defined as a series of technical and management
activities performed by an expert in a field other than the system developer with the
objective to improve the quality of a system [86].
57
5.3 The Need for a Software Quality Model for Evaluating
Medical Simulation Tools The proposed software quality model assesses decision makers to evaluate medical
simulation tools by embedding it into the DSS. Modeling for decision-making development
is an effective method that assists in validating, testing and understanding complex
simulation tools. So, one of the main thrust of this research is to evaluate and improve these
simulation tools especially in early stages. Developing such a software quality model can
be used as a pre-process, since it provides health care providers the ability to evaluate the
tool before approving it to be used in the hospital, university or institute. Stakeholders will
be involved who would include problem owners in order to assess in the software quality
model development for the medical simulation tools.
5.4 Aim and scope of Software Quality Model Study The aim of this study is to investigate the proposed medical simulation software quality
model by describing the techniques, findings and evaluations results. An experimental
study was conducted to evaluate the medical simulation software quality model including a
variety of quality measures embedded in a web-based DSS. A selected number of
participants were selected to take part in this study. In addition, a selected number of
medical simulation tools were used in the study.
5.5 Exploratory Study In the following sections, we describe the context of the experimental study including the
participants, apparatus, and tasks assigned to the users.
5.5.1 Participants This study included seven participants considered as subject matter experts in the
medical simulation field. The participants were physicians who are varied in their
specialty and are using medical simulation tools. The participant’s technical
experience and demographic information is shown in Table 17 and Table 18.
58
Table 17 Participants' Technical Experience
Participants Technical experience and background
Medical simulation tools usage/familiarity
High Medium Low High Medium Low
P1 x
x
P2 x
x
P3 x x
P4 x
x
P5 x x
P6 x x
P7 x x
Table 18 Participants' Demographic Information
Participants Demographic information
Age Education level
Job title Organization
P1 32 Senior Registrar
Dermatologist National Guard Hospital
P2 22 Bachelor degree
Medical Student King Saud University
P3 65 Consultant Surgeon Prince Sultan Military Hospital
P4 21 Bachelor degree
Dental student King Saud University
P5 25 Bachelor degree
Dentist Riyadh College of Dentistry and Pharmacy
P6 26 Master’s degree
Researcher KACST
P7 25 Master’s degree
Software engineer Tamkeen technologies
59
5.5.2 Apparatus and Materials All the sessions were conducted at the hospital with the physicians. The apparatus
and materials used are a Macbook Pro device for running the DSS, and an iPad to
demonstrate the medical simulation tools.
5.5.3 Test Materials
• A brief presentation of the system was presented to the users to give an
overview of the system and its functionality.
• Participants' Consent Form (Appendix A).
• A timer, paper and pencil.
• Task scenario for participants.
• SUS questionnaire (Appendix B).
5.5.4 Task Scenarios A list of tasks was given to the participants to test the system’s usability. The results
of applying these tasks were used to assist us in the validation and verification
process. The participants are stakeholders in the healthcare sector with diversity in
specialties and positions. The list of tasks contains various functions that could be
performed in the system. A collection of medical simulation tools was presented to
the participants. The participants selected one tool to be evaluated by the system.
5.5.5 Task List A task list was provided to the user to be applied on the medical simulation DSS.
The list of tasks is shown in Table 19.
Table 19 Task List
Test Case Test Step Expected Result
Select medical simulation tool.
1. Choose one tool from a variety of tools to be evaluated.
The tool will open and used for simulation purposes.
Verify the home page. 1. Go to localhost\medicalsimulationtools
The DSS home page will contain two buttons,
1. Evaluation a medical simulation tool
2. Review medical simulation tool
60
Evaluate a medical simulation tool.
1. Go to localhost\medicalsimulationtools
2. Click the “Evaluate a medical simulation tool button”
The system should display a page containing the software quality model embedded in the system.
Timer 1. Before using the medical simulation tool to be evaluated by the system, a timer should be opened to determine the tool usage time.
The amount of time a participants used the medical simulation tool to perform the needed tasks.
Use the selected tool 1. Perform the needed actions to complete a task.
After viewing the DSS, the needed measurements (e.g. error frequency and time to complete a task) will be written down by the participants in order to be entered in the software quality model.
Enter the values needed for each measurement
1. Fill in all the fields with values.
After filling all the needed fields and submitting the evaluation form, the system will display a page containing the final results for decision-making and will have the ability to compare with other tools.
Review previous evaluated tool
1. Go to localhost\medicalsimulationtools
2. Click the “Medical simulation tools review”
The system should display a page containing the all previous submitted tools with the ability to view all their measurements. In addition the participant can also view tool based on a certain measure.
5.6 Objective Measures of Satisfaction In the study we have used the System Usability Scale (SUS). It is considered an
inexpensive and an effective method used for assessing the usability of a system [87]. The
SUS is 10 statements, each one of them have a 5 point scale ranges from Strongly Disagree
to Strongly Agree [88]. It provides a scale from 0 to 100, where 0 is considered as negative
61
and 100 is considered as positive [88]. ISO 9241-11 suggests that usability measures should
cover efficiency, effectiveness, and satisfaction [87].
The session started by presenting a brief explanation of the system to the participants along
the list of task that should be applied on the DSS, Participants Consent Form, and the SUS.
At the end of the session, the participants filled the SUS forms. The results of the SUS for
all the participants are shown in Figure 27. The chart in Figure 27 shows that the majority
of participants had relatively high SUS score, which suggests that the system’s usability is
perceived to be high as reported by participants in the evaluation study.
Figure 26 SUS Scores
As shown in Figure 27, P3 has the lowest SUS score in the sample. Notably, this
participants reported having low technical skills as shown in Figure 28. Although he had
low technical skills and could not easily use the system, he provided positive feedback on
the perceived value of using this system in the context of medical simulation.
0
10
20
30
40
50
60
70
80
90
100
0 1 2 3 4 5 6 7 8
SUS score
SUS
scor
e
Participants in the usability study
62
Figure 27 Relationship between Technical and Medical Simulation
The relation between the participant’s technical experience and their usage of medical
simulation tools is shown in
Figure 27. The participants were asked to evaluate their self out of the score 9 based on two
aspects, their technical background and medical simulation tool uses. If the score was from
1-3 it indicated a low value, if the score was from 4-6 it indicated a medium value, if the
score was from 7-9 it indicated a high value [32]. This categorization helped us to
understand the participants experience and how it relates in the interpretation of findings
from the usability study.
The chart in Figure 29 shows the variation between average scores of the subject matter
experts and healthcare professionals. The average score is a comparison between how the
0
1
2
3
4
5
6
7
8
9
10
P1 P2 P3 P4 P5 P6 P7
Technical experience and background
Medical simulation tools uses
Participants in the usability study
Parti
cipa
nt’s
eva
luat
ion
63
subject matter experts see the system and how the healthcare professional also perceive its
usability in the context of healthcare simulation. Notably, Figure 28suggests that the
experts and actual users are relatively close in their perception of usability according to the
SUS guidelines [87].
Figure 28 SUS Average Score for All Participants
5.7 Subject Matter Experts Reviews In this usability testing study, two domain experts were involved in cognitive walkthroughs
to evaluate the system. The Subject Matter Experts (SME) were domain experts where their
involvement was sought to help in eliciting opportunities for improvement of the system’s
functionality. Their insights and recommendation are summaried in Table 20.
Table 20 Subject Matter Experts Review
Subject matter expert Insights and recommendations
SME1 SME1 applied the tasks in the list provided.
Insights about the DSS are:
0 20 40 60 80 100
Subject matter experts
Healthcare professionals
SUS Average
SUS Average
Types of participants in the usability study
SUS
aver
age
64
• The system is easy to use and beneficial for decision-making. Recommendations: None.
SME2 SME2 applied the tasks in the list provided.
Insights about the DSS are;
• The system is straight to the point, easy to use, and covers the needed functionality.
Recommendations:
• Since the software quality model is very long and comprehensive to use, the SME suggested dividing the software quality model into separate measures where the user has the ability to evaluate a tool based on certain measures and skip other quality measures (setting a default or null value where applicable).
• To improve system visualizations and decision making the evaluation results should be also provided in charts.
5.8 Chapter Summary In this chapter, we described the cognitive walkthrough sessions, which were conducted
with SMEs and professionals and the insights obtained from them with regards to the
system's functionality, interface, and context of use. The verification and validation of the
DSS with actual users was essential to reflect on the design features (e.g. interface,
accessibility, and usability). Findings suggested an addition to the growing technology
assistance tools for medical simulation in the DSS, as participants perceived it as an
effective repository for examining quality metrics across medical simulation tools.
65
Chapter Six: Conclusion
66
In this thesis, we examined designing and developing a software quality model for
evaluating medical simulation tools.
The research objectives were achieved across work phases. In work phase 1, a systematic
literature review was conducted to gain an insight on medical simulation tools, challenges,
and best practices to overcome these challenges. In work phase 2, the software quality
model has been designed and presented. In work phase 3, the web-based DSS was designed
and developed along with the software quality model. In addition, the system was
independently verified and validated in cognitive walkthroughs with experts and use testing
scenarios with expert and practitioners.
Figure 29 Medical Simulation DSS
6.1 Research Questions The research questions were addressed in this thesis and reported in the document. For the
first research question of “Why do professionals need to evaluate medical simulation
tools?” was addressed in chapter 2, section 1-3 in which the importance of evaluation was
highlighted for medical simulation. For the second research question “What are the DSSs
used in medical domains?”, was addressed in chapter 2 section 2 in which the medical
67
simulation DSS types were highlighted. For the third research question “What are the
software development models used to design DSS in medical simulation domains?” was
addressed in chapter 2 section 3 in which software quality models were described. For the
fourth research question, “How can we test the usability, performance, robustness, and
accessibility of the medical simulation tools?” was addressed in chapter 3 section 2 in
which software quality model and measurement is presented. Furthermore, the independent
verification and validation sessions provided insights into testing the quality of medical
simulation tools which was described in chapters 3 and 4. For the fifth research question
“What are the widely accepted Software quality models used among Software Engineers?”
the software quality models widely used was addressed in chapter 2 section 3
6.2 Future Work In our future work, the medical simulation DSS will be amended with more software
quality metrics and interaction functionalities. These features will include adding more
visualization and analytics for the quality measurements in the software quality model
which is embedded in the DSS. Continuous integration of human expert’s knowledge will
also be considered as for the system by keeping track of usage logs in the DSS. In addition,
Artificial Intelligence algorithms will be applied to the system in order to develop further
analytics for the knowledge-base.
6.3 Publications The research conducted as part of this thesis was disseminated in scientific venues of
publication in the form of work-in-progress during the work phases and is planned for
reporting as a journal publication for the final work phase described in chapters 3 and 4.
Two conference publications were accepted for presentation in 2015; as noted in the
enclosed list.
• Al-Romi, Norah (July 2015). Human Factors in the Design of Medical Simulation
Tools, 6th International Conference on Applied Human Factors and Ergonomics
(AHFE 2015) and the Affiliated Conferences, AHFE 2015, USA.
68
• Al-Romi, Norah (Augest 2015). A Quality Model Used in Evaluating Medical
Simulation Tools, 17th International conference On Human Computer Interaction,
HCII 2015.
6.4 References [1] Y. Okuda, E. O. Bryson, S. Demaria, L. Jacobson, J. Quinones, B. Shen, And A. I. Levine, “The Utility Of Simulation In Medical Education: What Is The Evidence?,” Mount Sinai Journal Of Medicine, Vol. 76, No. 4. Pp. 330–343, 2009.
[2] M. I. T. Opencourseware, “Introduction To Modeling And Simulation,” In Proceedings Of The 2004 Winter Simulation Conference R .G. Ingalls, M. D. Rossetti, J. S. Smith, And B. A. Peters, Eds., 2008, Pp. 21–23.
[3] J. Barjis, “Healthcare Simulation And Its Potential Areas And Future Trends,” Scs M&S Mag. –, Vol. 1, No. January, Pp. 1–6, 2011.
[4] B. Mielczarek And J. Uzialko-Mydlikowska, “Application Of Computer Simulation Modeling In The Health Care Sector: A Survey,” Simulation, Vol. 88, No. 2, Pp. 197–216, 2010.
[5] J. Bosire, “Comparing Simulation Alternatives Based On Quality Expectations,” 2007, No. 1, Pp. 1579–1585.
[6] D. C. C. Peixoto, R. F. Resende, And C. I. P. S. Pádua, “An Educational Simulation Model Derived From Academic And Industrial Experiences,” Proc. - Front. Educ. Conf. Fie, Pp. 691–697, 2013.
[7] P. Naraharisetty And S. Vanka, “Effectiveness Of Computer Based Management Simulations - A Case Study,” In Proceedings - 2012 Ieee 4th International Conference On Technology For Education, T4e 2012, 2012, Pp. 31–37.
[8] L. C. Dukes, J. Bertrand, M. Gupta, R. Armstrong, T. Fasolino, S. Babu, And L. F. Hodges, “Poster: Comparing Usability Of A Single Versus Dual Interaction Metaphor In A Multitask Healthcare Simulation,” In Ieee Symposium On 3d User Interface 2013, 3dui 2013 - Proceedings, 2013, Pp. 133–134.
[9] D. M. Gaba, “The Future Vision Of Simulation In Healthcare.,” Simul. Healthc., Vol. 2, No. 2, Pp. 126–135, 2007.
[10] J. A Sokolowski And C. M. Banks, “A Proposed Approach To Modeling And Simulation Education For The Medical And Health Sciences,” In 2010 Summer Simulation Multiconference, 2010, Pp. 284–289.
[11] J. Liu, X. Wang, And M. Cheng E, “Simulation Modeling And Analysis On Asset Planning For Emergency Medical System (Ems),” In Ieem2010 - Ieee International Conference On Industrial Engineering And Engineering Management, 2010, Pp. 1353–1357.
[12] C. C. Chen, J. S. Daponte, And M. D. Fox, “Fractal Feature Analysis And Classification In Medical Imaging.,” Ieee Trans. Med. Imaging, Vol. 8, No. 2, Pp. 133–42, 1989.
[13] G. Mani And W. Li, “3d Web Based Surgical Training Through Comparative Analysis,” In Proceedings Of The 18th International Conference On 3d Web Technology - Web3d ’13, 2013, P. 83.
69
[14] T. R. Coles, D. Meglan, And N. W. John, “The Role Of Haptics In Medical Training Simulators: A Survey Of The State Of The Art,” Ieee Trans. Haptics, Vol. 4, No. 1, Pp. 51–66, 2011.
[15] E. Samur, L. Flaction, And H. Bleuler, “Design And Evaluation Of A Novel Haptic Interface For Endoscopic Simulation,” Ieee Trans. Haptics, Vol. 5, No. 4, Pp. 301–311, 2012.
[16] X. Wang, X. Shen, And X. Liu, “Improving Patient Flow At Hospital Emergency Services; A Simulation Study,” In Icsssm11, 2011, Pp. 1–6.
[17] I. W. Gibson And B. L. Lease, “An Approach To Hospital Planning And Design Using Discrete Event Simulation,” In 2007 Winter Simulation Conference, 2007, Pp. 1501–1509.
[18] S. C. Brailsford, “Tutorial: Advances And Challenges In Healthcare Simulation Modeling,” In Proceedings Of The 2007 Winter Simulation Conference, 2007, Pp. 1436–1448.
[19] M. Passiment, H. Sacks, And G. Huang, “Medical Simulation In Medical Education: Results Of An Aamc Survey,” Education, No. September, 2011.
[20] S. B. Issenberg, W. C. Mcgaghie, I. R. Hart, J. W. Mayer, J. M. Felner, E. R. Petrusa, R. A. Waugh, D. D. Brown, R. R. Safford, I. H. Gessner, D. L. Gordon, And G. A. Ewy, “Simulation Technology For Health Care Professional Skills Training And Assessment.,” Jama, Vol. 282, No. 9, Pp. 861–866, 1999.
[21] A. Nishisaki, R. Keren, And V. Nadkarni, “Does Simulation Improve Patient Safety?: Self-Efficacy, Competence, Operational Performance, And Patient Safety,” Anesthesiology Clinics, Vol. 25, No. 2. Pp. 225–236, 2007.
[22] P. N. Lowe And M. W. Chen, “System Of Systems Complexity: Modeling And Simulation Issues,” Simul. Interoperability Stand. Organ. - Siso Eur. Simul. Interoperability Work. Euro Siw 2008, No. 2, Pp. 299–308, 2008.
[23] R. Blaser, M. Schnabel, D. Mann, P. Jancke, K. Kuhn, And R. Lenz, “Potential Prevention Of Medical Errors In Casualty Surgery By Using Information Technology,” In Proceedings Of The Acm Symposium On Applied Computing, 2004, Pp. 285–290.
[24] T. Eldabi, “Implementation Issues Of Modeling Healthcare Problems: Misconceptions And Lessons,” Proc. - Winter Simul. Conf., Pp. 1831–1839, 2009.
[25] A. Kushniruk, E. Borycki, J. G. Anderson, And M. M. Anderson, “Combining Two Forms Of Simulation To Predict The Potential Impact Of Interface Design On Technology-Induced Error In Healthcare,” 2008, Pp. 497–504.
[26] S. C. Brailsford, T. Bolt, C. Connell, J. H. Klein, And B. Patel, “Stakeholder Engagement In Health Care Simulation,” In Proceedings - Winter Simulation Conference, 2009, Pp. 1840–1849.
[27] M. Saunders, P. Lewis, And A. Thornhill, Research Methods For Business Students. Financialtimesprentice-Hall, 2009.
[28] C. Abras, D. Maloney-Krichmar, And J. Preece, “User-Centered Design,” Bainbridge, W. Encycl. Human-Computer Interact. Thousand Oaks Sage Publ., Vol. 37, No. 4, Pp. 445–56, 2004.
[29] K. Vredenburg, J.-Y. Mao, P. W. Smith, And T. Carey, “A Survey Of User-Centered Design Practice,” In Proceedings Of The Sigchi Conference On Human Factors In Computing Systems Changing Our World, Changing Ourselves - Chi ’02, 2002, No. 1, P. 471.
[30] M. Zarour, A. Abran, J.-M. Desharnais, And A. Alarifi, “An Investigation Into The Best Practices For The Successful Design And Implementation Of Lightweight Software Process Assessment Methods: A Systematic Literature Review,” J. Syst. Softw., Vol. 101, Pp. 180–192, 2015.
70
[31] M. Maguire, “Methods To Support Human-Centred Design,” Int. J. Hum. Comput. Stud., Vol. 55, No. 4, Pp. 587–634, 2001.
[32] I. E. C. Jtc, “Systems And Software Engineering - Systems And Software Quality Requirements And Evaluation ( Square ) – Measurement Of Quality In Use,” 2012.
[33] J. Pruitt And J. Grundin, “Personas : Practice And Theory,” In Proceedings Of The 2003 Conference On Designing For User Experiences, 2003, Pp. 1–15.
[34] D. Samadhiya, S.-H. Wang, And D. Chen, “Quality Models: Role And Value In Software Engineering,” In International Conference On Software Technology And Engineering, 2010, Pp. 320–324.
[35] C. D. Buckingham, A. Ahmed, And A. Adams, “Designing Multiple User Perspectives And Functionality For Clinical Decision Support Systems,” In 2013 Federated Conference On Computer Science And Information Systems U6 - Ctx_Ver=Z39.88-2004&Ctx_Enc=Info%3aofi%2fenc%3autf-8&Rfr_Id=Info:Sid/Summon.Serialssolutions.Com&Rft_Val_Fmt=Info:Ofi/Fmt:Kev:Mtx:Book&Rft.Genre=Proceeding&Rft.Title=2013+Federate, 2013, Pp. 211–218.
[36] E. S. Binstadt, R. M. Walls, B. A. White, E. S. Nadel, J. K. Takayesu, T. D. Barker, S. J. Nelson, And C. N. Pozner, “A Comprehensive Medical Simulation Education Curriculum For Emergency Medicine Residents,” Ann. Emerg. Med., Vol. 49, No. 4, 2007.
[37] K. R. Rosen, “The History Of Medical Simulation,” J. Crit. Care, Vol. 23, No. 2, Pp. 157–166, 2008.
[38] S. M. Sanchez, T. Ogazon, D. M. Ferrin, And T. J. Ward, “Emerging Issues In Healthcare Simulation,” In Proceedings - 2000 Winter Simulation Conference, 2000, Pp. 1999–2003.
[39] K. Anderson, “Modeling And Simulation Grand Challenges: An Or/Ms Perspective R. Pasupathy, S.-H. Kim, A. Tolk, R. Hill, And M. E. Kuhl, Eds,” In Proceedings Of The 2013 Winter Simulation Conference, 2013, No. Johns 2008, Pp. 3059–3065.
[40] K. Kunkler, “The Role Of Medical Simulation: An Overview.,” Int. J. Med. Robot., Vol. 2, No. 3, Pp. 203–210, 2006.
[41] J. A. Aucar, N. R. Groch, S. A. Troxel, And S. W. Eubanks, “A Review Of Surgical Simulation With Attention To Validation Methodology.,” Surg. Laparosc. Endosc. Percutan. Tech., Vol. 15, No. 2, Pp. 82–89, 2005.
[42] J. H. Magee, “Validation Of Medical Modeling & Simulation Training Devices And Systems,” In Studies In Health Technology And Informatics, 2003, Vol. 94, Pp. 196–198.
[43] L. Lin, R. Isla, K. Doniz, H. Harkness, K. J. Vicente, And D. J. Doyle, “Applying Human Factors To The Design Of Medical Equipment: Patient-Controlled Analgesia.,” J. Clin. Monit. Comput., Vol. 14, No. 4, Pp. 253–263, 1998.
[44] J. R. Callan And J. W. Gwynne, “Human Factors Principles For Medical Device Labeling,” 1993.
[45] “App Store Downloads On Itunes.” [Online]. Available: Https://Itunes.Apple.Com/En/Genre/Ios/Id36?Mt=8. [Accessed: 03-May-2015].
[46] A. Adla And P. Zarate, “A Cooperative Intelligent Decision Support System,” Service Systems And Service Management, 2006 International Conference On, Vol. 1. Pp. 763–769, 2006.
[47] F. Z. F. Zhou, B. Y. B. Yang, L. L. L. Li, And Z. C. Z. Chen, “Overview Of The New Types Of Intelligent Decision Support System,” In 2008 3rd International Conference On Innovative Computing Information And Control, 2008, Pp. 1–4.
71
[48] D. J. Power And R. Sharda, “Model-Driven Decision Support Systems: Concepts And Research Directions,” Decis. Support Syst., Vol. 43, No. 3, Pp. 1044–1061, 2007.
[49] G. Desanctis, R. B. Gallupe, M. Science, And N. May, “A Foundation For The Study Of Group Decision Support Systems,” 2007, Vol. 33, No. 5, Pp. 589–609.
[50] C. Schultewolter, “Towards A Framework For The Generic Specification Of Model-Driven Decision Support Systems : Classification Criteria Of Model Relationships,” Pp. 1–10, 2010.
[51] D. J. Power, “A Brief History Of Decision Support Systems,” Decis. Support Syst., Vol. 4, No. 1969, Pp. 1–18, 2007.
[52] D. J. Power And N. Iowa, “Web-Based And Model-Driven Decision Support Systems : Concepts And Issues,” In Americas Conference On Information Systems, 2000, Pp. 1–4.
[53] N. I. B. Adnan And Z. Tasir, “Online Social Learning Model,” In 2014 International Conference On Teaching And Learning In Computing And Engineering, 2014, Pp. 143–144.
[54] W. Cheung And C. Hsu, “The Model-Assisted Global Query System For Multiple Databases In Distributed Enterprises,” Acm Trans. Inf. Syst., Vol. 14, No. 4, Pp. 421–470, 1996.
[55] R. Stern And B. Sagot, “Population Of A Knowledge Base For News Metadata From Unstructured Text And Web Data,” In Proceedings Of The Joint Workshop On Automatic Knowledge Base Construction And Web-Scale Knowledge Extraction, 2012, Pp. 35–40.
[56] M. A Shwe, B. Middleton, D. E. Heckerman, M. Henrion, H. E.J., H. P. Lehman, And G. F. Cooper, “Probabilistic Diagnosis Using A Reformulation Of The Internist-1/ Qmr Knowledge Base,” In Methods Of Information In Medicine, 1991, Vol. 30, Pp. 241–255.
[57] S. U. N. Yu And X. U. Tianwei, “A Formal Model Of User Knowledge Base Systems In Intelligent Tutoring Systems,” 2010, No. 60903131.
[58] H. Wang, “Medcase : A Template Medical Case Store For Case-Based Reasoning In Medical Decision Support,” 2013, Pp. 962–967.
[59] M. Siahbani, R. Vadlapudi, M. Whitney, And A. Sarkar, “Knowledge Base Population And Visualization Using An Ontology Based On Semantic Roles,” In Proceedings Of The 2013 Workshop On Automated Knowledge Base Construction - Akbc ’13, 2013, Pp. 85–90.
[60] K. S. Kadambi, J. Li, And A. H. Karp, “Near-Field Communication-Based Secure Mobile Payment Service,” In Proceedings Of The 11th International Conference On Electronic Commerce - Icec ’09, 2009, P. 142.
[61] T. X. Bui And M. Jarke, “Communications Design For Co-Op: A Group Decision Support System,” Acm Trans. Inf. Syst., Vol. 4, No. 2, Pp. 81–103, 1986.
[62] A. Steck, A. Lotz, And C. Schlegel, “Model-Driven Engineering And Run-Time Model-Usage In Service Robotics,” In Proceedings Of The 10th Acm International Conference On Generative Programming And Component Engineering - Gpce ’11, 2011, P. 73.
[63] H.-T. Wang And A. U. Tansel, “Medcase: A Template Medical Case Store For Case-Based Reasoning In Medical Decision Support,” In Proceedings Of The 2013 Ieee/Acm International Conference On Advances In Social Networks Analysis And Mining - Asonam ’13, 2013, Pp. 962–967.
[64] I. Fatima, M. Fahim, D. Guan, Y.-K. Lee, And S. Lee, “Socially Interactive Cdss For U-Life Care,” Proc. 5th Int. Confernece Ubiquitous Inf. Manag. Commun., Pp. 1–8, 2011.
72
[65] S. Subramanian, S. Hoover, B. Gilman, T. S. Field, R. Mutter, And J. H. Gurwitz, “Computerized Physician Order Entry With Clinical Decision Support In Long-Term Care Facilities: Costs And Benefits To Stakeholders.,” J. Am. Geriatr. Soc., Vol. 55, No. 9, Pp. 1451–7, Sep. 2007.
[66] J. A. Osheroff, Md, Facp, Facmi, And Editor-In-Chief, Improving Medication Use And Outcomes With Clinical Decision Support:: A Step By Step Guide. Himss, 2009.
[67] D. J. Desilets, S. Banerjee, B. A. Barth, V. Kaul, S. R. Kethu, M. C. Pedrosa, P. R. Pfau, J. L. Tokar, S. Varadarajulu, A. Wang, L. M. W. K. Song, And S. A. Rodriguez, “Endoscopic Simulators,” Gastrointest. Endosc., Vol. 73, No. 5, Pp. 861–867, 2011.
[68] B. W. O’malley, G. S. Weinstein, W. Snyder, And N. G. Hockstein, “Transoral Robotic Surgery (Tors) For Base Of Tongue Neoplasms.,” 2006.
[69] D. Jamwal, “Analysis Of Software Quality Models For Organizations,” Int. J. Latest Trends Comput. (E-Issn 2045-5364), Vol. 1, No. 2, Pp. 19–23, 2010.
[70] M. Kläs, C. Lampasona, And J. Münch, “Adapting Software Quality Models: Practical Challenges, Approach, And First Empirical Results,” In Proceedings - 37th Euromicro Conference On Software Engineering And Advanced Applications, Seaa 2011, 2011, Pp. 341–348.
[71] J. A. Mccall, P. K. Richards, And G. F. Walters, “Factors In Software Quality,” Nat’l Tech. Inf. Serv., Vol. 1, 2 And 3, No. November, 1977.
[72] B. Singh And S. P. Kannojia, “A Review On Software Quality Models,” Proc. - 2013 Int. Conf. Commun. Syst. Netw. Technol. Csnt 2013, Pp. 801–806, 2013.
[73] B. W. Boehm, J. R. Brown, And M. Lipow, “Quantitative Evaluation Of Software Quality,” Proc. 2nd Int. Conf. Softw. Eng., Pp. 592–605, 1976.
[74] A. B. Al-Badareen, M. H. Selamat, And M. A Jabar, “Software Quality Models : A Comparative Study,” 2011, Pp. 46–55.
[75] R. G. Dromey, “Cornering The Chimera,” Ieee Softw., Vol. 13, No. 1, Pp. 33–43, 1996.
[76] R. G. Dromey, “A Model For Software Product Quality,” Ieee Trans. Softw. Eng., Vol. 21, No. 2, Pp. 146–162, 1995.
[77] P. Botella, X. Burgués, J.-P. Carvallo, X. Franch, G. Grau, J. Marco, And C. Quer, “Iso/Iec 9126 In Practice: What Do We Need To Know?,” In Software Measurement European Forum 2004, 2004, Pp. 297–306.
[78] H. Pedram, D. K. Moghaddam, And Z. Asheghi, “Applying The Iso 9126 Model To The Evaluation Of An E-Learning System In Iran,” Iran. J. Inf. Process. Manag., Vol. 27, No. 2, Pp. 495–517, 2012.
[79] D. St-Louis And W. Suryn, “Enhancing Iso/Iec 25021 Quality Measure Elements For Wider Application Within Iso 25000 Series,” In Iecon Proceedings (Industrial Electronics Conference), 2012, Pp. 3120–3125.
[80] Z. Cooper And C. Fairburn, “The Eating Disorder Examination: A Semi-Structured Interview For The Assessment Of The Specific Psychopathology Of Eating Disorders,” Int. J. Child Adolesc. Heal., Vol. 6, No. 1, Pp. 1–8, 1987.
[81] L. Z. Aspiazu, “De Madrid Development Of A Model For Security And Usability Master Thesis,” 2013.
[82] G. Miller And L. Williams, “Personas : Moving Beyond Role-Based Requirements Engineering.”
73
[83] J. Grudin And J. Pruitt, “Personas , Participatory Design And Product Development : An Infrastructure For Engagement,” Design, Vol. 2002, Pp. 144–161, 2002.
[84] R. Mack And J. Nielsen, “Usability Inspection Methods,” Acm Sigchi Bull., Vol. 25, No. 1, Pp. 28–33, 1993.
[85] D. Mayhew, “The Usability Engineering Lifecycle,” In Chi’99 Extended Abstracts On Human Factors In Computer Systems, 1999, Pp. 147–148.
[86] D. T. Brunner, J. D. Arthur, R. E. Nanee, And V. Tech, “Independent Verification And Validation: A Missing Link In Simulation Methodology?,” In Proceedings Of The 1996 Winter Simulation Conference, 1996.
[87] A. Bangor, T. Staff, P. Kortum, J. Miller, And T. Staff, “Determining What Individual Sus Scores Mean : Adding An Adjective Rating Scale,” Vol. 4, No. 3, Pp. 114–123, 2009.
[88] J. Brooke, Sus - A Quick And Dirty Usability Scale. 1996.
74
Appendix A: Consent Form for
Participation in a Research Study
Consent Form for Participation in a Research Study
Prince Sultan University
Title of Study Usability testing for a medical simulation decision support system
Description of the research and your participation
You are invited to participate in a research study conducted by Norah AlRomi and Dr.
Areej AlWabil. The purpose of this research is to apply usability testing on a medical
simulation Decision Support System. Your participation will involve in applying a list of
tasks on the system.
Risks and discomforts
There are no known risks associated with this research.
Potential benefits
This research may help us to understand and evaluate the Decision Support System by
subject matter experts and stakeholders
Protection of confidentiality
We will do everything we can to protect your privacy. Your identity will not be revealed in
any publication resulting from this study.
Voluntary participation
Your participation in this research study is voluntary. You may choose not to participate
and you may withdraw your consent to participate at any time. You will not be penalized in
any way should you decide not to participate or to withdraw from this study.
75
Contact information
If you have any questions or concerns about this study or if any problems arise, please
contact Norah AlRomi at Prince Sultan University. If you have any questions or concerns
about your rights as a research participant, please contact the Prince Sultan University.
Consent
I have read this consent form and have been given the opportunity to ask questions. I
give my consent to participate in this study.
Participant’s signature_______________________________ Date:_________________
A copy of this consent form should be given to you.
76
Appendix B: System Usability Scale System Usability Scale
Strongly Strongly
disagree agree
1. I think that I would like to
use this system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
77
Appendix C: Personas Personas
78
Appendix D: DSS Data Dictionary