![Page 1: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/1.jpg)
Evaluating health informatics projects
•Reasons for and problems of evaluation
•Objective model
•Subjective model
![Page 2: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/2.jpg)
Definitions
• Evaluate – to determine the value of (Chambers)
• To examine and judge carefully (Dictionary.com)
![Page 3: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/3.jpg)
Reasons for evaluation(Friedman/ Wyatt)
• Promotional – – encouraging people to use systems
• Scholarly – – study of the impact etc. of HI systems
• Pragmatic (practical) – – finding out what is good and bad, improving future
systems
• Ethical –– like any medical intervention, safe and effective
• Legal – – same reason. Also to inform users so they know when
and when not to use it
![Page 4: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/4.jpg)
Perspectives
• Stakeholders– Developers– Users– Patients– Managers– Sources of funding
![Page 5: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/5.jpg)
Effects
• Structure – environment, staff, money
• Processes – diagnosis, investigation, treatments
• Outcomes – success of treatment, survival, continuing health
![Page 6: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/6.jpg)
Complexity
• Combination of– Medicine & health care– Information systems / IT– Evaluation methodology
• Each of these is a huge area
• Arguably IT is the simplest, or at least the most structured
![Page 7: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/7.jpg)
Of Medicine• Extremely large and growing area of
knowledge• Complex structure
– Equipment, staff, regulation
• Processes– Treatments etc.
• Outcomes– Long term, difficult to measure– Knock-on effects of innovation– Effect of IT particularly hard to measure
![Page 8: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/8.jpg)
Of Information systems
• Difficult to fully test– Combinatorial explosion
• Multi-function– Has a range of effects
• System itself vs. impact on health care
![Page 9: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/9.jpg)
Of Evaluation
• Have to measure impact– This means impact on people - difficult to study
• Need patients and staff to perform tests– May not be enough willing to cooperate
• Range of things that can be evaluated, ranging from– ‘Does it work?’ to– ‘Does it help patients?’
![Page 10: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/10.jpg)
Evaluation
• In theory, – study situation before & after
• In practice, – don’t know what changes would have occurred
without innovation– don’t know what interesting questions will arise
during study
![Page 11: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/11.jpg)
Tips
• Tailor study to problem– Not research – specific to this project
• Collect useful data– Data which inform final decision
• Look for side-effects– Effects not related to intended purpose
• Formative & summative– Study during & after development
![Page 12: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/12.jpg)
Tips (continued)
• In vitro vs. in vivo– Evaluate on-site & off-site
• Don’t accept developer’s view
• Take account of environment – context
• Let questions appear during study
• Be prepared to use a range of methods
![Page 13: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/13.jpg)
What can be studied• Need for resource
– What does it give us that we didn’t have before
• Development process– What methods do developers use to design their
solution?
• Structure of resource– What does the program & spec look like?
• Functions of resource– How well does it work?
• Impact– How does it affect HCPs and patients?
![Page 14: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/14.jpg)
Study features• Focus
– As previous slide
• Setting– Laboratory or hospital
• Data– Real or simulated
• Users– Developers, evaluators, end-users
• Decisions– None, simulated or real
![Page 15: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/15.jpg)
Types of study
• Need validation• Design validation• Structure validation• Laboratory function• Field function• Laboratory user impact• Field user impact• Clinical impact
![Page 16: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/16.jpg)
Objectivist or quantitative approach
• Can measure things objectively and without affecting thing being measured
• What to measure can be agreed rationally
• Can use numerical data
• Draw definite conclusions
![Page 17: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/17.jpg)
Objectivist approaches
• Comparison-based– Like randomised clinical trial
• Objectives-based– Does it do what the designers said?
• Decision facilitation– Answers questions posed by managers
• Goal-free approach– Evaluators not aware of project goals
![Page 18: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/18.jpg)
Methods
• Measurement
• Demonstration studies– Descriptive– Comparative– Correlational
• Statistical analysis
![Page 19: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/19.jpg)
Measurement studies
• Terminology for measurements– Object e.g. patient– Object class e.g. patient group– Attribute e.g. temperature– Instrument e.g. thermometer– Observation e.g. temperature at one time
• Validation – calibration of thermometer
![Page 20: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/20.jpg)
Demonstration studies
• Demonstrate effect– ‘Do patients who have been inoculated have a
higher temperature?’
• Object -> subject (patient)
• Attribute -> variable (temperature)
![Page 21: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/21.jpg)
Descriptive
• ‘The patients in this study have a rather high temperature’.
• Mean, standard deviation etc.
![Page 22: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/22.jpg)
Comparative
• ‘The patients in this study have a higher temperature than a control group’
• Controlled environment (usually)
• T-test etc.
![Page 23: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/23.jpg)
Correlational
• ‘We are seeing more patients with fever since we introduced inoculation’
• Live situation
• Could still be a t-test
• Trying to associate one factor with another in a real situation
![Page 24: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/24.jpg)
Subjectivist or qualitative
• Observations depend on observer
• Observations only meaningful in context
• Different points of view may be valid
• Descriptions as valuable as numbers
• Discussion of results
![Page 25: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/25.jpg)
Subjectivist approaches
• Quasi-legal– Cf ethical debate
• Art criticism– Expert review
• Professional review– Site visit
• Responsive/illuminative– Immersion in environment– Questions evolve over time
![Page 26: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/26.jpg)
Qualitative approach
• Attempts to understand why as well as measure differences e.g.– Is system working as intended?– How can it be improved?– Does it make a difference– Are differences beneficial?– Are the effects those expected?
![Page 27: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/27.jpg)
Stages in qualitative study
• Negotiation of ground rules
• Immersion into environment
• Initial data collection to focus questions
• Iteration
• Report and feedback
• Final report
![Page 28: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/28.jpg)
Methods in qualitative study
• Observation
• Interviews
• Document analysis
• Others, e.g. structured questionnaires
![Page 29: Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model](https://reader036.vdocuments.mx/reader036/viewer/2022070307/551ae636550346b2288b65c7/html5/thumbnails/29.jpg)
Mixed study
• Can combine qualitative and quantitative approaches