Jytte BrenderJytte BrenderAssociate Research ProfessorAssociate Research Professor

Virtual Centre for Health InformaticsVirtual Centre for Health Informatics & Aalborg UniversityAalborg University

An Overview of An Overview of Evaluation Evaluation MethodsMethods

Evaluation …is the act of measuring or exploring properties of a health information system (in planning, development, implementation, or operation),

the result of which informs a decision to be made concerning that system in a specific context

[Ammenwerth et al. 2004]

What is ’evaluation’?What is ’evaluation’?

measuring or exploring properties

the result of which informs a decision to be made

Metaphorically speaking, evaluation is like a torch being waved in the dark in front of you, • screening for options, obstacles and barriers, • diagnosing problems within a vast of observations, • prognosing risks, while • monitoring success and failure indicators,

but NOT:

… for treating identified obstacles and symptoms of failure

Evaluation is seen as a means to:• maximise success • minimize failure

My intention is to view available methods in the perspective

of the nature and needs of evaluation

• nature of different systems

• success & failure factors

• nature of evaluation methods

• pitfalls & perils in evaluation

inventory ofevaluation methods

low mental load(repetitive activities)

high mental load(cognitive activities)

high mental load(ergonomic activities)

EHCREHCREduc.Educ.

RIS/RIS/PACSPACS

LISLISKBSKBS

ES/DSSES/DSS

HISHIS PASPAS

• circumscribed tasks• repetitive tasks• many users• invariant staff

• complex tasks• data phlethora• decision support• fewer users• invariant staff

• complex tasks within a dynamically changing context• unique cases• decision support • many users• composite and variable staff

speed / effe

ctiveness

compre

hens

ibility

PP

assessment

focus

type ofsystemHIS/

PASLISRIS

EHCReducation

ergonomic ass.

cognitive ass.

KBSES/DSS

functionality ass.

impact ass.

techn. verification

Where are the success and failure indicators?Where are the success and failure indicators?

Everywhere!– but mainly within the many processes that lead to an IT-based solution

• Functionality• Organisation • Managerial• Behavioural• Political• Cultural• Technical• Publicity• Economy• Education

What are the barriers to success ?What are the barriers to success ?

Major Obstacles to Success Major Obstacles to Success [Crosswell 1991][Crosswell 1991]

Organisational coordination and conflictsData & software standardsPlanning / management supportTraining / understanding of technologyDatabase structure & source materialFunding availabilityData communication and networkingApathy / fear of changeSW complexity/maturityStaffing availability / recruitment

39 cases

5% affected

54% affected

20% affected

What are the barriers to success ?What are the barriers to success ?Obstacles to Success Obstacles to Success

[Price Waterhouse 1997][Price Waterhouse 1997]Insufficient management of change skillsCompeting resource prioritiesPresence of functional barriersLack of middle management supportInsufficient communicationEmployee oppositionInsufficient training and coachingNo perceived need for changeLong lead-time on IT solutionsInitiative fatigueBottom line benefits not understoodInsufficient involvement of employeesInsufficient focus on people issuesInappropriate leadership styleLack of clarity of objectives and vision

40% affected

20% affected

500 cases

Critical success factors !Critical success factors !

Critical Success CriteriaCritical Success Criteria[Price Waterhouse 1997][Price Waterhouse 1997]

Ensuring sponsoring from the topTreating employees fairlyInvolving employees throughoutQuality communicationQuality trainingClear performance measuresBuilding teams after changeFocus on culture / skill changesRewarding successInternal champions/change agentsManaging stakeholdersFormal project managementPhasing the changes

500 cases

rated highly important by 82%

rated highly important by 44%

Critical success factors !Critical success factors !

Top Indicators of Success Top Indicators of Success [Bikson & Evelyn 1989][Bikson & Evelyn 1989]

• User participation• Flexible planning balancing social and technical issues• Long-term learning support• Users identify the benefit from change for them• The change perceived as leading edge

55 cases

The people and organisational issuesseem to be a key figure to success

A system is”All the components, attributes and relationships needed to accomplish an objective”

[Haimes & Schneiter 1996]

Available methods and toolsAvailable methods and tools... introduction to an inventory of methods

icons that together characterises the

assessment method’sapplication area

within the lifecycle of the IT-system,

including the applicability for the

mentioned purpose.

The method’s application areais concerned with what it may be applied for.

What does the method include and how is this accomplished.

Necessary (non-trivial) conditionsfor application of the method with a useful result.

Aspects– like the philosophy behind the method,

including hidden assumptions – of importancefor the accomplishment or the interpretation of results,

Where, how and why one may risk specific pitfalls or perils.

Small tricks/advices that may help the accomplishment of the investigation or how to modify the method.

A complete list of references used from the literature, and of supplementary valuable references.

☺☺

Pitfalls and perils:

Tips and comments:

Assumptions for application:

Description:

References:

Perspectives:

Application area:<method name>

Frame of reference: What might serve as the frame of reference.

Questionnaireshas been used to ask for almost anything, but

should primarily be used to elicit qualitative measures of subjective aspects

☺☺ €(€)§(§§)

may different types of questions that may be combined at random

validation of the questions, and use of the right analysis tools

”Everyone can formulate questions”

Pitfalls and perils:

Tips and comments:

Assumptions for application:

Description:

References:

Perspectives:

Application area:

Frame of reference: Depends on the purpose of the study

• internal validity• context of response• psychological aspects• linguistic aspects• cultural aspects• who to ask

• validation of objectives fulfilment• impact/effect assessment• id. of problems in IT <-> organisation interaction

☺☺ €€€§§

Pitfalls and perils:

Tips and comments:

Assumptions for application:

Description:

References:

Perspectives:

Application area:

Frame of reference:

Functionality Assessment

• information access• diagramming experience

• the organisation changes dynamically• Leavitt’s model for organisational change• Mumford’s model for Participative Design

Built into the method

1. Other frameworks may be applied, like structure, process, outcome

2. Any diagramming technique may be applied

Brender J. Methodology for Assessment of Medical IT-Based Systems, IOS Press Vol 42, 1997

The causal analysis is vulnerable, because it may be difficult to access/elicit/verify the

true causal mechanisms

23

4

environmenttechnology

actors

task

structure

re-designof system

causalanalysis

=?

Divergences

Normativesystem

Step 1

Realsystem

Step 2

Step 3

Step 4

Step 5

• Assessment of system design and early prototypes• Assessment of demos

Cognitive walkthrough

Measurement of diagnostic accuracy and precisionClinical/diagnostic performance

Strategic managementBalanced Scorecard

A set of techniques for comparative assessment of bidsAssessment of bids

• Identification of where and why operation errors occur• Identification of focus point for improvement of user friendliness and usability

Cognitive assessment

• Trends assessment • Assessment of the future

Delphi

Critical, subjective assessment of an existing practiceBIKVA

Application purposeMethod

Inventory of methodsInventory of methods

Situation analysis aimed at identification of focus points for change

Future Workshop

Text-analysisGrounded Theory

Assessment of objectives fulfilment, and identification of focus points for change

Functionality assessment

Comparative judgement of development strategiesFramework for assessment of strategies

Observation of and within an organisation to identify practice and mechanisms controlling change

Field study

Combined assessment of degree of objectives fulfillment and of side effects (impact assessment)

Effect assessment

• Elicitation of patterns of attitudes, ways / modes of acting within/among social groups• Elicitation of problem areas within the organisation or the IT-based solution

Focus Group Interview

Application purposeMethod

Qualitative investigations of subjective aspects, like attitudes and opinions

Questionnaires

Measurement of trends, in particular effect of a specific initiative

Prospective time studies

Assessment of a User Requirements SpecificationRequirements Assessment

Identification of future scenariosPardizipp

Impact AssessmentModelling of work procedures

Incremental optimization of outcome to comply with policy and aim.

KUBI

(anything)Heuristic evaluation

Situation analysis, aiming at identifying focus points for change

Logical Framework Approach

Exploration of individuals’ attitudes, opinions, and perception of phenomena and observations

Interview

Application purposeMethod

Assessment of opinions, attitudes and perceptionsUser acceptance and satisfaction

Situation analysisSWOT

Assessment of relations between elements in an organisation, influencing acceptance and use of future solutions

Social network analysis

Evaluation of efficacy, i.e. that the IT-system (under ideal circumstances) improves patient care

RCT

Monitoring of risk aspectsRisk Assessment

Assessment of who to involveStakeholder analysis

• Analysis of complex patterns of interaction• Data acquisition and documentation in general

Video recording

Assessment of user friendliness in terms of ergonomic aspects

Usability

Systematic assessment of requirements fulfilment as part of an acceptance trial

Technical verification

Application purposeMethod

There are plenty of methods and frameworks, and a fair number of methodologies to address users’ need for evaluation

... but are they sufficient?

Metaphorically speaking, evaluation is like a torch being waved in the dark in front of you, • screening for options, obstacles and barriers, • diagnosing problems within a vast of observations, • prognosing risks, while • monitoring success and failure indicators,

My intention is to view available methods in the

perspective of the nature and needs of

evaluation

• nature of different systems

• success & failure factors

• nature of evaluation methods

• pitfalls & perils in evaluation

Inventory of evaluation methods

technicalaspects

people aspects

organisationalaspects

organisationalaspects

technicalaspects

people aspects

technicalaspects

people aspects

organisationalaspects

2D 3D

4D

... takes into account the complexity, like natural variation in daily work and practice

... takes into account both the complexity and the dynamically changing situation, as well as their interactions

structure

technology

actors

taskmanagem.main-

tenance

requestinganalytic.chemical

work

product.

calibr./calculat.

technic.inspect.

qualitymanagem.

needs,schedules

samples

rerun-information

rawdata interim

results interimresults

quality control criteria

change parameters

productioncontrol procedures

information

laboratory service order information

order

priority

So,

the challenge implied by the Information Age is the increasingly complex IT-solutions and the implied change management

... and

the challenge for the evaluation professionals is to continue developing appropriate methods,

while continuing to identify and exploreindicators for succes & failure

technicalaspects

people aspects

organisationalaspects

Another challenge: (typical) pitfalls and perilsAnother challenge: (typical) pitfalls and perils

Pitfalls and perils were identified within themedical and social sciences

- all have analogues at experimental assessment of IT-based solutions within the healthcare sector- very few evaluation studies for IT-based solutions within the healthcare sectorsucceed to avoid even the simplest ones

Study design biasesStudy design biases

Aspects of culture

Circumscription of study objectives

Selecting the study methodology/method

Defining methods & materials

(User) Recruitment

(Case) Recruitment

The frame of reference

Outcome measures or end-points

Construct validity of methods, techniques and tools

Prophylactic grounding

Prognostic measures

The (un-)used user

Diagnostic suspicion

The placebo effectHawthorne Effect

Co-interventionA before-and-after study

Carry-over EffectCheck-list Effect

Human cognition/recognitionUsability and user acceptability(Adverse) confounding effects

More than one population included as the subject of investigation

Compliance between application domain and target domain

Use of artificial scenarios or set-up

Subjectivity versus objectivityAssumptions for methods’ application

Circular inferenceApplication range

The Substitution Game

Comparing apples and pears

Technical verification of outcome measures

Constructive assessment

Complex causal relations

Delimitation of the investigation spaceAmbiguous or changing study objectives

Stakeholder analysisTargeting the methodology/method to the study aim

Biases related to data collection, Biases related to data collection, analysis and interpretationanalysis and interpretation

Beyond the developer’s point of viewUnstable technology

Hindsight or insight biasJudgement of the probability of eventsJudgement of performanceMiscellaneous judgemental biases

Assessment BiasEvaluation paradox

Constructive assessmentAction-case research

Half of the storyPresentation of the future

Exhaustion of the data material

A publication dilemma

Matching of control and study groupsImplicit domain assumptions

Competing risks

False-Negative StudiesFalse-Positive studies

Validation is not black-and-whiteRandonised controlled trialsAction-case Research

The developers’ actual engagementIntra- and Inter-person (or -case) variabilityIllicit UseFeedback EffectExtra WorkJudgmental biasesPost-rationalisationVerification of implicit assumptionsNovelty of the technology – technophile or technophobeSpontaneous regressFalse conclusionsIncomplete studies or study reportsHypothesis fixationThe ‘Intension to treat’ principleImpact

... and then what is ...The practical implication?The practical implication?

It may meanIt may mean

-- everything!everything!-- or nothing!or nothing!

Flaws may completely ruin a studyFlaws may completely ruin a study-- examplesexamples

Two presentations on the same IT-system had opposite conclusions

Triangulation proves that one cannot trust the users’ statements

Some uses t-test for verification of the significance at questionnaire studies with ordinal ratings

You cannot – with sense – compare just anything

The users don’t do as they say they do, ... but does this matter when it is their reality that has to be judged for the IT-system?

If the users express that an IT-system is good, does it then matter that someone can prove the opposite?

A lot of the pitfalls and perils have no or marginal effect in a given evaluation study

Flaws may have no or minor implicationFlaws may have no or minor implication-- examplesexamples

The issue of evaluation is not simple!

jyttejytte..brenderbrender@v@v--chi.chi.dkdk