mie2009 good evaluation practice workshop pirkko nykanen 1 guidelines for good evaluation practices...

1MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Guidelines for Good Evaluation Practices in

Health Informatics - A shared networked initiative

EFMI and IMIA

CoordinatorsPirkko Nykänen, Tampere University

Jytte Brender, Aalborg University


Motivation• Need to have good practice guidance

for evaluation • No single, global methodology exists• Good practice guidelines should

serve as:– Framework to design evaluation studies, to

select methodologies, to conduct studies– Support for health professionals and users to

understand evaluation and contribute to evaluation studies


Objectives• Generic practical guidelines that

provide evaluators, users, health professionals with structured, scientifically justified, comprehensive and understandable set of rules for good practice– To design and carry out evaluation studies in

health informatics domain


Objectives – practically• Guidelines:

– Criteria and aspects, both in quantitative and qualitative terms, to consider at each evaluation stage

– Carry out evaluation of the specific aspect, criteria at specific stage

– Design and manage the evaluation study

• Guidelines from all stakeholders viewpoints: third party evaluators, users, health professionals, managers, decision makers, health economists,...


How GEP-HI guidelines have been developed

• Existing guidelines development principles• Existing literature and material on evaluation

studies, methodologies, reported evaluation experiences, guidelines for good clinical practice, codes of ethics, good implementation practices critical study and assessment, review

• Spiral, consensus making approach: – Coordinators - editorial team - external experts by

HISEVAL website / mailing list – Revised versions – comments – input – revisions…– Presentations / workshops in MIE, MEDINFO-

confereneces


ExplorationReport Buyer

Team: iterative discussion

and consensus

Selection of MethodsInternalReport

First Study DesignReport Buyer

Study DesignIncluding:

Goal and Mandate Report Buyer

Participants

InternalReports

InterimReports

Buyer

(?) Participants

Final Study Report Buyer

Participants

Pub

lica

tion

1...

n

Study ImplementationThe Study

Project Controlling& Risk Management

Receiver

C Weßel 2006-12-06 GEP-HI Suggestion for a Flowchart

First start GEP-HI guidelines


First studydesign

Studyexploration

Operational.of methods

Detailed Planning

EvaluationImplement.

Funder/ client

Proj.Managem.

Proj.Managem.

Proj.Managem.

Proj.Managem.

Eval.consultant

Eval.consultant

Eval.consultant

Eval.consultant

Objectorganisation

Evaluationteam

Evaluationteam

Evaluationteam

Evaluationteam

Informationneed

Agreement Designoutline

Method Plans &procedures

Finalisation of the evaluation study.

GEP_HI guidelines flowchart

Project control and risk management


GEP-HI guidelines• Study exploration

– Starting question • First study design

– Preliminary design• Operationalisation of methods

– methodological approach, methodical aspects• Detailed study plan

– detailed planning • Evaluation study implementation

– Actual accomplishment of the planned evaluation study• Project control and risk management

– Good project management practices• Finalisation of the evaluation study

– Accounting, archiving, reporting of evaluation study results


Status now and progress

• Guidelines-paper has been accessible on iig.umit.at/efmi-website since 2006

• Comments and input received• Final revision is ongoing

– Final version will be available by end 2009

• Comments and input received will be incorporated and the guidelines finalised for publication


Contact

• Feedback and input [email protected] [email protected]

• paper accessible: iig.umit.at/efmi

Editorial team: Elske Ammenwerth, Nicolette de Keizer, Jan Talmon, Michael Rigby, Jytte Brender, Pirkko Nykänen


A walk-through of the key guidelines issues

Pirkko Nykänen, Tampere UniversityJytte Brender, Aalborg University


First studydesign

Studyexploration

Operational.of methods

Detailed Planning

EvaluationImplement.

Funder/ client

Proj.Managem.

Proj.Managem.

Proj.Managem.

Proj.Managem.

Eval.consultant

Eval.consultant

Eval.consultant

Eval.consultant

Objectorganisation

Evaluationteam

Evaluationteam

Evaluationteam

Evaluationteam

Informationneed

Agreement Designoutline

Method Plans &procedures

Finalisation of the evaluation study.

GEP_HI guidelines flowchart

Project control and risk management


Guidelines summary

1. Study exploration2. First study design3. Operationalisation of methods4. Detailed study plan and project plan5. Evaluation study implementation

5.1 Project control and risk management

6. Finalisation of the evaluation study


Phase 1: Study explorationfocuses on the starting question of the evaluation study

1.1 The information need1.2 Primary audience 1.3 Identification of the buyer / sponsor / study funding party1.4 The context of the evaluation study1.5 A first identification of stakeholders1.6 A first identification of (external) consultants1.7 A first sketch of the setting1.8 First exploration of evaluation methods to be used 1.9 Exploring the restrictions of study execution and publication 1.10 Budget1.11 Ethical, moral and legal issues

1.12 Result of Study Exploration= Outline, draft design of the study

1.13 Formal accept to proceed to the next phase


Phase 2: First study design

focuses on the preliminary design of the evaluation study

2.1 Elaboration of the rationale for the study2.2 Key evaluation issues/questions2.3 Budget2.4 Establishment of the design team2.5 Stakeholder analysis/Social Network analysis2.6 Study constraints2.7 Methods2.8 Organisational setting, the study context2.9 Technical setting, the study context2.10 Participants from the organisational setting2.11 Material and immaterial resources2.12 Time and timing2.13 Risk analysis2.14 Ethical, moral and legal issues2.15 Strategy for reporting and dissemination of results

2.16 Result of First Study Design = a more detailed evaluation study plan


Phase 3: Operationalisation of methods

focuses on making the design and evaluation methods concrete and compliant with the organizational setting and the information need, while taking into account the known pitfalls and perils

3.1 Study type 3.2 Approach3.3 Assumptions3.4 Pitfalls and perils3.5 Skills3.6 Frame of reference3.7 Timing3.8 Justification3.9 Usability studies3.10 Outcome measures3.11 Quality Control on data (measures)3.12 Participants3.13 Study flow

3.14 Result of Operationalisation of Methods = Decisions on the methodological and methodical approach

3.15 Ethical, moral and legal issues = required approvals, ethical committee permissions achievable and

prepared


Phase 4: Detailed study plan and project plan

focuses on providing plans, prescriptions and procedures detailed to the level necessary for the specific study

4.1 Project management4.2 Evaluation activity mapping4.3 Quality management4.4 Risk management4.5 Communication strategy4.6 Recruitment of necessary additional staff

4.7 Result of Detailed Study Plan and Project Plan =

Practical plans with incorporated tools. Upon approval of plans the evaluation study can now be started


focuses on activities related with the actual accomplishment of the designed evaluation study

5.1 Establishment of the frame of reference5.2 Observation of changes5.3 Quality control of findings5.4 Interpretation of observations5.5 Continuous project management, quality management and

risk management5.6 Regular reports

5.7 Final result of Evaluation Study Implementation = The final evaluation report summarising the study plan and results, and presenting conclusions

Phase 5: Evaluation study implementation


Phase 5.5 Project controlling and risk management

Focuses on the good project management practices and risk management specifically for an evaluation study

- Dedicated project management tools - Quality management in relation to the objectives

and milestones of the study > Develop a quality management plan

- Risk management - Prospectively during planning- Retrospectively with the risks realised


Phase 6: Finalisation of the evaluation study

focuses on accounting, archiving of materials, and reporting of evaluation studies in terms of the STARE-HI guidelines

6.1 Accountings6.2 Reports and publications6.3 Archiving6.4 Reporting guidelines6.5 Reporting scope6.6 The reporting message6.7 Authorship6.8. Ethical and moral aspects6.9 Preparation of reports / publications

- Follow STARE-HI guidelines / Statement on reporting of evaluation studies in health informatics, IJMI 78, 2009, 1-9

6.10 Usability documentation in the context of the EU regulations and standards


Generalisability of the guidelines

• Designed for all types of evaluation study types, also applicable on usability evaluation study planning

• The guidelines are for– to plan an evaluation study– to carry out an evaluation study

• The guidelines are NOT for– Which methods to use– How to use the methods

• Handbooks available, see e.g. Brender 2006

mie2009 good evaluation practice workshop pirkko nykanen 1 guidelines for good evaluation practices...

Documents

evaluation study guidelines

evaluation studies

good evaluation practices

evaluation stage

evaluation experiences

good practice guidelines

good clinical practice

good practice guidance