Eindhoven, 5-12-2012

M.J.A.J. (Michel) Koeken Student ID: 0722653

in partial fulfillment of the requirements for the degree of

Master of Science

in Operations Management and Logistics

Supervisors: Dr. P.M.E. (Pieter) Van Gorp TU/e - IE&IS Dr. M. (Marco) Comuzzi TU/e - IE&IS Mr. C.J. (Cor) Rumping Siemens – Director IT Healthcare

Identifying and ranking the selection criteria in European tenders regarding the procurement of HIS/EHR systems in Dutch academic hospitals

TUE. School of Industrial Engineering. Series Master Theses Operations Management and Logistics Subject headings: European tender, academic hospital, Hospital Information System, Electronic Health Record

- iv -

Preface The report lying in front of you is the result of my graduation project to obtain the degree of Master of Science in Operational Management and Logistics at the Eindhoven University of Technology. This graduation project is done in cooperation with Siemens Den Haag. I take this opportunity to thank some people for their contribution to this graduation project. First of all I would like to thank my first supervisor, Pieter Van Gorp, for his input, nonstop attention and energy he put into my project. He was always available to help whenever I got questions and provided useful feedback. Furthermore, I would like to thank Marco Comuzzi for being my second supervisor and providing me with useful advice. In the third place, I would like to thank my supervisor from Siemens, Cor Rumping, as he was always available for questions, help, advice, and input. Fourthly, I would like to thank my friends for their support, advice and sympathy. Last but not least, I would like to give a special thanks to my family, especially my parents, who gave me constant support and motivation and made it possible to complete my study without any troubles. Michel Koeken December 2012

- v -

Management summary The thesis focused on the procurement of Hospital Information System / Electronic Health Record (HIS/EHR) systems at Dutch academic hospitals. As Dutch academic hospitals are considered to be a public institution and the value of the procurement of a HIS/EHR system exceeds the threshold value of €200,000 they are obliged by the BAO (Besluit Aanbestedingsregels voor Overheidsopdrachten) to conduct a European tender. The BAO is the Dutch implementation of the European guideline 2004/18/EG. Each European tender contains a Research For Proposal (RFP). The RFP provides a detailed set of requirements of what the academic hospital wants to acquire and the score model that the academic hospital will use to evaluate the proposals of the vendors. The detailed set of requirements is divided into subtopics and these subtopics are considered as the selection criteria. The score model states how much points can be scored by the vendors by fulfilling a specific selection criterion. The vendor with the highest score and therefore the most economically advantageous offer (i.e. best price-quality ratio) wins the tender. In The Netherlands are three European tenders conducted by the Dutch academic hospitals in order to acquire a HIS/EHR system. All three European tenders were won by different vendors and this is interesting when one take into account that:

- a vendor had to be selected through a strict tendering process which is based on the three principles (equal treatment, transparency, and non-discrimination) and therefore ensures for total transparency since all decisions are open for involved parties, no possibility for negotiation, and an equally review of every vendor;

- the Dutch academic hospitals did not differ that much; - and there were only five (six at the time of the first two European tenders) available vendors

from which the Dutch academic hospitals could choose from. One would expect that at least two out of the three European tenders were won by the same vendor, but this was not the case. Therefore it is interesting to analyze these European tenders to investigate how Dutch academic hospitals used the European tenders to come to their results. To answer the raised issue the thesis started with analyzing the three score models in the RFPs in order to identify and rank the selection criteria in terms of importance. The three included RFPs were those from the European tenders conducted by Leiden University Medical Center + University Medical Center Utrecht, University Medical Center St. Radboud, and Erasmus Medical Center + University Medical Center Groningen. This analysis showed that the three European tenders could not be compared one on one as they used slightly different terminology and wording for their selection criteria. This problem was tackled by selecting one European tender as the reference tender and the selection criteria of the other two European tenders were matched with the selection criteria of the reference tender. This matching was done by two vendor experts and followed the two round Delphi method (Skulmoski, Hartman, & Krahn, 2007). Now the three European tender could be compared as they contained the same selection criteria. The comparison showed that the European tenders scored their selection criteria differently. Table 1 lists the top 10 selection criteria based on the last issued European tender. The table shows the names of the selection criteria and the percentage of the total score a specific selection criterion accounted for per European tender.

Table 1 – Top 10 selection criteria based on the last European tender

Name selection criteria Tender B Tender A Tender C

Wet en regelgeving 0,00% 1,47% 10,08%

Klinische dossiervoering 4,78% 5,17% 7,67%

Innovatie 10,81% 0,80% 6,98%

Administratie Medisch wetenschappelijk onderzoek 5,65% 5,17% 5,35%

Administratie en monitoring onderwijs 2,84% 0,52% 4,88%

Medicatie 0,27% 5,17% 4,65%

Communicatie / Correspondentie 1,49% 0,52% 4,36%

Portalen 1,24% 5,17% 4,21%

Documentatie en templates 7,76% 1,33% 4,13%

Conversie 3,24% 1,33% 4,07%

- vi -

As academic hospitals are often an example for general hospitals this identification and ranking could be an important insight for general hospitals. It is even possible that general hospitals use this as a blueprint for determining their own selection criteria. Obviously also other academic hospitals can use this insight to their advantage and consultancy companies can use it in advising their clients. The differences between the percentages were analyzed with the help of a vendor expert and as the tenders occurred sequentially trends were determined. It is important to state that the trends could be based on coincidence as there were only three available data sets. Nonetheless it is interesting and useful to discuss the possible trend and especially the most important ones. These are listed in table 2 with their possible explanation.

Selection criteria Explanation

Wet en regelgeving The academic hospitals tried to achieve more commitment from the vendors; therefore this selection criterion became more important over time.

Decision Support This selection criterion scored surprisingly low, the vendor expert thought it would be much more important as hospitals spoke a lot about it.

Medicatie The rising trend was due to the fact that law changed over time. Per January 1st 2013 a law will be activated which states that the medication has to be prescribed electronically. The first tender did not had to consider this law, while the other two tenders had to. Therefore it became more important over time.

Portalen This selection criterion just entered the market at the time of the first tender and it became more important since then.

Beheer ‘ Monitoring en reporting

In the past the academic hospitals did it themselves, but the trend is that they want to outsource as much as possible to relieve themselves and to be more efficient, therefore both selection criteria became more important over time.

Implementatie en bedrijf gerede oplevering ’ Plan van aanpak voor de implementatie en bedrijf gerede oplevering

In the past the academic hospitals wanted full flexibility during their client specific implementation. Nowadays the academic hospitals want to make use of standard implementations; therefore both selection criteria show a declining trend.

Contractvoorwaarden project

The academic hospitals tried to achieve more commitment from the vendors by making the selection criteria more important; the last tender even used it as a knockout criterion.

Table 2 – The most important trends

The possible trends can be beneficial for every stakeholder in The Netherlands or elsewhere as they show which topics are hot or not. Vendors could use it to determine their business strategy, the consultancy companies can advise their clients about it, and the hospitals can take it into account in their procurement processes.

- vii -

Table of contents Preface .................................................................................................................................................... iv

Management summary ............................................................................................................................ v

1 Introduction .................................................................................................................................... 1

1.1 Procurement procedures ........................................................................................................ 1

1.1.1 Direct order ..................................................................................................................... 1

1.1.2 Tendering ........................................................................................................................ 1

1.1.3 Quotation-based methods .............................................................................................. 3

1.2 Information Systems in the healthcare industry .................................................................... 4

1.3 Hospital Information System / Electronic Health Record system ........................................... 6

1.4 Research questions ................................................................................................................. 9

1.5 Thesis structure ....................................................................................................................... 9

2 Project context .............................................................................................................................. 10

2.1 Scope ..................................................................................................................................... 10

2.2 Design alternatives ................................................................................................................ 10

2.3 Research method and model ................................................................................................ 11

2.3.1 Delphi method .............................................................................................................. 11

2.3.2 Research model ............................................................................................................. 12

3 Literature study and backgrounds ................................................................................................ 14

3.1 Lack of consensus .................................................................................................................. 14

3.2 Literature study method ....................................................................................................... 14

3.3 Results first and second phase .............................................................................................. 15

3.4 Reference Concept Map ....................................................................................................... 16

3.4.1 RCM for EHR definition ................................................................................................. 18

3.4.2 RCM for EMR definition ................................................................................................ 20

3.4.3 RCM for HIS definition .................................................................................................. 20

3.5 Results of Reference Concept Maps ..................................................................................... 20

4 Tender analyses ............................................................................................................................ 22

4.1 Analysis of tender A .............................................................................................................. 22

4.2 Analysis of tender B .............................................................................................................. 22

4.3 Analysis of tender C .............................................................................................................. 23

4.4 Match selection criteria ........................................................................................................ 23

4.5 Matching results ................................................................................................................... 25

5 Result tender analyses .................................................................................................................. 27

5.1 Rank the selection criteria .................................................................................................... 27

5.2 Analyze the matching results ................................................................................................ 29

5.3 Fairness of tender ................................................................................................................. 34

5.4 Contribution of TCO on the total score................................................................................. 35

5.5 Opinions regarding the use of tenders ................................................................................. 35

- viii -

6 Conclusions and future work ........................................................................................................ 36

6.1 Conclusion ............................................................................................................................. 36

6.2 Limitations............................................................................................................................. 37

6.3 Future work ........................................................................................................................... 38

Bibliography .......................................................................................................................................... 39

Appendices ............................................................................................................................................ 44

Appendix A Interview University Medical Center St Radboud ..................................................... 45

Appendix B Activity diagram research method HIS ..................................................................... 46

Appendix C Results Publish or Perish EMR + HIS ......................................................................... 47

Appendix D Rephrased definition regarding EHR (figure 9) ......................................................... 48

Appendix E Rephrased definition regarding EMR (figure 22) ...................................................... 51

Appendix F Reference Concept Map regarding EMR ................................................................... 54

Appendix G Rephrased definition regarding HIS (figure 23) ........................................................ 55

Appendix H Reference Concept Map regarding HIS ..................................................................... 57

Appendix I Score of considered (sub) topics tender A ................................................................ 58

Appendix J Score of considered (sub) topics tender B ................................................................ 59

Appendix K Score of considered (sub) topics tender C ................................................................ 60

Appendix L Matching tender B with tender A ............................................................................. 61

Appendix M Matching tender C with tender A ............................................................................. 62

Appendix N Result matching tender B with tender A ................................................................... 63

Appendix O Result matching tender C with tender A ................................................................... 65

Appendix P Ranking selection criteria of tender A before the matching..................................... 66

Appendix Q Ranking selection criteria of tender B before the matching ..................................... 67

Appendix R Ranking selection criteria of tender C before the matching ..................................... 68

Appendix S Ranking selection criteria after matching based on tender A .................................. 69

Appendix T Ranking selection criteria after matching based on tender B ................................... 70

Appendix U Ranking averaged selection criteria of the tenders after the matching ................... 71

Appendix V Explanation of trends ................................................................................................ 72

- ix -

List of Figures Figure 1 – Open tender ........................................................................................................................... 2 Figure 2 – Restricted tender ................................................................................................................... 2 Figure 3 – Competitive dialogue ............................................................................................................. 3 Figure 4 – Modular design Hospital Information System in The Netherlands ........................................ 6 Figure 5 – Structure of the RFP of tender A ............................................................................................ 8 Figure 6 – Delphi method ..................................................................................................................... 11 Figure 7 – Research model .................................................................................................................... 13 Figure 8 – Activity diagram literature research method EHR and EMR ................................................ 15 Figure 9 – RCM regarding the definition of an EHR .............................................................................. 18 Figure 10 – Percentage subtopic Wet en regelgeving accounted for ................................................... 29 Figure 11 – Percentage subtopic Decision support accounted for ....................................................... 30 Figure 12 – Number of published articles in PubMed per year with topic Decision support ............... 30 Figure 13 – Percentage subtopic Medicatie accounted for .................................................................. 31 Figure 14 – Percentage subtopics Implementatie en bedrijfsgerede oplevering and Plan van aanpak voor implementatie en bedrijfsgerede oplevering accounted for ........................................................ 31 Figure 15 – Percentage subtopic Portalen accounted for .................................................................... 32 Figure 16 – Number of published articles in PubMed per year with topic Portal hospital .................. 32 Figure 17 – Percentage subtopics Beheer and Monitoring en Reporting accounted for ...................... 33 Figure 18 – Number of published articles in PubMed per year with topic Monitoring + Reporting .... 33 Figure 19 – Percentage subtopic Contractvoorwaarden project accounted for .................................. 34 Figure 20 – Percentage subtopic TCO accounted for ........................................................................... 35 Figure 21 – Activity diagram research method HIS............................................................................... 46 Figure 22 – RCM regarding the definition of an EMR ........................................................................... 54 Figure 23 – RCM regarding the definition of an HIS ............................................................................. 57

List of Tables Table 1 – Top 10 selection criteria based on the last European tender ................................................. 5 Table 2 – The most important trends ..................................................................................................... 6 Table 3 – Threshold values that the contract has to exceed to make tendering mandatory ................. 1 Table 4 – Dutch academic hospitals ........................................................................................................ 7 Table 5 – Example tender C subtopic matching.................................................................................... 12 Table 6 – Matching by HIS vendor expert 1 .......................................................................................... 12 Table 7 – Matching by HIS vendor expert 2 .......................................................................................... 12 Table 8 – Example tender A subtopic matching ................................................................................... 12 Table 9 – Best matching discussed by HIS expert 1 and 2 .................................................................... 12 Table 10 – Similar matching after first round ....................................................................................... 12 Table 11 – Example tender A subtopic ................................................................................................. 12 Table 12 – Results Publish or Perish EHR search queries ..................................................................... 16 Table 13 – Google hits regarding the definition of an EHR ................................................................... 18 Table 14 – Quick EHR reference list ...................................................................................................... 19 Table 15 – Points and relative score per subtopic ................................................................................ 22 Table 16 – Starting information tender T1 ........................................................................................... 24 Table 17 – Starting information tender T2 ........................................................................................... 24 Table 18 – Starting information tender T3 ........................................................................................... 24 Table 19 – Matching of tender T2 with tender T1 ................................................................................ 24 Table 20 – Matching of tender T3 with tender T1 ................................................................................ 24 Table 21 – Result matching tender T2 with tender T1 ......................................................................... 25 Table 22 – Result matching tender T3 with tender T1 ......................................................................... 25 Table 23 – End result example .............................................................................................................. 25

- x -

Table 24 – Preview result matching tender C with tender A ................................................................ 26 Table 25 – Top five selection criteria tender A ..................................................................................... 27 Table 26 – Top five selection criteria tender B ..................................................................................... 27 Table 27 – Top five selection criteria tender C ..................................................................................... 27 Table 28 – End result: ranking selection criteria after matching based on tender C ........................... 28 Table 29 – Fairness tender .................................................................................................................... 34 Table 30 – The nine most important trends ......................................................................................... 37 Table 31 – Interview with strategic purchaser of University Medical Center St Radboud ................... 45 Table 32 – Results Publish or Perish EMR search queries..................................................................... 47 Table 33 – Results Publish or Perish HIS search queries ....................................................................... 47 Table 34 – Original and rephrased definitions of an EHR ..................................................................... 50 Table 35 – Original and rephrased definitions of an EMR .................................................................... 53 Table 36 – Google hits regarding the definition of an EMR .................................................................. 54 Table 37 – Original and rephrased definitions of a HIS ........................................................................ 56 Table 38 – Google hits regarding the definition of a HIS ...................................................................... 57 Table 39 – Score considered (sub) topics in tender A........................................................................... 58 Table 40 – Score considered (sub) topics in tender B ........................................................................... 59 Table 41 – Score considered (sub) topics in tender C ........................................................................... 60 Table 42 – Matching subtopics of tender B with subtopics of tender A .............................................. 61 Table 43 – Matching subtopics of tender C with subtopics of tender A............................................... 62 Table 44 – Result matching tender B with tender A ............................................................................. 64 Table 45 – Result matching tender C with tender A ............................................................................. 65 Table 46 – Ranking original selection criteria of tender A .................................................................... 66 Table 47 – Ranking original selection criteria of tender B .................................................................... 67 Table 48 – Ranking original selection criteria of tender C .................................................................... 68 Table 49 – Ranking selection criteria after matching based on tender A ............................................. 69 Table 50 – Ranking selection criteria after matching based on tender B ............................................. 70 Table 51 – Ranking average selection criteria after matching .............................................................. 71 Table 52 – Explanation differences ....................................................................................................... 73

- 1 -

1 Introduction The thesis focuses on the use of procurement procedures by hospitals to determine the most appropriate information system. How this topic is selected as research topic, what it means and the potential benefits for the reader will become clear in the following paragraphs. First the different procurement procedures are explained followed by the information systems in the healthcare industry and finally the research questions and thesis structure are presented.

1.1 Procurement procedures In order to procure the best possible product/service/works, e.g. information systems, at the right time and on the most favorable terms (e.g. price, quality, delivery time) it is vital that companies (client(s) in the remaining of the thesis) select the right procurement procedure (Council, 2009, Temple-Bird & Parsons, 2006). The three common procedures are (Council, 2009, Temple-Bird & Parsons, 2006):

- Direct order - Tendering - Quotation-based methods

1.1.1 Direct order Direct ordering is the most informal and easiest procedure and is used when clients know exactly what they want, which vendor to approach, and how to negotiate conditions. This procedure is typically used when the value of the procurement is up to ± €5,000.00 (Council, 2009, Temple-Bird & Parsons, 2006). Disadvantages are that the client is unable to compare prices of other vendors, no opportunity to see what other products and deals were available, and it is most open to corruption (Temple-Bird & Parsons, 2006).

1.1.2 Tendering Whenever the value of the procurement exceeds a defined threshold value, see table 3 (EU-Commission, 2004), and the client is considered to be a public institution he is obliged by law to undertake a tendering process. The thesis is limited for procurement procedures within Europe to ensure that clients are obliged to the same regulations. In Europe tendering processes are regulated by the European guideline 2004/18/EG. A client is considered a public institution when it meets the following three criteria (EU-Commission, 2004):

- The client is established for the specific purpose of meeting needs in the general public interest, and;

- The client has legal personality, and; - Or the activities of the client are for more than 50% funded by the government, or the

management of the client is supervised by the government, or at least 50% of the board of the client is designated by the government.

2012-2013 2010-2011 2008-2009 2006-2007

€200,000.00 €193,000.00 €206,000.00 €211,000.00 Table 3 – Threshold values that the contract has to exceed to make tendering mandatory

The general goal of tendering is to select the most economically advantageous offer (i.e. the best price-quality ratio). In order to achieve this, the European tenders (tender(s) in the remaining of the thesis) are based on the following three principles (EU-Commission, 2004):

- Equal treatment (the public institution must act fairly: all vendors must have an equal opportunity to compete for the contract)

- Transparency (use of transparent decision making in order to show that the public institution is following the principle of equal treatment)

- Non-discrimination (prohibit discrimination on the basis of nationality, grant freedom of establishment, and establish the freedom to provide services)

- 2 -

There are different tender types (Canton, de Bas, Meindert, & others, 2012, EU-Commission, 2004): - Open tender - Restricted tender - Competitive dialogue - Negotiated procedure

The most used tender types are the open and restricted tender. The competitive dialogue and negotiated procedure are used in particularly complex cases, when clients expect that the open and restricted tender types are not possible, e.g. when there is no standard solution for a particular problem (Canton et al., 2012). Open and restricted tender Open tenders are intended for every interested vendor and therefore the client can receive many responses. In a restricted tender the number of vendors is restricted by the client who eliminates vendors who are unsuitable (EU-Commission, 2004). The open tender is by far the most used tender type (Canton et al., 2012). These two types are visualized in figure 1 and figure 2.

Figure 1 – Open tender

Figure 2 – Restricted tender

The open tender procedure starts with the announcement by the client in trade journals and/or websites. Interested vendors have to call for a Request For Proposal (RFP in the remaining of the thesis). A RFP provides detailed specifications of what the client wants to buy and the evaluation criteria that the client will use to evaluate the proposals of the vendors (Mandell, 1986). The vendor with the most economically advantageous offer wins the tender. The restricted tender adds a step between the announcement and the RFP, namely the Request For Information (RFI in the remaining of the thesis). In the RFI the client request potential vendors to provide general business information and based on the RFI the client can decide to invite or exclude a vendor for the RFP (EU-Commission, 2004). (EU-Commission, 2004) requires that (assuming that there are five or more vendors) at least five vendors should be invited for the RFP. Competitive dialogue The competitive dialogue is a tender type where clients conduct a dialogue with candidate vendors with the aim of developing one or more suitable alternatives capable of meeting the clients’ requirements. Vendors are invited for the RFP based on this dialogue (EU-Commission, 2004). The tender type is visualized in figure 3. The competitive dialogue starts with the announcement by the client in trade journals and/or websites, where all vendors may request to participate. Only those vendors invited by the client may start the dialogue and on basis of the dialogue the client invites vendors to call for the RFP. The contract is awarded to the winner of the tender.

- 3 -

Figure 3 – Competitive dialogue

Negotiated procedure In the negotiated procedure clients consult the vendors of their choice and negotiate the terms of contract with one or more vendors (EU-Commission, 2004). This is the least used tender type as it may only be used in very special cases (Canton et al., 2012). The negotiated procedure is left out of scope during the thesis as it is rarely used in special cases. (Dis) advantages of tendering The advantages of tendering are that it is formal and regulated, it limits the risk for corruption as it exclude negotiation and it provides total transparency since all decisions are open for involved parties, it ensures that each vendor’s proposal is threatened confidential, and it includes formal written procedures and rules to which vendors confirm to agree to. Disadvantages are that it requires a high level of purchasing expertise as any weak element in the tender might result in the forced selection of a vendor supplying questionable product/service. It is also a slow and expensive process in terms of administration (Temple-Bird & Parsons, 2006).

1.1.3 Quotation-based methods Whenever the value of the procurement is above ± €5,000.00 and less than the threshold values of table 3, a number of proposals will be obtained to select the appropriate vendor (Council, 2009). The three common quotation-based methods are (Temple-Bird & Parsons, 2006):

- Request for quotes - National competitive bids - Competitive negotiation

Request for quotes The request for quotes method is the most straightforward version as a minimum of three vendors has to be approached and invited to submit a proposal for specified goods and/or services. The proposals are reviewed by the client and the best most appropriate one is selected (Temple-Bird & Parsons, 2006). National competitive bids The national competitive bids method is similar to the open tender procedure, but it does not have to be transparent or provide evaluation criteria and the announcements are limited to the applicable country. This saves a lot of time (Temple-Bird & Parsons, 2006).

- 4 -

Competitive negotiation With the competitive negotiation method the client approaches potential vendors of the required goods and/or services and invites them to submit a proposal. The client may use multiple negotiation rounds to get better deals. The client is even allowed to inform vendors about the proposals of other vendors to gain better deals (Temple-Bird & Parsons, 2006). The advantages of the quotation-based methods are that they are quicker and less formal than tenders and there is room for negotiating the terms of the proposal directly with vendors. The disadvantages are the lack of transparency, risk of dispute about contract terms as the purchase document may be less detailed, and risk of accusations about unfair treatment from unsuccessful vendors (Council, 2009, Temple-Bird & Parsons, 2006).

1.2 Information Systems in the healthcare industry The described procurement procedures can be applied in many different industries, but this thesis focused on the healthcare industry. The healthcare industry is interesting as it is facing several pressing issues: the ageing population (in 1950 8% of the world population was 60 years and older, it was 10% in 2000 and it will be 21% in 2050) (Haux, 2006, UN, 2002), people demand better care (Haux, 2006), and the potential support ratio, i.e. the number of persons between 15 and 60 years which are available to support one person above 60 years, is decreasing. The ratio was 12:1 in 1950, it was 9:1 in 2000 and it is estimated for 2050 to be 4:1 (Haux, 2006, UN, 2002). The healthcare industry has to find ways to deal with these potential trends. One way is the implementation of information systems (IS in the remaining of the thesis) for hospitals as they can not only reduce errors and increase speed of care and accuracy but they also can lower health costs by coordinating services and improving quality of care, improve effectiveness and efficiency, and contribute to increased life expectancy (Malliarou & Zyga, 2009, Temple-Bird & Parsons, 2006, A. Winter et al., 2010, 2011, S. Wu et al., 2006). However not everyone is positive about IS in hospitals: (Boonstra & Broekhuis, 2010, Handel, Wears, Nathanson, & Pines, 2011, IGZ, 2011) state that poor IS implementation can actually lead to a lower quality of care, higher costs and that it does not automatically lead to improvements in information provision. Another potential drawback is the ease of operation. Users should be able to deal easily with the IS and the learning process should be as short as possible (Fledderus, 2007). But when the IS implementation is carried out appropriately a lot of these (potential) drawbacks can be minimalized (Boonstra & Broekhuis, 2010, IGZ, 2011). Therefore hospitals need to select the most appropriate IS vendor as it helps in minimalizing the drawbacks and maximizing the benefits of IS (Boonstra & Broekhuis, 2010, IGZ, 2011). That is why it is important that hospitals select the right procurement procedure in order to maximize the possibility of selecting the most appropriate IS vendor (Council, 2009, Temple-Bird & Parsons, 2006). Hence the thesis focused on the procurement of IS for hospitals. Hospitals first attempted to build their own IS1 (Appendix A), but these solutions became inferior as specialized vendors provided better solutions tailored to the application1 (Appendix A). Those specific solutions are called Hospital Information Systems (HIS). Although the literature state that HIS has both benefits (Malliarou & Zyga, 2009, Temple-Bird & Parsons, 2006, A. Winter et al., 2010, 2011, S. Wu et al., 2006) and drawbacks (Boonstra & Broekhuis, 2010, Handel et al., 2011, IGZ, 2011) the hospitals believed that HIS was the best-of-breed IS in the healthcare1 (Appendix A). The selection between HIS vendors is different across European countries. For instance Dutch hospitals are free to undergo a procurement procedure and choose a vendor, while in the UK the hospitals were forced by the government to use a specific vendor2 (NHS-Information-Governance, 2007).

1 http://www.zorgvisie.nl/Nieuws/06081/Eigen-epd-Erasmus-MC.htm Viewed: 01/12/2012 2 Siemens Healthcare

- 5 -

Before we can elaborate on the procurement of HIS there is an important difference between hospitals. In general there are two types of hospitals: academic hospitals and general hospitals3. An academic hospital provides clinical education and training to future and current health professionals in addition to the tasks provided at general hospitals like delivering medical care to patients3. Another difference is that academic hospitals are funded directly by the government and general hospitals not (at least not for more than 50%) (Knoester, 2009). Both hospital types are established for the specific purpose of meeting needs in the general interest and both hospital types have legal personality. This means that both hospital types apply to the first two criteria to be seen as a public institution. However the general hospitals do not apply to the third criteria as none of the three aspects mentioned in 1.1.2 are met (Gerechtshof-Arnhem, 2008). The academic hospitals do apply to the third criteria and therefore the academic hospitals are considered to be public institutions (Knoester, 2009). This means that if hospitals want to procure the best-of-breed HIS it is important to see if the hospital is an academic or a general hospital. As the academic hospitals are public institutions and the value of the procurement (> €750,000.004) easily exceeds the threshold values the academic hospitals are obliged by law to undertake a tendering process (EU-Commission, 2004), while general hospitals can choose freely between other procurement procedures as they are not considered public institutions (Gerechtshof-Arnhem, 2008). Another important difference that has to be discussed is the difference among European countries. Apparently these differences are such that the Portuguese HIS vendor Alert, who initially successfully entered the Dutch market as four general hospitals decided to use Alert, had to withdraw from the Dutch market as Alert failed to adapt their system to fit the Dutch laws and regulations5. Taking into account the already stated difference in freedom of selection between European countries4 (NHS-Information-Governance, 2007) the differences between European countries cannot be neglected. With respect to the available time and in order to assure that the analyses are based on the same starting point the scope is refined to the Dutch market only. Academic hospitals are obliged to undertake a strict tendering process as stated above. This also applies to the Dutch academic hospitals due to the BAO (Besluit Aanbestedingsregels voor Overheidsopdrachten). The BAO is the Dutch implementation of the European guideline 2004/18/EG. As a tender has to follow the three basic principles; equal treatment, transparency, and non-discrimination, it is well suited for performing valid analyses as it excludes negotiation, it provides total transparency since all decisions are open for involved parties, and every vendor is equally reviewed. The general hospitals are free to choose between the other procurement procedures. This means that there could be room for negotiation and the (review) procedure is not transparent, which makes it very difficult to perform analyses as it is not exactly clear how and if vendors are equally reviewed. Therefore the scope is limited to the Dutch academic hospitals. However, this does not mean that the thesis is irrelevant for general hospitals, vendors and/or consultancy companies in The Netherlands or elsewhere. The insights and conclusions of the thesis are interesting for these stakeholders as they provide better insight in the selection criteria that are used by the Dutch academic hospital and it provide insight in the conversion of hospital functionalities into modules.

3 http://www.zorgatlas.nl/zorg/ziekenhuiszorg/algemene-en-academische-ziekenhuizen/ Viewed: 12/07/2012 4 Siemens Healthcare 5 http://www.zorgvisie.nl/ICT/14124/Brabantse-ziekenhuizen-slepen-ICTbedrijf-Alert-voor-de-rechter.htm Viewed: 12/07/2012

- 6 -

1.3 Hospital Information System / Electronic Health Record system The Dutch HIS market uses a modular design. An example is presented in figure 4, which was drawn up by the University Medical Center St. Radboud during their tendering process. Figure 4 shows that the functionalities that Dutch hospitals want to achieve are converted into modules in the HIS solution. The vendors support the modular design as their systems are based on modules. The individual modules in the Dutch market are combined in the three main modules: Electronic Health Record (EHR), HIS, and Enterprise Resource Planning (ERP)6. The Dutch academic hospitals want to procure a Hospital Information System / Electronic Health Record system (HIS/EHR system(s) in the remaining of the thesis) as this includes the functionalities they require. It is not the intention of the terminology HIS/EHR system to state that these terms are the same. The terminology is used as the IS that Dutch academic hospitals want to acquire consists both the HIS and EHR module.

Figure 4 – Modular design Hospital Information System in The Netherlands

There are five HIS/EHR vendors (vendor(s) in the remaining of the thesis) that can provide a HIS/EHR system to the Dutch market: ChipSoft, EPIC, iSoft, McKesson and Siemens7. As mentioned Alert tried to enter the Dutch market as sixth vendor, but they eventually failed to successfully convert their system to the Dutch market. Nonetheless Alert was active at the time of the first three tenders. The eight Dutch academic hospitals are obliged to undertake a tendering process for the procurement of a HIS/EHR system due to the BAO. The Dutch academic hospitals are listed in table 4; this table shows the name, location and abbreviation of the academic hospital, if it collaborated with another academic hospital for its tender, the current status of the tender, if it is included in the thesis, which procedure type (PT) is used (C = competitive dialogue, O = open tender, and R = restricted tender), and its specific fake name that is used during the thesis. To ensure confidentiality the information in the last column is deleted in this public thesis.

6 Tender University Medical Center St. Radboud Viewed: 25/07/2012 7 http://www.mxi.nl/upload/epd-marktomschrijving_website_mi_v4.pdf Viewed: 01/12/2012

- 7 -

Table 4 – Dutch academic hospitals

Tender D did not only procure a HIS/EHR system, it also procured an ERP system, see figure 4. All other tenders only procured a HIS/EHR system and as it is necessary to maintain the same starting point tender D is left out of scope. Otherwise the results will be incorrect as tender D used the questionnaire to procure an ERP/HIS/EHR system and therefore the decision was not only based on the HIS/EHR system, but also on the ERP system. This will lead to skewed results as the set of requirements contained ERP requirements and therefore not all of the available points were meant for the procurement of a HIS/EHR system which was the case in the other tenders. The goal of the tendering process performed by Dutch academic hospitals is to select a HIS/EHR system which maximizes the benefits and minimizes the drawbacks mentioned in 1.2. Selecting the right vendor will contribute to this goal and this selection is ensured by selecting the most economically advantageous offer (i.e. the best price-quality ratio) (Council, 2009, Temple-Bird & Parsons, 2006). All included tenders used a RFP as they used either a competitive dialogue, or an open tender, or a restricted tender. A RFP consists of a large set of requirements, which are mentioned in a questionnaire (ranging from ±1600 questions in tender B up to ±2000 questions in tender A and C). The set of requirements, and therefore the questions, are drawn up by the Dutch academic hospital itself. The RFP of tender A used the structure as visualized in figure 5. The questionnaire consisted of twelve topics. These twelve topics were divided into 48 subtopics and

Name City Abbr. In collaboration with

Status of the tender

Included PT Used name in thesis

Maastricht University Medical Center

Maastricht MUMC - Finished in 2007, winner Siemens

No -

Leiden University Medical Center

Leiden LUMC UMCU (LUMC/UMCU)

Finished in 2010, winner ChipSoft

Yes C

University Medical Center Utrecht

Utrecht UMCU LUMC (LUMC/UMCU)

Finished in 2010, winner ChipSoft

Yes C

University Medical Center St. Radboud

Nijmegen UMCN - Finished in 2012, winner EPIC

Yes O

Erasmus Medical Center

Rotterdam EMCR UMCG (EMCR/UMCG)

Currently active

Yes R

University Medical Center Groningen

Groningen UMCG EMCR (EMCR/UMCG)

Currently active

Yes R

Academic Medical Center

Amsterdam AMC VUMC (AMC /VUMC)

Expected to start at the end of 2012

No -

Free University Medical Center

Amsterdam VUMC AMC (AMC /VUMC)

Expected to start at the end of 2012

No -

- 8 -

these were divided into 202 sub subtopics. The questions were placed under these 202 sub subtopics. The RFP of tender B and C used almost the same structure only both RFPs did not use the sub subtopics, the questions were placed directly under the subtopics. As all tenders provided the “subtopic level” the analyses were based on this level of detail. The “question level” was even more detailed and was provided by each tender, but it was unfeasible to perform analyses on this level due to the number of questions. In the remaining of the thesis the terms subtopics and selection criteria are used interchangeably and both refer to the red rectangle in figure 5.

Figure 5 – Structure of the RFP of tender A

All three included questionnaires consist of three types of questions in which the set of requirements has been incorporated: yes/no questions, open questions, and document evaluations. The first question type consists of a statement in the tender and the vendor can only answer this question with ‘yes’ or ‘no’. The second question type consists of a question and the vendor has to provide an answer within a specified maximum word count. The third question type usually consists of an elaborated question and the vendor is allowed to provide figures and tables in its answer without a specified maximum word count. Every question is either a knockout or an (important) wish of the Dutch academic hospital. If a vendor cannot fulfill a knockout question they are excluded from the tender and no points can be scored by fulfilling a knockout question. Vendors can only gain points by fulfilling the specified (important) wish questions. Up front the Dutch academic hospitals are obliged to provide a score model in their tender which states if a specific question is either a knockout or an (important) wish and how many points can be scored per (important) wish question. At the end of the RFP the completed questionnaires are reviewed by the Dutch academic hospital itself. If knockout questions are not fulfilled the Dutch academic hospital excludes the specific vendor from the tender and the answers to the (important) wish questions of the included vendors are scored. The vendor with the highest score (and therefore the most economically advantageous offer) is the winner. In order to minimalize the risk of accusations about unfair treatment from unsuccessful vendors the academic hospitals are obliged to provide feedback to the vendors. However it is not mandatory to provide a detailed insight in the scored points per questions, a relative score to the winner is already sufficient.

- 9 -

1.4 Research questions Table 4 shows that the two included tenders were won by two different vendors (Chipsoft and EPIC) and the excluded tender was won by even another vendor (Siemens). This is interesting when one takes the following into account:

- a vendor had to be selected through a strict tendering process which is based on the three principles (equal treatment, transparency, and non-discrimination) and therefore ensures for total transparency since all decisions are open for involved parties, no possibility for negotiation, and an equally review of every vendor;

- the Dutch academic hospitals did not differ that much; - and there were only five (six at the time of tender A, B, and D) available vendors from which

the Dutch academic hospitals could choose from. One would expect that at least two out of the three tenders were won by the same vendor, but this was not the case. Therefore it would be interesting to investigate the tendering processes to see how Dutch academic hospitals used the tenders to come to their results. As the inside information of the tenders was classified it was necessary to collaborate with one of the stakeholders to gain access to the information in order to perform analyses. The vendors were the only suited stakeholders as they were the only one who had the required information of all the tenders. Other stakeholders like the Dutch academic hospitals only had the information of their own tender and not one consultancy company was used at all tenders. An interested vendor was Siemens Healthcare (Siemens in the remaining of the thesis). Siemens lost the last two tenders to Chipsoft and EPIC and they were interested in a scientific analysis of the tenders in order to see where they could improve. Consequently the thesis was conducted with collaboration of Siemens. The research questions were drawn up in consultation with Siemens, Eindhoven University of Technology (TU/e in the remaining of the thesis) and the student. In this way the practical use for Siemens and the scientific use for the TU/e were taken into account. The main and sub-questions are listed below. Main question How do the Dutch academic hospitals use the European tenders to determine the best suited HIS/EHR system? Sub-questions

What are the selection criteria in the European tenders in terms of weightings?

Do the selection criteria differ per European tender (in terms of weightings)?

Is there fairness in the European tenders in terms of the percentage of the total score the open questions account for versus the percentage of the total score the closed questions account for?

The goal of the thesis was twofold. The practical goal of the project was to analyze the tenders to identify the selection criteria and rank them in order of importance to give Siemens an advantage above their competitors. At the academic level the project aimed at contributing to the understanding of Dutch procurement procedures for HIS/EHR systems in academic hospitals and to see how the details of these procurement procedures evolved over time.

1.5 Thesis structure The thesis is structured as follows. Chapter 2 describes the research design and explains the chosen research approach. Chapter 3 is based on the literature study conducted prior to the thesis. Chapter 4 defines and describes conducted analyses. Chapter 5 reflects on the analyses results and finally the conclusion and suggestions for future research will be presented.

- 10 -

2 Project context The project context will be explained in this chapter. The scope is described in the first paragraph. The second paragraph provides design alternatives and argues why the used design was selected. The research method and model are presented in the third paragraph.

2.1 Scope The scope of the thesis was on the procurement of HIS/EHR systems for Dutch academic hospitals due to the fact that Dutch academic hospitals are obliged to conduct a tender for the procurement of an HIS/EHR system. The Dutch general hospitals were left out of scope during the thesis as they were not obliged to conduct a tender and to make the thesis manageable the scope was limited to the Dutch market. Tender D was left out of scope as explained in 1.3.

2.2 Design alternatives HIS/EHR vendor Siemens is chosen as collaboration partner in the used design and the goal was to identify and rank the selection criteria, analyze these selection criteria to identify trends, and investigate the fairness of tenders and the importance of price. The selection criteria were analyzed on the subtopic level as all tenders included this level and therefore the selection criteria could be matched with each other. The benefits of this design are that it is a topical issue as the second last tender had just started at the start of the thesis and the last Dutch tender will start in January 2013. Therefore stakeholders are interested in the insights and conclusions of the research. Other benefits are the well-defined scope and that the research is based on objective information sources. A drawback is that there are only three information sources. The alternative designs will be presented in the next paragraphs. An alternative design could have been to research the level of implementation in hospitals which already conducted a procurement procedure regarding HIS/EHR systems. The goal of this research would be to investigate to what extent the people are actually using the system and to see how happy the hospitals are with their HIS/EHR system. The drawbacks of this alternative are that it will be difficult to maintain objectivity and that there is not much information available due to the fact that hospitals are still implementing or choosing HIS/EHR systems. Probably it will not be difficult to find a collaboration partner as this could be important information for every hospital. Although it would have been an interesting topic it is probably not the right timing for such a research, it is too soon. Therefore this alternative was not chosen. Another alternative design could be to investigate what the cost of ownership (TCO) of the procurement of HIS/EHR systems were at hospitals. A goal of this research could be to investigate if the advantages of a HIS/EHR system outweigh the total investments. Again the objectivity will be at risk as it is difficult to measure the advantages of a HIS/EHR system without introducing biased opinions. Together with the fact that it is probably not the right timing for such a research this alternative was not chosen. Another alternative design could be to focus on finding the best procurement procedure. This could be determined by conducting interviews with hospitals, vendors, and consultancy companies. The goal of the interviews are to determine how long a procurement procedure took, what the costs were of the procurement procedure and how the end result is in terms of price-quality ratio. A collaboration partner could be a consultancy company as they work with multiple hospitals and vendors. By using interviews as main information source it is again difficult to maintain objectivity through the biased opinions. Although this could be a nice topic for a research when the objectivity can be ensured it was decided to take a different direction as the used research provided already objective information sources.

- 11 -

2.3 Research method and model The available information of the three included tenders as listed in table 4 were used as initial information sources. The thesis focused initially on the two conducted tenders. Tender A provided the most detailed feedback, as tender B only provided a general feedback with relative scores to the winner. Obviously the currently active tender did not have the scores yet. As tender A provided the most detailed questionnaire, score model and feedback it was considered as the starting point of the thesis. As reproducibility is one of the cornerstones of science the thesis focused on the weightings and score models of the tenders who were identified in the RFPs. The analysis of tender A started with identifying the selection criteria according to the score model and the weightings. This identification was also done for tender B and C. The selection criteria were slightly different among the three tenders and therefore the selection criteria of tender B and C had to be matched to the selection criteria of tender A.

2.3.1 Delphi method This matching was based on the two round Delphi method (Skulmoski, Hartman, & Krahn, 2007). In the first round were two tender experts asked individually to match the selection criteria of tender B with the selection criteria of tender A. This matching was reviewed by the researcher and some differences emerged in the matching. These differences were tackled with a second round where both experts and the researcher sat together and argued which matching was correct. The same procedure has been applied to match the selection criteria of tender C with the selection criteria of tender A. The used Delphi method is visualized in figure 6.

Figure 6 – Delphi method

- 12 -

The following example explains the used Delphi method. The selection criteria of tender C had to be matched with selection criteria of tender A, for instance the two selection criteria of tender C in table 5.

Table 5 – Example tender C subtopic matching

In the first round matched expert 1 subtopic Z.2.10 with subtopics 1.2.8, 1.2.12, and 1.2.13 (table 6) while expert 2 matched subtopic Z.2.10 only with subtopics 1.2.8 and 1.2.12 (table 7). The percentage next to the subtopic means that, for instance in table 7, subtopic Z.2.10 was related to subtopic 1.2.8 for 20% and to subtopic 1.2.12 for 80%.

Table 6 – Matching by HIS vendor expert 1

Table 7 – Matching by HIS vendor expert 2

The subtopics of tender A that has been used by one or both experts to match subtopic Z.2.10 are listed in table 8.

Table 8 – Example tender A subtopic matching

The results of the first round were not similar as expert 1 matched subtopic Z.2.10 to three subtopics and expert 2 only to two. As the researcher was not an expert in this field a second round was conducted. In this second round both experts and the researcher argued what would be the best matching. For this subtopic the best matching is presented in table 9.

Table 9 – Best matching discussed by HIS expert 1 and 2

Subtopic Z.2.11 was already in the first round matched to subtopic 1.2.6 by both experts, table 10. Therefore a second round was unnecessary. The used subtopic of tender A is listed in table 11.

2.3.2 Research model The ranking of the selection criteria in order of importance followed from the analysis on the three tenders. The criterion which had the most influence on the total score was considered as the most important criterion. A comparison of the raking of the selection criteria among the three tenders followed. To make sure that the thesis did not focus solely on the analyses of tender documents an interview with an independent consultancy companies has been held. Next the fairness of the tenders was determined. Was the fairness in the tenders in terms of the percentage of the total score the open questions account for versus the percentage of the total score the closed questions account for preserved or not? The explained research method is visualized in figure 7.

Subtopic Name subtopic

Z.2.10 Management informatie

Z.2.11 Medicatie

Tender C

Z.2.10 1.2.8  40% 1.2.12 50% 1.2.13 10%

Tender A

Tender C

Z.2.10 1.2.8  20% 1.2.12 80%

Tender A

Subtopic Name subtopic

1.2.8 Registratie van prestatie-indicatoren

1.2.12 Rapportages

1.2.13 Administratie Medisch wetenschappelijk onderzoek

Tender C

Z.2.10 1.2.8  30% 1.2.12 60% 1.2.13 10%

Tender A

Tender C

Z.2.11 1.2.6  100%

Tender A Subtopic Name subtopic

1.2.6 Medicatie

Table 10 – Similar matching after first round Table 11 – Example tender A subtopic

- 13 -

Figure 7 – Research model

- 14 -

3 Literature study and backgrounds The conducted literature study has focused on the terms EHR and HIS as the academic hospitals used a HIS/EHR system. The literature study on the procurement procedures is already mentioned in 1.1 and therefore will not be displayed again. First the lack of consensus among the terms is described followed by the literature study method. Next the Reference Concept Maps are explained and the results are presented.

3.1 Lack of consensus One of the major problems in the available healthcare literature is the different terminology and the definition of the terms: EHR, electronic medical record (EMR) and HIS (Jha et al., 2006). Another issue is the discussion if the terms EHR and EMR differ from each other (Garets & Davis, 2006, Hinman & Ross, 2010, Sonoda, 2011, Zhang & Liu, 2010) or not (Hoffman & Podgurski, 2009). These issues were addressed in the literature study conducted prior to the thesis.

3.2 Literature study method There was a lot of available literature in the field of Electronic Health Records; a simple search for “electronic health record“ provided more than 25 thousand hits in Google Scholar and more than three million hits in Google. Obviously not all articles could be read for the literature study and therefore a strategy was determined to go through all the available literature in a structured manner. This strategy divided the literature study into two phases (Holly, Salmond, & Saimbert, 2011). In the first phase the results of a specific search query were ranked, where the authority of the articles was the key factor as the articles were ranked by the number of citations in Google Scholar. In the second phase the articles found in the first phase were used to conduct a citation chase. This is a search technique that involved taking an article and search “backward in time,” using the references found at the end of that article to see what authors and publications are covering on a topic of interest (Holly et al., 2011). The search technique “forward in time” was not used as recent articles already received more weight in the first phase. In summary:

Phase 1: Articles ranked by the number of citations in Google Scholar;

Phase 2: Citation chase on the top ranked articles to find missing articles. An encountered problem during the first phase was that older articles could be cited more than newer articles, not necessary because they are more important, but simply because they are older. To tackle this problem and to determine how many articles should be included: top-3 ranked, top-5 ranked, top-10 ranked, or even more, there has been made use of the Hirsch contemporary h-index. The original h-index is a way to perform fair ranking of scientists as it accounts for both productivity and impact (Sidiropoulos, Katsaros, & Manolopoulos, 2007). A drawback of the original h-index is its “inability to differentiate between active and inactive scientists” (Sidiropoulos et al., 2007). (Sidiropoulos et al., 2007) stated that their contemporary h-index, which gives more weight to recent articles and thus rewards academics who maintain a steady level of activity, solved this problem. The tool used to generate the contemporary h-index and the ranking of the articles by the number of citations is called Publish or Perish (Harzing, 2007). Publish or Perish is a tool which automatically ranks articles through the number of citations in Google Scholar. The tool generated per search query the contemporary h-index which showed how many articles should be included in the short list of the first phase. These articles were called the nominated articles; unfortunately not all of the nominated articles were relevant (e.g. the search query definition of ehr provided a contemporary h-index of ten, but only seven out of the ten nominated articles provided a definition of EHR). Therefore the first phase was extended with a step where the relevance of the nominated articles was verified.

- 15 -

The second phase, the citation chase on the top ranked articles to find missing articles, was done manually as there was no available tool. All references found at the end of the nominated articles had to be crosschecked if there was any overlap and therefore missing articles. The activity diagram regarding the used research method is summarized and visualized in figure 8 (for EHR and EMR). The same method was used for HIS and is visualized in figure 21 (Appendix B).

Figure 8 – Activity diagram literature research method EHR and EMR

3.3 Results first and second phase The result of the first and second phase regarding EHR is listed in table 12. The results regarding EMR and HIS are listed in similar tables, table 32 and table 33 (Appendix C). These tables show which search queries were used for the first phase and entered in the tool Publish or Perish, how many articles were found with a particular search query, how many articles should be included according to the contemporary h-index by Publish or Perish, how many nominated articles of this contemporary h-index were relevant for this literature study, and how many different definitions were provided by these relevant articles per search query. Articles that were labeled not relevant either could not be found or did not provide a definition. The second last row in each table shows how many articles were found during the second phase and how many were relevant.

- 16 -

Search query Total number of articles

Contemporary h-index

Relevant articles

Number of different definitions

"definition of ehr" 55 10 7 7

"definition of electronic health record" OR "definition of electronic health records"

22 5 2 2

"definition of emr" 32 7 1 1

"ehr is defined" 71 6 2 2

"electronic health record is defined" OR "electronic health records are defined"

16 5 4 4

Second phase: citation chase 38 0 0

234 33 15 15

Table 12 – Results Publish or Perish EHR search queries

The results of the first and second phase showed nine different definitions of the term EHR, twelve different definitions of the term EMR and five different definitions of the term HIS. The numbers for EHR and HIS do not match the numbers in the tables, but that is due to the fact that some articles provided the same definition. As all articles passed the first and/or second phase you could say that they all contributed significantly to the healthcare community. This immediately raised the following questions:

How to reach consensus of the definition of an EHR as there are nine equally important definitions?

How to reach consensus of the definition of an EMR as there are twelve equally important definitions?

How to reach consensus of the definition of a HIS as there are six equally important definitions?

3.4 Reference Concept Map An answer to the raised questions was the use of a so called Reference Concept Map (RCM). A RCM was first introduced by (Rodriguez-Priego, Garc’\ia-Izquierdo, & Rubio, 2010) in the context of lacking definitions and contradictory literature. As the raised questions corresponded with this context a RCM was the answer to the questions. (Rodriguez-Priego et al., 2010) explained a RCM as: ‘RCM helps to identify the key concepts that appear in each of the term definitions and their relationships. A RCM shows main concepts related to other concepts, within a definition, to the main concepts. Links between the concepts include the numbers of the references where the term definitions are given.’ In other words the RCM consisted of several graph nodes which, all originated from the same start node, were linked through directed edges. Every graph node consisted of one or more words. The directed edges linked the graph nodes in a specific order resulting in a path and every path represented a definition. Each reference was given a number and these numbers were placed on the directed edges. Every directed edge between graph nodes was represented by an arc and the weight of this arc was determined by the number of references that used this directed edge. The RCM was a way to reach consensus of definitions and it was the answer to the raised questions. But as could be expected the original definitions did not use the same terminology. Therefore the original definitions had to be rephrased into general graph nodes. This process is visible in table 34 (Appendix D) for EHR, table 35 (Appendix E) for EMR and table 37 (Appendix G) for HIS.

- 17 -

The paper by (Rodriguez-Priego et al., 2010) only rephrased the original definition, but the literature study added another dimension for both the EHR as the EMR process. This dimension was the forming of seven different classes where every class consisted of equivalent graph nodes. The reason for this addition was to make the RCM more structured and the number of classes followed from the first version of the RCM. Each original definition was rephrased in such a way that the sequence of the graph nodes was the same for every rephrased definition. If the original definition did not provide a graph node in a specific class, the class was skipped. The general sequence regarding the graph nodes of the rephrased definition followed these seven classes:

1. Record vs. System; 2. Content; 3. Subject; 4. Format; 5. Security; 6. Target audience; 7. Other.

The color in front of the class is the same color as can be found in the RCMs regarding the definition of an EHR and an EMR, figure 9 and figure 22 (Appendix F), and the corresponding rephrase tables, table 34 and table 35. The RCM regarding the definition of a HIS, figure 23 (Appendix H), and the corresponding table, table 37, has no classes and colors as it was not feasible to introduce them. The RCMs regarding EMR and HIS are placed in the appendices as they are similar to the one regarding EHR.

- 18 -

3.4.1 RCM for EHR definition

Figure 9 – RCM regarding the definition of an EHR

Table 13 – Google hits regarding the definition

of an EHR

- 19 -

The RCM in figure 9 was used to determine the universal definition of an EHR. For the explanation we start at the graph node EHR. Six arcs depart out of this graph node; this means that the rephrased definitions of table 34 (Appendix D) provided six different solutions for the graph node direct after EHR. Every arc contains one of more numbers and these numbers correspond to the reference that used that particular graph node in its definition. The references and their corresponding numbers are stated in table 14 (quick reference list) and table 34 (Appendix D) (full list including definitions).

# Source # Source

1 (Jha et al., 2006) 9 (Garde & Knaup, 2006) 2 (Häyrinen, Saranto, & Nykänen, 2008) 10 (Zhang & Liu, 2010) 3 (Iakovidis, 1998) 11 (Garde, Knaup, Hovenga, & Heard, 2007) 4 (Simon et al., 2007) 12 (Garets & Davis, 2006) 5 (Katehakis, Sfakianakis, Kavlentakis,

Anthoulakis, & Tsiknakis, 2007) 13 (Knaup, Bott, Kohl, Lovis, & Garde, 2007)

6 (Hoffman & Podgurski, 2009) 14 (Alhaqbani & Fidge, 2007) 7 (Terry, 2008) 15 (Hinman & Ross, 2010) 8 (DesRoches et al., 2010) Table 14 – Quick EHR reference list

An example of a path: starting from EHR, to repository [2,5,6,9,11,13], to health status [5,6,9,11,13], to information [5,6,9,11,13], to subject of care [5,6,9,11,13], to computer processable form [5,6,9,11,13], to stored and transmitted securely [2,5,6,9,11,13], to multiple authorized users [2,5,6,9,11,13] (in this example the arc goes first to health status and second to information, while this was the other way around in the definition, however this is no problem as both graph nodes are in the same class). All other paths can be determined in the same manner. All definitions/paths can be found in table 34 (Appendix D). The little words were left out of the RCMs and only the important graph nodes were taken into account to keep it orderly. All references contributed significantly to the healthcare literature, so the first way to determine the universal definition regarding an EHR was taking the path with the highest number of references as the universal definition. This path followed the red line and green boxes and the definition was: an EHR is a repository of information regarding the health status of a subject of care in computer processable form, stored and transmitted securely, and accessible by multiple authorized users. A second way to determine the universal definition regarding an EHR was through the seven different classes. Each swim lane represents a class and the graph nodes in this class are related to each other. For example the class Format contains the graph nodes: electronically maintained, digitally stored, computer processable form and digital form. Each graph node is related to each other as they all defined which format was used for the EHR. The graph node with the most incoming arcs was the most important graph node in a specific class. It received a blue dotted oval and should be included in the universal definition. With this reasoning the definition should be: an EHR is a repository of information regarding a subject of care / individual / lifetime in computer processable form, stored and transmitted securely, and accessible by multiple authorized users. A third way to determine the universal definition was to see which complete path, starting at the right conducting a backward pass with the reference number as guard, had the most hits in Google. The first two ways used the scientific database created in the first and second phase. The third way showed the definition which was used most in practice, table 13. The definition had to make use of at least three classes, therefore path I was not included and path VI was the highest ranked. With this reasoning the definition should be: an EHR is an electronic record of health-related information on an individual that is created, gathered, managed, and consulted by authorized health care clinicians and staff.

- 20 -

3.4.2 RCM for EMR definition The RCM showed in figure 22 was used to determine the definition of an EMR. The RCM is similar to the one explained in 3.4.1 and therefore it will not be explained again. The numbers in figure 22 correspond to the reference that used that particular graph node in its definition. The references and their corresponding numbers are stated in table 35 (Appendix E). The three different ways that were used to determine the universal definition regarding an EHR were also executed to determine the universal definition regarding an EMR. The first way provided the following definition: an EMR is an application environment which supports the medical record of a patient in an electronic manner, is owned by and used in a single Care Delivery Organization (CDO). According the second way the definition should be: an EMR is a medical record which contains patient information of a patient in an electronic / computer-based manner, is owned by CDO and used in a single Care Delivery Organization (CDO). And the third way delivered: an EMR is a medical record for each patient that is computer-based.

3.4.3 RCM for HIS definition The RCM showed in figure 23 was used to determine the definition of an HIS. The RCM is similar to the one explained in 3.4.1 and therefore it will not be explained again. The numbers in figure 23 correspond to the reference that used that particular graph node in its definition. The references and their corresponding numbers are stated table 37 (Appendix G). As this RCM did not consist of classes the second way could not be used to determine the universal definition. The first way provided the following definition: a HIS is a socio-technical subsystem which comprises all information processing as well as the associated human or technical actors in their respective information processing role. The other way delivered: a HIS is a subsystem which comprises all information processing as well as the associated human or technical actors in their respective information processing role.

3.5 Results of Reference Concept Maps After examining the results of the RCM in figure 9 the universal definition of an EHR is: an EHR is a repository of information regarding the health status of a subject of care in computer processable form, stored and transmitted securely, and accessible by multiple authorized users. As this definition scored the first place in both the first and second way and a respectively third place in the third way it was considered as the best universal definition of EHR. The universal definition of an EMR is: an EMR is a medical record of a patient in an electronic manner, is owned by and used in a single Care Delivery Organization (CDO). The first way provided exactly this definition. The second way stated that six graph nodes should be included, but this was impossible as no path considered all six graph nodes in its definition. The best path considered five graph nodes was the same path as the winner of the first way. The universal definition scored a fifth place in the third way. Overall it was considered as the best universal definition of EMR. The universal definition of a HIS is: a HIS is a socio-technical subsystem which comprises all information processing as well as the associated human or technical actors in their respective information processing role. It was the winner of the first and scored second in the third way and therefore the best universal definition of HIS.

- 21 -

After comparison of the universal definitions it is hard to state if the terms EHR and EMR differ. The definitions state that an EHR is intended for multiple authorized users, while an EMR is intended for one single CDO. It seems that the terms differ, but one could argue that the multiple authorized users are from one single CDO as the used literature is not clear if the multiple authorized users can be from different CDOs or not. Therefore no hard conclusion can be drawn. Further analysis (e.g. conducting a literature study focusing only on this problem) is required in order to tackle the raised problem and state a final conclusion. The universal definition of HIS is final.

- 22 -

4 Tender analyses The tender analyses will be explained in this chapter. First the analysis on tender A, considered as the reference tender as it provided the most detailed questionnaire, score model and feedback, is displayed. Next the analysis on tender B and C is presented followed by the matching procedure and finally the matching results.

4.1 Analysis of tender A As visualized in figure 7 the thesis initially focused on tender A. The structure of tender A is displayed in figure 5. The RFP consisted of a questionnaire with twelve topics. These twelve topics were divided into 48 subtopics and these were divided into 202 sub subtopics. The 1919 questions were placed in these 202 sub subtopics. The questions could be either a knockout of an (important) wish. The analysis was based on the points that a question could contribute to the total score, therefore the 43 knockout questions were left out of scope. The remaining 1876 questions were divided as 1566 important wishes and 310 wishes. An important wish question could gain twice the points a wish question could. All questions were either a closed or an open question. Due to the RFP it was known how many points could be scored per specific question and which questions belonged to a specific subtopic. Therefore it was also known how many points could be scored per subtopic. This was verified with the provided score model. Table 15 is a preview of one topic in tender A. It shows the four subtopics (1.1.1 – 1.1.4) that belonged to the topic Algemene functionele eisen. The third column shows how many points could be scored per subtopic, and the fourth column calculates the percentage of the total score that a specific subtopic accounted for. For example: answering all questions in subtopic Wet en regelgeving correctly would lead to a score of 1.10 point and this accounted for 1.47% of the total score. The total score is italic, because the calculations were not done using the actual total score. The subtopics price, contract, vision and demonstration were left out of the total score to focus on the functional aspects. Obviously these subtopics were also left out in the calculations of the other two tenders. A total of eight topics and 44 subtopics were considered in further analysis. The points and percentages of all considered (sub) topics of tender A can be found in table 39 (Appendix I). The points and percentages are crosschecked with hospital A.

Table 15 – Points and relative score per subtopic

4.2 Analysis of tender B The RFP of tender B was not as nicely documented as the one of tender A and therefore not every specific question could be placed under a subtopic. However, the score model was complete and therefore it was known how many points could be scored per subtopic. Tender B consisted of a questionnaire with eleven topics. These eleven topics were divided into 53 subtopics and these were not further divided into sub subtopics. The questionnaire consisted of 1581 questions. A total of nine subtopics were left out of the total score to focus on the functional aspects, thus 44 subtopics were considered for further analysis. The points and percentages of all considered (sub) topics of tender B can be found in table 40 (Appendix J).

Subtopics of topic 1.1

(Algemene functionele eisen)

Name subtopic Points that can be

scored per subtopic

Percentage of

total score

1.1.1  Wet en regelgeving 1,10 1,47%

1.1.2  Gebruiksvriendelijkheid 2,75 3,67%

1.1.3  Beschikbaarheid en performance 1,10 1,47%

1.1.4  Beveiliging van informatie 0,55 0,73%

- 23 -

4.3 Analysis of tender C The analysis of tender C encountered the same problem as tender B as the RFP of tender C was not as nicely documented as the one of tender A. Again the score model was complete and therefore it was known how many points could be scored per subtopic. Tender C consisted of a questionnaire with eleven topics and 22 subtopics and these were not further divided into sub subtopics. The questionnaire consisted of 1956 questions. After deleting the subtopics that were out of scope a total nineteen subtopics were considered in further analysis. The points and percentages of all considered (sub) topics of tender C can be found in table 41 (Appendix K).

4.4 Match selection criteria After the analysis of the three tenders it became clear that the selection criteria were similar, but not exactly the same as the tenders used different wording and terminology. In order to analyze how the selection criteria changed over time, one tender had to be selected as reference and the subtopics of the other two tenders had to be matched with the subtopics of the reference tender. As mentioned before tender A was chosen as the reference tender. The following example explains what happened during the matching procedure. It consists of three tenders just like in the real situation: tender T1, T2, and T3 which have respectively five, four, and three subtopics, see table 16, table 17, and table 18. These tables work the same as the tables of the real tenders. The first column contains the subtopics, the second column shows how many actual points could be scored per subtopic, and the third column calculates the percentage of the total score that a specific subtopic accounted for. In the example tender T1 is the reference tender (similar to tender A). In order to simplify the example let’s assume a tender is required to buy a bicycle. The bicycle has the following components:

wheels consisting of tubes and spokes;

frame;

brakes;

material;

and a color. The three tenders sometimes use slightly differed wording and terminology as listed in table 16, table 17, and table 18 just like in the real situation.

- 24 -

For matching the selection criteria of tender T2 and T3 with the selection criteria of reference tender T1 the explained two round Delphi method (Skulmoski et al., 2007) is used. The result of this matching is visualized in table 19 and table 20.

The first column of table 20 shows the selection criteria (subtopics) of tender T2 and the second column which selection criteria of reference tender T1 are matched with that specific selection criterion of tender T2. For instance: subtopic Material of tender T2 is matched with subtopics Frame (for 75%) and Spokes (for 25%) of reference tender T1. In other words: 75% of the questions in subtopic Material relates with the questions in subtopic Frame and 25% of the questions in subtopic Material relates with the questions in subtopic Spokes. Another example: subtopic Wheels is matched with subtopics Tubes (for 5%), Color (for 90%), and Spokes (for 5%). In other words: 5% of the questions in subtopic Wheels relates with the questions in subtopic Tubes, 90% relates with the questions in subtopic Color, and 5% relates with the questions in subtopic Spokes. Apparently tender T3 wants a special color for his/her wheels. The same method was used in the real scenario, only on a much larger scale. The matching of the tender B (44 subtopics) with the reference tender A (44 subtopics) is visualized in table 42 (Appendix L). The matching of the tender C (19 subtopics) with the reference tender A is visualized in table 43 (Appendix M). All subtopics were matched 100% to other subtopics, therefore no problems occurred.

Subtopic Name subtopic Points that can be

scored per subtopic

Percentage of

total score

T1.1 Tubes 45 30,0%

T1.2 Frame 30 20,0%

T1.3 Brakes 25 16,7%

T1.4 Color 35 23,3%

T1.5 Spokes 15 10,0%

Subtopic Name subtopic Points that can be

scored per subtopic

Percentage of

total score

T2.1 Material 15 15,0%

T2.2 Brakes 30 30,0%

T2.3 Wheels 20 20,0%

T2.4 Color 35 35,0%

Subtopic Name subtopic Points that can be

scored per subtopic

Percentage of

total score

T3.1 Wheels 60 50,0%

T3.2 Frame 35 29,2%

T3.3 Brakes 25 20,8%

Tender T2 Tender T1

Material Frame 75% Spokes 25%

Brakes Brakes 100%

Wheels Tubes 80% Spokes 20%

Color Color 100%

Tender T3 Tender T1

Wheels Tubes 5% Color 90% Spokes 5%

Frame Frame 50% Color 50%

Brakes Brakes 100%

Table 16 – Starting information tender T1

Table 17 – Starting information tender T2

Table 18 – Starting information tender T3

Table 20 – Matching of tender T3 with tender T1 Table 19 – Matching of tender T2 with tender T1

- 25 -

4.5 Matching results The stated example in 4.4 is also used to explain the matching results. In order to analyze how the selection criteria change over time the tenders has to be compared. For this comparison another step is made which is visualized in table 21 and table 22.

The matching of tender T2 with tender T1 is done in table 20 where subtopic Material is matched with subtopic Frame (for 75%) and Spokes (for 25%). As stated in table 17, subtopic Material accounts for 15% of the total score in tender T2. The goal is to convert the subtopics of tender T2 into the subtopics of tender T1. Therefore the 15% that subtopic Material account for in tender T2 has to be allocated to the subtopics it is matched to, namely subtopic Frame and Spokes. The calculations that follows are for subtopic Frame: 15% X 75% = 11.25% and for subtopic Spoke: 15% X 25% = 3.75%. The same calculations are done for every subtopic of tender T2 and T3. These values are placed under the correct subtopic of tender T1, because it is possible that multiple subtopics of tender T2 of tender T3 are matched with the same subtopic of tender T1. In the example both subtopics Material and Wheels of tender T2 are matched with subtopic Spokes. In the end all percentages that belong to a specific subtopic of tender T1 are added for the final results. For tender T2 the results are: Tubes accounts for 16% of the total score, Frame for 11.25%, Brakes for 30%, Color for 35%, and Spokes for 7.75%. The same procedure is performed for tender T3 and this provides the end result in table 23. In this table the first column shows the subtopics of the reference tender T1. The second, third, and fourth column show the percentage that a specific subtopic accounts for of the total score in a specific tender. For example: subtopic Tubes accounts for 30% of the total score in tender T1, for 16% in tender T2, and for 2.5% in tender T3.

Table 23 – End result example

The same procedure was used in the real scenario, only on a much larger scale. The result of the matching of tender B with as reference tender A is visualized in table 44 (Appendix N). The result of the matching of tender C with as reference tender A is visualized in table 45 (Appendix O). A preview of the tables in the appendices is visualized in table 24. The first column states the subtopic numbers of tender C (Z.1.1, Z.1.2, etc.) and the first row shows some of the subtopic numbers of tender A (1.1.1, 1.1.2, etc.). In the middle are the allocated percentages of a specific subtopic listed and the last row show the results. The end result of the matching is shown in table 28 which shows that the selection criteria differed among the three tenders.

T2 / T1 Tubes Frame Brakes Color Spokes

Material 0,00% 11,25% 0,00% 0,00% 3,75%

Brakes 0,00% 0,00% 30,00% 0,00% 0,00%

Wheels 16,00% 0,00% 0,00% 0,00% 4,00%

Color 0,00% 0,00% 0,00% 35,00% 0,00%

16,00% 11,25% 30,00% 35,00% 7,75%

T3 / T1 Tubes Frame Brakes Color Spokes

Wheels 2,50% 0,00% 0,00% 45,00% 2,50%

Frame 0,00% 14,58% 0,00% 14,58% 0,00%

Brakes 0,00% 0,00% 20,83% 0,00% 0,00%

2,50% 14,58% 20,83% 59,58% 2,50%

Subtopic

Tender T1 Tender T2 Tender T3

Tubes 30,00% 16,00% 2,50%

Frame 20,00% 11,25% 14,58%

Brakes 16,67% 30,00% 20,83%

Color 23,33% 35,00% 59,58%

Spokes 10,00% 7,75% 2,50%

Data

Table 21 – Result matching tender T2 with tender T1 Table 22 – Result matching tender T3 with tender T1

- 26 -

Table 24 – Preview result matching tender C with tender A

C / A 1.1.1  1.1.2  1.1.3  1.1.4  1.2.1  1.2.2  1.2.3  1.2.4  1.2.5  1.2.6  1.2.7  1.2.8  1.2.9 1.2.10

Z.1.1 9,30% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.1.2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.1 0,00% 0,47% 0,00% 0,00% 0,47% 0,47% 0,00% 0,00% 0,47% 0,00% 0,47% 0,00% 0,00% 0,47%

Z.2.2 0,00% 0,70% 0,70% 0,93% 0,47% 0,00% 0,00% 0,47% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.3 0,00% 0,23% 0,00% 0,00% 0,00% 0,00% 0,70% 0,23% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.4 0,00% 0,70% 0,00% 0,23% 1,16% 0,00% 0,00% 0,00% 1,16% 0,00% 0,00% 0,00% 0,35% 0,00%

Z.2.5 0,00% 0,35% 0,00% 0,00% 0,93% 2,33% 0,00% 0,35% 0,35% 0,00% 0,35% 0,00% 0,00% 0,00%

Z.2.6 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,93% 0,00% 0,00% 0,00% 0,00% 3,26%

Z.2.7 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.8 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.9 0,00% 0,00% 0,00% 0,00% 4,65% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.10 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,40% 0,00% 0,00%

Z.2.11 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 4,65% 0,00% 0,00% 0,00% 0,00%

Y.1 0,00% 0,00% 0,29% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Y.2 0,29% 0,00% 0,29% 0,58% 0,00% 0,00% 0,00% 0,00% 1,45% 0,00% 0,00% 0,00% 0,00% 0,00%

Y.3 0,48% 0,00% 0,48% 0,97% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,48% 0,00% 0,00% 0,48%

X.1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

X.2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

X.3 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

100,00% 10,08% 2,44% 1,76% 2,71% 7,67% 2,79% 0,70% 1,05% 4,36% 4,65% 1,30% 1,40% 0,35% 4,21%

- 27 -

5 Result tender analyses In this chapter the concluding results of the analyses will be presented as the tender analyses and matching results were finished in chapter 4. First the selection criteria of the three tenders will be ranked before the matching occurred, followed by the ranking of the matched selection criteria. Next the matched selection criteria are plotted in time and the possible trends are discussed. Next the fairness in the tenders is checked, the contribution of the topic price is presented and the opinions of experts regarding tenders are described.

5.1 Rank the selection criteria The ranking of the selection criteria was done in order of importance. The criterion which had the most influence on the total score was considered the most important criterion. The rankings were done before the matching with the actual subtopics per tender and after the matching with the allocated percentages per subtopic. The top five selection criteria before the matching for tender A are listed in table 25, for tender B in table 26, and for tender C in table 27. The totals ranking can be found in table 46 (Appendix P) for tender A, in table 47 (Appendix Q) for tender B, and in table 48 (Appendix R) for tender C.

Table 25 – Top five selection criteria tender A

Table 26 – Top five selection criteria tender B

Table 27 – Top five selection criteria tender C

Subtopics Name subtopic Points that can be

scored per subtopic

Percentage of

total score

2.2.2 Plan van aanpak voor de implementatie en bedrijfsgerede oplevering 4,00 5,33%

2.3 Contractvoorwaarden project 4,00 5,33%

1.2.1  Klinische dossiervoering 3,88 5,17%

1.2.3  Ordercommunicatie 3,88 5,17%

1.2.11 Patiënt monitoring en datacollectie (PDMS) 3,88 5,17%

Subtopic Name subtopic Points that can be

gained per subtopic

Percentage of

total score

1-2-1 Functionele rijkheid wanneer beschikbaar 80 10,81%

1-2-15 ICT - Beschikbaarheiden betrouwbaarheid 60 8,11%

1-2-5 Dossier – onderzoek 40 5,41%

2-2-1 Kwaliteit van de best practices 30 4,05%

1-2-3 Dossier – gebruiksgemak 28 3,78%

Subtopic Name subtopic Points that can be

gained per subtopic

Percentage of

total score

Z.1.1 Mate van compleetheid huidig product 80 9,30%

X.3 Implementatieplan en nazorg 80 9,30%

Z.1.2 Productenkalender en visie op doorontwikkeling 60 6,98%

Y.1 Technische flexibiliteit 50 5,81%

Y.2 Beheersbaarheid 50 5,81%

- 28 -

The ranking of the selection criteria after the matching based on tender C is listed in table 28. This table shows the percentage of the total score that each selection criterion accounted for. The table is ranked based on the percentages of tender C as this was the last tender in time. The selection criterion with the most influence on the total score in tender C is on top. The percentages of the other two tenders are also mentioned in order to make comparisons in a later stadium. The ranking based on tender A is listed in table 49 (Appendix S), the ranking of tender B in table 50 (Appendix T), and the ranking of the average selection criteria in table 51 (Appendix U).

Table 28 – End result: ranking selection criteria after matching based on tender C

Name subtopic

Tender B Tender A Tender C

Wet en regelgeving 0,00% 1,47% 10,08%

Klinische dossiervoering 4,78% 5,17% 7,67%

Innovatie 10,81% 0,80% 6,98%

Administratie Medisch wetenschappelijk onderzoek 5,65% 5,17% 5,35%

Administratie en monitoring onderwijs 2,84% 0,52% 4,88%

Medicatie 0,27% 5,17% 4,65%

Communicatie / Correspondentie 1,49% 0,52% 4,36%

Portalen 1,24% 5,17% 4,21%

Documentatie en templates 7,76% 1,33% 4,13%

Conversie 3,24% 1,33% 4,07%

Open vragen – integratie 1,08% 2,93% 3,16%

Zorgpaden en zorgplannen 2,00% 0,52% 2,79%

Rapportages 0,65% 0,52% 2,79%

Beveiliging van informatie 1,81% 0,73% 2,71%

Gebruiksvriendelijkheid 3,35% 3,67% 2,44%

Serviceondersteuning en –contract (service level agreement) 4,05% 2,80% 2,44%

Implementatie en bedrijfsgerede oplevering 10,54% 4,00% 2,44%

Plan van aanpak voor de implementatie en bedrijfsgerede oplevering 16,49% 5,33% 2,44%

Beheer 0,77% 1,10% 2,23%

Monitoring en reporting 0,77% 1,10% 2,03%

Opleiding en instructie 3,78% 1,33% 1,86%

Beschikbaarheid en performance 1,46% 1,47% 1,76%

Eisen Service Oriented Architecture 0,54% 4,40% 1,74%

Registratie van prestatie-indicatoren 0,32% 0,52% 1,40%

Medische bibliotheek / knowledge base 0,70% 5,17% 1,30%

Decision Support 2,11% 0,52% 1,05%

Beveiliging 1,10% 1,10% 0,95%

Functionaliteiten organisatorische werkeenheden / ketens 0,68% 0,52% 0,81%

Anesthesie, OK en IC 0,68% 0,46% 0,81%

Data opslag / Database 0,77% 1,10% 0,78%

Ordercommunicatie 0,81% 5,17% 0,70%

Afspraken planning 0,81% 4,58% 0,70%

Opnameplanning en -registratie 0,81% 4,58% 0,70%

Beddenplanning 0,81% 0,46% 0,70%

Applicatie 0,77% 1,47% 0,48%

Platform 0,77% 0,73% 0,48%

Patiëntenlogistiek 0,54% 0,46% 0,47%

Wachtlijstbeheer 0,54% 0,46% 0,47%

Spoedeisende hulp 0,14% 0,52% 0,35%

Patiënt monitoring en datacollectie (PDMS) 0,14% 5,17% 0,35%

Open vragen - techniek 0,77% 0,73% 0,29%

Eisen met betrekking tot de leverancier 0,00% 1,60% 0,00%

Commerciële en juridische eisen 0,00% 2,80% 0,00%

Contractvoorwaarden project 1,35% 5,33% 0,00%

Tender

- 29 -

5.2 Analyze the matching results The differences between the percentages of a specific selection criterion in the three tenders were analyzed with the help of an expert within Siemens. It is important to state beforehand that the trends could be based on coincidence as there were only three available data sets. Nonetheless it is interesting and useful to discuss the possible trends, especially the most important ones. An explanation that returned a couple of times was that the (very straightforward) IS that hospital A used before the HIS/EHR system was worse than the one that was used by hospitals B, C, D, and E. Hospital A used Eclypsis and it was iSoft for hospitals B, C, D, and E 8. Some subtopics scored almost the same at tender B and C and different for tender A. This could be explained by the difference in the prior used IS as tender B and C believed the subtopic was a commodity, while tender A missed the subtopic in their IS. The complete list of trend explanation is listed in table 52 (Appendix V). A total of nine subtopics were considered as eye-openers to Siemens and therefore these subtopics are visualized in the seven graphs below. There are only seven graphs, because the explanation of subtopics Beheer and Monitoring en Reporting and of subtopics Implementatie en bedrijfsgerede oplevering and Plan van aanpak voor implementatie en bedrijfsgerede oplevering was the same. The graphs show the percentage of the total score the subtopics accounted for per tender. Below each graph is an explanation why this subtopic was an eye-opener. When it was applicable the trend was supported (or not) by a search to the number of published articled in PubMed (Medline) per year.

Figure 10 – Percentage subtopic Wet en regelgeving accounted for

The graph in figure 10 shows that subtopic Wet en regelgeving became more important over time. According to the expert, this was because the academic hospitals tried to achieve more commitment from the vendors. In order to achieve this commitment the subtopic became more important.

8 http://www.zorgvisie.nl/Nieuws/06072/Overzicht-van-ZISEPDleveranciers-aan-ziekenhuizen.htm

- 30 -

Figure 11 – Percentage subtopic Decision support accounted for

The eye-opener in subtopic Decision support is not the trend in figure 11, but the percentage of the total score it accounted for. The expert expected that this subtopic would have been much more important as the academic hospitals spoke a lot about this subtopic. This hunch of the expert is supported by the trend and the number of published articles in PubMed, see figure 12. Nonetheless the analysis showed that subtopic Decision support did not account for a high percentage to the total score.

Figure 12 – Number of published articles in PubMed per year with topic Decision support

- 31 -

Figure 13 – Percentage subtopic Medicatie accounted for

The trend in figure 13 is due to the fact that the law changed over time. At the time of tender B there was no law regarding medication. But per January 1st 2013 a law will be activated which states that medication has to be prescribed electronically in The Netherlands9. Tender A and C had to take this law into account and therefore subtopic Medicatie became more important.

Figure 14 – Percentage subtopics Implementatie en bedrijfsgerede oplevering and Plan van aanpak voor implementatie en bedrijfsgerede oplevering accounted for

The trend in figure 14 supports the hunch of the expert that academic hospitals try to make use of standard implementations. In the past the academic hospitals wanted full flexibility during their client specific implementation. Nowadays the trend is that academic hospitals want to make use of standard implementations. This trend is supported by the result of the performed analysis.

9 http://medischcontact.artsennet.nl/Nieuws-26/Nieuwsbericht/108012/Nieuw-in-2012-Elektronisch-voorschrijven-moet.htm

- 32 -

Figure 15 – Percentage subtopic Portalen accounted for

Subtopic Portalen was just introduced at the time of tender B and therefore it did not account for a high percentage of the total score in that tender. As visualized in figure 15 it became more important over time and this is supported by the trend in the number of published articles in PubMed, see figure 16.

Figure 16 – Number of published articles in PubMed per year with topic Portal hospital

- 33 -

Figure 17 – Percentage subtopics Beheer and Monitoring en Reporting accounted for

The trend of subtopics Beheer and Monitoring en Reporting is an eye-opener as it supports the hunch of the expert that academic hospitals try to outsource more and more regarding these subtopics. In the past the academic hospitals did it themselves, but the trend is that they want to outsource as much as possible to relieve themselves and to be more efficient. This trend is also supported by the trend in the number of published articles in PubMed, see figure 18.

Figure 18 – Number of published articles in PubMed per year with topic Monitoring + Reporting

- 34 -

Figure 19 – Percentage subtopic Contractvoorwaarden project accounted for

The graph in figure 19 shows that subtopic Contractvoorwaarden project was more important in tender A then in tender B, but it did not exist anymore in tender C. This is due to the fact that the subtopic became a knockout in tender C. If a vendor did not agree with the contract terms in tender C it was excluded from the tender. Therefore you may say that the subtopic became more important over time and this was because the academic hospitals tried to achieve more commitment from the vendors.

5.3 Fairness of tender The fairness in the tenders was checked through the determination of the percentage of the total score the open questions accounted for versus the percentage of the total score the closed questions accounted for. This resulted in the information presented in table 29.

Table 29 – Fairness tender

Table 29 shows the three tenders, the number of open and closed questions and the total points that could be scored in a specific tender. The sixth column shows how many points could be scored through open questions and obviously all other points could be scored through closed questions which are stated in the seventh column. The eighth column states the percentage that the open questions accounted for per tender. For example: vendors could score a total of 75 points in tender A. From these 75 points, 14.6 points could be scored by the open questions. This means that the open questions accounted for 19.5% of the total score in tender A. In the other two tenders the percentage of the total score the open questions accounted for was higher; in tender B it was 48.0% and 46.1% in tender C. This means that almost 50% of the points were scored through subjective judgment of the open questions in tender B and C. Due to this high percentage it could be that the fairness of these two tenders is at stake.

Tender Total #

questions

# open

questions

# closed

questions

Total

score

Open

scored

Closed

scored

% of score

open

% of score

closed

Tender A 1919 171 1748 75,0 14,6 60,4 19,5% 80,5%

Tender B 1581 242 1339 720,0 345,9 374,1 48,0% 52,0%

Tender C 1956 798 1158 860,0 396,5 463,5 46,1% 53,9%

- 35 -

5.4 Contribution of TCO on the total score Het percentage of the actual total score that the total cost of ownership (TCO) accounted for per tender is visualized in figure 20. Actual is italic, because this calculation was done with the total score including all subtopics, thus including price, contract, vision and demonstration.

Figure 20 – Percentage subtopic TCO accounted for

As expected by the expert the graph shows a slowly declining average trend as functional aspects are becoming more important than the price. Academic hospitals rather have a more expensive system scoring good on functional aspects than a cheaper system that scores less on functional aspects.

5.5 Opinions regarding the use of tenders The information of the interviews that were held with a strategic purchaser of hospital A and with the director of D&A Medical Group (a consultancy company) was used to verify the selection criteria and to check what they thought of tenders in general. It was interesting to see that the strategic purchaser believed that tendering was a good procedure while the director did not share that opinion. They both agreed on the fact that it is good that, when performing the tender people are obliged to think about what they really want and what should be distinctive for them. These things are often neglected if people are not obliged to carefully think about it. The strategic purchaser only missed a bit negotiation in the tender in order to see if all parties are on the same page. According the director there are more drawbacks to a tender. It is difficult to see what the effect will be of your questions, and it is hard to formulate the questions in such a way that what is really important for you also is the most important in your questions. Besides that she thought it is a waste of time for both parties. It would be better to conduct a quotation-based method, without all the strict tendering rules, but with some kind of obligation that people of the hospital distinguishes what should be distinctive, otherwise this part will be neglected.

- 36 -

6 Conclusions and future work First the main insights and conclusions of the thesis are presented. The next paragraph discusses the limitations of the thesis and finally some future work directions are presented.

6.1 Conclusion Although the scope of the thesis was limited to the procurement of HIS/EHR systems for Dutch academic hospitals and the thesis was conducted in collaboration with Siemens Healthcare the insights and conclusions are not only interesting for Siemens. Other Dutch stakeholders like general hospitals, academic hospitals, vendors, and/or consultancy companies can definitely use the insights and conclusions to their advantage. The insights and conclusions could even be beneficial for stakeholders in other countries. One of the insights was the identification and ranking of the selection criteria in terms of weightings of the three conducted European tenders by Leiden University Medical Center + University Medical Center Utrecht, University Medical Center St. Radboud, and Erasmus Medical Center + University Medical Center Groningen. As academic hospitals are often an example for general hospitals this could be an important insight for general hospitals. The presented tables in Appendix P, Appendix Q, and Appendix R show which selection criteria were the most important for the academic hospitals and this could give general hospital a head start. It is even possible that general hospitals use the tables as a blueprint for determining their own selection criteria. Obviously also other academic hospitals can use this insight to their advantage and consultancy companies can use it in advising their clients. Due to the RFPs it was known how many points could be scored per specific selection criterion per European tender. To investigate if the selection criteria differed per European tender in terms of weightings one tender was selected as the reference tender and the selection criteria of the other two tenders were matched with that reference tender. The end result is presented in table 28. This table shows the percentage of the total score that each selection criterion accounted for per European tender. It is clear that the percentage that some selection criteria accounted for differed among the conducted European tenders. These differences were analyzed with the help of an expert within Siemens and several trends were determined. It is important to state that the trends could be based on coincidence as there were only three available data sets. Nonetheless it is interesting and useful to discuss the possible trends and especially the nine most important ones as listed in table 30. The first column state the name of the selection criterion and the second column provides an explanation of the trend.

- 37 -

Selection criteria Explanation

Wet en regelgeving The academic hospitals tried to achieve more commitment from the vendors; therefore this subtopic became more important over time.

Decision Support This subtopic scored surprisingly low, the expert of Siemens thought it would be much more important as hospitals spoke a lot about it.

Medicatie The increase was due to the fact that law changed over time. Per January 1st 2013 a law will be activated which states that the medication has to be prescribed electronically. Tender B did not had to consider this law, while the other two tenders had to. Therefore it became more important over time.

Portalen This subtopic just entered the market at the time of tender B and it became more important since then.

Beheer ‘ Monitoring en reporting

In the past the academic hospitals did it themselves, but the trend is that they want to outsource as much as possible to relieve themselves and to be more efficient, therefore both subtopics became more important over time.

Implementatie en bedrijf gerede oplevering ’ Plan van aanpak voor de implementatie en bedrijf gerede oplevering

In the past the academic hospitals wanted full flexibility during their client specific implementation. Nowadays the academic hospitals want to make use of standard implementations; therefore both subtopics show a declining trend.

Contractvoorwaarden project

The academic hospitals tried to achieve more commitment from the vendors by making the selection criteria more important; the last tender even used it as a knockout criterion.

Table 30 – The nine most important trends

The other possible trends are presented with an explanation in Appendix V. The possible trends can be beneficial for every stakeholder in The Netherlands or elsewhere as they could show what the future will be. Upward trends may indicate that the healthcare industry is giving more attention to that selection criterion and vice versa. Vendors could use it to determine their business strategy, the consultancy companies can advise their clients about it, and the hospitals can take it into account in their procurement processes. Concerning the fairness of the European tender the percentage of the total score the open questions accounted for was calculated. In tender B and C almost 50% of the points were scored through subjective judgment of the open questions and in tender A almost 20%. Although expected by the expert one could argue that the fairness is at stake in tender B and C. Not much can be done regarding this conclusion as the rules regarding the European tender does not specify which is the maximum percentage of the total score that open questions may account for.

6.2 Limitations A limitation of this thesis is that the analyses are based on the information of only three tenders. Therefore it could be that the insights and conclusions are based on coincidence. Perhaps the researcher could have used tender D as extra information source and/or he could have acquired tenders from foreign academic hospital. Although both suggestions were left out of scope with a good reason, it could have provided new insights and conclusions would have been based on more information. Another limitation is the matching procedure. The researcher assumed the matching was correct as it was performed by two vendor experts and the Delphi method was used to minimalize the potential matching errors. Although the risk was minimalized it cannot be excluded.

- 38 -

6.3 Future work One obvious expansion of the current research would be the addition of tender E. This will mean an additional data set that could support or reject the recognized trends in this research. Or even other trends will appear or become important while they were not recognized in the current research. Another addition to the current research could be the determination of the level of implementation at (the academic) hospitals, to what extent are the people using the system and how happy are they with the chosen HIS/EHR system? Probably it is too soon to perform a research like that as the academic hospitals are still implementing or choosing the HIS/EHR system, but it will be an interesting topic in the future. Performing an investigation what the real cost of ownership (TCO) of the procurements of HIS/EHR systems were at (academic) hospitals would also be nice addition to the current research as it could determine if the advantages of a HIS/EHR system outweigh the total investments. This research topic encounters the same problem as the last one as it is too soon, but it will be interesting for future research.

- 39 -

Bibliography Alhaqbani, B. & Fidge, C. (2007). Access control requirements for processing electronic health

records. Proceedings of the 2007 international conference on Business process management, 371–382.

Anwar, F. & Shamim, A. (2011). Barriers in Adoption of Health Information Technology in Developing Societies. Int. J Adv. Comput. Sci. Appl, 2(8), 40–45.

Boonstra, A. & Broekhuis, M. (2010). Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Services Research, 10(1), 231. BioMed Central Ltd.

Canton, E., de Bas, P., Meindert, L. & others. (2012). Openbaar aanbesteden: Een vergelijking tussen procedures en technieken in Nederland en de rest van Europa. Veilen van laagwaardig radiospectrum in een VOA-procedure........ 21, 49.

Council, N. S. (2009). Doing business with. Waverley Council.

DesRoches, C. M., Campbell, E. G., Vogeli, C., Zheng, J., Rao, S. R., Shields, A. E., Donelan, K., et al. (2010). Electronic health records’ limited successes suggest more targeted uses. Health Affairs, 29(4), 639–646. Health Affairs.

EU-Commission. (2004). Directive 2004/18/EC. Offical Journal of the European Communities.

Fledderus, M. (2007). Elektronisch zorgdossier: een zorg?: een evaluatiestudie van de invoering van het elektronisch zorgdossier in een verpleeghuis ten behoeve van de verdere implementatie daarvan. University of Twente.

Garde, S. & Knaup, P. (2006). Requirements engineering in health care: the example of chemotherapy planning in paediatric oncology. Requirements Engineering, 11(4), 265–278. Springer.

Garde, S., Knaup, P., Hovenga, E. J. S. & Heard, S. (2007). Towards Semantic Interoperability for Electronic Health Records-Domain Knowledge Governance for open EHR Archetypes. Methods of information in medicine, 46(3), 332–343. Citeseer.

Garets, D. & Davis, M. (2006). Electronic medical records vs. electronic health records: yes, there is a difference. HIMSS Analytics, 2–14.

Gerechtshof-Arnhem. (2008). LJN: BG4586. Gerechtshof Arnhem.

- 40 -

Handel, D. A., Wears, R. L., Nathanson, L. A. & Pines, J. M. (2011). Using information technology to improve the quality and safety of emergency care. Academic Emergency Medicine, 18(6), e45–e51. Wiley Online Library.

Harzing, A. W. (2007). Publish or Perish. Retrieved from http://www.harzing.com/pop.htm

Haux, R. (2006). Health information systems-past, present, future. International Journal of Medical Informatics, 75(3-4), 268–281. Elsevier.

Häyrinen, K., Saranto, K. & Nykänen, P. (2008). Definition, structure, content, use and impacts of electronic health records: a review of the research literature. International journal of medical informatics, 77(5), 291–304. Elsevier.

Hinman, A. R. & Ross, D. A. (2010). Immunization registries can be building blocks for national health information systems. Health Affairs, 29(4), 676–682. Health Affairs.

Hoffman, S. & Podgurski, A. (2009). E-Health hazards: provider liability and electronic health record systems.

Holly, C., Salmond, S. & Saimbert, M. K. (2011). Comprehensive Systematic Review for Advanced Nursing Practice. Springer Publishing Company.

Iakovidis, I. (1998). Towards personal health record: current situation, obstacles and trends in implementation of electronic healthcare record in Europe1. International journal of medical informatics, 52(1-3), 105–115. Elsevier.

IGZ. (2011). Staat van de gezondheidszorg. IGZ.

Jha, A. K., Ferris, T. G., Donelan, K., DesRoches, C., Shields, A., Rosenbaum, S. & Blumenthal, D. (2006). How common are electronic health records in the United States? A summary of the evidence. Health Affairs, 25(6), w496–w507. Health Affairs.

Kaiser, F. (2003). Visualization of process flows in hospital information systems.

Katehakis, D. G., Sfakianakis, S. G., Kavlentakis, G., Anthoulakis, D. N. & Tsiknakis, M. (2007). Delivering a lifelong integrated electronic health record based on a service oriented architecture. Information Technology in Biomedicine, IEEE Transactions on, 11(6), 639–650. IEEE.

Knaup, P., Bott, O., Kohl, C., Lovis, C. & Garde, S. (2007). Electronic patient records: moving from islands and bridges towards electronic health records for continuity of care. Yearb Med Inform, 34–46.

- 41 -

Knoester, T. (2009). Europese aanbesteding. Contractmanagement in de praktijk, 69–77. Springer.

Kuhn, K. & Giuse, D. (2001). From hospital information systems to health information systems. Methods of information in medicine, 40(4), 275–287. FK SCHATTAUER VERLAGSGESELLSCHAFT MBH.

Lenhart, J. G., Honess, K., Covington, D. & Johnson, K. E. (2000). An analysis of trends, perceptions, and use patterns of electronic medical records among US family practice residency programs. Fam Med, 32(2), 109–14.

Malliarou, M. & Zyga, S. (2009). Advantages of Information Systems in Health Services. SMIJ, 5(2).

Mandell, S. F. (1986). The Request for Proposal (RFP). Journal of Medical Systems, 10(1), 31–39. Springer.

Mehrotra, A., Epstein, A. M. & Rosenthal, M. B. (2006). Do integrated medical groups provide higher-quality medical care than individual practice associations? Annals of Internal Medicine, 145(11), 826. Am Coll Physicians.

NHS-Information-Governance. (2007). Guidance on Legal and Professional Obligations. Department of Health.

Park, R. W., Shin, S. S., Choi, Y. I., Ahn, J. O. & Hwang, S. C. (2005). Computerized physician order entry and electronic medical record systems in Korean teaching and general hospitals: results of a 2004 survey. Journal of the American Medical Informatics Association, 12(6), 642–647. BMJ Publishing Group Ltd.

Rodriguez-Priego, E., Garc’\ia-Izquierdo, F. & Rubio, Á. (2010). Modeling issues: a survival guide for a non-expert modeler. Model Driven Engineering Languages and Systems, 361–375. Springer.

Sidiropoulos, A., Katsaros, D. & Manolopoulos, Y. (2007). Generalized Hirsch h-index for disclosing latent facts in citation networks. Scientometrics, 72(2), 253–280. Akadémiai Kiadó, co-published with Springer Science+ Business Media BV, Formerly Kluwer Academic Publishers BV.

Simon, S. R., Kaushal, R., Cleary, P. D., Jenter, C. A., Volk, L. A., Poon, E. G., Orav, E. J., et al. (2007). Correlates of electronic health record adoption in office practices: a statewide survey. Journal of the American Medical Informatics Association, 14(1), 110–117. Elsevier.

Skulmoski, G. J., Hartman, F. T. & Krahn, J. (2007). The Delphi method for graduate research. Journal of information technology education, 6, 1. Informing Science Institute.

- 42 -

Sonoda, E. (2011). Evolution of electronic medical record solutions. Fujitsu Sci. Tech. J, 47(1), 19–27.

Tange, H. J., Hasman, A., de Vries Robbé, P. F. & Schouten, H. C. (1997). Medical narratives in electronic medical records. International journal of medical informatics, 46(1), 7–29. Elsevier.

Temple-Bird, C. & Parsons, R. (2006). “How to manage”series for healthcare technology. IEE.

Terry, N. (2008). Personal Health Records: Directing More Costs and Risks to Consumers? Saint Louis U. Legal Studies Research Paper No. 2008-20, Drexel Law Review, Vol. 1, p. 216, 2008.

Tsai, J. & Bond, G. (2008). A comparison of electronic records to paper records in mental health centers. International Journal for Quality in Health Care, 20(2), 136–143. ISQHC.

UN, U. (2002). World Population Ageing: 1950-2050. United Nations, New York.

Wells, L. S. M. & others. (2009). Getting evidence to and from general practice consultations for cardiovascular risk management using computerised decision support.

Winter, A., Brigl, B. & Wendt, T. (2003). Modeling hospital information systems (part 1): the revised three-layer graph-based meta model 3LGM2. Methods of Information in Medicine-Methodik der Information in der Medizin, 42(5), 544–551. Stuttgart [etc.] FK Schattauer [etc.].

Winter, A., Haux, R., Ammenwerth, E., Brigl, B., Hellrung, N. & Jahn, F. (2010). Health Information Systems: Architectures and Strategies. Springer Verlag.

Winter, A., Haux, R., Ammenwerth, E., Brigl, B., Hellrung, N. & Jahn, F. (2011). Health Information Systems. Health Information Systems, 33–42. Springer.

Winter, AF, Ammenwerth, E., Bott, O., Brigl, B., Buchauer, A., Gräber, S., Grant, A., et al. (2001). Strategic information management plans: the basis for systematic information management in hospitals. International Journal of Medical Informatics, 64(2), 99–109. Elsevier.

Wu, R. C. & Straus, S. E. (2006). Evidence for handheld electronic medical records in improving care: a systematic review. BMC medical informatics and decision making, 6(1), 26. BioMed Central Ltd.

Wu, S., Chaudhry, B., Wang, J., Maglione, M., Mojica, W., Roth, E., Morton, S. C., et al. (2006). Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Annals of internal medicine, 144(10), 742–752.

- 43 -

Zhang, R. & Liu, L. (2010). Security models and requirements for healthcare application clouds. Cloud Computing (CLOUD), 2010 IEEE 3rd International Conference on, 268–275.

- 44 -

Appendices Appendix A Interview University Medical Center St Radboud 45 Appendix B Activity diagram research method HIS 46 Appendix C Results Publish or Perish EMR + HIS 47 Appendix D Rephrased definition regarding EHR (figure 9) 48 Appendix E Rephrased definition regarding EMR 51 Appendix F Reference Concept Map regarding EMR 54 Appendix G Rephrased definition regarding HIS (figure 23) 55 Appendix H Reference Concept Map regarding HIS 57 Appendix I Score of considered (sub) topics tender 58 Appendix J Score of considered (sub) topics tender 59 Appendix K Score of considered (sub) topics tender 60 Appendix L Matching tender B with tender 61 Appendix M Matching tender C with tender 62 Appendix N Result matching tender B with tender 63 Appendix O Result matching tender C with tender A 65 Appendix P Ranking selection criteria of tender A before the matching 66 Appendix Q Ranking selection criteria of tender B before the matching 67 Appendix R Ranking selection criteria of tender C before the matching 68 Appendix S Ranking selection criteria after matching based on tender A 69 Appendix T Ranking selection criteria after matching based on tender B 70 Appendix U Ranking averaged selection criteria of the tenders after the matching 71 Appendix V Explanation of trends 72

- 45 -

Appendix A Interview University Medical Center St Radboud In this appendix it is demonstrated that nothing can be found regarding the homemade Information System of University Medical Center St Radboud and therefore an interview has been conducted. Table 31 shows that the used search queries in order to try to find academic literature did not provide any papers. In order to maintain confidentiality the contact person will remain classified and only the questions that are used for the thesis are provided here.

Questionnaire

Organization: University Medical Center St Radboud Date: 10/10/12 Time: 10:00

Contact person: Classified

Question 1: What was the starting point of the UMCN with respect to Information Systems and

how did it develop?

The UMCN had developed an Information System itself, called Eclipses, but it was unfeasible to keep the ICT management under control. It became too big and therefore the decision was made to start a procurement to go for the best-of-breed HIS solution.

Question 2: Did UMCN look at the conducted tender of LUMC/UMCU and what aspects were

taken into consideration?

The LUMC/UMCU tender used a competitive dialogue procedure to procure their Information System. LUMC/UMCU used this procedure as the HIS/EHR system was not a standard solution at the time. UMCN wanted to procure an Information System as the own developed Eclipses could not compete with the better HIS/EHR systems that were the standard solution. As there was already a standard solution the UMCN could use the open or restricted tender. As there were only six vendors (Chipsoft, iSoft, EPIC, Siemens, McKesson, and Alert) and it is obliged that minimum five vendors are included in the RFP, UMCN did choose for an open tender. The benefits of a restricted tender in terms of reviewing five instead of six vendors did not outweigh the extra work of creating a RFI. The EMCR/UMCG tender only asked how UMCN handled the procedure, further no information was exchanged.

Search query Number of articles

Search query Number of articles

Information system(s) + UMCN 0 Eclipses + UMCN 0

Information system(s) + Nijmegen 0 Eclipses + Nijmegen 0

Information system(s) + St. Radboud 0 Eclipses + St. Radboud 0

Information system(s) + Eclipses 0 Table 31 – Interview with strategic purchaser of University Medical Center St Radboud

- 46 -

Appendix B Activity diagram research method HIS

Figure 21 – Activity diagram research method HIS

- 47 -

Appendix C Results Publish or Perish EMR + HIS

Search query Total number of articles

Contemporary h-index

Relevant articles

Number of different definitions

"definition of emr" 32 7 3 3

"definition of electronic medical record" OR "definition of electronic medical records"

11 4 3 3

"ehr is defined" 71 6 1 1

"electronic health record is defined" OR "electronic health records are defined"

16 5 1 1

"emr is defined" 85 11 1 1

"electronic medical record is defined" OR "electronic medical records are defined"

7 3 2 2

Second phase: citation chase 38 0 1 1

260 26 12 12

Table 32 – Results Publish or Perish EMR search queries

Search query Total number of articles

Contemporary h-index

Relevant articles

Number of different definitions

"definition of hospital information systems" OR "definition of hospital information system"

12 6 5 4

"hospital information system is defined" OR "hospital information systems are defined"

3 1 1 1

Second phase: citation chase 2 1 1

17 7 7 6

Table 33 – Results Publish or Perish HIS search queries

- 48 -

Appendix D Rephrased definition regarding EHR (figure 9)

# Source Original definition (EHR is/are … ) Rephrased definition

1 (Jha et al., 2006) key components like electronic documentation of vendors’ notes, electronic viewing of laboratory and radiology results, and electronic prescribing

an EHR has key components like several clinical functions

2 (Häyrinen et al., 2008) a repository of patient data in digital form, stored and exchanged securely, and accessible by multiple authorized users

an EHR is a repository of patient data in digital form, stored and exchanged securely, and accessible by multiple authorized users

3 (Iakovidis, 1998) digitally stored health care information about an individual’s lifetime with the purpose of supporting continuity of care education and research, and ensuring confidentiality at all times

an EHR is health care information about an individual's lifetime stored digitally ensuring confidentiality at all times with the purpose of supporting continuity of care education and research

4 (Simon et al., 2007) an integrated clinical information system that tracks patient health data and may include such functions as visit notes, prescriptions, lab orders, etc.

an EHR is a clinical information system that tracks health data of a patient including clinical functions

5 (Katehakis et al., 2007) a repository of information regarding the health status of a subject of care in computer processable form, stored and transmitted securely, and accessible by multiple authorized users

an EHR is a repository of information regarding the health status of a subject of care in computer processable form, stored and transmitted securely, and accessible by multiple authorized users

6 (Hoffman & Podgurski, 2009) a repository of electronically maintained information about an individual‘s lifetime health status and health care

an EHR is a repository of the health status and health care information of an individual's lifetime maintained electronically

7 (Terry, 2008) an electronic record of health-related information on an individual that is created, gathered, managed, and consulted by authorized health care clinicians and staff

an EHR is an electronic record of health-related information on an individual that is created, gathered, managed, and consulted by authorized health care clinicians and staff

- 49 -

# Source Original definition (EHR is/are … ) Rephrased definition

8 (DesRoches et al., 2010) adoption of twenty-four clinical functions across all major clinical units in the hospital

an EHR is an EHR if it has adopted twenty-four clinical functions across all major clinical units in the hospital

9 (Garde & Knaup, 2006) a repository of information regarding the health status of a subject of care in computer processable form, stored and transmitted securely, and accessible by multiple authorized users

an EHR is a repository of information regarding the health status of a subject of care in computer processable form, stored and transmitted securely, and accessible by multiple authorized users

10 (Zhang & Liu, 2010) a subset of EMR record maintained by each CDO and is created and owned by the patient. An EHR typically has patient input and access that spans episodes of care across multiple CDOs within a community, region, or state

an EHR is a subset of EMR record which is created and owned by the patient and has patient input and access that spans episodes of care across multiple CDOs

11 (Garde et al., 2007) a repository of information regarding the health status of a subject of care in computer processable form, stored and transmitted securely, and accessible by multiple authorized users

a repository of information regarding the health status of a subject of care in computer processable form, stored and transmitted securely, and accessible by multiple authorized users

12 (Garets & Davis, 2006) A subset of each care delivery organization s EMR, presently assumed to be summaries like ASTM s Continuity of Care Record (CCR) or HL7 s Continuity of Care Document (CCD), is owned by the patient and has patient input and access that spans episodes of care across multiple CDOs within a community, region, or state

an EHR is a subset of each CDO's EMR which is owned by the patient and has patient input and access that spans episodes of care across multiple CDOs

13 (Knaup et al., 2007) a repository of information regarding the health status of a subject of care in computer processable form, stored and transmitted securely, and accessible by multiple authorized users

an EHR is a repository of information regarding the health status of a subject of care in computer processable form, stored and transmitted securely, and accessible by multiple authorized users

- 50 -

# Source Original definition (EHR is/are … ) Rephrased definition

14 (Alhaqbani & Fidge, 2007) digitally stored healthcare information about an individual’s lifetime with the purpose of supporting continuity of care, education and research, and ensuring confidentiality at all times

an EHR is health care information about an individual's lifetime stored digitally ensuring confidentiality at all times with the purpose of supporting continuity of care education and research

15 (Hinman & Ross, 2010) an electronic record of the range of services received by a single patient within his or her lifetime from various vendors and across a series of institutions AND/OR health-related information on an individual that conforms to nationally recognized interoperability standards and that can be created, managed, and consulted by authorized clinicians and staff across more than one health care organization

an EHR is an electronic record containing health-related information of a patient lifetime created, managed, and consulted by authorized health care clinicians and staff

Table 34 – Original and rephrased definitions of an EHR

- 51 -

Appendix E Rephrased definition regarding EMR (figure 22)

# Source Original definition (EMR is/are … ) Rephrased definition

1 (Mehrotra, Epstein, & Rosenthal, 2006) a computerized database containing a medical record for each patient

an emr is a medical record for each patient that is computer-based

2 (Park, Shin, Choi, Ahn, & Hwang, 2005) a computer application with which health care personnel enter all of the medical records related to the patient care. It is a comprehensive system that includes all the patient’s health care records, such as admission and progress notes, operation notes, anesthesia notes, discharge summaries, and nurses’ records

an emr is a medical record for each patient that is computer-based and entered by health care personnel

3 (Zhang & Liu, 2010) the legal record of what happened to the patient during their encounter at a Care Delivery Organization (CDO) across inpatient and outpatient environments and is owned by the CDO

an emr is a legal record of what happened to the patient, owned by CDO, during their encounter at a single Care Delivery Organization (CDO) across inpatient and outpatient environments

4 (R. C. Wu & Straus, 2006) the computerization of health record content and associated processes usually referring to an electronic medical health record in a physician office setting or a computerized system of files

an emr is a medical record containing health record content in an electronic manner at a single Care Delivery Organization (CDO)

5 (Wells & others, 2009) computer-based clinical data of an individual that are location specific and kept by a single physician office or practice, community health center or possibly ambulatory clinic

an emr is clinical data of an individual that is computer-based at a single Care Delivery Organization (CDO)

- 52 -

# Source Original definition (EMR is/are … ) Rephrased definition

6 (Sonoda, 2011) 1. Level 1 Electronic patient information that is handled within a department 2. Level 2 Electronic patient information that is handled across multiple departments 3. Level 3 Patient information that is (mostly) handled within a single medical institution 4. Level 4 Patient information that is handled across multiple medical institutions 5. Level 5 Healthcare-related information that is handled in addition to medical-care information

an emr is patient information that is handled within a single Care Delivery Organization (CDO) (JAHIS states that level 3 is the most acquired level)

7 (Garets & Davis, 2006) An application environment composed of the clinical data repository, clinical decision support, controlled medical vocabulary, order entry, computerized vendor order entry, pharmacy, and clinical documentation applications. This environment supports the patient s electronic medical record across inpatient and outpatient environments, and is used by healthcare practitioners to document, monitor, and manage health care delivery within a care delivery organization (CDO). The data in the EMR is the legal record of what happened to the patient during their encounter at the CDO and is owned by CDO.

an emr is an application environment which supports the medical record of a patient in an electronic manner, is owned by CDO and used in a single Care Delivery Organization (CDO) across inpatient and outpatient environments

- 53 -

# Source Original definition (EMR is/are … ) Rephrased definition

8 (Hinman & Ross, 2010) an electronic record of health-related information on an individual that can be created, gathered, managed, and consulted by authorized clinicians and staff within one health care organization

an emr is an electronic record of health-related information on an individual that can be created, gathered, managed, and consulted by authorized clinicians and staff within a single Care Delivery Organization (CDO)

9 (Lenhart, Honess, Covington, & Johnson, 2000) an interoffice electronic software and hardware system that captures the essential components of a patient’s medical encounter with the vendor, including subjective, objective, assessment, and plan

an emr is a system that captures the essential components of a medical encounter by a patient in an electronic manner with the vendor

10 (Boonstra & Broekhuis, 2010) a computerized medical information systems that collect, store and display patient information

an emr is patient information that is computer-based

11 (Tsai & Bond, 2008) a medical records located on a shared computer network that are both read and written electronically on a relational database through a graphic user interface.

an emr is a medical record that is read and written in an electronic manner located on a shared computer network

12 (Tange, Hasman, de Vries Robbé, & Schouten, 1997)

a repository for patient information within one health-care enterprise (e.g. within one hospital, author’s note) that is supported by direct computer input and integrated with other information sources

an emr is a repository for patient information within a single Care Delivery Organization (CDO)

Table 35 – Original and rephrased definitions of an EMR

- 54 -

Appendix F Reference Concept Map regarding EMR

Figure 22 – RCM regarding the definition of an EMR

Table 36 – Google hits regarding the definition of

an EMR

- 55 -

Appendix G Rephrased definition regarding HIS (figure 23)

# Source Original definition (EMR is/are … ) Rephrased definition

1 (Kuhn & Giuse, 2001) The hospital information system is that socio-technical subsystem of a hospital which allows constructing and managing communication and interoperation by presenting information at the right time, in the right place to the right people.

A HIS is a socio-technical subsystem which allows constructing and interoperation the communication and interoperation by presenting information at the right time, in the right place to the right people

2 (AF Winter et al., 2001) A hospital information system is that socio-technical subsystem of a hospital, which comprises all information processing actions as well as the associated human or technical actors in their respective information processing role.

A HIS is a socio-technical subsystem which comprises all information processing actions as well as the associated human or technical actors in their respective information processing role

3 (A. Winter et al., 2011) A hospital information system is the socio-technical subsystem of a hospital, which comprises all information processing as well as the associated human or technical actors in their respective information processing roles.

A HIS is a socio-technical subsystem which comprises all information processing as well as the associated human or technical actors in their respective information processing role

4 (A. Winter et al., 2010) A hospital information system is the socio-technical subsystem of a hospital, which comprises all information processing as well as the associated human or technical actors in their respective information processing roles.

A HIS is a socio-technical subsystem which comprises all information processing as well as the associated human or technical actors in their respective information processing role

5 (Anwar & Shamim, 2011) HIS is an N-tier application suit built for a single location or multi location environment. Important features of an effective and functional HIS should include easy, friendly and ready to use, well integrated, customization property and possible tracking and alert facility. Least but not last automation back up is necessary so that no data loss should occur.

A HIS is a N-tier application which should include easy, friendly and ready to use, well integrated, customization property and possible tracking and alert facility and has an automated back up

- 56 -

# Source Original definition (EMR is/are … ) Rephrased definition

6 (Kaiser, 2003) A hospital information system is defined as a subsystem of a hospital which comprises all information processing as well as the associated human or technical actors in their respective information processing roles

A HIS is a subsystem which comprises all information processing as well as the associated human or technical actors in their respective information processing role

7 (A. Winter, Brigl, & Wendt, 2003) The hospital information system is that socio-technical subsystem of a hospital which allows constructing and managing communication and interoperation by presenting information at the right time, in the right place to the right people.

A HIS is a socio-technical subsystem which allows constructing and interoperation the communication and interoperation by presenting information at the right time, in the right place to the right people

Table 37 – Original and rephrased definitions of a HIS

- 57 -

Appendix H Reference Concept Map regarding HIS

Figure 23 – RCM regarding the definition of an HIS

Table 38 – Google hits regarding the definition

of a HIS

- 58 -

Appendix I Score of considered (sub) topics tender A

Table 39 – Score considered (sub) topics in tender A

Topic Name topic Points that can be

scored per topic

Percentage of

total score

1.1 Algemene functionele eisen 5,50 7,33%

1.2 Functionele eisen perceel 1 30,25 40,33%

1.3 Functionele eisen perceel 2 8,25 11,00%

1.4 Technische eisen 5,50 7,33%

1.5 Integratie 5,50 7,33%

2.1 Eisen dienstverlening service 6,00 8,00%

2.2 Eisen dienstverlening implementatie 10,00 13,33%

2.3 Contractvoorwaarden project 4,00 5,33%

Subtopic Name subtopic Points that can be

scored per subtopic

Percentage of

total score

1.1.1  Wet en regelgeving 1,10 1,47%

1.1.2  Gebruiksvriendelijkheid 2,75 3,67%

1.1.3  Beschikbaarheid en performance 1,10 1,47%

1.1.4  Beveiliging van informatie 0,55 0,73%

1.2.1  Klinische dossiervoering 3,88 5,17%

1.2.2  Zorgpaden en zorgplannen 0,39 0,52%

1.2.3  Ordercommunicatie 3,88 5,17%

1.2.4  Decision Support 0,39 0,52%

1.2.5  Communicatie / Correspondentie 0,39 0,52%

1.2.6  Medicatie 3,88 5,17%

1.2.7  Medische bibliotheek / knowledge base 3,88 5,17%

1.2.8  Registratie van prestatie-indicatoren 0,39 0,52%

1.2.9 Spoedeisende hulp 0,39 0,52%

1.2.10 Portalen 3,88 5,17%

1.2.11 Patiënt monitoring en datacollectie (PDMS) 3,88 5,17%

1.2.12 Rapportages 0,39 0,52%

1.2.13 Administratie Medisch wetenschappelijk onderzoek 3,88 5,17%

1.2.14 Administratie en monitoring onderwijs 0,39 0,52%

1.2.15 Functionaliteiten organisatorische werkeenheden / ketens 0,39 0,52%

1.3.1  Afspraken planning 3,44 4,58%

1.3.2  Opnameplanning en -registratie 3,44 4,58%

1.3.3  Beddenplanning 0,34 0,46%

1.3.4  Patiëntenlogistiek 0,34 0,46%

1.3.5  Wachtlijstbeheer 0,34 0,46%

1.3.6  Anesthesie, OK en IC 0,34 0,46%

1.4.1  Applicatie 1,10 1,47%

1.4.2  Platform 0,55 0,73%

1.4.3  Data opslag / Database 0,83 1,10%

1.4.4  Beveiliging 0,83 1,10%

1.4.5  Beheer 0,83 1,10%

1.4.6  Monitoring en reporting 0,83 1,10%

1.4.7  Open vragen - techniek 0,55 0,73%

1.5.1  Eisen Service Oriented Architecture 3,30 4,40%

1.5.2  Open vragen – integratie 2,20 2,93%

2.1.1 Eisen met betrekking tot de leverancier 1,20 1,60%

2.1.2 Commerciële en juridische eisen 2,10 2,80%

2.1.3 Serviceondersteuning en –contract (service level agreement) 2,10 2,80%

2.1.4 Innovatie 0,60 0,80%

2.2.1 Implementatie en bedrijfsgerede oplevering 3,00 4,00%

2.2.2 Plan van aanpak voor de implementatie en bedrijfsgerede oplevering 4,00 5,33%

2.2.3 Documentatie en templates 1,00 1,33%

2.2.4 Opleiding en instructie 1,00 1,33%

2.2.5 Conversie 1,00 1,33%

2.3 Contractvoorwaarden project 4,00 5,33%

- 59 -

Appendix J Score of considered (sub) topics tender B

Table 40 – Score considered (sub) topics in tender B

Topic Name topic Points that can be

scored per topic

Percentage of

total score

1-2 Zeer belangrijke wensen 400 54,05%

2-1 Implementatieplan 200 27,03%

2-2 Best practices 50 6,76%

2-3 Risico’s 20 2,70%

2-4 Onderhoud en Beheer 30 4,05%

3-1 Presentatie Implementatieplan 40 5,41%

Subtopic Name subtopic Points that can be

scored per subtopic

Percentage of

total score

1-2-1 Functionele rijkheid wanneer beschikbaar 80 10,81%

1-2-2 Integraliteit (presenteren, gebruik van gegevens) 20 2,70%

1-2-3 Dossier – gebruiksgemak 28 3,78%

1-2-4 Dossier – delen van gegevens 20 2,70%

1-2-5 Dossier – onderzoek 40 5,41%

1-2-6 Dossier – onderwijs 20 2,70%

1-2-7 Zorgprocessen – toegankelijkheid en overzicht 16 2,16%

1-2-8 Zorgprocessen – ondersteuning van het zorgproces 16 2,16%

1-2-9 Integrale planning – integraliteit deelplanningen 28 3,78%

1-2-10 Integrale planning – planningsvoorkeuren 12 1,62%

1-2-11 Transmurale communicatie – functionaliteit 6 0,81%

1-2-12 Transmurale communicatie – openheid van het syteem 6 0,81%

1-2-13 Stuurinformatie, operationele rapportages 8 1,08%

1-2-14 Zorgadministratie, DBC’s en medische facturatie 20 2,70%

1-2-15 ICT - Beschikbaarheiden betrouwbaarheid 60 8,11%

1-2-16 ICT - Bevoegdheden en veiligheid 20 2,70%

2-1-1 Projectorganisatie en projectmanagement 6 0,81%

2-1-2 Werkwijze, methode en tools 6 0,81%

2-1-3 Fasering 16 2,16%

2-1-4 Tijdsplanning 8 1,08%

2-1-5 Capaciteitsplanning 6 0,81%

2-1-6 Taakverdeling 4 0,54%

2-1-7 Opleiding en training van projectmedewerkers en beheerders 16 2,16%

2-1-8 Opleiding eindgebruikers 12 1,62%

2-1-9 Testen en acceptatie 16 2,16%

2-1-10 Verandermanagement 4 0,54%

2-1-11 Interfaces en integratie 6 0,81%

2-1-12 Pakketaanpassingen en maatwerk 6 0,81%

2-1-13 Conversie en schoning 16 2,16%

2-1-14 Rapportages 6 0,81%

2-1-15 Go-live en nazorg 20 2,70%

2-1-16 Kwaliteitszorg 6 0,81%

2-1-17 Documentatie 6 0,81%

2-1-18 Randvoorwaarden aan de Opdrachtgever 10 1,35%

2-1-19 Samenwerking UMC's 10 1,35%

2-1-20 Plan van aanpak na de beslisimplementatie 20 2,70%

2-2-1 Kwaliteit van de best practices 30 4,05%

2-2-2 Bruikbaarheid van de best practices 10 1,35%

2-2-3 Toekomstvastigheid van de best practices 10 1,35%

2-3 Risico’s 20 2,70%

2-4-1 Inrichting organisatie 15 2,03%

2-4-2 Service Level Agreement 15 2,03%

3-1-1 Competenties projectteam 20 2,70%

3-1-2 Toelichting projectaanpak 20 2,70%

- 60 -

Appendix K Score of considered (sub) topics tender C

Table 41 – Score considered (sub) topics in tender C

Topic Name topic Points that can be

scored per topic

Percentage of

total score

Z.1 Compleetheid van Product 140 16,28%

Z.2 Aangeboden functionaliteit 440 51,16%

Y.1 Technische flexibiliteit 50 5,81%

Y.2 Beheersbaarheid 50 5,81%

Y.3 Architectuur 50 5,81%

X.1 Conversie 30 3,49%

X.2 Testen 20 2,33%

X.3 Implementatieplan en nazorg 80 9,30%

Subtopic Name subtopic Points that can be

scored per subtopic

Percentage of

total score

Z.1.1 Mate van compleetheid huidig product 80 9,30%

Z.1.2 Productenkalender en visie op doorontwikkeling 60 6,98%

Z.2.1 Integrale functionaliteit 40 4,65%

Z.2.2 Het helpt de gebruiker 40 4,65%

Z.2.3 Integrale planning, ordering en signalering 40 4,65%

Z.2.4 Een dossier van een patient 40 4,65%

Z.2.5 Zorgtrajecten en zorgpaden 40 4,65%

Z.2.6 Transmuraal 40 4,65%

Z.2.7 Onderzoek 40 4,65%

Z.2.8 Onderwijs 40 4,65%

Z.2.9 Zorgadministratie en financien 40 4,65%

Z.2.10 Management informatie 40 4,65%

Z.2.11 Medicatie 40 4,65%

Y.1 Technische flexibiliteit 50 5,81%

Y.2 Beheersbaarheid 50 5,81%

Y.3 Architectuur 50 5,81%

X.1 Conversie 30 3,49%

X.2 Testen 20 2,33%

X.3 Implementatieplan en nazorg 80 9,30%

- 61 -

Appendix L Matching tender B with tender A

Table 42 – Matching subtopics of tender B with subtopics of tender A

Tender B

1-2-1 2.1.4 100%

1-2-2 1.1.2  5% 1.2.1  10% 1.2.2  10% 1.2.5  15% 1.2.7  10% 1.2.10 10% 1.2.13 5% 1.2.14 5% 1.2.15 10% 1.5.2  20%

1-2-3 1.1.2  40% 1.1.3  10% 1.1.4  5% 1.2.1  20% 1.2.4  20% 2.2.3 5%

1-2-4 1.1.2  5% 1.1.4  20% 1.2.1  15% 1.2.5  20% 1.2.6  10% 1.2.9 5% 1.2.11 5% 1.2.15 15% 1.3.6  5%

1-2-5 1.2.13 100%

1-2-6 1.2.14 100%

1-2-7 1.1.2  5% 1.2.1  15% 1.2.2  40% 1.2.4  25% 1.2.5  5% 1.2.7  10%

1-2-8 1.1.2  5% 1.2.1  15% 1.2.2  40% 1.2.4  25% 1.2.5  5% 1.2.7  10%

1-2-9 1.1.2  5% 1.2.3  15% 1.2.4  5% 1.3.1  15% 1.3.2  15% 1.3.3  15% 1.3.4  10% 1.3.5  10% 1.3.6  10%

1-2-10 1.1.2  5% 1.2.3  15% 1.2.4  5% 1.3.1  15% 1.3.2  15% 1.3.3  15% 1.3.4  10% 1.3.5  10% 1.3.6  10%

1-2-11 1.2.5  20% 1.2.10 60% 1.4.4  20%

1-2-12 1.2.5  20% 1.2.10 60% 1.4.4  20%

1-2-13 1.2.8  30% 1.2.12 60% 1.2.13 10%

1-2-14 1.2.1  100%

1-2-15 1.1.2  10% 1.1.3  10% 1.1.4  10% 1.4.1  7% 1.4.2  7% 1.4.3  7% 1.4.4  7% 1.4.5  7% 1.4.6  7% 1.4.7  7% 1.5.1  5% 1.5.2  5% 2.2.5 10%

1-2-16 1.1.2  10% 1.1.3  10% 1.1.4  10% 1.4.1  7% 1.4.2  7% 1.4.3  7% 1.4.4  7% 1.4.5  7% 1.4.6  7% 1.4.7  7% 1.5.1  5% 1.5.2  5% 2.2.5 10%

2-1-1 2.2.2 100%

2-1-2 2.2.1 100%

2-1-3 2.2.1 100%

2-1-4 2.2.1 100%

2-1-5 2.2.1 100%

2-1-6 2.2.2 100%

2-1-7 2.2.4 100%

2-1-8 2.2.4 100%

2-1-9 2.2.1 100%

2-1-10 2.2.2 100%

2-1-11 2.2.1 100%

2-1-12 2.2.2 100%

2-1-13 2.2.5 100%

2-1-14 2.2.2 100%

2-1-15 2.2.1 100%

2-1-16 2.2.2 100%

2-1-17 2.2.3 100%

2-1-18 2.3 100%

2-1-19 2.2.2 100%

2-1-20 2.2.2 100%

2-2-1 2.2.3 100%

2-2-2 2.2.3 100%

2-2-3 2.2.3 100%

2-3 2.2.2 100%

2-4-1 2.1.3 100%

2-4-2 2.1.3 100%

3-1-1 2.2.2 100%

3-1-2 2.2.2 100%

Tender A

- 62 -

Appendix M Matching tender C with tender A

Table 43 – Matching subtopics of tender C with subtopics of tender A

Tender C

Z.1.1 1.1.1  100%

Z.1.2 2.1.4 100%

Z.2.1 1.1.2  10% 1.2.1  10% 1.2.2  10% 1.2.5  10% 1.2.7  10% 1.2.10 10% 1.2.13 5% 1.2.14 5% 1.2.15 10% 1.5.2  20%

Z.2.2 1.1.2  15% 1.1.3  15% 1.2.1  10% 1.2.4  10% 1.1.4  20% 2.2.3 30%

Z.2.3 1.1.2  5% 1.2.3  15% 1.2.4  5% 1.3.1  15% 1.3.2  15% 1.3.3  15% 1.3.4  10% 1.3.5  10% 1.3.6  10%

Z.2.4 1.1.2  15% 1.1.4  5% 1.2.1  25% 1.2.5  25% 1.2.9 8% 1.2.11 8% 1.2.15 8% 1.3.6  8%

Z.2.5 1.1.2  8% 1.2.1  20% 1.2.2  50% 1.2.4  8% 1.2.5  8% 1.2.7  8%

Z.2.6 1.2.5  20% 1.2.10 70% 1.4.4  10%

Z.2.7 1.2.13 100%

Z.2.8 1.2.14 100%

Z.2.9 1.2.1  100%

Z.2.10 1.2.8  30% 1.2.12 60% 1.2.13 10%

Z.2.11 1.2.6 100%

Y.1 1.1.3 5% 1.4.3  5% 1.4.6  25% 1.4.7  5% 1.5.1  30% 1.5.2  30%

Y.2 1.1.1 5% 1.1.3  5% 1.1.4  10% 1.2.5  25% 1.4.5  20% 1.4.6  10% 2.1.3 10% 2.2.3 15%

Y.3 1.1.1 8% 1.1.3  8% 1.1.4  8% 1.2.7  8% 1.2.10 8% 1.4.1  8% 1.4.2  8% 1.4.3  8% 1.4.4  8% 1.4.5  8% 1.4.7  8% 1.5.2  8%

X.1 2.2.5 100%

X.2 2.2.1 25% 2.2.2 25% 2.2.5 25% 1.4.5  25%

X.3 2.1.3 20% 2.2.1 20% 2.2.2 20% 2.2.3 20% 2.2.4 20%

Tender A

- 63 -

Appendix N Result matching tender B with tender A

B / A 1.1.1  1.1.2  1.1.3  1.1.4  1.2.1  1.2.2  1.2.3  1.2.4  1.2.5  1.2.6  1.2.7  1.2.8  1.2.9 1.2.10 1.2.11 1.2.12 1.2.13 1.2.14 1.2.15 1.3.1  1.3.2  1.3.3 

1-2-1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-2 0,00% 0,14% 0,00% 0,00% 0,27% 0,27% 0,00% 0,00% 0,41% 0,00% 0,27% 0,00% 0,00% 0,27% 0,00% 0,00% 0,14% 0,14% 0,27% 0,00% 0,00% 0,00%

1-2-3 0,00% 1,51% 0,38% 0,19% 0,76% 0,00% 0,00% 0,76% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-4 0,00% 0,14% 0,00% 0,54% 0,41% 0,00% 0,00% 0,00% 0,54% 0,27% 0,00% 0,00% 0,14% 0,00% 0,14% 0,00% 0,00% 0,00% 0,41% 0,00% 0,00% 0,00%

1-2-5 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 5,41% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-6 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,70% 0,00% 0,00% 0,00% 0,00%

1-2-7 0,00% 0,11% 0,00% 0,00% 0,32% 0,86% 0,00% 0,54% 0,11% 0,00% 0,22% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-8 0,00% 0,11% 0,00% 0,00% 0,32% 0,86% 0,00% 0,54% 0,11% 0,00% 0,22% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-9 0,00% 0,19% 0,00% 0,00% 0,00% 0,00% 0,57% 0,19% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,57% 0,57% 0,57%

1-2-10 0,00% 0,08% 0,00% 0,00% 0,00% 0,00% 0,24% 0,08% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,24% 0,24% 0,24%

1-2-11 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,16% 0,00% 0,00% 0,00% 0,00% 0,49% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-12 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,16% 0,00% 0,00% 0,00% 0,00% 0,49% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-13 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,32% 0,00% 0,00% 0,00% 0,65% 0,11% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-14 0,00% 0,00% 0,00% 0,00% 2,70% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-15 0,00% 0,81% 0,81% 0,81% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-16 0,00% 0,27% 0,27% 0,27% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-3 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-4 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-5 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-6 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-7 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-8 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-9 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-10 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-11 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-12 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-13 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-14 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-15 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-16 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-17 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-18 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-19 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-20 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-2-1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-2-2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-2-3 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-3 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-4-1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-4-2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

3-1-1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

3-1-2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

100,00% 0,00% 3,35% 1,46% 1,81% 4,78% 2,00% 0,81% 2,11% 1,49% 0,27% 0,70% 0,32% 0,14% 1,24% 0,14% 0,65% 5,65% 2,84% 0,68% 0,81% 0,81% 0,81%

- 64 -

Table 44 – Result matching tender B with tender A

B / A 1.3.4  1.3.5  1.3.6  1.4.1  1.4.2  1.4.3  1.4.4  1.4.5  1.4.6  1.4.7  1.5.1  1.5.2  2.1.1 2.1.2 2.1.3 2.1.4 2.2.1 2.2.2 2.2.3 2.2.4 2.2.5 2.3

1-2-1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 10,81% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,54% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-3 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,19% 0,00% 0,00% 0,00%

1-2-4 0,00% 0,00% 0,14% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-5 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-6 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-7 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-8 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-9 0,38% 0,38% 0,38% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-10 0,16% 0,16% 0,16% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-11 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,16% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-12 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,16% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-13 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-14 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

1-2-15 0,00% 0,00% 0,00% 0,58% 0,58% 0,58% 0,58% 0,58% 0,58% 0,58% 0,41% 0,41% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,81% 0,00%

1-2-16 0,00% 0,00% 0,00% 0,19% 0,19% 0,19% 0,19% 0,19% 0,19% 0,19% 0,14% 0,14% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,27% 0,00%

2-1-1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,81% 0,00% 0,00% 0,00% 0,00%

2-1-2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,81% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-3 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,16% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-4 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,08% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-5 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,81% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-6 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,54% 0,00% 0,00% 0,00% 0,00%

2-1-7 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,16% 0,00% 0,00%

2-1-8 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,62% 0,00% 0,00%

2-1-9 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,16% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-10 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,54% 0,00% 0,00% 0,00% 0,00%

2-1-11 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,81% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-12 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,81% 0,00% 0,00% 0,00% 0,00%

2-1-13 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,16% 0,00%

2-1-14 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,81% 0,00% 0,00% 0,00% 0,00%

2-1-15 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,70% 0,00% 0,00% 0,00% 0,00% 0,00%

2-1-16 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,81% 0,00% 0,00% 0,00% 0,00%

2-1-17 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,81% 0,00% 0,00% 0,00%

2-1-18 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,35%

2-1-19 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,35% 0,00% 0,00% 0,00% 0,00%

2-1-20 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,70% 0,00% 0,00% 0,00% 0,00%

2-2-1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 4,05% 0,00% 0,00% 0,00%

2-2-2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,35% 0,00% 0,00% 0,00%

2-2-3 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,35% 0,00% 0,00% 0,00%

2-3 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,70% 0,00% 0,00% 0,00% 0,00%

2-4-1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,03% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

2-4-2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,03% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

3-1-1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,70% 0,00% 0,00% 0,00% 0,00%

3-1-2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 2,70% 0,00% 0,00% 0,00% 0,00%

100,00% 0,54% 0,54% 0,68% 0,77% 0,77% 0,77% 1,10% 0,77% 0,77% 0,77% 0,54% 1,08% 0,00% 0,00% 4,05% 10,81% 10,54% 16,49% 7,76% 3,78% 3,24% 1,35%

- 65 -

Appendix O Result matching tender C with tender A

Table 45 – Result matching tender C with tender A

C / A 1.1.1  1.1.2  1.1.3  1.1.4  1.2.1  1.2.2  1.2.3  1.2.4  1.2.5  1.2.6  1.2.7  1.2.8  1.2.9 1.2.10 1.2.11 1.2.12 1.2.13 1.2.14 1.2.15 1.3.1  1.3.2  1.3.3 

Z.1.1 9,30% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.1.2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.1 0,00% 0,47% 0,00% 0,00% 0,47% 0,47% 0,00% 0,00% 0,47% 0,00% 0,47% 0,00% 0,00% 0,47% 0,00% 0,00% 0,23% 0,23% 0,47% 0,00% 0,00% 0,00%

Z.2.2 0,00% 0,70% 0,70% 0,93% 0,47% 0,00% 0,00% 0,47% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.3 0,00% 0,23% 0,00% 0,00% 0,00% 0,00% 0,70% 0,23% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,70% 0,70% 0,70%

Z.2.4 0,00% 0,70% 0,00% 0,23% 1,16% 0,00% 0,00% 0,00% 1,16% 0,00% 0,00% 0,00% 0,35% 0,00% 0,35% 0,00% 0,00% 0,00% 0,35% 0,00% 0,00% 0,00%

Z.2.5 0,00% 0,35% 0,00% 0,00% 0,93% 2,33% 0,00% 0,35% 0,35% 0,00% 0,35% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.6 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,93% 0,00% 0,00% 0,00% 0,00% 3,26% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.7 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 4,65% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.8 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 4,65% 0,00% 0,00% 0,00% 0,00%

Z.2.9 0,00% 0,00% 0,00% 0,00% 4,65% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.10 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,40% 0,00% 0,00% 0,00% 2,79% 0,47% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.11 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 4,65% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Y.1 0,00% 0,00% 0,29% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Y.2 0,29% 0,00% 0,29% 0,58% 0,00% 0,00% 0,00% 0,00% 1,45% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Y.3 0,48% 0,00% 0,48% 0,97% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,48% 0,00% 0,00% 0,48% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

X.1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

X.2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

X.3 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

100,00% 10,08% 2,44% 1,76% 2,71% 7,67% 2,79% 0,70% 1,05% 4,36% 4,65% 1,30% 1,40% 0,35% 4,21% 0,35% 2,79% 5,35% 4,88% 0,81% 0,70% 0,70% 0,70%

C / A 1.3.4  1.3.5  1.3.6  1.4.1  1.4.2  1.4.3  1.4.4  1.4.5  1.4.6  1.4.7  1.5.1  1.5.2  2.1.1 2.1.2 2.1.3 2.1.4 2.2.1 2.2.2 2.2.3 2.2.4 2.2.5 2.3

Z.1.1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.1.2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 6,98% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,93% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,40% 0,00% 0,00% 0,00%

Z.2.3 0,47% 0,47% 0,47% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.4 0,00% 0,00% 0,35% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.5 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.6 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,47% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.7 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.8 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.9 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.10 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Z.2.11 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Y.1 0,00% 0,00% 0,00% 0,00% 0,00% 0,29% 0,00% 0,00% 1,45% 0,29% 1,74% 1,74% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

Y.2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,16% 0,58% 0,00% 0,00% 0,00% 0,00% 0,00% 0,58% 0,00% 0,00% 0,00% 0,87% 0,00% 0,00% 0,00%

Y.3 0,00% 0,00% 0,00% 0,48% 0,48% 0,48% 0,48% 0,48% 0,00% 0,00% 0,00% 0,48% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00%

X.1 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 3,49% 0,00%

X.2 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,58% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,58% 0,58% 0,00% 0,00% 0,58% 0,00%

X.3 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 1,86% 0,00% 1,86% 1,86% 1,86% 1,86% 0,00% 0,00%

100,00% 0,47% 0,47% 0,81% 0,48% 0,48% 0,78% 0,95% 2,23% 2,03% 0,29% 1,74% 3,16% 0,00% 0,00% 2,44% 6,98% 2,44% 2,44% 4,13% 1,86% 4,07% 0,00%

- 66 -

Appendix P Ranking selection criteria of tender A before the matching

Table 46 – Ranking original selection criteria of tender A

Name subtopic Points that can be

scored per subtopic

Percentage of

total score

Plan van aanpak voor de implementatie en bedrijfsgerede oplevering 4,00 5,33%

Contractvoorwaarden project 4,00 5,33%

Klinische dossiervoering 3,88 5,17%

Patiënt monitoring en datacollectie (PDMS) 3,88 5,17%

Administratie Medisch wetenschappelijk onderzoek 3,88 5,17%

Ordercommunicatie 3,88 5,17%

Medicatie 3,88 5,17%

Medische bibliotheek / knowledge base 3,88 5,17%

Portalen 3,88 5,17%

Afspraken planning 3,44 4,58%

Opnameplanning en -registratie 3,44 4,58%

Eisen Service Oriented Architecture 3,30 4,40%

Implementatie en bedrijfsgerede oplevering 3,00 4,00%

Gebruiksvriendelijkheid 2,75 3,67%

Open vragen – integratie 2,20 2,93%

Commerciële en juridische eisen 2,10 2,80%

Serviceondersteuning en –contract (service level agreement) 2,10 2,80%

Eisen met betrekking tot de leverancier 1,20 1,60%

Wet en regelgeving 1,10 1,47%

Beschikbaarheid en performance 1,10 1,47%

Applicatie 1,10 1,47%

Documentatie en templates 1,00 1,33%

Conversie 1,00 1,33%

Opleiding en instructie 1,00 1,33%

Data opslag / Database 0,83 1,10%

Beveiliging 0,83 1,10%

Beheer 0,83 1,10%

Monitoring en reporting 0,83 1,10%

Innovatie 0,60 0,80%

Platform 0,55 0,73%

Open vragen - techniek 0,55 0,73%

Beveiliging van informatie 0,55 0,73%

Functionaliteiten organisatorische werkeenheden / ketens 0,39 0,52%

Rapportages 0,39 0,52%

Decision Support 0,39 0,52%

Administratie en monitoring onderwijs 0,39 0,52%

Zorgpaden en zorgplannen 0,39 0,52%

Communicatie / Correspondentie 0,39 0,52%

Registratie van prestatie-indicatoren 0,39 0,52%

Spoedeisende hulp 0,39 0,52%

Beddenplanning 0,34 0,46%

Patiëntenlogistiek 0,34 0,46%

Wachtlijstbeheer 0,34 0,46%

Anesthesie, OK en IC 0,34 0,46%

- 67 -

Appendix Q Ranking selection criteria of tender B before the matching

Table 47 – Ranking original selection criteria of tender B

Name subtopic Points that can be

scored per subtopic

Percentage of

total score

Functionele rijkheid wanneer beschikbaar 80 10,81%

ICT - Beschikbaarheiden betrouwbaarheid 60 8,11%

Dossier – onderzoek 40 5,41%

Kwaliteit van de best practices 30 4,05%

Dossier – gebruiksgemak 28 3,78%

Integrale planning – integraliteit deelplanningen 28 3,78%

Integraliteit (presenteren, gebruik van gegevens) 20 2,70%

Dossier – delen van gegevens 20 2,70%

Dossier – onderwijs 20 2,70%

Zorgadministratie, DBC’s en medische facturatie 20 2,70%

ICT - Bevoegdheden en veiligheid 20 2,70%

Go-live en nazorg 20 2,70%

Plan van aanpak na de beslisimplementatie 20 2,70%

Risico’s 20 2,70%

Competenties projectteam 20 2,70%

Toelichting projectaanpak 20 2,70%

Zorgprocessen – toegankelijkheid en overzicht 16 2,16%

Zorgprocessen – ondersteuning van het zorgproces 16 2,16%

Fasering 16 2,16%

Opleiding en training van projectmedewerkers en beheerders 16 2,16%

Testen en acceptatie 16 2,16%

Conversie en schoning 16 2,16%

Inrichting organisatie 15 2,03%

Service Level Agreement 15 2,03%

Integrale planning – planningsvoorkeuren 12 1,62%

Opleiding eindgebruikers 12 1,62%

Randvoorwaarden aan de Opdrachtgever 10 1,35%

Samenwerking UMC's 10 1,35%

Bruikbaarheid van de best practices 10 1,35%

Toekomstvastigheid van de best practices 10 1,35%

Stuurinformatie, operationele rapportages 8 1,08%

Tijdsplanning 8 1,08%

Transmurale communicatie – functionaliteit 6 0,81%

Transmurale communicatie – openheid van het syteem 6 0,81%

Projectorganisatie en projectmanagement 6 0,81%

Werkwijze, methode en tools 6 0,81%

Capaciteitsplanning 6 0,81%

Interfaces en integratie 6 0,81%

Pakketaanpassingen en maatwerk 6 0,81%

Rapportages 6 0,81%

Kwaliteitszorg 6 0,81%

Documentatie 6 0,81%

Taakverdeling 4 0,54%

Verandermanagement 4 0,54%

- 68 -

Appendix R Ranking selection criteria of tender C before the matching

Table 48 – Ranking original selection criteria of tender C

Name subtopic Points that can be

scored per subtopic

Percentage of

total score

Mate van compleetheid huidig product 80 9,30%

Implementatieplan en nazorg 80 9,30%

Productenkalender en visie op doorontwikkeling 60 6,98%

Technische flexibiliteit 50 5,81%

Beheersbaarheid 50 5,81%

Architectuur 50 5,81%

Integrale functionaliteit 40 4,65%

Het helpt de gebruiker 40 4,65%

Integrale planning, ordering en signalering 40 4,65%

Een dossier van een patient 40 4,65%

Zorgtrajecten en zorgpaden 40 4,65%

Transmuraal 40 4,65%

Onderzoek 40 4,65%

Onderwijs 40 4,65%

Zorgadministratie en financien 40 4,65%

Management informatie 40 4,65%

Medicatie 40 4,65%

Conversie 30 3,49%

Testen 20 2,33%

- 69 -

Appendix S Ranking selection criteria after matching based on tender A

Table 49 – Ranking selection criteria after matching based on tender A

Name subtopic

Tender B Tender A Tender C

Plan van aanpak voor de implementatie en bedrijfsgerede oplevering 16,49% 5,33% 2,44%

Contractvoorwaarden project 1,35% 5,33% 0,00%

Klinische dossiervoering 4,78% 5,17% 7,67%

Ordercommunicatie 0,81% 5,17% 0,70%

Patiënt monitoring en datacollectie (PDMS) 0,14% 5,17% 0,35%

Administratie Medisch wetenschappelijk onderzoek 5,65% 5,17% 5,35%

Medicatie 0,27% 5,17% 4,65%

Medische bibliotheek / knowledge base 0,70% 5,17% 1,30%

Portalen 1,24% 5,17% 4,21%

Afspraken planning 0,81% 4,58% 0,70%

Opnameplanning en -registratie 0,81% 4,58% 0,70%

Eisen Service Oriented Architecture 0,54% 4,40% 1,74%

Implementatie en bedrijfsgerede oplevering 10,54% 4,00% 2,44%

Gebruiksvriendelijkheid 3,35% 3,67% 2,44%

Open vragen – integratie 1,08% 2,93% 3,16%

Commerciële en juridische eisen 0,00% 2,80% 0,00%

Serviceondersteuning en –contract (service level agreement) 4,05% 2,80% 2,44%

Eisen met betrekking tot de leverancier 0,00% 1,60% 0,00%

Wet en regelgeving 0,00% 1,47% 10,08%

Beschikbaarheid en performance 1,46% 1,47% 1,76%

Applicatie 0,77% 1,47% 0,48%

Documentatie en templates 7,76% 1,33% 4,13%

Conversie 3,24% 1,33% 4,07%

Opleiding en instructie 3,78% 1,33% 1,86%

Data opslag / Database 0,77% 1,10% 0,78%

Beveiliging 1,10% 1,10% 0,95%

Beheer 0,77% 1,10% 2,23%

Monitoring en reporting 0,77% 1,10% 2,03%

Innovatie 10,81% 0,80% 6,98%

Platform 0,77% 0,73% 0,48%

Open vragen - techniek 0,77% 0,73% 0,29%

Beveiliging van informatie 1,81% 0,73% 2,71%

Functionaliteiten organisatorische werkeenheden / ketens 0,68% 0,52% 0,81%

Decision Support 2,11% 0,52% 1,05%

Rapportages 0,65% 0,52% 2,79%

Zorgpaden en zorgplannen 2,00% 0,52% 2,79%

Communicatie / Correspondentie 1,49% 0,52% 4,36%

Registratie van prestatie-indicatoren 0,32% 0,52% 1,40%

Spoedeisende hulp 0,14% 0,52% 0,35%

Administratie en monitoring onderwijs 2,84% 0,52% 4,88%

Beddenplanning 0,81% 0,46% 0,70%

Patiëntenlogistiek 0,54% 0,46% 0,47%

Wachtlijstbeheer 0,54% 0,46% 0,47%

Anesthesie, OK en IC 0,68% 0,46% 0,81%

Tender

- 70 -

Appendix T Ranking selection criteria after matching based on tender B

Table 50 – Ranking selection criteria after matching based on tender B

Name subtopic

Tender B Tender A Tender C

Plan van aanpak voor de implementatie en bedrijfsgerede oplevering 16,49% 5,33% 2,44%

Innovatie 10,81% 0,80% 6,98%

Implementatie en bedrijfsgerede oplevering 10,54% 4,00% 2,44%

Documentatie en templates 7,76% 1,33% 4,13%

Administratie Medisch wetenschappelijk onderzoek 5,65% 5,17% 5,35%

Klinische dossiervoering 4,78% 5,17% 7,67%

Serviceondersteuning en –contract (service level agreement) 4,05% 2,80% 2,44%

Opleiding en instructie 3,78% 1,33% 1,86%

Gebruiksvriendelijkheid 3,35% 3,67% 2,44%

Conversie 3,24% 1,33% 4,07%

Administratie en monitoring onderwijs 2,84% 0,52% 4,88%

Decision Support 2,11% 0,52% 1,05%

Zorgpaden en zorgplannen 2,00% 0,52% 2,79%

Beveiliging van informatie 1,81% 0,73% 2,71%

Communicatie / Correspondentie 1,49% 0,52% 4,36%

Beschikbaarheid en performance 1,46% 1,47% 1,76%

Contractvoorwaarden project 1,35% 5,33% 0,00%

Portalen 1,24% 5,17% 4,21%

Beveiliging 1,10% 1,10% 0,95%

Open vragen – integratie 1,08% 2,93% 3,16%

Ordercommunicatie 0,81% 5,17% 0,70%

Afspraken planning 0,81% 4,58% 0,70%

Opnameplanning en -registratie 0,81% 4,58% 0,70%

Beddenplanning 0,81% 0,46% 0,70%

Applicatie 0,77% 1,47% 0,48%

Platform 0,77% 0,73% 0,48%

Data opslag / Database 0,77% 1,10% 0,78%

Beheer 0,77% 1,10% 2,23%

Monitoring en reporting 0,77% 1,10% 2,03%

Open vragen - techniek 0,77% 0,73% 0,29%

Medische bibliotheek / knowledge base 0,70% 5,17% 1,30%

Functionaliteiten organisatorische werkeenheden / ketens 0,68% 0,52% 0,81%

Anesthesie, OK en IC 0,68% 0,46% 0,81%

Rapportages 0,65% 0,52% 2,79%

Patiëntenlogistiek 0,54% 0,46% 0,47%

Wachtlijstbeheer 0,54% 0,46% 0,47%

Eisen Service Oriented Architecture 0,54% 4,40% 1,74%

Registratie van prestatie-indicatoren 0,32% 0,52% 1,40%

Medicatie 0,27% 5,17% 4,65%

Spoedeisende hulp 0,14% 0,52% 0,35%

Patiënt monitoring en datacollectie (PDMS) 0,14% 5,17% 0,35%

Wet en regelgeving 0,00% 1,47% 10,08%

Eisen met betrekking tot de leverancier 0,00% 1,60% 0,00%

Commerciële en juridische eisen 0,00% 2,80% 0,00%

Tender

- 71 -

Appendix U Ranking averaged selection criteria of the tenders after the matching

Table 51 – Ranking average selection criteria after matching

Name subtopic Average

Tender B Tender A Tender C

Plan van aanpak voor de implementatie en bedrijfsgerede oplevering 16,49% 5,33% 2,44% 8,09%

Innovatie 10,81% 0,80% 6,98% 6,20%

Klinische dossiervoering 4,78% 5,17% 7,67% 5,88%

Implementatie en bedrijfsgerede oplevering 10,54% 4,00% 2,44% 5,66%

Administratie Medisch wetenschappelijk onderzoek 5,65% 5,17% 5,35% 5,39%

Documentatie en templates 7,76% 1,33% 4,13% 4,41%

Wet en regelgeving 0,00% 1,47% 10,08% 3,85%

Portalen 1,24% 5,17% 4,21% 3,54%

Medicatie 0,27% 5,17% 4,65% 3,36%

Gebruiksvriendelijkheid 3,35% 3,67% 2,44% 3,15%

Serviceondersteuning en –contract (service level agreement) 4,05% 2,80% 2,44% 3,10%

Conversie 3,24% 1,33% 4,07% 2,88%

Administratie en monitoring onderwijs 2,84% 0,52% 4,88% 2,75%

Open vragen – integratie 1,08% 2,93% 3,16% 2,39%

Medische bibliotheek / knowledge base 0,70% 5,17% 1,30% 2,39%

Opleiding en instructie 3,78% 1,33% 1,86% 2,33%

Contractvoorwaarden project 1,35% 5,33% 0,00% 2,23%

Eisen Service Oriented Architecture 0,54% 4,40% 1,74% 2,23%

Ordercommunicatie 0,81% 5,17% 0,70% 2,23%

Communicatie / Correspondentie 1,49% 0,52% 4,36% 2,12%

Afspraken planning 0,81% 4,58% 0,70% 2,03%

Opnameplanning en -registratie 0,81% 4,58% 0,70% 2,03%

Patiënt monitoring en datacollectie (PDMS) 0,14% 5,17% 0,35% 1,88%

Zorgpaden en zorgplannen 2,00% 0,52% 2,79% 1,77%

Beveiliging van informatie 1,81% 0,73% 2,71% 1,75%

Beschikbaarheid en performance 1,46% 1,47% 1,76% 1,56%

Beheer 0,77% 1,10% 2,23% 1,37%

Rapportages 0,65% 0,52% 2,79% 1,32%

Monitoring en reporting 0,77% 1,10% 2,03% 1,30%

Decision Support 2,11% 0,52% 1,05% 1,22%

Beveiliging 1,10% 1,10% 0,95% 1,05%

Commerciële en juridische eisen 0,00% 2,80% 0,00% 0,93%

Applicatie 0,77% 1,47% 0,48% 0,91%

Data opslag / Database 0,77% 1,10% 0,78% 0,88%

Registratie van prestatie-indicatoren 0,32% 0,52% 1,40% 0,75%

Functionaliteiten organisatorische werkeenheden / ketens 0,68% 0,52% 0,81% 0,67%

Platform 0,77% 0,73% 0,48% 0,66%

Beddenplanning 0,81% 0,46% 0,70% 0,66%

Anesthesie, OK en IC 0,68% 0,46% 0,81% 0,65%

Open vragen - techniek 0,77% 0,73% 0,29% 0,60%

Eisen met betrekking tot de leverancier 0,00% 1,60% 0,00% 0,53%

Patiëntenlogistiek 0,54% 0,46% 0,47% 0,49%

Wachtlijstbeheer 0,54% 0,46% 0,47% 0,49%

Spoedeisende hulp 0,14% 0,52% 0,35% 0,33%

Tender

- 72 -

Appendix V Explanation of trends Selection criteria Explanation

Wet en regelgeving The academic hospitals tried to achieve more commitment from the vendors; therefore this subtopic became more important over time.

Klinische dossiervoering The trend was rising and this is interesting for Siemens as they could focus on this subtopic regarding future tenders.

Ordercommunicatie Eclypsis vs. iSoft. The academic hospitals which used iSoft believed this was a commodity, while hospital A missed this subtopic in their current situation. Therefore this subtopic was more important for hospital A than for the others.

Decision Support This subtopic scored surprisingly low, Siemens thought it would be much more important. Therefore a learning point.

Communicatie / Correspondentie

Eclypsis vs. iSoft. Tender A scored the lowest as Eclypsis was a more open system than the iSoft system. Tender C scored the highest as communication was more developed and therefore more important in that region.

Medicatie The increase was due to the fact that the law changed over time. Per January 1st

2013 a law will be activated which states that the medication has to be prescribed electronically. Tender B did not had to consider this law, while the other two tenders had to.

Medische bibliotheek / knowledge base

Unexpected, best explanation is Eclypsis vs. iSoft

Registratie van prestatie-indicatoren

Demonstrability of the quality of care became more important over time due to the insurance companies.

Portalen This subtopic just entered the market at the time of tender B and it became more important over time.

Patiënt monitoring en datacollectie (PDMS)

Eclypsis vs. iSoft. The academic hospitals which used iSoft believed this was a commodity, while hospital A missed this subtopic in their current situation. Therefore this subtopic was more important for hospital A than for the others.

Rapportages Demonstrability of the quality of care became more important over time due to the insurance companies.

Administratie en monitoring onderwijs

This subtopic depended on the academic hospital itself as opinions were divided how it should be incorporated in the HIS/EHR system.

Afspraken planning Eclypsis vs. iSoft. The academic hospitals which used iSoft believed this was a commodity, while hospital A missed this subtopic in their current situation. Therefore this subtopic was more important for hospital A than for the others.

Opnameplanning en -registratie

Eclypsis vs. iSoft. The academic hospitals which used iSoft believed this was a commodity, while hospital A missed this subtopic in their current situation. Therefore this subtopic was more important for hospital A than for the others.

Beheer In the past the academic hospitals did it themselves, but the trend is that they want to outsource as much as possible to relieve themselves and to be more efficient, therefore it became more important over time.

Monitoring en reporting

In the past the academic hospitals did it themselves, but the trend is that they want to outsource as much as possible to relieve themselves and to be more efficient, therefore it became more important over time.

Eisen Service Oriented Architecture

Became more important over time due to the fact that hospitals wanted standard technologies instead of client specific closed systems.

Open vragen – integratie

Became more important over time due to the fact that hospitals wanted standard technologies instead of client specific closed systems.

Implementatie en bedrijfsgerede oplevering

In the past the academic hospitals wanted full flexibility during their client specific implementation. Nowadays the academic hospitals want to make use of standard implementations; therefore it is a declining trend.

Plan van aanpak voor de implementatie en bedrijfsgerede oplevering

In the past the academic hospitals wanted full flexibility during their client specific implementation. Nowadays the academic hospitals want to make use of standard implementations; therefore it is a declining trend.

- 73 -

Selection criteria Explanation

Opleiding en instructie

Hospitals became less insecure about this topic over time due to E-learning.

Conversie Hospitals are insecure, no good explanation.

Contractvoorwaarden project

The academic hospitals tried to achieve more commitment from the vendors; the last tender even used it as a knockout criteria.

Table 52 – Explanation differences