· d4.6 pilots evaluation results – third round page 2 of 215 deliverable factsheet project...
TRANSCRIPT
Page 1 of 215
OpenGovIntelligence
Fostering Innovation and Creativity in Europe through Public
Administration Modernization towards Supplying and Exploiting
Linked Open Statistical Data
Deliverable 4.6
Pilots Evaluation Results – Third Round
Leading partner: Technical University of Delft (TUDelft)
Participating partners: TUDelft
Version-Status: V1.0
Dissemination level: CO
D4.6 Pilots Evaluation Results – Third Round
Page 2 of 215
Deliverable factsheet
Project Number: 693849
Project Acronym: OpenGovintelligence
Project Title: Fostering Innovation and Creativity in Europe through Public Administration Modernization towards Supplying and Exploiting Linked Open Statistical Data
Deliverable title: Pilots and Evaluation Results – Second Round
Deliverable number: D4.6
Official submission date: 31/01/2019
Actual submission date: 31/01/2019
Editor(s): Ricardo Matheus (TUDelft)
Author(s): Ricardo Matheus (TUDelft), Marijn Janssen (TUDelft), Dhata Praditya (TUDelft)
Reviewer(s) All partners
Abstract: This report presents the evaluation results for the OGI project. It was updated taking into consideration the 2nd evaluation report (D4.4) and the 3rd round of Pilots plan (D4.5). The evaluation has four areas: Co-creation, ICT Toolkit, Acceptance of ICT Toolkit and Outcomes.
D4.6 Pilots Evaluation Results – Third Round
Page 3 of 215
Effort of Participating Partners Consortium
Name Short
Name
Role Person
Months
1. Centre for Research & Technology - Hellas CERTH Coordinator 2
2. Delft University of Technology TUDelft R&D Partner 4.0
3. National University of Ireland, Galway NUIG R&D Partner 12
4. Tallinn University of Technology TUT R&D Partner 5
5. ProXML bvba ProXML R&D Partner 0.2
6. Swirrl IT Limited SWIRRL R&D Partner 0.1
7. Trafford council TRAF Pilot Partner
8. Flemish Government VLO Pilot Partner
9. Ministry of Administrative Reconstruction MAREG Pilot Partner 0.55
10. Ministry of Economic Affairs and Communication MKM Pilot Partner
11. Marine Institute MI Pilot Partner 0.5
12. Public Institution Enterprise Lithuania EL Pilot Partner
D4.6 Pilots Evaluation Results – Third Round
Page 4 of 215
Revision History
Version Date Revised by Reason
0.1 01/11/2018 TUDelft Initial setup by Ricardo Matheus
0.4 15/01/2019 TUDelft Internal Review by Marijn Janssen
0.9 25/01/2019 Internal reviewer Internal reviewer
1.0 31/01/2019 TUDelft Final Version
Statement of originality:
This deliverable contains original unpublished work except where clearly indicated otherwise. Acknowledgement of previously published material and of the work of others has been made through appropriate citation, quotation or both.
D4.6 Pilots Evaluation Results – Third Round
Page 5 of 215
Table of Contents
DELIVERABLE FACTSHEET ............................................................................................................2
EFFORT OF PARTICIPATING PARTNERS CONSORTIUM ..................................................................3
REVISION HISTORY ......................................................................................................................4
TABLE OF CONTENTS ...................................................................................................................5
LIST OF FIGURES ..........................................................................................................................7
LIST OF TABLES ...........................................................................................................................9
EXECUTIVE SUMMARY .............................................................................................................. 11
1 INTRODUCTION ................................................................................................................. 13
1.1 SCOPE .................................................................................................................................................... 13
1.2 AUDIENCE............................................................................................................................................... 13
1.3 STRUCTURE ............................................................................................................................................. 13
2 EVALUATION OVERVIEW .................................................................................................... 14
3 EVALUATION METHODOLOGY ............................................................................................ 16
3.1 PILOTS’ CO-CREATION EVALUATION METHODS .............................................................................................. 20
3.1.1 Selected Co-Creation Tools and Methods Overview ....................................................................... 21
3.2 PILOTS’ ACCEPTANCE AND OUTCOMES EVALUATION METHODOLOGY ................................................................. 25
3.2.1 Pilots’ Acceptance Selected Methodology ...................................................................................... 25
3.3 OGI ICT TOOLKIT EVALUATION METHODOLOGY ............................................................................................ 34
3.3.1 OGI Pilots’ Data Quality Evaluation Methodology .......................................................................... 38
3.3.2 The standard for Systems and Software Quality Requirements and Evaluation – ISO 25010 and ISO
25012 39
3.3.3 User Experience (UX) and User Interface (UI) testing ..................................................................... 43
4 OGI EVALUATION RESULTS ................................................................................................. 46
4.1 PILOTS’ EVALUATION RESULTS .................................................................................................................... 47
4.1.1 Pilot 1 – The Greek Ministry of Administrative Reconstruction (Greece) ........................................ 48
4.1.2 Pilot 2 – Enterprise Lithuania (Lithuania) ........................................................................................ 65
4.1.3 Pilot 3 – Tallinn Real Estate (Estonia) ............................................................................................. 81
4.1.4 Pilot 4 – Trafford Council Worklessness (England) ........................................................................ 103
4.1.5 Pilot 5 – The Flemish Environment Agency (Belgium) ................................................................... 122
4.1.6 Pilot 6 – Marine Institute (Ireland) ................................................................................................ 137
4.1.7 Overall Results From Pilots’ Evaluation ........................................................................................ 159
4.2 OGI ICT TOOLKIT EVALUATION RESULTS .................................................................................................... 163
4.2.1 Internal Evaluation ........................................................................................................................ 164
4.2.2 External Evaluation ....................................................................................................................... 178
5 CONCLUSIONS ................................................................................................................. 182
6 REFERENCES .................................................................................................................... 184
D4.6 Pilots Evaluation Results – Third Round
Page 6 of 215
7 ANNEXES ......................................................................................................................... 187
7.1 END-USER QUESTIONNAIRES .................................................................................................................... 187
7.1.1 Pilot 1 – The Greek Ministry of Administrative Reconstruction (Greece) End-User Questionnaire
187
7.1.2 Pilot 2 – Enterprise Lithuania (Lithuania) End-User Questionnaire ............................................... 190
7.1.3 Pilot 3 – Tallinn Real Estate (Estonia) End-User Questionnaire .................................................... 194
7.1.4 Pilot 4 – Trafford Council Worklessness (England) End-User Questionnaire ................................. 197
7.1.5 Pilot 5 – The Flemish Environment Agency (Belgium) End-User Questionnaire ............................ 201
7.1.6 Pilot 6 – Marine Institute (Ireland) End-User Questionnaire ......................................................... 205
7.2 DATA QUALITY QUESTIONNAIRE ............................................................................................................... 209
7.3 OGI ARCHITECTURE QUESTIONNAIRE ......................................................................................................... 211
D4.6 Pilots Evaluation Results – Third Round
Page 7 of 215
List of Figures FIGURE 1- INTERCONNECTIONS AND INTERDEPENDENCIES BETWEEN OGI WORKING PACKAGES AND
DELIVERABLES ............................................................................................................................................. 15 FIGURE 2 - EVALUATION DIMENSIONS ................................................................................................................... 16 FIGURE 3 - HIGH-LEVEL PROCESSES OF PILOT PLAN ............................................................................................. 18 FIGURE 4 - PILOTS' TIMELINE ................................................................................................................................ 19 FIGURE 5 – EXPLORATORY AND EXPLANATORY APPROACHES AT CO-CREATION EVALUATION SURVEYS ............ 22 FIGURE 6 - ADAPTED TRANSPARENCY MODEL FOR OGI EVALUATION ................................................................. 28 FIGURE 7- OGI FOCAL POINT ................................................................................................................................ 29 FIGURE 8- OGI TOOLS AND WORKING FLOW ........................................................................................................ 36 FIGURE 9 - UX STRUCTURE AND LAYERS OF PRODUCT AND INFORMATION ........................................................... 43 FIGURE 10 - THE RIPPLE EFFECT ............................................................................................................................ 44 FIGURE 11 - UX IMPLEMENTATION AND EVALUATION STEPS ................................................................................ 45 FIGURE 12 – COMMON FLOW OF PILOTS ACTIVITIES .............................................................................................. 47 FIGURE 13 – GREEK PILOT CO-CREATION STEPS .................................................................................................. 53 FIGURE 14 – GREEK PILOT APPLICATION CALLIMACHUS ...................................................................................... 54 FIGURE 15 – VEHICLES DATA CUBES SCREENSHOT ............................................................................................... 58 FIGURE 16 - PIE SORTED GRAPH VEHICLE TYPE ................................................................................................... 64 FIGURE 17 - BAR GRAPH FILTERED BY AREA (MUNICIPALITY) ............................................................................. 64 FIGURE 18 – LOSD USER FAMILIARITY – THIRD YEAR ........................................................................................ 76 FIGURE 19 – USER ACCEPTANCE – THIRD YEAR ................................................................................................... 77 FIGURE 20 – DATASETS ACCEPTANCE – THIRD YEAR ........................................................................................... 78 FIGURE 21 – RESULTS – THIRD YEAR .................................................................................................................... 79 FIGURE 22 - TABLE WITH BAR CHART FUNCTIONALITY ......................................................................................... 80 FIGURE 23 - TABLE WITH HEATMAP FUNCTIONALITY ............................................................................................ 80 FIGURE 24 – AUDIENCE OF TALLINN REAL ESTATE HACKATHON ......................................................................... 90 FIGURE 25 – DEVELOPERS CODING FOR THE TALLINN REAL ESTATE HACKATHON ............................................... 91 FIGURE 26 TALLINN APP RESPONDENTS’ FAMILIARITY WITH LOSD ..................................................................... 97 FIGURE 27 TALLINN APP RESPONDENTS USER ACCEPTANCE .................................................................................. 98 FIGURE 28 TALLINN APP DATASETS QUALITY RESULTS ......................................................................................... 99 FIGURE 29 TALLINN APP PERCEIVED OUTCOME ................................................................................................... 100 FIGURE 30 – TALLINN REAL ESTATE APPLICATION ............................................................................................. 101 FIGURE 31 - BAR CHART OF AVERAGE COST ....................................................................................................... 101 FIGURE 32 - PIE CHART DISTRIBUTION OF CRASHES ............................................................................................ 102 FIGURE 33 – LOSD FAMILIARITY – TRAFFORD PILOT ......................................................................................... 117 FIGURE 34 – USER ACCEPTANCE – TRAFFORD PILOT .......................................................................................... 118 FIGURE 35 – DATASETS QUALITY – TRAFFORD PILOT ......................................................................................... 119 FIGURE 36 – PERCEIVED OUTCOMES – TRAFFORD PILOT .................................................................................... 120 FIGURE 37 – TRAFFORD COUNCIL WORKLESSNESS APPLICATION ....................................................................... 121 FIGURE 38 - METADATA OF BELGIUM PILOT DATA SET ...................................................................................... 130 FIGURE 39 – LOSD FAMILIARITY – FLEMISH PILOT ............................................................................................ 132 FIGURE 40- USER ACCEPTANCE – FLEMISH PILOT .............................................................................................. 133 FIGURE 41 – DATASETS QUALITY – FLEMISH PILOT ............................................................................................ 134 FIGURE 42 – PERCEIVED OUTCOMES – FLEMISH PILOT ....................................................................................... 135 FIGURE 43 – THE FLEMISH ENVIRONMENT AGENCY APPLICATION ..................................................................... 136 FIGURE 44 - PERCENTAGE BREAKDOWN OF CO-CREATION CONTRIBUTIONS ........................................................ 145 FIGURE 45 – LOSD FAMILIARITY – MARINE INSTITUTE PILOT ........................................................................... 154 FIGURE 46 – USER ACCEPTANCE – MARINE INSTITUTE PILOT............................................................................. 155
D4.6 Pilots Evaluation Results – Third Round
Page 8 of 215
FIGURE 47 – DATASETS QUALITY – MARINE INSTITUTE PILOT ........................................................................... 156 FIGURE 48 – PERCEIVED OUTCOMES – MARINE INSTITUTE PILOT ....................................................................... 157 FIGURE 49 - AVERAGE BAR CHART OF IRISH NATIONAL TIDE GAUGE NETWORK DATA SET ............................... 158 FIGURE 50 – OVERALL LOSD FAMILIARITY ....................................................................................................... 159 FIGURE 51 – OVERALL USER ACCEPTANCE ......................................................................................................... 160 FIGURE 52 –OVERALL DATASETS QUALITY ........................................................................................................ 161 FIGURE 53 – OVERALL PERCEIVED OUTCOMES ................................................................................................... 162 FIGURE 54 – INTERNAL AND EXTERNAL EVALUATION PROCESSES ..................................................................... 163 FIGURE 55 – DISCOVERABILITY AND USABILITY – OVERALL OF ALL PILOTS ..................................................... 165 FIGURE 48 – GRANULARITY SECTION – OVERALL OF ALL PILOTS ...................................................................... 166 FIGURE 57 – INTELLIGIBILITY SECTION – OVERALL OF ALL PILOTS.................................................................... 167 FIGURE 58 – TRUSTWORTHINESS SECTION – OVERALL OF ALL PILOTS ............................................................... 168 FIGURE 59 –LINKABLE TO OTHER DATASETS SECTION – OVERALL OF ALL PILOTS ............................................ 169 FIGURE 60 – OPEN DATA PRINCIPLES SECTION – OVERALL OF ALL PILOTS ........................................................ 170 FIGURE 61 – USE BEHAVIOUR OF LOSD OGI ARCHITECTURE ............................................................................ 172 FIGURE 62 – PERCEIVED USEFULNESS OF LOSD OGI ARCHITECTURE .............................................................. 173 FIGURE 63 – PERCEIVED EASE OF USE OF LOSD OGI ARCHITECTURE .............................................................. 174 FIGURE 64 – COMPUTER SELF-EFFICACY, PERCEPTIONS OF EXTERNAL CONTROL, COMPUTER PLAYFULNESS,
COMPUTER ANXIETY, PERCEIVED ENJOYMENT, OBJECTIVE USABILITY OF LOSD OGI ARCHITECTURE ... 175 FIGURE 65 –SUBJECTIVE NORM, VOLUNTARINESS, IMAGE, JOB RELEVANCE, OUTPUT QUALITY OF LOSD OGI
ARCHITECTURE........................................................................................................................................... 176 FIGURE 66 –RESULT DEMONSTRABILITY, BEHAVIOURAL INTENTION, USE, PRESENT SATISFACTION, FUTURE
DESIRE OF LOSD OGI ARCHITECTURE ...................................................................................................... 177 FIGURE 67 – NTTS HANDS-ON OGI ICT TOOLKIT WORKSHOP .......................................................................... 178
D4.6 Pilots Evaluation Results – Third Round
Page 9 of 215
List of Tables TABLE 1 - CO-CREATION FRAMEWORK STAGES, METHODS FOR DATA COLLECTION AND TOOLS FOR EVALUATION
..................................................................................................................................................................... 20 TABLE 2 - USER WORKSHOP PROTOCOL FOR CO-CREATION ................................................................................. 21 TABLE 3 – OVERVIEW OF PILOTS’ SELECTED CO-CREATION TOOLS AND METHODS ............................................ 24 TABLE 4 - USER ACCEPTANCE EVALUATION FOR OGI .......................................................................................... 25 TABLE 5 - ADAPTED TRANSPARENCY MODEL FOR OGI EVALUATION .................................................................. 27 TABLE 6 - TAXONOMY FOR BENEFITS FOR OGI INNOVATION ECOSYSTEM STAKEHOLDERS .................................. 29 TABLE 7 – PILOTS’ OUTCOMES EVALUATION QUESTIONNAIRE ............................................................................. 32 TABLE 8 - OGI TOOLKIT RELEASES AND TOOLS PER YEAR ................................................................................... 35 TABLE 9 - OGI TOOLS USED IN PILOTS .................................................................................................................. 37 TABLE 10 – PILOTS’ DATA QUALITY QUESTIONNAIRE .......................................................................................... 38 TABLE 11 - CUBE DESIGN AND BUILDING BLOCKS DATA COLLECTION AND METHODOLOGY METHODS OF
EVALUATION ................................................................................................................................................ 39 TABLE 12 - OGI TOOLKIT REQUIREMENTS FOR EVALUATION ............................................................................... 39 TABLE 13 - CRITERIA FOR EVALUATION OF QUALITY IN USE ................................................................................ 42 TABLE 14 - CO-CREATION USER WORKSHOP QUESTIONS ..................................................................................... 49 TABLE 15 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS ................................................................ 50 TABLE 16 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS – FIRST YEAR ......................................... 51 TABLE 16 - SUMMARY OF ACTIONS TAKEN ABOUT IDEAS AND SOLUTIONS FROM PARTICIPANTS – THIRD YEAR
STATUS ......................................................................................................................................................... 52 TABLE 17 – DATA QUALITY SURVEY RESULTS – THIRD YEAR ............................................................................. 55 TABLE 18 – DATA QUALITY IMPROVEMENT PROGRESS ........................................................................................ 56 TABLE 19 – DATA QUALITY IMPROVEMENT PROGRESS ........................................................................................ 57 TABLE 21 - CO-CREATION USER WORKSHOP QUESTIONS ..................................................................................... 66 TABLE 22 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS – FIRST YEAR ......................................... 67 TABLE 23 - SUMMARY OF PROBLEMS, EXPECTED SOLUTIONS AND IMPLEMENTATION - SECOND YEAR ............... 68 TABLE 24 - SUMMARY OF PROBLEMS, EXPECTED SOLUTIONS AND IMPLEMENTATION - THIRD YEAR .................. 69 TABLE 23 – DATA QUALITY SURVEY RESULTS – THIRD YEAR ............................................................................. 71 TABLE 24 – DATA QUALITY IMPROVEMENT PROGRESS ........................................................................................ 72 TABLE 25 – DATA QUALITY IMPROVEMENT PROGRESS – THIRD YEAR ................................................................. 73 TABLE 26 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS ................................................................ 83 TABLE 27 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS ................................................................ 88 TABLE 28 – DATA QUALITY SURVEY RESULTS ..................................................................................................... 92 TABLE 29 – DATA QUALITY IMPROVEMENT PROGRESS ........................................................................................ 93 TABLE 30 – DATA QUALITY IMPROVEMENT PROGRESS ........................................................................................ 94 TABLE 31 - CO-CREATION USER WORKSHOP QUESTIONS ................................................................................... 105 TABLE 32 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS .............................................................. 106 TABLE 33 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS – THIRD YEAR ...................................... 108 TABLE 34 – CO-CREATION USER WORKSHOP AGENDA ............................................................................................ 109 TABLE 35 – DATA QUALITY SURVEY RESULTS – THIRD YEAR ........................................................................... 113 TABLE 36 – DATA QUALITY IMPROVEMENT PROGRESS – THIRD YEAR ............................................................... 114 TABLE 37 – DATA QUALITY IMPROVEMENT PROGRESS – THIRD YEAR ............................................................... 115 TABLE 38 - QUESTIONS MADE TO THE AUDIENCE ................................................................................................ 123 TABLE 39 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS .............................................................. 124 TABLE 40 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS – THIRD YEAR ...................................... 126 TABLE 41 – DATA QUALITY SURVEY RESULTS – THIRD YEAR ........................................................................... 128 TABLE 42 – DATA QUALITY IMPROVEMENT PROGRESS – THIRD YEAR ............................................................... 129
D4.6 Pilots Evaluation Results – Third Round
Page 10 of 215
TABLE 43 - SAMPLE DATA INFRASTRUCTURE BARRIERS .................................................................................... 139 TABLE 44 - SAMPLE USER CHALLENGES BARRIERS ............................................................................................ 140 TABLE 45 - SERVICE CREATION - SAMPLE OPTIONS FOR OVERCOMING BARRIERS ............................................... 141 TABLE 46 - SERVICE ENGINEERING - SAMPLE OPTIONS FOR OVERCOMING BARRIERS ......................................... 142 TABLE 47 - SERVICE MANAGEMENT - OPTIONS FOR OVERCOMING BARRIERS ..................................................... 142 TABLE 48 - SERVICE CREATION - OPTIONS FOR OVERCOMING BARRIERS ............................................................ 143 TABLE 49 - SERVICE ENGINEERING - OPTIONS FOR OVERCOMING BARRIERS ....................................................... 143 TABLE 50 - SERVICE MANAGEMENT - OPTIONS FOR OVERCOMING BARRIERS ..................................................... 144 TABLE 51 - SAMPLE CO-CREATION CONTRIBUTIONS ............................................................................................ 146 TABLE 52 - SUMMARY OF IDEAS AND SOLUTIONS FROM PARTICIPANTS – THIRD YEAR ...................................... 148 TABLE 53 – DATA QUALITY SURVEY RESULTS – THIRD YEAR ........................................................................... 150 TABLE 54 – DATA QUALITY IMPROVEMENT PROGRESS ...................................................................................... 151 TABLE 55 – DATA QUALITY IMPROVEMENT PROGRESS ...................................................................................... 152 TABLE 58 – PILOTS’ DATA QUALITY QUESTIONNAIRE ........................................................................................ 209
D4.6 Pilots Evaluation Results – Third Round
Page 11 of 215
Executive Summary
OpenGovIntelligence (OGI) project aims to go beyond traditional top-down approaches and
proposes public participation in terms of the co-initiation, co-design, co-implementation, and co-
evaluation for providing innovative and data-driven public services.
During the project, building blocks for the use of Linked Open Statistical Data (LOSD) are developed.
Then, the participation processes and building blocks were implemented in six pilot projects which
have different objectives, take different approaches and look at various aspects of Linked Open
Statistical Data (LOSD). Each pilot exclusively addresses specific society’s needs by exploitation of
public sector statistical data.
The evaluation is focussed on the four main areas of this project; 1) Co-Creation Framework, 2) OGI
ICT Toolkit, 3) Acceptance of the pilots and, 4) the outcomes of the pilots (public-value creation). The
Co-creation process was evaluated in the first and second-year reports (provided in detail in D4.2
and D4.4) and in this report, we included the summary of the co-creation development.
The main focus of this 3rd-year report is on the evaluation of pilots acceptance and outcomes, which
could only be properly evaluated after the pilots have been used in practice by sufficient users. In
the final year of the project,, the final version of the pilots were delivered, and the number of the
apps users are sufficient to be evaluated. As presented in Section 3.2, the user acceptance and
perceived outcomes evaluation are divided into 4 parts: first, LOSD familiarity, user acceptance,
datasets quality and perceived outcomes.
In total, there are 219 respondents of user-questionnaire from six pilots. Most of the respondents
found the pilots’ applications are useful for them, including the creation of transparency, reduction
the administrative burden by providing a more efficient search of information and better visualizing
the results at a glance. We considered strongly agree, agree and neutral answers as positive
reactions, while negative reactions are disagree and strongly disagree. Below, we provided the key
findings of descriptive statistics of user acceptance and outcomes expectation evaluation:
The overall acceptance of the pilots’ apps was 90%;
92% pointed out an increase of Transparency after accessing the Pilots’ Apps;
The Pilots’ apps helped to 94% of surveyed people have better decision-making;
93% of people answered there is a better interpretation of Data;
For 92% of respondents, Pilots’ app reduced the time spent searching for information;
94% of end-users identified a cost reduction when using OGI pilots’ apps; and,
There is an increase in efficiency to 91% of OGI Pilots’ users.
D4.6 Pilots Evaluation Results – Third Round
Page 12 of 215
In addition, the OGI ICT Toolkit was evaluated using a mixed qualitative approach (described in
section 3.3) by the developers. The OGI ICT toolkit consists of building blocks. All pilots used the
toolkit as their reference in designing and building their apps architecture. However, due to the
different objectives and approaches, some pilots did not need to use all OGI building blocks.
Technical partners and pilots’ leaders were interviewed and surveyed to collect information about
fundamental requirements and parameters for an ICT Toolkit development. The result shows the
potential and limits of the LOSD OGI Toolkit.
From the project we can learned that developing the ICT-toolkits took longer than expected, as this
is a new field in which there are hardly any examples. The development of the building block
encountered main challenge in deciding which level of granularity is acceptable for the both
developers and users as this requires a trade-off between the flexibility given to the developer and
ease-of-use of manipulating data. User-friendliness and having right granularity is a trade-off and
developers have different requirements in this regard.
Co-creation remains challenging in government. Co-creation is bottom-up, whereas government
traditionally takes a more top-down approach by developing pilots and providing the services.
Realizing co-creation is not only a technological problem but rather a cultural problem.
The end-users regard the pilots as successful as shown in the figure below. The pilots show that data
quality is a key aspect as garbage in is garbage out. Realizing high-data quality often took more time
than expected, whereas the OGI ICT-toolkit enables to develop the pilots within a short time frame.
They pilots help end-users to find information faster and in a more efficient way resulting in
transparency and better decision-making quality.
D4.6 Pilots Evaluation Results – Third Round
Page 13 of 215
1 Introduction
This document reports the evaluation results of OpenGovIntelligence (OGI) project. The OGI
environment provides an ICT toolkit comprising easy-to-use and user-centric tools to facilitate
realizing a Linked Open Statistical Data (LOSD) innovation ecosystem. Pilots were deployed to
validate and prove the usability and effectiveness of OGI ICT toolkit to co-create and innovate
ecosystems.
The first part of the evaluation is related to the co-creation development which was already covered
in previous reports and will only be summarized here. The second part concerns the evaluation of
the OGI Toolkit by developers. The third part is the acceptance of the OGI ICT Toolkit by end-users.
The fourth part is the evaluation of the outcomes of pilots’, which includes Transparency,
Administrative Burden Reduction, and Costs Reduction.
1.1 Scope
The present document is the Deliverable 4.6 “D4.6 – Pilots Evaluation Results - Third Round”
(henceforth referred to as D4.6) of the OGI project. The main objective of D4.6 is to describe the
pilot’s results in this third year of OGI project and elicit the lessons learned and conclusions from the
evaluation results.
1.2 Audience
The audience for this deliverable is as follows:
European Commission (EC);
The audience interested on the exploitation of Linked Open Statistical Data (LOSD) and Data
Cubes for public service delivery
The audience interested in public service delivery based on co-creation from external and
internal stakeholders; and,
OGI Project partners.
1.3 Structure
In the next chapter, we start by presenting the pilots’ Evaluation Overview in accordance with Pilots’
Implementation Plan in the report D4.5. The structure of this document is as follows:
Section 2 provides the Evaluation Overview;
Section 3 describes the Pilots Evaluation Methodology;
Section 4 presents the OGI Evaluation Results;
Section 5 presents Conclusions;
Section 6 provides the References; and,
Section 7 shows the annexes.
D4.6 Pilots Evaluation Results – Third Round
Page 14 of 215
2 Evaluation Overview
The OGI toolkit is implemented in 6 pilot projects. The pilots are deploying the OGI ICT Toolkit to
implement an application that can be used by others. To develop the pilots, the OGI ICT Toolkit was
used and tested and the co-creation framework is guiding the relationship between service
providers and users. Each pilot made only use of the relevant OGI building blocks. The results of the
evaluation provides insights for further improvement of the OGI toolkit and for improving the
evaluation instruments.
OGI to was evaluated by conducting six pilot projects, described in the report D1.1- Challenges and
Needs (https://www.slideshare.net/OpenGovIntelligence/deliverable-11-ogi-challenges-and-needs).
The projects use the OGI toolkit for co-creation, used LOSD to provide services and ultimately
contribute to social impact.
1. The Greek Ministry of Administrative Reconstruction (Greece)
a. http://wapps.islab.uom.gr/CubeVisualizer/vehicles/;
2. The Enterprise Lithuania (Lithuania)
a. http://vmogi03.deri.ie:8080/superset/dashboard/5/
b. http://vmogi03.deri.ie:8080/superset/dashboard/7/
3. The Tallinn Real Estate (Estonia)
a. https://rnd-tut.shinyapps.io/Estonian_Pilot/
4. Trafford Council Worklessness (England)
a. http://www.trafforddatalab.io/opengovintelligence/
5. The Flemish Environment Agency (Belgium)
a. https://www.milieuinfo.be/emissiepunten/
6. The Marine Institute (Ireland)
a. http://vis.marine.ie/DashboardsTest/#/wave_spectral
b. http://vis.marine.ie/DashboardsTest/#/wave_zero
c. http://vis.marine.ie/DashboardsTest/#/weather
The OGI environment provides an ICT toolkit comprising easy-to-use and user-centric tools to
facilitate the processing and use of Linked Open Statistical Data (LOSD). The source code of OGI
toolkit can be found at Github (https://github.com/OpenGovIntelligence)
The pilots were deployed to validate and prove the usability and effectiveness of OGI ICT toolkit to
co-create and innovate ecosystems. The resulting apps should be used by a variety of users which in
turn should result in long term effects (outcomes).
The WPs and evaluation are closely related to each other. Figure 1 summarizes the
interdependencies and interconnections between the working packages (WP) of the OGI project.
D4.6 Pilots Evaluation Results – Third Round
Page 15 of 215
Figure 1- Interconnections and Interdependencies between OGI Working Packages and Deliverables
In an ecosystem, there are different stakeholders who view the pilots from their own perspectives.
Developers might want to evaluate the pilots based on their ability to meet the end-user
requirements using the OGI toolkit and building blocks. End-users want easy-to-use interfaces and
pilots satisfying their needs. Decision-makers might look at how participation by users and other
forms of co-creation result in an impact of the applications on the number of users and return on
investment. Policy-makers would be interested in societal impact generated by reducing
administrative burden, the creation of transparency and its contribution to solving societal
problems. Our evaluation will take into account the multiple stakeholders’ perspectives.
D4.6 Pilots Evaluation Results – Third Round
Page 16 of 215
3 Evaluation Methodology
Based on the four stakeholder perspectives, four areas to be evaluated were derived In :
1. The first area is the co-creation framework (co-initiation, co-design, co-implement, and co-
evaluation). The evaluation is relevant for those decision-makers who want to innovate and
ensure that there is a high level of user participation.
2. The second area is the OGI Toolkit (ICT building blocks and cubes design). The evaluation is
primarily relevant for developers who build applications.
3. The third area is the acceptance of OGI Pilot Applications. End-users should accept and use
the OGI Toolkit deployed in the pilots.
4. The fourth area is the outcomes (estimative money savings, time reduction, efficiency
perception of service delivery and transparency). The outcomes are of relevance for the
policy-makers, but also the society as a whole. In the end, the purpose is to create societal
value.
These four areas are visualized in Figure 2 together with co-creation. The second area about OGI
solution platforms (in blue) contains the development of the building blocks contributing to the
platforms and the use of the platforms, and its building block for developing the apps in the pilots.
Both are closely connected to each other and therefore are integrated together.
Figure 2 - Evaluation dimensions
D4.6 Pilots Evaluation Results – Third Round
Page 17 of 215
The co-creation framework is described in detail in Deliverable D2.1, in the WP2 Framework
Creation. The OGI toolkit is also described in detail in Deliverable D3.5, in the WP3 ICT Tools
Development.
Beyond that, this report is linked to D1.1 OGI Challenges and Needs, as part of the WP1 Challenges
and needs identification the D4.4 Pilots Evaluation results - Second round in the WP4- Pilots Planning
and Evaluation. Those interconnections are visualized in Figure 1.
Pilots are organized in three main iterations. After each iteration, the OGI toolkit will be more
advanced and enable further development in the pilot. This agile way of working enables relatively
short cycle-times and quick improvements. Furthermore, this enabled to evaluate the OGI Toolkit
and building blocks and improve them each time.
1. The first (initial) iteration resulted in an early version of the evaluation of OGI services and
tools. This feedback was used to further improve the OGI ICT Toolkit
2. The second iteration was used to develop a more advanced version. Again, this feedback
was used to further improve the OGI toolkit; and,
3. The final iteration of pilots benefitted from the lessons learned in the first and second years
of pilot iterations and resulted in a mature OGI toolkit.
Figure 3 illustrates the tasks involved in planning for and conducting a pilot and shows the high-level
process. Although the activities in this figure are presented in a waterfall manner, this is only for
communication and planning purposes, and the actual way of working was agile embracing co-
creation and focussed on short iteration resulting in working software and meeting end-user
requirements.
D4.6 Pilots Evaluation Results – Third Round
Page 18 of 215
Figure 3 - High-Level Processes of Pilot Plan
The pilot design team was responsible for creating the pilot implementation and evaluation plans
(Deliverables 4.1, 4.3 and 4.5) as well as pilot evaluation reports (Deliverables 4.2, 4.4 and 4.6). This
team consists of the Research and Development (R&D) Partners in the OGI consortium.
The pilot implementation team was formed by the technical partners and, the people in charge of
each of the six pilots. The Pilot implementation team is responsible for executing the pilot projects
based on the plan created by the pilot design team described on this report D4.5 and the previous
version, D4.3 and D4.1.
The pilot implementation was divided into three main actions:
1) Preparation: the part that deals with collecting needed information from the pilots to fill the
implementation template;
2) Implementation: the part that executes the implementation of the OGI toolkit and co-
creation framework on the pilots by technical partners; and,
3) Evaluation: the part that measures the success of outputs and outcomes after
implementation of OGI toolkit and co-creation framework.
The findings of the evaluation of the second step were analyzed by the OGI Consortium. The result
of this analysis was used to create the pilot plan for the next iterative cycle. These steps aimed to
D4.6 Pilots Evaluation Results – Third Round
Page 19 of 215
identify challenges and needs to improve the implementation and evaluation of OGI ICT Toolkit, and
the OGI Co-Creation Framework in the OGI pilots.
The pilots’ reports provided the processes of each pilot and evaluation on four evaluation
dimensions for each pilot (described at Figure 4), and was the source for the pilot plan of next
iteration, for example D4.4 (Evaluation results in 2nd round) was the source for D4.5 (Pilot and
Evaluation Plan 3rd release), influencing D4.6 (Evaluation results 3rd round – this report). Below,
Figure 4 describes the expected reports aforementioned.
Figure 4 - Pilots' Timeline
D4.6 Pilots Evaluation Results – Third Round
Page 20 of 215
3.1 Pilots’ Co-Creation Evaluation Methods
Taking into consideration the types of co-creation and participant contribution an evaluation
method was developed to collect data and to analyze the data and feedback of participants. This is
explained at D2.1 OGI Framework from WP2 Framework Creation.
The objective of this section is to explain the methods to collect data and ICT tools used to evaluate
the feedback from the identified types of co-creation and participant contributions during the three
years of the OGI project. Table 2 summarises the methods to collect data and ICT tools identified as
useful to collect and analyze feedback from a participant on the four different co-creation stages.
Table 1 - Co-Creation Framework Stages, Methods for Data Collection and Tools for Evaluation
Co-Creation Step
Participant role Source to collect data Methods and Tools for evaluation
Co-Initiation
Problem & needs identification
- Social Media - R statistical analytics
- Weka
- Other social media analytics
Idea generation for ways to solve problems (informed by data)
- User workshops
- Public meetings
- Social Media
Co-Design The input to service design
- User workshop
- Continued participation
- Focus groups
- User Experience and User Interface testing
- Survey (Questionnaire and interviews)
Co-Implementation
Uploading user data - Web and Phone Statistics (Number of access, download, etc.)
- Web Analytics
- Survey (Questionnaire and Interviews)
- R statistical analysis
- Weka
- Other social media analytics
Suggesting changes to data sets
- Portal’s Feedback channels
Data creation for a service
- Web and Phone Statistics (Number of access, download, etc.)
Co-Evaluation
Providing feedback to service quality, usefulness, etc.
- Social Media - Portal’s Feedback channels
- Web Analytics
- Survey (Questionnaire and Interviews)
- R statistical analysis
- Weka - Other social media analytics
Reporting data on service operation
- Web and Phone Statistics (Number of access, download, etc.)
The selected sources and methods are described below in the next section 3.1.1.
D4.6 Pilots Evaluation Results – Third Round
Page 21 of 215
3.1.1 Selected Co-Creation Tools and Methods Overview
3.1.1.1 User workshop for co-creation
A workshop is an activity that aims to introduce something (idea, skill, product, etc.) to a potential
interested people. Workshops range from short workshops (45 minutes or less) to one or more days.
A critical aspect of user workshop feedback process is the inclusion of end-users in the creation of
the new data-driven public services.
The overall structure user workshop planned to be conducted on the pilots is described in Table 3:
Table 2 - User Workshop Protocol for Co-Creation
£ Stage Number of questions
1 Introduction This stage has the aim to describe the background to participants and clarify
questions. A general objective is given to participants
2 Silent
Ideation
In this stage, participants brainstorm to produce ideas. They can take notes
and be prepared to share the ideas with other people on the workshop.
3 Group
Discussion
In this stage, there is a group discussion of all the participants, presenting
the ideas that they had during the silent ideation. It is allowed to
participants to give commentaries or insert inputs from other participants
ideas presentations (discussion).
The three stages can be repeated as many times as needed. This way is possible for all individuals to
provide valuable inputs on the design and structure of the new public service. The user workshop
can be used on all the four stages of Co-Creation framework.
The user workshops can produce outcomes like :
• List of issues (problems) with the new service;
• List of potential solutions and alternatives for improvement;
• Basic thoughts on the usability and functionality of the service;
• User stories;
• List of user personas of individuals who could use the service, and
• Any other information which may come out of the workshop organically.
After participating in the user workshop, a survey was conducted to identify the participants'
feedback.
D4.6 Pilots Evaluation Results – Third Round
Page 22 of 215
3.1.1.2 Survey Research
Whereas the workshops were focussed on deep-understanding and gaining feedback, surveys’ were
conducted to collect qualitative (interviews) and quantitative (questionnaires) data about the pilot's
performance.
Figure 5 – Exploratory and Explanatory approaches at Co-Creation Evaluation Surveys
A survey is a systematic poll of questions made to some group, or individually, to collect answers
about some problem, observation, etc. Glasow (2005) considers two types of data collection
methods:
1. Written (questionnaire); and,
2. Verbal (interviews).
Both types will be conducted with different objectives and different periods of co-creation
evaluation.
Based on the co-creation evaluation, survey research was used in all of the co-creation types.
Interviews were used to examine the co-initiation and co-evaluation, while questionnaire will be
used on the co-design and co-implementation.
The interviews used open-ended questions to seek understanding and interpretation in a different
situation. In the co-initiation stage, interviews were aimed at identifying problems and generating
ideas for problem-solving. In the co-evaluation stage, interviews were aimed at understanding the
questionnaire results as well as to seek in-depth information on the several issues found in the
questionnaire. Normally, only a specific group of stakeholders was involved, taking into
consideration there is a low degree of statistical validity due to low number and homogenous profile
of participants. In addition, there is a higher chance of bias on the answers in comparison of
questionnaires. Confidential information can be collected, the need of resource, time, etc. are spent,
and fewer errors happen on this type of survey.
Questionnaires are using closed-ended questions to gather highly standardized data. It was designed
in such so the target respondents can give more generic inputs of design and implementation on the
co-creation of public policies processes. Normally questionnaires are given to several people to
D4.6 Pilots Evaluation Results – Third Round
Page 23 of 215
reach substantial statistical validity of a hypothesis, for example already observed on the qualitative
approach of the survey (interview).
Before conducting the survey (questionnaire or interview), Glasow (2005) suggests creating a model
that identifies the expected relationships among the variables (independent and dependent).
Variables are used to define the scope of the study, however, cannot be explicitly controlled by the
researcher. Then is possible to test the model against observations of the phenomena analyzing the
data collected on interview or questionnaire.
The survey design
In designing the survey research, Levy and Ellis (2006) suggest two steps 1) sampling strategy and
the procedure to obtain the representativeness of the population, including ensuring reliability and
2) validity. The nature of this evaluation process is in between exploratory and explanatory.
Explorative to find issues and explanatory to explain the effects of the pilots on the user-satisfaction
and outcomes. For this purpose, a the mixed method, including qualitative and quantitative method
was used. The sampling strategy should follow these methods. Population for this survey research
will be all stakeholders in each pilot, or in general, units of observation will be the Public
Administration’s employees, citizens and companies’ employees who use the OGI innovation
ecosystem. Participants of this co-creation survey should represent these units of observation.
The sample techniques were different for each co-creation type. The co-initiation and co-design
used the non-probability sampling, and the co-implementation and co-evaluation used the
probability sampling. The non-probability sampling is used because, in the two first types of co-
creation, the respondents will be selected by the ones that use the OGI tools and framework to
identify the problems and propose the improvement of the public services based on the LOSD from
PAs, citizens, and companies.
For co-implementation and co-evaluation, survey participants were selected randomly, to reach
stronger analysis to justify the use of OGI innovation ecosystem. The challenges of this technique
were to minimize sampling bias and achieve good representativeness. To deal with these issues,
each pilot partner needs to carefully acknowledge the stakeholders of the system, for example,
number of employee, the demography of users (citizens and businesses), structure of the
companies, etc. The list of questions helped partners addressing this issue.
D4.6 Pilots Evaluation Results – Third Round
Page 24 of 215
3.1.1.3 Overview of Pilots’ Co-Creation Steps, Tools, Methods, and Results Description
Table 3 summarizes the selected co-creation tools and methods chosen by each of pilots to
implement and evaluate the co-creation steps. Further, a brief description of results from each of
tools and method.
Table 3 – Overview of Pilots’ Selected Co-Creation Tools and Methods
Pilot Co-Creation Steps
in Year 2
Co-Creation Methods and Tools
for Evaluation Brief Description of Results
Pilot 1
The Greek
Ministry of
Administrative
Reconstruction
(Greece)
Co-
Implementation
Continued participation from User
workshop
Update in application design and
data quality improvement.
Co-Evaluation End-User acceptance and
outcomes survey
Statistical analysis of Pilot
feedback from participants
Pilot 2
Enterprise
Lithuania
(Lithuania)
Co-
Implementation
Online continued participation
from User workshop
Update in application design, new
data sets included and data quality
improvement.
Co-Evaluation End-User acceptance and
outcomes survey
Statistical analysis of Pilot
feedback from participants
Pilot 3
Tallinn Real
Estate
(Estonia)
Co-
Implementation
Hackathon event to exploit the
code and data sets
List of ideas from Hackathon
participants
Online discussion via Github
about code and data sets by
Hackathon participants
Github statistical analysis (views,
forks, changes) and list of
suggestions in code and data sets
Co-Evaluation End-User acceptance and
outcomes survey
Statistical analysis of Pilot
feedback from participants
Pilot 4
Trafford Council
Worklessness
(England)
Co-
Implementation User workshop
Update in application design and
data quality improvement.
Co-Evaluation End-User acceptance and
outcomes survey
Statistical analysis of Pilot
feedback from participants
Pilot 5
The Flemish
Environment
Agency
(Belgium)
Co-
Implementation User workshop
Update in application design and
data quality improvement.
Co-Evaluation Didn’t evaluated because the
application is under review.
Didn’t evaluated because the
application is under review.
Pilot 6
Marine Institute
(Ireland)
Co-
Implementation User workshop
Update in application design and
data quality improvement.
Co-Evaluation Didn’t evaluated because the
application is under review.
Didn’t evaluated because the
application is under review.
D4.6 Pilots Evaluation Results – Third Round
Page 25 of 215
3.2 Pilots’ Acceptance and Outcomes Evaluation Methodology
3.2.1 Pilots’ Acceptance Selected Methodology
A combination of theories was employed to create to evaluate the user perspective. The focus was on an evaluation of user acceptance and intention of use for OGI. The list of main theories is described in the report D4.3 and is listed below:
1. TAM 1 by Davis (1989);
2. TAM 2 by Venkatesh and Davis (2000);
3. TAM 3 by Venkatesh and Bala (2008);
4. UTAUT Framework by Venkatesh, Morris et al. (2003); and,
5. IS Success Framework by Delone and McLean (2003).
The presents variables and (general) measured items for the pilot's evaluation. However, as each pilot has its characteristics in term of objectives, target audiences, type of data, type of services, OGI building blocks uses, and visualization, The questionnaire created for the evaluation was slightly different to each pilot. We used the same foundation. However, we added specific questions for the pilot. The questionnaires of each pilot are described in Section 7.1.
Table 4 - User Acceptance Evaluation for OGI
Variable Measured Items Source
Job Relevance (JR)
OGI toolkit makes mine and my colleagues job tasks easier to be accomplished.
In my job, usage of the system is important.
In my job, usage of the system is relevant.
Venkatesh and Davis (2000)
Output Quality (OQ)
The quality of the output I get from the OGI toolkit is higher compared to the previous system.
I have no problem with the quality of the OGI toolkit's output.
Venkatesh and Davis (2000)
Result Demonstrability (RD)
I have no difficulty telling others about the results of using the OGI toolkit.
I believe I could communicate to others the consequences of using the OGI toolkit.
The results of using the OGI toolkit are apparent to me.
I would have difficulty explaining why using the OGI toolkit may or may not be beneficial.
Venkatesh and Davis (2000)
Perceived Ease of Use (PEU)
My interaction with OGI toolkit is clear and understandable.
OGI toolkit usage does not require a lot of skills.
I find it easy to get the OGI toolkit to do what I want it to do.
I find the OGI toolkit to be easy to use.
(Davis 1989)
Perceived Usefulness (PU)
Using OGI toolkit improves my performance in my job tasks.
Using OGI toolkit enhances my effectiveness in my job tasks.
Using OGI toolkit in my job increases my productivity.
I find the OGI toolkit to be useful in my job.
(Davis 1989)
Intention to Use (IU)
If I have access, I will use OGI toolkit. (Davis 1989)
D4.6 Pilots Evaluation Results – Third Round
Page 26 of 215
3.2.1.1 Pilots’ Outcomes Selected Methodology
The outcomes evaluation is divided into a contribution to transparency, Reduction of Administrative Burden and Costs. These will be discussed next.
3.2.1.2 Transparency
Transparency has been used as a magic concept by the government and public managers (Ward
2014). The usage of transparency is sometimes used as a synonym for accountability (Bovens 2007),
openness (Coglianese 2009), and even open government data (Frank and Oztoprak 2015) as an
example. There are no clear definitions that are generally accepted by scholars in this field. This
project takes into consideration transparency as a concept of a unilateral process of disclosure of
data, information, and actions that an organization has been conducting to the public (Peixoto
2013).
Transparency is a phenomenon that can lead to accountability but does not guarantee any concrete
result of justice or mobilization, just public exposure to scrutiny (Fox 2007). In addition to the
discussion of what is and what is not transparency, only a few models are trying to explain and
evaluate transparency. Matheus and Janssen (2013) proposed an evaluation model for transparency
initiatives considering transparency as a multidimensional object based on many factors influencing
two main dimensions: interpretation and accessibility. Facilitating conditions were added
considering the background and profile of users influencing transparency.
The OGI project will test if factors and facilitating conditions influence transparency dimensions
positively or negatively. These factors designed as propositions with suggesting the direction the
factors contribute to, however, there is no proof for this in the literature. Our evaluation of the
pilots contributes to understanding these propositions. The transparency model used in OGI
evaluation is described in detail in the section 3.6.1 of D4.3. The table below summarizes the
attributes evaluated and described each proposition in detail. Figure 6 then shows the graphical
view of Transparency Model of Matheus and Janssen (2013).
D4.6 Pilots Evaluation Results – Third Round
Page 27 of 215
Table 5 - Adapted Transparency Model for OGI Evaluation
Dimension Factor P. # Proposition
Interpretation
Ibid P1 easier interpretation of data results in higher transparency
Examples of usage P2 presence of examples of the website product, the higher has a positive influence on interpretation.
Simple Language used P3 simple language has significant positive influence on transparency
Data Quality P4 higher information quality has a significant influence on the interpretation
Updatedness of Information
P4a higher updated information has a significant influence on data quality.
Completeness P4b higher data completeness has a significant influence on data quality
Accuracy P4c higher data accuracy has a significant influence on data quality Accessibility ibid P5 higher accessibility has a significant positive influence on
transparency Simple Language P5a simple language has significant positive influence on
transparency Data Overload P5b data overload has a significant negative influence on accessibility Adhesion to Standards P5c adhesion to standards has a significant positive influence on
accessibility Unified Technology P5d unified use of technology has a positive influence on accessibility
Facilitating Conditions
Experience 6a the influence of interpretation on transparency will be moderated by experience
6b the influence of accessibility on transparency will be moderated by experience
Age 7a the influence of interpretation on transparency will be moderated by age
Level of Education 8a the influence of interpretation on transparency will be moderated by the level of education
8b the influence of accessibility on transparency will be moderated by the level of education
D4.6 Pilots Evaluation Results – Third Round
Page 28 of 215
Figure 6 - Adapted Transparency Model for OGI Evaluation
Matheus and Janssen (2013)
3.2.1.3 Administrative Burden
Governments are facing issues to deliver more and better public services with less financial
resources, time and people. Due to these reasons, OGI evaluates how the OGI ICT toolkit can help to
reduce the administrative burden. The evaluation methods are based on the Cost-Benefit
Assessment (CBA) approach, following the Study on eGovernment and the Reduction of
Administrative Burden - SMART 2012/0061 (Gallo, Giove, et al. 2014). The taxonomy of costs and
benefits of Gallo et al. (2014) has been adapted.
The dimension considered for both Public Sector and Users Benefits was divided into two categories,
e.g. direct and indirect benefits. The reason to focus only on Benefits is that the OGI consortium is
interested oi the benefits that OGI innovation ecosystem can provide. The figure below summarises
the OGI focal point which is aimed at creating both advantages for the public sector and end-users.
D4.6 Pilots Evaluation Results – Third Round
Page 29 of 215
Figure 7- OGI Focal Point
Table 6 summarises the adapted taxonomy for Costs and Benefits of OGI innovation ecosystem for
stakeholders.
Table 6 - Taxonomy for Benefits for OGI innovation ecosystem stakeholders
Dimension Categories Sub-Categories Description
Benefits for Public Sector
Direct Benefits
Time Savings
It includes all monetizable benefits arising from improvements on public service delivery, including time savings before/after OGI innovative ecosystem implementation.
Indirect Benefits
Efficiency perception It encompass non-magnetisable benefits related to
better service delivery and the enhancement of the decision-making process.
Quality of service
Benefits to Users (citizens, businesses)
Direct Benefits
Time-Saving It includes all monetizable benefits arising from improvements on public service delivery, including time savings before/after OGI innovative ecosystem implementation.
Indirect Benefits
Estimative Money Saving
Quality of service It encompass non-magnetisable benefits related to better service delivery and the enhancement of the decision-making process.
There were two ways to understand "benefits for Public Sector." First as an organization, and second
as people involved in the processes. Taking into consideration the OGI innovation ecosystem, we
decided to take approaches depending on pilots. Some pilots have a clear boundary between Public
Sector (organization and people – civil servants, policy makers, etc.) and users (citizens, businesses,
D4.6 Pilots Evaluation Results – Third Round
Page 30 of 215
etc.). However, other pilots didn't have a clear boundary between Public Sector and users, because
the users are on the Public Sector (civil servants, policy makers, etc.).
The summary of data collection methods is presented in Table 6, and the planned questions of
questionnaires and interviews are at section 7.1.
1. Time savings
Time savings can be determined in two types of scenarios. If the public service already existed, it is
possible to measure how much time in days or hours a civil servant or citizen/business person
performed the task or receive the public service delivery. The measure can be collected using the log
observations of the service delivery, researcher observation of process (in person) or via a survey
(questionnaire and interview).
If the public service did not exist yet and a new service was created, measures comparing similar
processes or collect data from users (civil servants, citizens, business person, etc.) from a perception
or expectation perspective will be used. Next, a Likert scale based on the level of satisfaction with
time spent on public service (1- very dissatisfied to 5-very satisfied) will be used to measure the
perceived time savings.
2. Efficiency Perception
Efficiency perception considers the measuring of the perception of people who use public sector
data in the OGI pilots. People might have the experience to use open data and can compare this with
the use of the data in the OGI pilot. This parameter is a perception of the users, and it does not
mean quantifying the time savings like in the previous measure. The objective of this measure is to
identify if public sector users (civil servants, public managers, etc.) perceive if there is an
improvement in efficiency. To measure efficiency perception, a questionnaire will be conducted
among pilot users. The questionnaire uses a Likert scale for measuring the level of agreement (1
strongly disagree to 5- strongly agree) with an increase in efficiency.
3. Estimative Money Savings
Money savings look like a hard measurement, but is often hard to quantify and has to rely on
estimates. Taking this into consideration, the approach used a perception or expectation perspective
for expenses saving b comparing the situations before and after implementation of OGI pilots. A
questionnaire was used to collect data based on a Likert scale of agreement (1 strongly disagree to
5- strongly agree), and an open question was used to ask uses how much expenses were avoided.
4. Quality of Service
Quality of service considers two potential scenarios. If the public service already existed, it is
possible to measure the level of quality using a Likert scale (1-poor to 5- excellent) of the current
service and compare the outcomes with the ranking on the past service (1- much worse to 5- much
better). If any issue was identified on this questionnaire, further interviews were conducted to
further analyze the issue.
If the public service doesn't do exist and a new one was created, perception data about the quality if
the new service was collected from the users (civil servants, citizens, business person, etc.) using a
D4.6 Pilots Evaluation Results – Third Round
Page 31 of 215
survey. A Likert scale was used measuring the level of satisfaction with the quality of public service
(1- very dissatisfied to 5-very satisfied).
D4.6 Pilots Evaluation Results – Third Round
Page 32 of 215
3.2.1.4 Pilots’ Acceptance and Outcomes Evaluation Questionnaire
In this section, we surveyed Pilots’ end-users about their perceptions of using the OGI pilots
Applications. The categories collected and evaluated are described in Table 7:
Table 7 – Pilots’ Outcomes Evaluation Questionnaire
Outcomes
Categories Description
Number of
questions Source
User
Information
This section aims to introduce and explain
the pilot application, describe the survey
and identify general background
information. For example, key questions if
end-user is aware of Linked Open Data, the
OGI Project and the specific pilot end-user is
using.
Two
questions Created by authors
User
Acceptance
This section aims to identify if End-Users
accepts the functionalities that pilot
applications offer. For example, if the
functionalities are by end-users needs, or if
the app does not crash.
Thirteen
questions and
an open
question in
the final for
general
comments.
Adapted from Davis (1989);
Venkatesh and Davis (2000);
Venkatesh and Bala (2008);
Venkatesh, Morris et al.
(2003); and,
Delone and McLean (2003)
Data Sets
This section aims to identify if End-Users
accepts the available data sets in the pilot
application has the desired or needed data
quality standard.
Five questions
and an open
question in
the final for
general
comments.
Adapted from Dodds (2016)
and Matheus and Janssen
(2013)
Results
This section aims to identify if End-Users
accepts that pilot application can deliver
expected results and outcomes. For
example, if the pilot application can
increase Transparency or can reduce the
Administrative Burden and Costs.
Ten questions
and an open
question in
the final for
general
comments.
Adapted from Gallo, Giove,
et al. (2014)
D4.6 Pilots Evaluation Results – Third Round
Page 33 of 215
For each pilot, an end-user questionnaire was developed and distributed among users.
1. Pilot 1 – The Greek Ministry of Administrative Reconstruction (Greece)
a. End-User Questionnaire (Section 7.1.1).
2. Pilot 2 – Enterprise Lithuania (Lithuania)
a. End-User Questionnaire (Section 7.1.2)
3. Pilot 3 – Tallinn Real Estate (Estonia)
a. End-User Questionnaire (Section 7.1.3)
4. Pilot 4 – Trafford Council Worklessness (England)
a. End-User Questionnaire (Section 7.1.4)
5. Pilot 5 – The Flemish Environment Agency (Belgium)
a. End-User Questionnaire (Section 7.1.5)
6. Pilot 6 – Marine Institute (Ireland)
a. End-User Questionnaire (Section 7.1.6)
D4.6 Pilots Evaluation Results – Third Round
Page 34 of 215
3.3 OGI ICT Toolkit Evaluation Methodology
To evaluate the implementation of OGI ICT Toolkit in the third Year, we created the ICT Toolkit
Questionnaire to measure the OGI Data Quality, OGI Architecture, System Quality of Application,
and to externally evaluate the OGI ICT Toolkit.
The OGI ICT toolkit consists of building blocks that were developed for performing a certain
functionality. Table 8 shows the evolution of the tools (building blocks) names and the merges of
tools over the three years of projects and the accompanying deliverables (D3.1, D3.2, D3.3, D3.4,
D3.5, and D3.6). Figure 8 summarizes the OGI data cycle and the tools for each step.
Table 8 - OGI Toolkit Releases and Tools per year
1st Release
(Based on D3.1 and D3.2)
2nd Release
(Based on D3.3 and D3.4)
Full version
(Expected D3.5 and D3.6)
A- Previous
Project Results
1. Grafter 1. Grafter 1. Grafter
- 2. Data Cube Builder 2. Data Cube Builder
B- OGI Toolkit
2. Data Cube OLAP
Browser 3. Data Cube OLAP Browser
3. Cube Explorer 3. Data Cube Explorer 4. Data Cube Explorer
4. Data Cube Visualizer 5. Data Cube Visualizer
1. JSON API for Data Cube 1. JSON API for Data Cube 1. CubiQL API
2. Table2QB 2. Table2QB 2. Table2QB
3. Data Cube Aggregator 3. Data Cube Aggregator 3. Cube Aggregator
- 4. LOSD Machine Learning
Component
4. LOSD Machine Learning
Component
- 5. SPARQL connector for
Exploratory
5. SPARQL connector for
Exploratory
- - 6. DataScience SPARQL plugin
- - 7. Data Cube RDF Validator
C- Pilot
Specification
Tools
- 1. Assisted Cube Schema
Creator
1. Assisted Cube Schema
Creator
- 2. QB Multi-Dimensional
Charting
2. QB Multi-Dimensional
Charting
- 3. RDF Data Cube Geo-Data
Supported for Dashboard
3. RDF Data Cube Geo-Data
Supported for Dashboard
- - 4. Data Cleansing Tools
- - 5. Data Collection Tools
- - 6. Custom Mappings and Scripts
D4.6 Pilots Evaluation Results – Third Round
Page 35 of 215
- - 7. ShinyR
- - 8. Superset + Druid
Table 8 - OGI Toolkit Releases and Tools per year
1st Release
(Based on D3.1 and D3.2)
2nd Release
(Based on D3.3 and D3.4)
Full version
(Expected D3.5 and D3.6)
A- Previous
Project Results
1. Grafter 1. Grafter 1. Grafter
- 2. Data Cube Builder 2. Data Cube Builder
B- OGI Toolkit
2. Data Cube OLAP
Browser 3. Data Cube OLAP Browser
3. Cube Explorer 3. Data Cube Explorer 4. Data Cube Explorer
4. Data Cube Visualizer 5. Data Cube Visualizer
1. JSON API for Data Cube 1. JSON API for Data Cube 1. CubiQL API
2. Table2QB 2. Table2QB 2. Table2QB
3. Data Cube Aggregator 3. Data Cube Aggregator 3. Cube Aggregator
- 4. LOSD Machine Learning
Component
4. LOSD Machine Learning
Component
- 5. SPARQL connector for
Exploratory
5. SPARQL connector for
Exploratory
- - 6. DataScience SPARQL plugin
- - 7. Data Cube RDF Validator
C- Pilot
Specification
Tools
- 1. Assisted Cube Schema
Creator
1. Assisted Cube Schema
Creator
- 2. QB Multi-Dimensional
Charting
2. QB Multi-Dimensional
Charting
- 3. RDF Data Cube Geo-Data
Supported for Dashboard
3. RDF Data Cube Geo-Data
Supported for Dashboard
- - 4. Data Cleansing Tools
- - 5. Data Collection Tools
- - 6. Custom Mappings and Scripts
- - 7. ShinyR
- - 8. Superset + Druid
D4.6 Pilots Evaluation Results – Third Round
Page 36 of 215
Figure 8 shows how the various OGI Tools can be used in the LOSD workflow. The workflow starts
with the raw data that might need to be prepared and ends with the visualization.
Figure 8- OGI Tools and Working Flow
D4.6 Pilots Evaluation Results – Third Round
Page 37 of 215
Table 9 shows the list of tools, the links to the OGI Github code source
(https://github.com/OpenGovIntelligence) and the pilots which are using these tools. Important to
highlight is that the colors used are the same as in Table 8 to show the connection between them.
D4.6 Pilots Evaluation Results – Third Round
Page 38 of 215
Table 9 - OGI Tools used in Pilots
Tools
Pilots
MAREG
(Greece)
Lithuania
Enterprise
Trafford
(England)
The Flemish
Government
(Belgium)
Marine
Institute
(Ireland)
Ministry of
Economics
(Estonia)
Grafter (Private Tool)
www.goo.gl/vaGzLc X X X
Tarql – Cube Builder
https://goo.gl/fQ9YQx X
X X
Cube Explorer (NEW)
www.goo.gl/WLC6zh X
X
Cube Browser
www.goo.gl/b57zpX X X
Cube Visualizer
www.goo.gl/hc9QGj X
X
CubiQL API (NEW)
www.goo.gl/tbZuPg X X x
X X
Table2QB (NEW)
www.goo.gl/JgMKvx X X X X X X
Cube Aggregator
www.goo.gl/urSj5V X X
X X
LOSD Machine Learning
Component (NEW)
https://goo.gl/c2gkV9
X
SPARQL connector for
Exploratory (NEW)
www.goo.gl/7AeYVF
X
Validator (NEW)
www.goo.gl/Kn6Bhh X X X X X
Assisted Cube Schema
Creator (NEW)
https://goo.gl/BCLW2s
X X
QB Multi-Dimensional
Charting X
RDF Data Cube Geo-Data
Supported for Dashboard
Data Cleansing Tools X X X X X X
Data Collection Tools X X X X X X
Custom Mappings and
Scripts
X X X X X X
ShinyR X
SuperSet + Druid X X
D4.6 Pilots Evaluation Results – Third Round
Page 39 of 215
3.3.1 OGI Pilots’ Data Quality Evaluation Methodology
To evaluate the implementation of OGI Data Quality in the third year, we created a Data Quality
Questionnaire. The Data Quality Questionnaire is inspired by the Open Data Institute blog post
“Exploring Open Data Quality (https://theodi.org/blog/exploring-open-data-quality) by Leigh Dodds
(Dodds, 2016). Table 10 describes the data quality attributes, their descriptions, the number of
questions in the questionnaire and the source from where they were collected or inspired. The
questionnaire with all the questions is presented in Table 58. The link to the online form is
https://docs.google.com/forms/d/e/1FAIpQLSeCl3flPZIGYEWhhASTEnwF6GkMERIxZ8OWjkKpJCmqRl
LwoA/viewform.
Table 10 – Pilots’ Data Quality Questionnaire
Data Quality
Section Description
Number of
questions Source
Discoverability
and Usability
This section aims to identify the application characteristics
that will influence some attributes of data qualities below.
For example, if the website or application has an option for
downloading data. This parameter influences the level of
granularity.
Seven
questions
Created by
authors.
Granularity
This section aims to identify the level of granularity. The
desired level of granularity varies according to the problem
being addressed. For example, if the data is in aggregated
level (average sex instead of individual data) or timely
option for download (per year average, per month, per
week, per day).
Three
questions
Frank and
Walker
(2016) Intelligibility
This section aims to identify immediate intelligibility. This
parameter gives quality to the data in the sense of
readability and increases immediate understand by users.
For example, if there is documentation giving context for
humans understand the data and see how it is possible to
use it in human or machine ways.
1 question
and open for
commentary
Trustworthiness
This section aims to identify data quality attributes that
increase the trust in data. For example, if the source is
trusted, if who is in charge of data is aware of how it was
collected and created.
Five
questions
Linkable to other
data
(5 Stars LOD)
This section aims to identify data quality attributes to
Linking the data to other opened datasets. For example, if
the data is in CSV or RDF format.
Four
questions
Bizer et al.
(2009)
15 Open Data
Principles
This section aims to identify data quality attributes related
to the 15 Open Data Principles such as timely, accessible
and primary.
Nine
questions
OPENGOV
DATA
(2018)
D4.6 Pilots Evaluation Results – Third Round
Page 40 of 215
3.3.2 The standard for Systems and Software Quality Requirements and Evaluation – ISO 25010 and ISO 25012
Evaluation of OGI ICT toolkits is divided into two categories. The first of building blocks and the second for evaluation of cubes design. The summary of data collection and methodology of evaluation is presented at Table 11.
Table 11 - Cube Design and Building blocks data collection and methodology methods of Evaluation
Category Target groups Data Collection Approach Methodology of Evaluation
Product Quality
ICT Partners and IT Department of PAs
Questionnaire and Structured observation of application/website
ISO/IEC 25010
Quality in Use
System’s Data Quality
ISO/IEC 25012
Since the beginning, criteria for evaluation the OGI toolkit needs to be defined. Scientific literature review couldn't provide us with an extensive list of standards and requirements organized and structured. On the other hand, ISO/IEC 25010:2011, the standard for Systems and Software Quality Requirements and Evaluation (ISO/IEC, 2010), presents a structured list of requirements for building blocks and systems, which we considered for cubes design.
ISO 25010 is adopted as the evaluation method for OGI ICT toolkit. ISO 25010 is organized in 8 parameters which are divided into 30 measurement variables presented at Table 12.
Table 12 - OGI Toolkit Requirements for Evaluation
No Parameter Description Measured by Description
1 Functionality
the degree to which the OGI solution platform provides functions that meet stated and implied needs when used under specified conditions
Functional completeness
the set of functions covers all the specified tasks and user objectives
Functional correctness
the correct results with the needed degree of precision
Functional appropriateness
the accomplishment of specified tasks and objectives
2 Performance
the degree to which the OGI solution platform performs relative to the number of resources used under stated conditions
Time behavior
the response and processing times and throughput rates of a product or system, when performing its functions, meet requirements.
Resource Utilization
the amounts and types of resources used by a product or system, when performing its functions, meet requirements.
Capacity the maximum limits of a product or system parameter meet requirements.
3 Compatibility the degree to which the OGI solution Coexistence perform its required functions
D4.6 Pilots Evaluation Results – Third Round
Page 41 of 215
platform can exchange information with other products, systems or components, and/or perform its required functions while sharing the same hardware or software environment.
efficiently while sharing a common environment and resources with other products, without detrimental impact on any other product.
Interoperability exchange information and use the information that has been exchanged.
4 Usability
the degree to which the OGI solution platform can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use
Appropriateness recognizability
users can recognize whether a product or system is appropriate for their needs.
Learnability
can be used by specified users to achieve specified goals of learning to use the product or system with effectiveness, efficiency, freedom from risk and satisfaction in a specified context of use.
Operability has attributes that make it easy to operate and control.
User error protection
protects users against making errors.
User interface Aesthetics
the user interface enables pleasing and satisfying interaction for the user.
Accessibility
can be used by people with the widest range of characteristics and capabilities to achieve a specified goal in a specified context of use.
5 Reliability
The degree to which the OGI solution platform performs specified functions under specified conditions for a specified period.
Maturity meets needs for reliability under normal operation.
Availability operational and accessible when required for use.
Fault tolerance operates as intended despite the presence of hardware or software faults.
Recoverability recover the data directly affected and re-establish the desired state of the system.
6 Security
the degree to which the OGI solution platform protects information and data so that persons or other products or systems have the degree of data access appropriate to their types and levels of authorization
Confidentiality ensures that data are accessible only to those authorized to have access
Integrity prevents unauthorized access to, or modification of, computer programs or data
Non-repudiation proven to have taken place, so that the events or actions cannot be repudiated later
Accountability actions of an entity can be traced uniquely to the entity
Authenticity the identity of a subject or resource can be proved to be the one claimed
7 Maintainability the degree to which the OGI solution Modularity composed of discrete components
D4.6 Pilots Evaluation Results – Third Round
Page 42 of 215
platform can be modified to improve it, correct it or adapt it to changes in the environment, and in requirements
such that a change to one component has minimal impact on other components
Reusability an asset can be used in more than one system, or in building other assets
Analysability
possible to assess the impact on a product or system of an intended change to one or more of its parts, or to diagnose a product for deficiencies or causes of failures, or to identify parts to be modified
Modifiability effectively and efficiently modified without introducing defects or degrading existing product quality
8 Portability
the degree to which the OGI solution platform can be transferred from one hardware, software or other operational or usage environment to another
Adaptability
can effectively and efficiently be adapted for different or evolving hardware, software or other operational or usage environments
Installability can be successfully installed and uninstalled in a specified environment
Replaceability can replace another specified software product for the same purpose in the same environment
Source: (ISO/IEC, 2010)
The quality in use relates to the impact or outcome of the product when used in a particular context and consists of 5 parameters which are divided into 11 measurement variables as shown in Table 13.
D4.6 Pilots Evaluation Results – Third Round
Page 43 of 215
Table 13 - Criteria for Evaluation of Quality in Use
No Parameter Description Measured by Description
1 Effectiveness accuracy and completeness with which users achieve specified goals - -
2 Efficiency resources expended about the accuracy and completeness with which users achieve goals - -
3 Satisfaction
the degree to which the user needs are satisfied when using the OGI solution platform in a specific context of use
Usefulness
the degree to which a user is satisfied with their perceived achievement of pragmatic goals, including the results of use and the consequences of use
Trust
the degree to which a user or other stakeholder has confidence that a product or system will behave as intended
Pleasure
the degree to which a user obtains pleasure from fulfilling their personal needs
Comfort the degree to which the user is satisfied with physical comfort
4 Freedom from Risk
the degree to which the OGI solution platform mitigates the potential risk of the usage
Economic Risk Mitigation
the potential risk to financial status, efficient operation, commercial property, reputation or other resources in the intended contexts of use
Health and Safety Risk Mitigation
the potential risk to people in the intended contexts of use
Environmental Risk Mitigation
the potential risk to property or the environment in the intended contexts of use
5 Context coverage
the degree to which the OGI solution platform can be used with effectiveness, efficiency, freedom from risk and satisfaction in both specified contexts of use and contexts beyond those initially explicitly identified
Context Completeness
can be used with effectiveness, efficiency, freedom from risk and satisfaction in all the specified contexts of use
Flexibility
can be used with effectiveness, efficiency, freedom from risk and satisfaction in contexts beyond those initially specified in the requirements
Source: (ISO/IEC, 2010)
During the evaluation, each measurement variable was assessed from the user perspective, such as public administration offices, citizens and businesses. Their input was used to improve the OGI solution platform by the consortium, and the revision was evaluated again during the next stage of the pilot.
D4.6 Pilots Evaluation Results – Third Round
Page 44 of 215
3.3.3 User Experience (UX) and User Interface (UI) testing
While user experience (UX) is a term that has been used on practice and scientific literature, but is
hardly deep described or conceptualized uniformly. After conducting 275 interviews on the UX area
and deep literature review, Law, Roto et al. (2009) realized the reasons due an ill-description and ill-
conceptualization of UX:
1. First, because of the broad range of fuzzy and dynamic concepts, including emotional, affective, experiential, hedonic, and aesthetic variables.
2. Second, because of flexibility on analysis since single point to a holistic process. 3. Third, due to the fragmented and theoretical models involved on the UX domain.
Hassenzahl and Tractinsky (2006) conceptualize UX as "a term associated with a wide variety of
meanings ranging from traditional usability to beauty, hedonic, affective or experiential aspects of
technology use" (Forlizzi and Battarbee 2004). Garrett (2010) structures user experience as a project
with five dimensions and two product layers (as functionality and as information), from the more
abstract to the more concrete: strategy, scope, structure, skeleton, and surface. This structured is
presented in Figure 9.
Figure 9 - UX Structure and layers of product and information
Source: Garret (2010)
To improve the usability of software and information systems, the paradigm of user-centered
design, International Organization for Standardization (ISO) 13407, Human-centred design processes
for interactive systems, is a standard that guides user-centered design (Jokela, Iivari et al. 2003).
D4.6 Pilots Evaluation Results – Third Round
Page 45 of 215
The ISO 9241-210 replaced the ISO 13407, which aimed to provide guidance on achieving quality in
use by incorporating user-centered design activities throughout the life cycle of interactive
computer-based systems. ISO 9241-210 standard describes six key principles that will ensure your
design is user centered (Travis 2011):
1. The design is based upon an explicit understanding of users, tasks, and environments. 2. Users are involved throughout design and development. 3. The design is driven and refined by user-centered evaluation. 4. The process is iterative. 5. The design addresses the whole user experience. 6. The design team includes multidisciplinary skills and perspectives.
ISO 9241-210 recommends the use of "ripple effect." It means to plan all the possibilities of tools
and scenarios of usage before implementing. After implementation, scenario, tools, activities, goals,
etc., can change, and influence the result. If the plan is well conceptualized, the plan is likely to
succeed. Figure 10 visualizes an example path taken due changes of plans made during the
implementation.
Figure 10 - The ripple effect
Source: Garret (2010)
Further to the explanations given on ISO 9241-210 (Travis 2011) and (13407 Travis 2011), on both
standards, there are no clear guidelines of steps to implement UX. Filling in this void, Jokela et al.
(1999) proposed a guideline to fill this blank comparing both standards. The guideline consists of 6
steps:
1. Identify the need for human-centered design;
D4.6 Pilots Evaluation Results – Third Round
Page 46 of 215
2. Understand and specify the context of use; 3. Specify the user and organizational requirements; 4. Produce design solutions; 5. Evaluate design against requirements (loop to step 1 if not reach the desired requirement);
and, 6. A system satisfies specified user and organizational requirements.
Figure 11 - UX implementation and evaluation steps
Source: Jokela et al. (2003)
Besides the steps to conduct implementation of UX, it was also identified by Jakola et al. (1999) that
measures are not created to identify efficiency or any goal that should be reached. For this, we are
using the ISO/IEC 25010:2011 (ISO/IEC 2010).
The ISO/IEC 25010:2011 has a parameter called "usability" in which six measures define if a system is
usable. If consortium identifies the need to improve this usability, the Albert and Tullis (2013)
evaluation method can be used as an auxiliary. We consider that User Interface (UI) is a
complementary effect of UX and associated with the look, feel and interactivity of system. It is
already measured on the UX standards and ISO 25010:2011, in special on the evaluation quality of
use.
D4.6 Pilots Evaluation Results – Third Round
Page 47 of 215
4 OGI Evaluation Results
The previous sections provided insights regarding plan and methodology used to evaluate co-
creation framework, OGI toolkit and its implementation in six pilots. This section presents the results
of the evaluation process.
This section is divided into two main sections. The first section 4.1 shows the Pilot’s Evaluation
Results. The first section includes the evaluation of:
1. Pilots’ Co-creation;
2. Pilots’ acceptance of OGI App; and,
3. Pilots’ outcomes
a. Transparency; and,
b. Administrative burden.
The second section 4.1.7 shows the OGI ICT Toolkit Evaluation Results. The second section includes
the evaluation of:
1. OGI Data Quality;
2. OGI Architecture;
3. System Quality of Application; and,
4. External evaluation of OGI ICT Toolkit.
D4.6 Pilots Evaluation Results – Third Round
Page 48 of 215
4.1 Pilots’ Evaluation Results
This section presents the evaluation results of the six pilots. Figure 12 below summarizes the
common flow of activities performed in pilots during implementation and evaluation.
Figure 12 – Common flow of pilots activities
D4.6 Pilots Evaluation Results – Third Round
Page 49 of 215
4.1.1 Pilot 1 – The Greek Ministry of Administrative Reconstruction (Greece)
4.1.1.1 Pilot Description and Expectation
The Greek pilot is under coordination of the Greek Ministry of Administrative Reconstruction
(MAREG) and has an initial objective to improve the monitoring and management of Government
Vehicles used by all Greek Public Agencies. The data that MAREG possesses for this monitoring and
management originate from different sources have not yet been properly defined, structured and
combined to be converted to meaningful information to facilitate internal decision-making and to
increase transparency towards the public.
Taking into consideration the results of the first iteration, the updated target groups (A) and
expected user scenarios (B) are presented below:
A- Target Groups:
1. Decision makers of MAREG
2. Public agencies operating government’s Vehicles
B- Expected User Scenarios:
1. As a user, I want to be able to have an overview of all Government Vehicles operating in
Greece based on descriptive statistical measures
2. As a user, I want to have an overview of all Government Vehicles operating in each Public
Agency
3. As a user, I want to have an overview of Government Vehicles per Municipality
4. As a user, I want to have an overview of Government Vehicles per Prefecture
5. As a user, I want to have an overview of Government Vehicles per Region
6. As a user, I want to search municipalities according to their population and have an overview
of the Government Vehicles operating in them
7. As a user, I want to search municipalities according to their altitude and have an overview of
the Government Vehicles operating in them
For a detailed description of MAREG and Greek pilot, please check report D1.1 and D4.1.
4.1.1.1.1 Identified Pilot Problems in the First Year
MAREG invited pilot’s stakeholders for this workshop with the objective to introduce them to the
pilot’s objective and extract from participants’ useful inputs for Greek pilot and OGI ICT toolkit
development. Important to highlight, all the workshops performed on this evaluation process were
conducted using the Delbecq et al. (1975) methodology. Below, the list of Greek workshop
objectives:
Optimize monitoring and management of Government Vehicles
Facilitate internal decision-making
Minimize operational costs related to Government Vehicles
Increase transparency on data related to Government Vehicles
D4.6 Pilots Evaluation Results – Third Round
Page 50 of 215
Table 14 provides the questions for the participants.
Table 14 - Co-Creation User Workshop Questions
Question
number Question
1
Describe the main problems encountered in your interaction with the
Ministry of Administrative Reconstruction regarding Government
Vehicles?
2 What do you think are the causes of these problems?
3 What are the proposed solutions to these problems?
4 How do you think the current pilot could help towards these solutions
by using linked open data?
5 What useful information would you like to be produced by combining
the data sets available?
6 How will this information facilitate your daily work?
7 What services do you think could be available to the public?
8 What data could be openly available to the public?
The Greek pilot is mainly addressed to internal government users as it aims to improve decision-
making regarding government matter. The use cases refer to the user’s need for accurate
information on government’s Vehicles. This section presents two examples of the need for such
reporting:
The Ministry of Administrative Report needs to reply to a parliamentary question about the
number of Government Vehicles that operated by all Public Agencies. For this direct access
to accurate data about the actual number of Government Vehicles, as well as statistical
measures, such as their average cubic capacity, or their average fuel consumption per
kilometer is needed.
Public Agency X requests the permission to acquire a new four-wheel drive 2.000 cc car.
User Y of the Department in charge of Government Vehicles needs to get a detailed
overview of how many Government Vehicles operate in each Public Agency, as well as a
summary of their technical specifications (e.g., average cubic capacity).
After 30 minutes of ideation, participants were invited to explain their perspectives, bringing ideas
and solutions for the issues identified on the questions at Table 14.
D4.6 Pilots Evaluation Results – Third Round
Page 51 of 215
Table 15 - Summary of Ideas and Solutions from participants
Problems Solutions
Data Quality Open API Solutions, Automatic dataset updates,
Manual data cleansing
Data Integrity Involve government agencies
Data is not open Open API solutions, common license template
Datasets are not
linked Use OGI Tools to link datasets
Reports from data
needed
Use visualization tools and link datasets to derive
reports
The main data set which contains descriptive data about Government Vehicles consists of data that
were collected regularly using a spreadsheet template that was sent to all Government Vehicles
beneficiaries and were not extracted from an Information System. This spreadsheet template did not
follow any standards, nor had any specific guidelines, which resulted in ad hoc data entry by users.
This is the most serious problem recognized at the workshop, and different solutions were suggested
to solve this issue by participants.
The problem of data integrity was also connected to the method of data collection, as well as with
the non-existence of an Information System available to end-users, who would also contribute in
data updates. In the current situation, Government Vehicles data needs to be updated manually by
users of the Ministry of Administrative Reconstruction. The OGI project will, therefore, serve as an
opportunity to review available datasets.
The problems of data not opened and not linked to each other, as well as the lack of standardized
reports that would facilitate internal decision-making are also connected with the absence of an
Information System for Government Vehicles and shall be addressed by the OGI project.
The list of organizations in order in English is presented below:
1. Ministry of Administrative Reconstruction (MAREG) – OGI Research team;
2. Ministry of Administrative Reconstruction (MAREG) – Government Vehicles Department;
3. Ministry of Infrastructure, Transport, and Networks;
4. Hellenic Police; and,
5. Ministry of Finance – Public Property Management Directorate.
D4.6 Pilots Evaluation Results – Third Round
Page 52 of 215
4.1.1.1.2 Solutions for Pilot problems in Second Year
Some of the problems identified were solved in the second year and marked as “done.” If not yet
solved, it was planned to be deployed in the third year and were considered as “in progress.”
Table 16 - Summary of Ideas and Solutions from participants – First Year
Problems Solutions Actions Taken Status
Data Quality
Open API Solutions,
Automatic dataset
updates, Manual data
cleansing
Considering the main issue for data quality is
the creation of proper ontology and
metadata, the Greek pilot has 60% done.
For example, not all the agencies agree in
terms used to label the data, while part of
agencies uses abbreviations for names and
other use the full name.
In Progress.
Data Integrity Involve government
agencies
In the first year of the project, three datasets
were found with similar data to describe the
government vehicles and their status,
performance, etc.
In the second year, it was decided which
would be the selected data set. After that, the
data quality to find a common sense of what
are the useful data, metrics, and ontology to
be used. This issue is connected with the issue
of Data Quality.
Done.
Data is not
open
Open API solutions,
common license template
Technical partners linked datasets and later,
data cubes were created as a preparation to
opening the data.
In Progress.
Datasets are
not linked
Use OGI Tools to link
datasets
Technical partners linked datasets and later,
data cubes were created as a preparation to
opening the data.
In Progress.
Reports from
data needed
Use visualization tools and
link datasets to derive
reports
Pivot table on the top of the CubiQL API.
There is no stable version for all pilots and is
still being developed by technical partners.
The visualization will be solved using the Pivot
Table on the top of the CubiQL API.
In Progress.
D4.6 Pilots Evaluation Results – Third Round
Page 53 of 215
4.1.1.1.3 Solutions for Pilot problems in Third Year
Part of the problems identified was solved in the second year and marked as “done” or description
of the last status achieved by the pilot.
Table 17 - Summary of actions taken about Ideas and Solutions from participants – Third Year Status
Problems Solutions Actions Taken Status
Data Quality
Open API Solutions,
Automatic dataset
updates, Manual data
cleansing
Considering the main issue for data quality is
the creation of proper ontology and
metadata, the Greek pilot has 100% done.
A lot of time was spent on this task because
the data cleansing was made manually
Done in the
Third Year of
Implementation
Data Integrity Involve government
agencies
In the first year of the project, three datasets
were found with similar data to describe the
government vehicles and their status,
performance, etc.
In the second year, it was decided which
would be the selected data set. After that,
the data quality to find a common sense of
what are the useful data, metrics, and
ontology to be used. This issue is connected
with the issue of Data Quality.
Done in the
Second Year of
Implementation
Data is not
open
Open API solutions,
common license template
Technical partners linked datasets and later,
data cubes were created as a preparation to
opening the data.
Done. Open via
SPARQL
endpoint
Datasets are
not linked
Use OGI Tools to link
datasets
Technical partners linked datasets and later,
data cubes were created as a preparation to
opening the data.
Done. Linked by
default.
Reports from
data needed
Use visualization tools
and link datasets to
derive reports
An application was created using the Cube
Visualizer, producing data visualization.
Check Figure 15.
Done.
D4.6 Pilots Evaluation Results – Third Round
Page 54 of 215
4.1.1.2 Pilot Co-Creation Framework Evaluation
The Co-Creation in the Greek pilot is connected with the section 4.1.1.3 about Data Quality
Evaluation. Taking into consideration the data quality situation of Greek pilot, technical partners and
the person in charge of pilot decided to focus on increasing the data quality before creating the
application. The main reason is due to the existent of datasets from different sources inside the
government with divergent data about the same vehicles. These datasets must be compared and the
only datasets that Greek pilot MAREG had was unstructured, resulting from ad hoc data entry and
full of errors. The complete list of other issues identified in the Greek pilot was already describe
above in the section 4.1.1.1. A blog post about the Greek pilot was also created in the OGI Medium
to summarise this scenario: https://medium.com/opengovintelligence/opengovintelligence-
showcase-the-greek-pilot-90639ae3fd3d.
Further, the first step was to perform data cleansing and data comparison along with CERTH, which
resulted in a total number of structured data records that amounted to 40% of the entire fleet. The
next step demanded the validation of the existing data and data collection for the remaining 60% of
the fleet. To encounter these problems, MAREG along with CERTH, elaborated a detailed plan for
data validation with input from end-users. The plan is including the development of a web app for
data validation, where users will log in, view their Government Vehicles data and validate them, by
editing, adding or deleting vehicles. This web app was created by CERTH using a free open source
tool for the development of Linked Data Applications created by the Callimachus project
(http://callimachusproject.org/), in honor of the Greek poet and scholar at the Library of Alexandria
(https://en.wikipedia.org/wiki/Callimachus).
The next step demands the contribution of end-users, mostly employees of Decentralized
Administrations in charge of Government Vehicles, who need to log in and provide feedback on the
validity of the initial data imported in the app. The success of this step demands the imposition of
the participation of all end-users by the top management in MAREG, preferably by the Minister
herself. An existing risk that needs the attention of the Greek Pilot team is the recent change in the
organization chart of MAREG, which brought by the change in the duties of the new structure, as
well as a change of people now in charge of the management and supervision of Government
Vehicles.
The following flowchart describes the main steps of data validation, as part of the co-creation
process:
Figure 13 – Greek Pilot Co-Creation Steps
The following screenshot shows the homepage of the data validation app:
D4.6 Pilots Evaluation Results – Third Round
Page 55 of 215
Figure 14 – Greek Pilot Application Callimachus
D4.6 Pilots Evaluation Results – Third Round
Page 56 of 215
4.1.1.3 Data Quality Evaluation
Table 18 – Data Quality Survey Results – Third Year
Data Quality
Attributes Results Overview
Discoverability and
Usability
1- App has links to the original data sets;
2- App has a searchable list of datasets used;
3- App allows users to download the data sets, including a partial download of selected
variables;
4- App does not have multiple format download option;
5- App allows to user preview and interacts with the data.
Granularity 1- Not only aggregate level of data (e.g., an average of cars) but individual unit data (raw
data of cars’ specifications)
Intelligibility 1- Supporting documentation exists, but has to be found separately
Trustworthiness 1- The data sets collected are known by developers, including how it was processed, with
reliable information and double checked with external sources.
Linkable to other
data
(5 Stars LOD)
1- The data sets are available in different formats to different audiences, from XLS to CSV
and RDF;
2- The RDF is linked to other data to provide context.
15 Open Data
Principles
1- Data is complete;
2- Data is primary;
3- There are still issues about timely availability;
4- Data is accessible to the widest range of users (XLS, CSV, and RDF)
5- Data is non-discriminatory because it is freely available;
6- Data has license-free (CLS, CSV, RDF);
7- Data is safe to open;
8- Data is not 100% designed with public input (decide format and parts of data sets to
visualize and download).
D4.6 Pilots Evaluation Results – Third Round
Page 57 of 215
Table 19 – Data Quality Improvement Progress
Data Quality
Attributes
Status
1st Year 2nd Year 3rd Year
Discoverability
and Usability
The app in development,
with few functionalities.
The app developed, almost
complete and with more
functionalities.
The app is complete with all the
demanded functionalities and
datasets.
Granularity
Diverse datasets with low
quality and only raw or
aggregate level.
Task-force for data quality
unified and enriched datasets.
Ontology for all variables.
Datasets have the demanded
granularity.
Intelligibility
Low level of intelligibility,
with no context. High level
of data sensitivity.
High level of intelligibility, lower
level of data sensitivity, with
data context.
High level of intelligibility, lower
level of data sensitivity, with
data context.
Trustworthiness
Poor data quality due
incomplete, not primary
and not linked data sets.
Data was not reliable.
Increase data quality due to
linking data and enriching with
context from external sources.
Data quality can be trusted with
context from external sources.
Linkable to
other data
(5 Stars LOD)
Not following all the five
stars LOD principles.
Following all the 5 Stars LOD
principles.
Following all the 5 stars LOD
principles
15 Open Data
Principles
Not following all the 15
Open Data Principles.
Following 7 of 8 principles that
match OGI needs.
Following 7 of 8 principles that
match OGI needs.
D4.6 Pilots Evaluation Results – Third Round
Page 58 of 215
Table 20 – Data Quality Improvement Progress
Data Quality
Attributes Summary of Benefits Achieved Summary of Challenges Faced
Discoverability
and Usability
Greek pilot improves the discoverability
and usability. Data quality was improved,
and visualizations were created on the
top of the data cubes.
Combining data quality and end-users surveys,
Greek pilot can increase the number of data
sets in the cubes and include new asked
functionalities.
Granularity
Despite the 1st
year, the second year
Greek applications brings both raw and
aggregate data level of granularity.
This attribute is maxed out since both options
are present (raw data and aggregate data).
Intelligibility
Data were linked, new ontology to
normalize all data sets combined,
metadata giving context and data cube
enabling data visualization.
After linking the data, new ontology, metadata
and data cubes, Greek pilot almost maxed out
the attributes to bring intelligibility.
Trustworthiness
From datasets with poor quality in the
first year to a second year with high data
quality and reliable content.
The Greek pilot improved the data quality by
achieving 100%.
Linkable to
other data
(5 Stars LOD)
The first year had data sets in
spreadsheets (XLS and XLSX). The second-
year increase from 2 stars to 5 stars.
The Greek pilot maxed out the 5 stars (RDF)
with an API endpoint to access the data using
automate machine-readable access.
15 Open Data
Principles
The second-year achieved 7 of 8
principles that should be followed in OGI
pilot.
The pilot is not following the design with
public input (users can select format and part
of data to download).
4.1.1.3.1 DataSets and Data Cubes
Below the list of datasets used on the Greek Pilot:
1. Data describing Government Vehicles
2. Data on the lifecycle of Government Vehicles
3. Data on the operation and maintenance of Government Vehicles
4. Statistical data on Greek Public Agencies and their personnel
5. Statistical data on Greek Municipalities, Prefectures, and Regions (describing their
population, their topography, their climate, etc.)
A sample of data sets in both not linked (CSV file) and linked (RDF) are stored here:
https://drive.google.com/open?id=0B0_fFuauifo0M1V2REl2OG13QTQ.
D4.6 Pilots Evaluation Results – Third Round
Page 59 of 215
Currently, the Greek data cubes with linked data have the following dimensions for search and
visualization:
1. Vehicles Data cube
a. Area: containing municipalities in both Greek and English characters (e.g., Athens,
Thessaloniki);
b. Fuel Type: containing the eleven types of fuel (e.g., Diesel, Benzene);
c. Vehicle Type: containing the 15 types of vehicles (e.g., Bus, Car, Motorcycle)
Figure 15 – Vehicles data Cubes Screenshot
D4.6 Pilots Evaluation Results – Third Round
Page 60 of 215
4.1.1.4 Outcomes Evaluation
To evaluate the application from the perspective of end-users, a questionnaire was distributed to
the end-users. In total, 45 people were surveyed in Greek Pilot using the OGI acceptance
questionnaire to evaluate the Greek Pilot Application
(http://wapps.islab.uom.gr/CubeVisualizer/vehicles/).
For this reason, the results are interpreted more qualitatively, making use of comments made by the
respondents.
1. Linked Open Statistical Data (LOSD) Familiarity;
2. User-Acceptance evaluation;
3. Datasets Quality; and,
4. Perceived Outcomes.
D4.6 Pilots Evaluation Results – Third Round
Page 61 of 215
4.1.1.4.1 Linked Open Statistical Data (LOSD) User Familiarity
Figure 16 – Familiarity of Greek App users with LOSD
In the first part of user-questionnaire, we asked about how familiar is the respondent with open
data, linked open statistical data, open data portals and datasets provided by the Greek app. From
the responses, we can assume that the respondents are highly familiar with open data portals and
Greek app datasets, as well as with open data. However, only 52% of the respondents is familiar with
LOSD. These results indicate that LOSD is a new concept that needs to be introduced specifically to
Greek civil servants (as targeted users of Greek Pilot app), so they can use it to improve their works
and public services.
D4.6 Pilots Evaluation Results – Third Round
Page 62 of 215
4.1.1.4.2 User Acceptance
Figure 17 – User acceptance of Greek app respondents
In the second part of user-questionnaire, we addressed an user acceptance test which aims to
validate the end to end business flow of the app. There are 12 questions (adopted from acceptance
theories and OSI standard) asked to the respondents perception in using the app, ranging from the
functionalities, technical issues, to the easiness and willingness to use as shown in Figure 17.
Regarding the functionalities, 86% of the respondents said that the app has a proper and well-
integrated functionalities. 98% of the respondents expressed that the functionalities of the app is
working properly when they are using it, however, 50% of the respondents experienced a crash
during using the app. 84% of respondents stated they have a clear and understandable interaction
with the app . Next, 77% of the respondents expressed that the Greek app have a proper design.
Regarding the easiness to use the app, we got conflicting results. 92% of the respondents claimed
that it is easy to use the app, with 98% stated that they have sufficient skills to use the app.
Furthermore, 100% of the respondents claimed that using the app does not require high level of
technical knowledge. However, in the next question, only 43% of the respondents stated that the
app is not difficult to use.
Finally, 89% of the respondents stated that the app is usefulness for their work, and 92% of the
respondents claimed that their colleagues will accept the app.
In summary, most respondents stated that the Greek application is accepted by them in terms of it
has a good level of technical functionality, is easy to use, and is useful.
D4.6 Pilots Evaluation Results – Third Round
Page 63 of 215
4.1.1.4.3 Datasets Quality
Figure 18 – Perception of data quality provided by Greek app
The third part of the user-questionnaire is addressed the data quality provided by the Greek app.
We were using 4 data quality parameters in this questionnaire: accessibility, completeness, accuracy,
and in accordance with user requirements from the co-creation process.
95% and 93% of the respondents claimed the Greek app already provided a proper data accessibility
and data accuracy, respectively. 68% of the respondents stated that the Greek app already provided
data as requested by users, and 75% of the respondents is satisfy with the data quality level of Greek
app. The only drawback is in term of completeness which only stated by 32% of the respondents.
D4.6 Pilots Evaluation Results – Third Round
Page 64 of 215
4.1.1.4.4 Perceived Outcomes
Figure 18 – Perception of data quality provided by Greek app
In the last part of the questionnaire, we addressed the outcome perceived by the respondents
regarding the Greek app. The respondents react positively in all of the expected outcome of the app:
89% for the better interpretation, 93% for the better decision, 91% in the efficiency (93% in reducing
time and 100% in reducing cost), 93% in increasing transparency, 93% in helping them achieving
goals. In addition, 89% of the respondents willing to use the app, while 93% asked for more
functions to improve transparency.
By this figure, we can conclude that the Greek app is useful in introducing the concept of LOSD to
the civil servants while also showing the benefits of using the LOSD in their daily routinies. In the
case of Greek app, the results align with the aims of the pilot to improve the decision making
regarding the use of government vehicles.
D4.6 Pilots Evaluation Results – Third Round
Page 65 of 215
4.1.1.5 Screenshots of The Greek Ministry of Administrative Reconstruction (MAREG) Application
Figure 16 and Figure 17 present the screenshots of OGI ICT toolkit using data sets from the Greek
Pilot.
Figure 16 - Pie Sorted Graph Vehicle Type
Figure 17 - Bar Graph Filtered by Area (Municipality)
D4.6 Pilots Evaluation Results – Third Round
Page 66 of 215
4.1.2 Pilot 2 – Enterprise Lithuania (Lithuania)
4.1.2.1 Pilot Description and Expectation
Enterprise Lithuania used OGI ICT Toolkit to extend service provided by Lithuanian pilot. The
objective was to identify the needs of business for exploiting LOSD, developing new user-friendly
tools for businesses to help them benchmark their business ideas in the overall context of Lithuania
business, providing tools for enabling businesses to create applications using LOSD, and helping
businesses create value from LOSD. For a detailed description of Enterprise Lithuania and Lithuanian
pilot, please check report D1.1 and D4.1. A Medium blog post was also created summarizing the
advances and challenges found by Lithuanian Pilot: https://medium.com/opengovintelligence/the-
lithuanian-pilot-a-market-research-solution-3330346a7131.
4.1.2.1.1 Identified Pilot Problems in the First Year
Aiming to identify what is the pilot situation, during the first year of implementation (2016) a scan
was performed into the pilots. This scan happened in a user workshop, as part of the OGI Co-
Creation Framework step “co-initiation” and “co-design.” The user workshop had the presence of
potential stakeholders presented below:
The government of the Republic of Lithuania;
Information Society Development Committee under the Ministry of Transport and
Communications (ISDC);
Vilnius city municipality administration;
Infobalt;
Kaunas University of Technology;
Mykolas Romeris University;
E TRADE, UAB; and,
Lithuanian Free Market Institute.
First, the OGI project was presented, explaining the potential solutions that LOSD and data cubes
could bring to the pilot. Then, questions were given to the participants as potential end-users,
presented in Table 21. These questions aimed to identify four situations:
1. What was the (open) data scenario?;
a. Are they in Open data format?
b. There is any data set not yet own and need to be accessed?
c. What are the barriers in the access or use of this data?
2. What was the system scenario?
a. How can data cubes (deployed as an application) help?
b. What are the functionalities needed?
D4.6 Pilots Evaluation Results – Third Round
Page 67 of 215
Table 21 - Co-Creation User Workshop Questions
Question
number Question
1 What type of data (open data) is available to be used? What are the forms to access them?
2 What are the Data quality issues on your datasets or accessing data sets already opened?
3 What are the legal barriers to use and access the data sets?
4 What are the needs of Lithuanian Pilot users, how could this service benefit them?
5 Which functionalities, features, and datasets could be useful for pilot users?
Second, two user scenarios were defined, based in the answers to the questions in Table 3.
1. “Julie has a children’s clothes shop in Vilnius, and she is doing so well that she decides to
expand her business to another city. She needs to know where is the best place to
open another shop and where is the highest concentration of children, the city with the
highest income and where is the lowest competition. She knows that the data is available in
the Lithuanian one-stop-shop for business. She uses the LOSD tool and finds out that the best
place to expand, according to the statistical data, is Siauliai city.”
2. “John has small IT company (IT services for medium and large companies) in London, and he
wants to open a new branch in Lithuania. He needs to know where is the best place to open
branch and where is the highest concentration of young people (21-35 years old) [from data
set – “Resident population at the beginning of the year (by age groups)“], what is average
earnings [from data set – „Average earnings (monthly (2015))”] and municipality with
highest rate of foreign direct investment [from data set –“Foreign direct investment per
capita at the end of the year“]. He knows that the data is available in the Lithuanian one-
stop-shop for business. He uses the LOSD tool and finds out that the best place to expand,
according to the statistical data, is Vilnius city municipality. But John wants more information
(e.g., about education (by sectors), unemployment rate (by sector), etc. ). He uses smart
feedback tool to inform the Lithuanian one-stop-shop for business about his needs to analyze
additional criteria (new data sets). After few days Lithuanian one-stop-shop for business adds
new criteria (new data sets) and informs John by email about these improvements”.
D4.6 Pilots Evaluation Results – Third Round
Page 68 of 215
The summary of problems and solutions from participants is presented in Table 22.
Table 22 - Summary of Ideas and Solutions from participants – First Year
Problems Solutions
Data Sets are not Linked Encourage the government to open more data sets. Funds to
support initiatives must be addressed to municipalities.
Data Quality Quality of data does not reflect reality due to a list of issues as for
example lack of collection, storage, and centralized databases.
Civil Servants Capacity Building
(Skills and experience)
Civil servants are not prepared to deal with LOSD and opening-up
data sets.
Data Integrity Involve government agencies to work together for better
interoperability between databases and data sets.
Data Format (not open) Opening-up more data sets in programming formats such as
Application Programming Interface (API)
Lack of Metadata and low quality
of datasets description
Metadata is still missing. Effort on creating metadata to give
context on data sets.
Interactive Functionalities on
Online Platform (Data Vis, Table
for comparison, etc.)
Management level has no skill to develop data analytics properly
and data visualization projects. The government can create an
environment helping them to data-driven decision-making.
Reports from data needed Use visualization tools and link datasets to derive reports
The answers from participants show that Lithuanian pilot ca be considered on an initial maturity
stage on opening-up and linking datasets. The initial maturity stage is clearly influencing the needs of
stakeholders. Demands like a low number of data sets opened represent a big issue on creating
“good” quality of LOSD. Without open data, it is not possible to create LOSD and even harder to
create more interactive functionalities on a potential online platform, such as OGI ICT toolkit can
provide.
The lack of metadata influences the level of transparency and accountability. If new business people
cannot understand what or where is the data sets, they are unlikely to create data-driven decision-
making. The provided examples of user scenarios represent this type of case. People are looking for
data, but cannot find or identify the relationship between them.
4.1.2.2 Solutions for Pilot challenges in Second Year
The problems identified were solved in the second year and considered as “done,” summarized in
Table 23. If not yet solved, it was planned to be deployed in the third year and were considered as
“in progress.”
D4.6 Pilots Evaluation Results – Third Round
Page 69 of 215
Table 23 - Summary of Problems, Expected Solutions and Implementation - Second Year
Problems Expected Solutions Actions are taken Status
Data Sets are not
Linked
Encourage the government to open
more data sets. Funds to support
initiatives must be addressed to
municipalities.
Data sets were linked by technical
partners, and data cubes were
created
Done.
Data Quality
Quality of data does not reflect
reality due to a list of issues as for
example lack of collection, storage,
and centralized databases.
Data quality was improved taking
into consideration the issues
identified.
Done
Data Integrity
Involve government agencies to work
together for better interoperability
between databases and data sets.
In regards to data quality, the actions
taken are presented in section 3.2.3.
Done
Lack of Metadata
and low quality of
datasets
description
Metadata is still missing. Effort on
creating metadata to give context on
data sets.
in regards to data quality, the actions
taken are presented in section 3.2.3.
Done
Data Format (not
open)
Opening-up more data sets in
programming formats such as
Application Programming Interface
(API)
In regards to data quality, the actions
taken are presented in section 3.2.3.
Done
Civil Servants
Capacity Building
(Skills and
experience)
Civil servants are not prepared to
deal with LOSD and opening-up data
sets.
Since the pilot app has not yet stable
version, the end-users were not
trained yet. The report “D4.7
Recommendations and tutorials for
opening-up and exploiting statistical”
data will be delivered in month 33.
In
Progress
Interactive
Functionalities on
Online Platform
(Data Vis, Table for
comparison, etc.)
Management level has no skill to
develop data analytics properly and
data visualization projects. The
government can create an
environment helping them to data-
driven decision-making.
Pivot table on the top of the CubiQL
API. There is no stable version for all
pilots and is still being developed by
technical partners.
In
Progress
Reports from data
needed
Use visualization tools and link
datasets to derive reports
Also will be solved using the Pivot
Table on the top of the CubiQL.
In
Progress
D4.6 Pilots Evaluation Results – Third Round
Page 70 of 215
4.1.2.3 Solutions for Pilots’ challenges in the Third Year
Table 24 summarize the actions taken in both second and third year to complete the issues
identified by Pilots’ participants.
Table 24 - Summary of Problems, Expected Solutions and Implementation - Third Year
Problems Expected Solutions Actions are taken Status
Data Sets are not Linked
Encourage the government to open more data sets. Funds to support initiatives must be addressed to municipalities.
Data sets were linked by technical partners, and data cubes were created
Done in the Second Year.
Data Quality
Quality of data does not reflect reality due to a list of issues as for example lack of collection, storage, and centralized databases.
Data quality was improved taking into consideration the issues identified.
Done in the Second Year.
Data Integrity Involve government agencies to work together for better interoperability between databases and data sets.
In regards to data quality, the actions taken are presented in section 3.2.3.
Done in the Second Year.
Lack of Metadata and low quality of datasets description
Metadata is still missing. Effort on creating metadata to give context on data sets.
In regards to data quality, the actions taken are presented in section 3.2.3.
Done in the Second Year.
Data Format (not open)
Opening-up more data sets in programming formats such as Application Programming Interface (API)
In regards to data quality, the actions taken are presented in section 3.2.3.
Done in the Second Year.
Civil Servants Capacity Building (Skills and experience)
Civil servants are not prepared to deal with LOSD and opening-up data sets.
Created the report D4.7 Recommendations and tutorials for opening-up and exploiting statistical and specific training were given to pilot leaders and users.
Done in the Third Year.
Interactive Functionalities on Online Platform (Data Vis, Table for comparison, etc.)
Management level has no skill to develop data analytics properly and data visualization projects. The government can create an environment helping them to data-driven decision-making.
The stable version was released in the third year and has visual functionalities demanded by users. The evaluation of these functionalities is in section 4.1.2.6.
Done in the Third Year.
Reports from data needed
Use visualization tools and link datasets to derive reports
The stable version was released in the third year and has visual functionalities demanded by users. The evaluation of these functionalities is in section 4.1.2.6.
Done in the Third Year.
D4.6 Pilots Evaluation Results – Third Round
Page 71 of 215
4.1.2.4 Pilot Co-Creation Framework Evaluation
4.1.2.4.1 Co-Implementation Step
The Lithuanian pilot preferred to have a continued workshop with online participation of
stakeholders. The participants of the first workshop in the first year were invited to evaluate the
application developed in the second year via an end-user survey (section 7.1.2).
The main suggestions from stakeholders are:
1. Increase the number of data sets for a more complete search; and
2. Increase the number and improve functionalities in the application.
The suggestions will be deep described in the next sections of Data Quality Evaluation and Outcomes
Evaluation in the Second Year. For example:
“Few datasets in www.opendata.gov.lt. A too small amount of open data sets, data
formats (often only pdf (not machine-readable data format)), datasets update
frequency, lack of metadata.”
Respondent B
“Lack of metadata or low quality of metadata are one of biggest problems to reuse
open data for business or public purposes. Another huge problem is the lack of
open data, too old data, and too much-aggregated data.”
Respondent D
“The Organisation X suggests adding more functionality to allow users to inform
our institutions and suggest solutions based on open data.”
Respondent F
“The Organisation Y needs more analytical functions of our pilot: possibility to
compare the same indicator at different regions, maybe to show aggregate
indicators, to rank regions by selected indicators, etc. The Organisation Y performs
ranking of regions and wants an online tool like is planned in the pilot.”
Respondent H
D4.6 Pilots Evaluation Results – Third Round
Page 72 of 215
4.1.2.5 Data Quality Evaluation
In the third year, a survey was conducted on technical partners to identify the stage data quality
Table 25 – Data Quality Survey Results – Third Year
Data Quality
Attributes Results Overview
Discoverability and
Usability
1- App has no links to the original data sets;
2- App has a searchable list of datasets used;
3- There is no option for users download the data sets;
4- App does not have multiple format download option;
5- App does not allow to user preview and interacts with the data.
Granularity 1- Not only aggregate level of data (e.g., an average of cars) but individual unit data (raw
data).
Intelligibility 1- There is no supporting documentation exists.
Trustworthiness
1- The data sets collected are not well known by developers, including how it was
processed. However, data is trusted and considered reliable, with minimum consistence
with external sources.
Linkable to other
data
(5 Stars LOD)
1- The data sets are available in different formats to different audiences, from XLS to CSV
and RDF;
2- The RDF is linked to other data to provide context.
15 Open Data
Principles
1- Data is complete;
2- Data is primary;
3- There are still issues about timely availability;
4- Data is accessible to the widest range of users (XLS, CSV, and RDF)
5- Data is non-discriminatory because it is freely available;
6- Data has license-free (CLS, CSV, RDF);
7- Data is safe to open;
8- Data is not 100% designed with public input (decide format and parts of data sets to
visualize and download).
D4.6 Pilots Evaluation Results – Third Round
Page 73 of 215
Table 26 – Data Quality Improvement Progress
Data Quality
Attributes
Status
1st Year 2nd Year 3rd Year
Discoverability
and Usability
The app in development,
with few functionalities
and few data sets.
App developed, still being
updated and without expected
functionalities asked by users.
Updated. App has all
functionalities demanded by
users.
Granularity
Diverse Datasets with low
quality and only aggregate
level.
A task-force for data quality to
unify and enrich data sets is
needed.
Updated. App has the
demanded datasets with the
initial expected granularity.
Intelligibility
Low level of intelligibility,
with no context. Low level
of data sensitivity.
Still same scenario with Low
level of intelligibility without
proper data context.
Updated. Minimum data
context with external sources,
however, without
documentation.
Trustworthiness
Poor data quality due
incomplete, not primary
and not linked data sets.
Data was not 100%
reliable.
Current scenario with some
improvements. Increase data
quality with data cubes linking
data sources however without
proper context from external
sources.
Updated. Data sources are
linked with external sources.
Linkable to
other data
(5 Stars LOD)
Not following all the 5
stars LOD principles.
Following all the 5 Stars LOD
principles.
Updated. Following the 5 stars
LOD Principles.
15 Open Data
Principles
Not following all the 15
Open Data Principles.
Following 6 of 8 OD principles
that match OGI needs.
Updated. Following 6 of 8 OD
principles that match OGI
needs.
D4.6 Pilots Evaluation Results – Third Round
Page 74 of 215
Table 27 – Data Quality Improvement Progress – Third Year
Data Quality
Attributes Summary of Benefits Achieved Summary of Challenges Faced
Discoverability
and Usability
Lithuanian pilot improved the
discoverability and usability. Data quality
was improved, and visualizations were
created on the top of the data cubes.
New 8 datasets and functionalities
demanded by end-users survey in the 2nd
Year.
Granularity There is still the same level of granularity. Preference for aggegate level (statistics),
however, unit level of data is also presented.
Intelligibility Data were linked and data cube enabling
data visualization.
Data is linked and new functionalities
provides new types of visualization.
Trustworthiness
A still similar scenario of poor and not
reliable data sets. The focus of 2nd
year was
given in other attributes.
Data is more reliable in comparison with the
2nd
year, but there is still room to improve
data quality.
Linkable to
other data
(5 Stars LOD)
The first year had data sets in spreadsheets
(XLS and XLSX). The second-year increase
from 2 stars to 5 stars.
The Lithuanian pilot maxed out the 5 stars
(RDF) with an API endpoint to access the
data using automate machine-readable
access.
15 Open Data
Principles
The second-year achieved 6 of 8 principles
that should be followed in OGI pilot.
The pilot is not following the design with
public input (users can select format and
part of data to download).
D4.6 Pilots Evaluation Results – Third Round
Page 75 of 215
4.1.2.5.1 Data Sets
Below the current list of data sets used on the Lithuanian Pilot:
1. Resident population at the beginning of the year (by age groups);
2. Average earnings (monthly (2015));
3. State Food and Veterinary Service;
4. State Social Insurance – Sodra;
5. Investment in tangible fixed assets at current prices (2015);
6. Economic entities in operation at the beginning of the year;
7. Resident population at the beginning of the year (by age groups);
8. Resident population at the beginning of the year (by gender);
9. Resident population at the beginning of the year (persons by municipality); and,
10. Foreign direct investment per capita at the end of the year (2014).
More datasets can be opened and linked taking into consideration the future needs of Lithuanian
pilot. The sample of data sets are here:
https://drive.google.com/open?id=0B0_fFuauifo0NmIzZFdqeTVDWjg
D4.6 Pilots Evaluation Results – Third Round
Page 76 of 215
4.1.2.6 Outcomes Evaluation
To evaluate the application from the perspective of end-users, a questionnaire was distributed to
the end users. Several different end-users were interviewed directly, using the questionnaire. The
results from these interviews are displayed below. A questionnaire was distributed to end-users of
the Lithuanian pilot. This questionnaire was filled in by 43 users, which does not allow for rigorous
quantitative analysis taking into consideration these two Apps:
a. http://vmogi03.deri.ie:8080/superset/dashboard/5/
b. http://vmogi03.deri.ie:8080/superset/dashboard/7/
For this reason, the results are interpreted more qualitatively, making use of comments made by the
respondents.
1. Linked Open Statistical Data (LOSD) Familiarity;
2. User-Acceptance evaluation;
3. Datasets Quality; and,
4. Perceived Outcomes.
D4.6 Pilots Evaluation Results – Third Round
Page 77 of 215
4.1.2.6.1 Linked Open Statistical Data (LOSD) User Familiarity
Surprisingly, the level of familiarity with Open Data and LOSD of Pilot App users was low.
Only 2% were Familiar with LOSD, 16% Familiar with Open Data, 23% Familiar with LOSD portals and
33% Familiar with the Datasets presented by the App (all presented in blue part of the stack). Figure
18 summarizes the results.
Figure 18 – LOSD User Familiarity – Third Year
D4.6 Pilots Evaluation Results – Third Round
Page 78 of 215
4.1.2.6.2 User Acceptance
In contrast with the familiarity with Open Data and LOSD, the Pilot App Acceptance is massively
positive. Major part pointed the functions have been working properly, few people experiencing a
crash during the use, the app interaction is acceptable, useful, and, easy to use. The people surveyed
also believe to have sufficient skills to use the app creates, your colleagues will accept the app, being
not difficult to use and having functions integrated.
However, a major part of the people understands the app still have a high level of technical
knowledge to use the app (100%), missing some functionalities (49%).
Figure 19 – User Acceptance – Third Year
D4.6 Pilots Evaluation Results – Third Round
Page 79 of 215
4.1.2.6.3 Datasets Quality
The surveyed people had a positive reaction from the data set quality, with an overall of 74%
satisfaction (data quality satisfaction). The data accessibility, accuracy, and reduced time to find the
data searched.
However, by 33% of the users, not all the datasets required were present in the app. Beyond that,
35% of respondents believe the data is incomplete.
Figure 20 – Datasets Acceptance – Third Year
D4.6 Pilots Evaluation Results – Third Round
Page 80 of 215
4.1.2.6.4 Perceived Outcomes
A major part of the dimensions surveyed about expected results had positive reactions. Al the
respondents accept the app. The main reasons to this positive reactions are associated with the
positive reactions in other dimensions such as 98% of users believing the app brought the better
interpretation of data and reduced time to look for the information they need. Further that, the app
helped 95% of surveyed people in increasing efficiency and made better decisions.
However, while 93% of them pointed an increase of transparency, users believe that more functions
can create even more transparency. This dimension can be exploited in the future with a new
qualitative search and a new round of co-creation within the users. Probably, after some rounds of
co-creation and evaluation, the expectation of users increased since the first year of the project, in
2016.
Figure 21 – Results – Third Year
D4.6 Pilots Evaluation Results – Third Round
Page 81 of 215
4.1.2.7 Screenshots of Lithuania Enterprise Application
The next figures present the screenshots of OGI ICT toolkit using data sets from the Pilot.
Figure 22 - Table with bar chart functionality
Figure 23 - Table with heatmap functionality
D4.6 Pilots Evaluation Results – Third Round
Page 82 of 215
4.1.3 Pilot 3 – Tallinn Real Estate (Estonia)
4.1.3.1 Pilot Description and Expectation
The Estonian Ministry of Economics will make use of the OGI ICT Toolkit to address issues in the
Estonian real estate market such as timely publication of data and information asymmetry. To best
identify the barriers facing transparency in the Estonian real estate sector interviews have been
carried out with all stakeholders from the private sector, the public sector, and the Estonian public at
large. In combination with these interviews, workshops have been conducted to gain a better
understanding of the current needs of those with interest in the real estate sector.
For a detailed description of Enterprise Lithuania and Lithuanian pilot, please check report D1.1 and
D4.1.
4.1.3.1.1 Identified Pilot Problems in the First Year
The purpose of the user workshop in Tallinn, Estonia, was defined by the following four points:
1. To get end-user feedback
2. To improve the initial offering of the public service
3. Involve stakeholders with the design of the service
4. Raise awareness of the Estonian Real Estate Pilot Program
Once the initial goals of the Estonian Real Estate Pilot Programs were discussed, a first of our two
sessions were performed. In the first session which was titled “Developing the Estonian Real Estate
Pilot Program” we focused on the following questions:
What problems do you see with this pilot program and its goals?
o What are the main areas in the real estate sector that you see issues?
What are the potential solutions for these problems?
o How could this pilot program better address these issues?
In what ways would this new public service be beneficial for Estonia?
All participants were given time to think carefully on these questions and write down their answers.
After the ideation process, a group discussion was held to elaborate each participants’ answers.
D4.6 Pilots Evaluation Results – Third Round
Page 83 of 215
The second session was titled “Constructing the Functionality of the Estonian Real Estate Pilot
Program.” For this session we wanted to focus on gathering participant created personas and user
stories. As we are following an agile development approach for the pilot program creation, we made
sure to co-generate user stories, and personas with the workshop attendants as these are key parts
of agile development. They were shown the following slide information on a slide:
After that, this task was addressed:
For this session, we split the participants into pairs and had each pair generate up to three personas.
After they had time to prepare their personas and user stories, these were discussed as a group.
What we had hoped to do with this user workshop was to get individuals from many different
agencies in the same room talking about Linked Open Statistical Data in Estonia, and in this end, we
succeeded. We had many participants from 7 public sector organizations and 2 private sector
companies.
User Stories As an X I want to do Y.
As a <type of user>, I want <some goal> so that <some reason>.
Personas
Demographics - who is this person?
Needs - what are their needs, how could this service benefit them?
Example: Priit is a 35 year old Estonian who has recently moved to the United Kingdom for work. He speaks English quite well and has a bachelor's degree in computer science. Since he just arrived he is having issues when it comes to finding a career which is related to his degree.
Why do these matter?
1. Prepare at least one persona:
Demographic Information
i. Who is this person? How old are they? What is their occupation?
What do they do?
ii. Needs
a. (What sort of reasons would this person have for using our
public service)
2. Using the previously generated persona, please try to write down 10 to 15 user stories
for this person.
Reminder: As a <type of user>, I want <some goal> so that <some reason>.
D4.6 Pilots Evaluation Results – Third Round
Page 84 of 215
Going into the workshop, we had hoped to receive information about:
Potential issues with the pilot
Potential solutions
User Stories
User Personas
Other organically generated information.
In the first session, we received the following information about potential problems and solutions
our pilot would face. They have been ordered by a number of occurrences/mentions going from the
most commonly mentioned problem/solution to the least commonly mentioned.
Table 28 - Summary of Ideas and Solutions from participants
Problems identified
by Users Expected Solutions from Users
Data Quality Open API Solutions, Automatic dataset updates
Confidentiality
issues Anonymize the data
Competition from
existing real estate
portal
Involve users in the design process, talk with
current users of existing portals
Data Integrity Update national registries involve
governmental agencies
Data is not open Open API solutions, common license template
Datasets are not
linked Use OGI ICT Toolkit to link data sets
Needed data not
collected
Get users to provide data.
The biggest issues which participants perceived matched what previously identified by pilots’
partners on initial research. In Estonia, the data quality is quite poor for many data sets. If the data
exist, it may not even be useable due to the “confidential” nature of such data.
When looking at the solutions, mostly they are out of our hands and require government
intervention or policy changes. The problems and solutions which we can best address are
D4.6 Pilots Evaluation Results – Third Round
Page 85 of 215
competition from other real estate portals, needed data not being collected, and, to some extent,
confidentiality issues.
When discussing the benefits that the Estonian Real Estate portal could have for Estonia, the main
five benefits were:
1. Increased transparency in the real estate sector;
2. Fairer pricing-model;
3. Happier citizens;
4. One stop shops for real estate data; and,
5. Increased availability and usage of real estate information.
In the second session, we had asked participants to generate user personas for potential users of the
Estonian Real Estate Portal. A list of the basic generated personas in order of discussion is below.
As a foreign IT specialist coming to Tallinn, I need a safe, environmentally friendly
place to live; information on recycling and public transportation, so that I could live
in a clean environment, recycle, move easily in the city.
As a family with 5 and 8-year old kids, I want to buy a house outside of Tallinn, we
need low living costs, information on schools and kindergartens, transportation
information, info on shops, sports and entertainment facilities, green areas and
parks so that I wouldn’t have to walk more than 20 minutes to school and
kindergarten
As a student moving to Tallinn, I need information on the location of universities,
public transportation, if it’s a tolerant & safe neighborhood, cheap living costs,
proximity of entertainment facilities; or using central heating.
Elderly Finnish couple, 65+, sold apartment in Helsinki, moved to Tallinn to spend
their golden years, looking for a 2-3-room apartment in very good condition with
everything necessary in walking distance or close by public transportation. Need
information on expenses related to the apartment (communication costs, mortgage,
loans taken by the house union), is there an elevator in the house, type of house,
location (safe, important PoI-s nearby)
Male, 35-years old, coming from South Estonia to find a job in Tallinn. Wants to
move his wife and 2 children to Tallinn. Looking to rent a 2-room apartment and has
a price limit, needs public transportation nearby and a municipal school in his area.
Mikk, a young graduate with a wife and baby on the way, looking for a family place
to move to, wants the portal to suggest areas that meet his requirements, and once
he finds the area, he wants to have more information about prices, quality of the
building, etc.
D4.6 Pilots Evaluation Results – Third Round
Page 86 of 215
A guy got married, wants to sell his apartment and move to a new one. He wants to
sell his apartment and get the highest possible price, wants to promote his area and
to check out what advantages he can mention in his selling advertisement.
A real estate company wants to develop their service of selling accommodation.
They would like to make personalized offers based on user preferences, so the
portal could offer them property in the areas that interest them.
As an exchange student coming to Estonia for 3-4 months, I want to rent an
apartment or a room in a shared apartment, need the apartment to be furnished,
close to the city center and/or university, an English-speaking landlord, sports
facilities nearby, affordable meals close-by, information about pollution, traffic, etc.
She’s 45 disabled and in a wheelchair, a heavy smoker, working as an office clerk,
wants to buy an apartment, needs public transport, handicap accessible, close to
work and hospital, and if there are any public smoking places, and clubs for people
with similar situations.
The 25-year old heterosexual couple wants to buy their apartment. No children yet
but maybe soon so they need a two-room apartment. Interested in police
complaints, noise in the area, energy efficiency level of the building. Need elevator,
good view from their window (no high buildings close to their windows), have
electric cars which they need to charge and if there are any vegetarian restaurants
nearby.
When looking at these personas and what needs they had, the following datasets stood out as the
most important:
1. Transport
2. Safety
3. Price of real estate
4. Environment
5. Points of Interest
6. Information on the property
The last point of discussion will be how this information benefits our pilot program and how we can
use it for the design of the pilot. What we now know is the main problems/solutions, who may use
our service, the benefits people expect, and what datasets we should use for our initial pilot
program. Regarding future research, we need to address the issues of data confidentiality as well as
finding out how/why people use other real estate portals.
Regarding initial pilot construction, we have outlined the following for our 1st iteration.
D4.6 Pilots Evaluation Results – Third Round
Page 87 of 215
Initial target groups:
1. Foreign students
2. Estonians new to Tallinn
Initial datasets:
1. Transport information
2. Safety statistics
3. Price statistics
4. Points of interest data (schools, doctors’ offices, etc.)
5. Information on the property (age, utilities, amenities, etc.)
Initial functionalities
1. As a user, I want to be able to search by address so that I can see the location on a map
2. As a user, I want to be able to see how safe my home is so that I can feel safe when moving
to a new location
3. As a user, I want to be able to measure the distance between my house and different points
of interest so that I can travel more efficiently
4. As a user, I want to be able to find addresses where I could live based on my criteria (like
price and safety) so that I don’t waste time searching for real estate that I don’t wish to see
5. As a user, I want to be able to find out how often public transport comes by my address so
that I can understand how much time I will have to spend commuting via public transport
In the workshop participants came up with ideas individually, and then these were discussed as a
group. The ideas put forth by every applicant were collected, and all feedback was recorded. We
believe that having individuals from many different agencies gives strength to the proposed initial
direction of the pilot program. Expected barriers are known now, giving the pilot a better
understanding of the solutions to overcome these barriers. In addition, a better understanding of
who may be the users of new Estonian Real Estate portal and how they will want to use it are
required.
It is important to highlight the importance of co-creation process, including end users in the process
of OGI ICT Toolkit development and dividing responsibility about the definition of pilots’ goals. As
noted, objectives changed from the initial common target users to international students and new
Estonians moving to Tallinn, as an example. It means a need to have further user workshops where
include more of the end users and trial our initial prototype service on them to get even further
input.
D4.6 Pilots Evaluation Results – Third Round
Page 88 of 215
Below, the List of organizations that participated on the user workshop in Tallinn (Estonia). All names
are translated in English:
1. European Commission
2. Mooncascade
3. Ministry of Economic Affairs and Communications
4. Tallinn University of Technology
5. Finance Ministry
6. Teleport
7. Land Board
8. Estonian Statistics Board
9. Tallinn City Government
10. National Registers and Information Systems Center
11. Land board
The link to a presentation given to participants is below:
https://docs.google.com/presentation/d/1r55VkEJxT5ECyYnJleCsHqEwjm55bg5OT8-
bPHPlVH0/edit?usp=sharing
D4.6 Pilots Evaluation Results – Third Round
Page 89 of 215
4.1.3.2 Solutions for Pilot problems in Second Year
The problems identified were solved in the second year and considered as “done.” If not yet solved,
it was planned to be deployed in the third year and were considered as “in progress.”
Table 29 - Summary of Ideas and Solutions from participants
Problems Solutions Actions Taken Status
Data Quality
Open API Solutions,
Automatic dataset
updates
The OGI CubiQL API was created in the second
year and can be as a solution for this issue. In Progress
Confidentiality
issues Anonymize the data The data set is completely anonymized. Done
Competition
from existing
real estate
portal
Involve users in the
design process, talk
with current users of
existing portals
A hackathon was hosted and attracted
students and developers. There is an
opportunity for new events with the same
characteristic to improve the user-friendly and
data quality.
Done
Data Integrity
Update national
registries involve
governmental agencies
Data sets are still being updated and enriched.
The third year will finish the data quality task-
force.
In Progress
Data is not
open
Open API solutions,
common license
template
Data sets are linked and in open format. Done
Datasets are
not linked
Use OGI ICT Toolkit to
link data sets
Data sets are linked however there is no data
cube. Potentially, in the third year can have
time to use data cubes and CubiQL API.
Done
Needed data
not collected
Get users to provide
data.
Some data there is no access yet due to
organizational issues. In Progress
D4.6 Pilots Evaluation Results – Third Round
Page 90 of 215
4.1.3.3 Pilot Co-Creation Framework Evaluation
The user workshop done in the first year decided to use five data sets in the initial prototype of
Tallinn Real Estate Application. The selected datasets were:
Public Transportation (buses stops and lines, traffic accidents in Tallinn);
Crime (type and position of crimes in Tallinn);
Real Estate pricing (average price to buy and rent houses in Tallinn);
Schools (positions and names of schools in Tallinn); and,
Property data (property information in Tallinn).
Because of some data quality issues, described in section 4.1.3.4, the five selected data sets required
a round of cleansing to make them proper to the Tallinn Real Estate Application. The initial cleansing
was conducted by staff members at Technical Tallinn University (https://www.tlu.ee/en). Further,
part of these data sets was improved and updated in a progressing hackathon sponsored by the
Ministry of Economic Affairs and Communications (MKM) (https://www.mkm.ee/en), the
organization in charge of this pilot.
Since the data sets were cleansed, the pilot started to link the data sets, create the data cubes and
web site. For this, a team was formed with members working in the Estonian pilot and members of
the big data science team from the private sector company Portal (https://nortal.com/).
After 48 hours, the initial porotype for the Tallinn Real Estate Pilot was presented to the audience. It
was awarded an honorary mention for providing valuable location-based information. From this
hackathon, goals were achieved by participating in this hackathon:
1. Open Government Data (OGD) can be used to create new services;
2. If OGD is freely available to the society, an initial prototype created in few days with relevant
public value to society;
3. Hackathons are one of the strategies that governments and private organizations can use to
aware people and developers; and
4. If reunited, individuals from different sectors can come together and use their free time to
achieve goals in a short time.
After the hackathon, with a prototype created, between 15 and 18 March 2017, TTU and MKM
started to develop a fairly simple and easy to understand user interface for the pilot project
(https://rnd-tut.shinyapps.io/Estonian_Pilot/).
The use of GitHub and developing the service in an agile and open source using the R package
helped the success in developing together with stakeholders (co-implementation and co-evaluation)
of the pilot program in three ways:
D4.6 Pilots Evaluation Results – Third Round
Page 91 of 215
1. GitHub allows Pilot developers to easily identify issues and share the original Pilot
Application code, asking external developers to help to solve the identified issues. Then,
external developers can view, copy the code with one click (fork) and work in their
environment without any prejudice for the original code created by the Pilot developers. If
external developers solve an issue, they can ask to merge the code into the source published
by a pilot, who approves or not the new solution.
2. GitHub is flexible and free, allowing anyone to creates new functionalities and not just
solve others’ issues. Creativity from external developers can lead to new insights and new
applications; and,
3. Since GitHub has free access, the code published by Tallinn Real Estate Pilot Application is
exportable. The code and functionalities created can be used by any other city, country, or
town in the world.
The pilot was also described in the blog post at OGI Medium account:
https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-estonian-pilot-
511fb4647e8 .
Figure 24 – Audience of Tallinn Real Estate Hackathon
D4.6 Pilots Evaluation Results – Third Round
Page 92 of 215
Figure 25 – Developers coding for the Tallinn Real Estate Hackathon
D4.6 Pilots Evaluation Results – Third Round
Page 93 of 215
4.1.3.4 Data Quality Evaluation
Table 30 – Data Quality Survey Results
Data Quality
Attributes Results Overview
Discoverability and
Usability
1- App has links to the original data sets;
2- App has a searchable list of data sets used;
3- App allows users to download the data sets, including a partial download of selected
variables;
4- App does not have multiple format download option;
5- App allows to user preview and interacts with the data.
Granularity 1- Only raw (individual) level of data is available.
Intelligibility 1- Supporting documentation can be found at the same time as the data (e.g., right next to
the link for the data). The metadata was created by a Pilot task-force.
Trustworthiness 1- The data sets collected are well-known by developers, including how it was processed;
2- The data is 100% reliable and is consistent with external sources.
Linkable to other
data
(5 Stars LOD)
1- The data sets are available in different formats to different audiences, from PDF, XLS to
CSV. RDF was not yet created by OGI developers in this pilot;
2- The RDF is not yet properly linked to other external data to provide context.
15 Open Data
Principles
1- Data is complete;
2- Data is primary;
3- There are issues about timely availability;
4- Data is accessible to the widest range of users (XLS, CSV, and static visualization)
5- Data is non-discriminatory because it is freely available;
6- Data has license-free (CLS, CSV, static visualization);
7- Data is safe to open
8- Data is partially permanent;
9- Data is not 100% designed with public input (decide format and parts of data sets to
visualize and download).
D4.6 Pilots Evaluation Results – Third Round
Page 94 of 215
Table 31 – Data Quality Improvement Progress
Data Quality
Attributes
Status
1st Year 2nd Year
Discoverability
and Usability
The app in development, with few
functionalities and few data sets.
The app is still under development, still being
updated and some expected functionalities
asked by users not yet implemented.
Granularity Diverse Datasets with room to improve
data quality and only raw level.
A task-force for data quality to unify and enrich
data sets is needed (Create data cube and RDF).
Intelligibility Low level of intelligibility, with no context.
Low level of data sensitivity.
An increased level of intelligibility however still
without proper data context.
Trustworthiness
Poor data quality due incomplete, not
primary and not linked data sets. Few
datasets published.
Current scenario with some improvements.
Increase data quality with however without
proper context from external sources.
Linkable to other
data
(5 Stars LOD)
Not following all the 5 stars LOD principles.
Some data sets were not published and
accessed via a static web site.
Not following all the 5 Stars LOD principles, still
missing the RDF.
15 Open Data
Principles
Not following all the 15 Open Data
Principles.
Following 6 of 9 OD principles that match OGI
pilot needs.
D4.6 Pilots Evaluation Results – Third Round
Page 95 of 215
Table 32 – Data Quality Improvement Progress
Data Quality
Attributes Summary of Benefits Achieved Summary of Challenges Faced
Discoverability
and Usability
Estonian pilot improved the
discoverability and usability. Data
quality was improved, and map
visualization was created in the web.
Combining data quality and end-users’ surveys,
Estonian pilot can increase the number of data sets
and include new asked functionalities by
stakeholders that tried the app.
Granularity There is still the same level of
granularity (raw data).
Estonian pilot can consider finding new data sets
that provide aggregate level but keeping the raw
data available (primary). It helps to reach a wide
audience and create new types of visualization.
Intelligibility
Data visualization was created, but
without using the linked datasets
(RDF) or data cube.
There is still room to improve Estonian pilot taking
into consideration co-creation participation from
stakeholders.
Trustworthiness
Data sets are reliable, from official
open data portals. The focus of 2nd
year was given in other attributes.
There is still room to improve the reliability of
Estonian data sets and external connections.
Linkable to
other data
(5 Stars LOD)
The first year had data sets in
spreadsheets (XLS and XLSX). The
second-year increase from 2 stars to
4 stars (RDF) and map visualization.
The Estonian pilot can improve it linking the data,
creating SPARQL endpoint (RDF) to reach the 5 LOD
stars.
15 Open Data
Principles
The second-year achieved 7 of 9
principles that should be followed in
this OGI pilot.
The pilot is not yet following the design with public
input (users can select format and part of data to
download) and only has an aggregate level (not
primary data).
D4.6 Pilots Evaluation Results – Third Round
Page 96 of 215
4.1.3.4.1 Data Sets
Public Transportation (buses stops and lines, traffic accidents in Tallinn);
Crime (type and position of crimes in Tallinn);
Real Estate pricing (average price to buy and rent houses in Tallinn);
Schools (positions and names of schools in Tallinn); and,
Property data (property information in Tallinn).
The data sets are already in RDF (LOD format). The link to access datasets:
https://drive.google.com/drive/folders/0B89NqGUhD5HxRGQ0ZXNwMU9vNkE
D4.6 Pilots Evaluation Results – Third Round
Page 97 of 215
4.1.3.5 Outcomes Evaluation
To evaluate the application from the perspective of end-users, a questionnaire was distributed to
the end users. Seventeen people responded to this survey for the Tallinn Real Estate pilot. The
results are displayed below. Since only seventeen people responded, the results are more qualitative
than quantitative as no statistical tests can be run on such a sample size. The attributes collected
and evaluated were:
1. Linked Open Statistical Data (LOSD) Familiarity;
2. User -acceptance evaluation;
3. Datasets Quality; and,
4. Perceived Outcomes.
D4.6 Pilots Evaluation Results – Third Round
Page 98 of 215
4.1.3.5.1 Linked Open Statistical Data (LOSD) Familiarity
Figure 26 Tallinn app respondents’ familiarity with LOSD
From Figure 26, we can assume that respondents of Tallinn app familiar with open data and open
data portal, however, less familiar with LOSD and Tallinn app datasets. Lack of familiarity with Tallinn
app datasets is understandable since most of the users of the Tallinn app is citizens, or international
visitor.
This results also shows that there is an opportunity to introduce LOSD to wider and varied public
group and use the LOSD to improve public services.
D4.6 Pilots Evaluation Results – Third Round
Page 99 of 215
4.1.3.5.2 User Acceptance
Figure 27 Tallinn app respondents user acceptance
In the second section of the questionnaire, we addressed respondents’ perception regarding user
acceptance of the app. Majority respondents agreed that the Tallinn app already provided a well-
integrated technical functionality and a proper design. 80% of the respondents also stated that the
functionalities of the app were working properly when they were using it and only 30% of the
respondents experienced with crash during using the app.
Majority of the respondents are also claimed that they have a clear and understandable interaction
using the app and they found the app very useful.
The percentage of respondents which perceive that the app is easy, does not require high level
technical knowledge and think that they have sufficient skills to use the app are even higher than
previous parameters. However, respondents that said the app is not difficult to use is only slightly
higher than respondents who think it is difficult. Last, 95% of respondents expressed that their
colleagues will also accept the Tallinn app.
To sum up, Tallinn app is well-accepted by majority of the respondents.
D4.6 Pilots Evaluation Results – Third Round
Page 100 of 215
4.1.3.5.3 Datasets Quality
Figure 28 Tallinn app datasets quality results
In regard to datasets quality, apart from completeness, majority of the respondents stated that
Tallinn app already provide a good datasets quality. 100% for data accuracy, 95% for time to find the
data, 80% for data accessibility and 70% for meeting the user requirements from the co-creation
process. Even though 85% of the respondents required improvement in data completeness, 80% of
the respondents already satisfied with the level of datasets quality provided by Tallinn app.
D4.6 Pilots Evaluation Results – Third Round
Page 101 of 215
4.1.3.5.4 Perceived Outcomes
Figure 29 Tallinn app perceived outcome
In the expected outcome part, the responses of questionnaire which disseminated randomly to
Tallinn app users are showing encouraging results. More than 80% of respondents claimed that the
Tallinn app bring benefits in term of better interpretation of data, improve decision making, improve
efficiency (time and cost), and increase transparency. As 85% of respondents claimed that the app
already helped them in achieving goals, majority of the respondents perceived that the app does not
require additional functions. Last, 70% of the respondents personally accept the app.
D4.6 Pilots Evaluation Results – Third Round
Page 102 of 215
4.1.3.6 Screenshots of Tallinn Real Estate Application
The next figure presents the screenshot of Estonian pilot application.
Figure 30 – Tallinn Real Estate Application
Figure 31 - Bar chart of Average Cost
D4.6 Pilots Evaluation Results – Third Round
Page 103 of 215
Figure 32 - Pie chart Distribution of crashes
D4.6 Pilots Evaluation Results – Third Round
Page 104 of 215
4.1.4 Pilot 4 – Trafford Council Worklessness (England)
4.1.4.1 Pilot Description and Expectation
Trafford’s Innovation and Intelligence Lab are working on using linked open statistical data to help
support decision making relating to worklessness. They are working closely with Swirrl, who are
handling the more technical aspects of modelling and storing the linked data. The goal is to build a
tool that will bring together data from a range of sources to help understand the factors that
contribute to, or are impacted by, worklessness. Representatives from the Department for Work and
Pensions; Trafford’s Economic Growth Team and the Greater Manchester Combined Authority are
also involved. For a detailed description of Enterprise Lithuania and Lithuanian pilot, please check
report D1.1 and D4.1.
4.1.4.1.1 Identified Pilot Problems in the First Year
Trafford Council provides services to around 226,000 people. It is part of Greater Manchester and
works actively with other neighbouring councils to share ideas, innovations and in some cases
services. A priority for Trafford is economic growth: to support businesses, to create jobs and to
tackle unemployment and the social challenges that brings.
Trafford is a leader in the UK in bringing digital innovations into its working processes. It was
identified by the UK government as one of the ‘Open Data Champions16’: a group of local
government organisations within the UK doing the most to promote and exploit open data. Trafford
Council has established an ‘Innovation and Intelligence Laboratory’17 to bring together data and
information specialists from many organisations who work on challenging problems for the public
sector in Trafford. The Innovation Lab has four priorities: Mental Health, Aging Population,
Unhealthy Weight and Worklessness.
The purpose of the workshop was to:
1. Raise awareness of the Trafford OpenGovIntelligence project;
2. Understand the current service provision and delivery environment;
3. Identify opportunities and challenges for future service delivery;
4. Engage stakeholders and end users in developing the project, including setting the project
objectives; and,
5. Develop a single multi-agency project delivery plan with key milestones, roles and
responsibilities.
D4.6 Pilots Evaluation Results – Third Round
Page 105 of 215
One scenario was selected to be part of OGI Trafford pilot. It will concentrate on worklessness,
within these priority areas, the objectives are to:
help to reduce demand on services;
help to redesign services;
improve people’s awareness and understanding of the area; and,
help to attract or retain investment.
Trafford is responsible for a varied area, incorporating both deprived inner city communities, rich
commuter-belt communities, Trafford Park – the largest industrial estate in Europe employing
around 35,000 people in 1400 companies, (and also ‘Old Trafford’, the home of Manchester United
football club). The pilot will focus on innovate generation and application of data from a range of
sources to tackle the problems of worklessness:
measuring and attempting to match demand for and supply of skills, gathering data from
job-seeking individuals and from businesses;
seeking to use data and digital technology to find new approaches to assisting workless
people; and,
profiling the economy, skills base and assets of the area to identify potential improvements,
and to help attract new companies to invest in the area.
The pilot will include investigation of new methods of data collection from local businesses and
citizens. The pilot will both generate requirements for and evaluate and exploit tools to be
developed in OGI for combining, analysing and visualizing statistical data. Trafford is the lead council
for the topic of worklessness in the Association of Greater Manchester Authorities, so innovations
arising from the project will have their impact enhanced by a direct route to replication in other
areas. The follow questions were made to the audience:
D4.6 Pilots Evaluation Results – Third Round
Page 106 of 215
Table 33 - Co-Creation User Workshop Questions
Question
number Question
1 Can you tell us a bit about your role in the project?
2 What do you feel are the key aspects of your pilot in the project?
3 Which audience do you want to reach with your pilot?
4 How are you going to reach your target audience for the pilot?
5 How are you going to ensure user engagement?
6 How are you going to publicise it?
7 What do you feel are the main advantages of using multidimensional statistical data?
D4.6 Pilots Evaluation Results – Third Round
Page 107 of 215
The summary of problems and solutions from participants is presented on the table below.
Table 34 - Summary of Ideas and Solutions from participants
Problems Solutions
Not live data in reports Interactive tool feeding in the most up-to-date information
Multiple reports required Window view with wide selection of data behind it enabling the
user to select multiple data sets
Spatial misalignment Long-term – the success of the project could influence change in
how data is collated
Accessing ‘other’ data There will be multiple data sets available across a range of
themes (work, housing, health, skills etc.)
Knowing who to go to for ‘other’
data
By acting as a first-point for all data, people will be able to build
relationships across agencies and sectors as they use each other’s
data
Demand on InfoTrafford team By making data more accessible and available, this will upskill
people to be able to self-serve
Qualitative over quantitative Available and accessible data will provide a balance to ‘local
knowledge’
Demand for information on
policy staff
It will be quicker to respond to queries (and should enable more
self-service by senior officers)
Time delay in official release of
data
Long-term – the success of the project could influence change in
when data is released
Comparing appropriate data The tool will make this easier and will explain when this is not
possible.
Long-term – it encourages more open data in standardised
formats
Taking in consideration the capacity building from Trafford Pilot, mainly the Innovation and
Intelligence Lab and the tradition to opening-up data sets, lead this pilot on a mature stage of
technical area. For example, demands like “new functionalities” and “high level of report
automation” reveals high expectation of this pilots’ stakeholders.
However, Trafford is starting a new project based on the OGI co-creation framework. It takes time to
learn and put on practice the co-creation procedures. For example, identifying all stakeholders, user
workshop for referendum of ideas, objectives, inclusion of new demands from stakeholders,
collection of new data sets, opening-up and linking them are processes that take more time than
D4.6 Pilots Evaluation Results – Third Round
Page 108 of 215
regular top-down approach, even based on data-driven services that Trafford Innovation and
Intelligence Lab is used to perform.
After the co-creation process, Trafford pilot will concentrate on worklessness. The Initial objectives
discussed are in bold format and italics added from co-creation user workshop. Within these priority
areas, the objectives are to:
Help to reduce demand on services
o using open shared data to direct more effective service interventions and enhance
partnership collaboration, reducing duplication
Help to redesign services
o enabling efficient resource planning through evidence-based decision-making
Improve people’s awareness and understanding of the area
o increasing local insight by providing an accessible, agile and easy-to-use data and
information tool
Help to attract or retain investment
o targeting the use of public sector people, funding, assets and resources to reduce
worklessness and increase skills
The list of participants:
1. Trafford Council – Economic Growth and Prosperity;
2. Trafford Council – InfoTrafford;
3. Trafford Council – Partnerships and Communities;
4. Department for Work and Pensions; and,
5. Greater Manchester Combined Authority.
D4.6 Pilots Evaluation Results – Third Round
Page 109 of 215
4.1.4.2 Solutions for Pilot problems in Second and Third Year
The problems identified were solved in the second year and considered as “done”. If not yet solved,
it was planned to be deployed in the third year and were considered as “in progress”.
Table 35 - Summary of Ideas and Solutions from participants – Third Year
Problems Solutions Actions Taken Progress
Not live data in
reports
Interactive tool feeding in the most
up-to-date information
The next ICT Tools to be developed
are related to visualisations (Pivot
Table).
Done in the
3rd Year.
Multiple reports
required
Window view with wide selection of
data behind it enabling the user to
select multiple data sets
The current application can have
different versions of report, but in an
interactive web site. Cannot
download results or part of data.
Done in the
3rd Year.
Spatial
misalignment
Long-term – the success of the
project could influence change in
how data is collated
Data quality task-force is improving
this type of issue (lack and low
quality ontologies).
Done in the
3rd Year.
Accessing ‘other’
data
There will be multiple data sets
available across a range of themes
(work, housing, health, skills etc.)
New data sets are being collected,
cleansed and stored properly before
be included in the Trafford App.
Done in the
3rd Year.
Knowing who to
go to for ‘other’
data
By acting as a first-point for all data,
people will be able to build
relationships across agencies and
sectors as they use each other’s data
New agencies and data owners were
identified in the second year.
Done in the
2nd Year.
Qualitative over
quantitative
Available and accessible data will
provide a balance to ‘local
knowledge’
The new application can be used in
two ways. Visualisations of maps or
downloading data sets for self-usage.
Done in the
2nd Year.
Comparing
appropriate data
The tool will make this easier and will
explain when this is not possible.
Long-term – it encourages more
open data in standardised formats
The new application can be used in
two ways. Visualisations of maps or
downloading data sets for self-usage.
Done in the
2nd Year.
Demand on
InfoTrafford team
By making data more accessible and
available, this will upskill people to
be able to self-serve
The new applications are easy to use.
Potential videos and manuals can be
created to reduce issues for users.
Done in the
2nd Year.
Demand for
information on
policy staff
It will be quicker to respond to
queries (and should enable more
self-service by senior officers)
The new applications are easy to use.
Potential videos and manuals can be
created to reduce issues for users.
Done in the
2nd Year.
Time delay in
official release of
data
Long-term – the success of the
project could influence change in
when data is released
Co-creation helped to unify data
providers and users. Data quality
task-force is improving this type of
issue (primary and timely)
Done in the
3rd Year.
D4.6 Pilots Evaluation Results – Third Round
Page 110 of 215
4.1.4.3 Pilot Co-Creation Framework Evaluation
The Trafford Council pilot has the objective of tackling worklessness in Greater Manchester by
providing relevant decision makers with information in the form of visualisations of Linked Open
Statistical Data (LOSD) that facilitate data-driven decision making and increases transparency
towards the public.
The workshop was divided into several group discussions that sought to explore participant views on
the use of data for decision making within their work, the challenges they encounter and what
outputs the pilot could provide to facilitate the use of data. For each discussion, the participants
were presented with themes for discussion, asking them to work in pairs or small groups to discuss
individual ideas and write them down. After these initial discussions, each participant shared their
ideas to the group and the written ideas were gathered on flip-charts for future reference.
The workshop included a presentation of tools by the technical partner Swirrl and example outputs
from the Trafford Council pilot. As part of the co-creation framework, a Data Literacy Survey was
sent via email to participants before the workshop. The survey was designed to assess familiarity
with different data formats, sources of open data and types of visualisation. Each part of the
workshop, the tools and output presentations and the Data Literacy Survey will be described in the
below. The agenda of the co-creation user workshop is presented in the Table 36.
Table 36 – Co-Creation User Workshop Agenda
Time Activity
1:30 to 1:40 Welcome and introductions
1:40 to 1:50 Introduction to the OGI Project
1:50 to 2:20 Understanding the environment
1:40 to 1:50 Current tools and systems
2:50 to 3:10 New tools and systems
3:10 to 3:20 Ideal outputs and summary
The workshop had the following participants:
Swirrl
Department of Work & Pensions
MCA
Trafford Council
o Trafford Data Lab
o Partnerships & Communities
o Performance team
o Commissioning Team
o Public Health
o Economic Growth
D4.6 Pilots Evaluation Results – Third Round
Page 111 of 215
The participants were introduced to the OGI Project and the aim of the Trafford Council pilot and the
technical partner Swirrl. The objectives of the workshop were also explained.
1. To raise awareness of the OGI project and the Trafford pilot;
2. To understand how data is used in decision-making;
3. To understand how data is currently accessed and the challenges involved; and,
4. To understand what tools would be needed to support effective use of data.
The participants were asked to discuss in small groups and write down in Post-it notes the decisions
or activities that they currently undertake that are informed by data. After the small group
discussions, the individual decisions/activities were presented to the wider group and the Post-it
notes were organised in flip-charts, one for each small group. After this initial discussion the
participants were asked to write down in their flip-chart next to each Post-it note the data they use
and the organisations related.
Current tools and systems
The participants were asked to share with the group what type of format they use and prefer when
using data. They wrote their preferred format on the flip-chart. The group discussion continued with
the challenges they encounter when obtaining, combining and analysing data.
New tools and systems
The technical partner Swirrl presented the GM Data Store and the Department for Communities and
Local Government's dashboard that visualise LOSD stored within an instance of their Publish My
Data system. GraphQL, part of the tools developed for the OGI project, could be used to access data
from GM Data Store. Afterwards, a prototype road traffic collisions application was demonstrated by
Trafford Council to show the type of interface and functionality that could be expected as a final
output from the Trafford Council pilot.
Ideal outputs
At the end of the workshop a summary was presented and the participants wrote on Post-it notes
what would they ideally like to help them with worklessness data/tasks. The Post-it notes were fixed
to a dedicated flip-chart for future reference.
Data literacy
Prior to the workshop, a survey was sent to the workshop participants to assess data literacy. The
respondents were presented with 8 questions created to evaluate the following aspects of data
literacy: Finding data, Reading data, Creating data, Cleaning data, Managing data, Visualising data,
Analysing data and Arguing with data.
Due to the variety of positions and organisations of the participants, a wide range of uses of data
were mentioned:
D4.6 Pilots Evaluation Results – Third Round
Page 112 of 215
Allocation of resources and training procurement for staff;
Procurement of supportive provision for Jobcentre Plus users;
Data to support and inform design of strategies and planning for year ahead and long term;
and,
Preparing reports for internal monitoring and statutory returns.
The data they use are:
Internal datasets; and,
Official sources (Office of National Statistics, DWP Data Explorer, NOMIS, Stat Xplore).
The preferred formats when analysing data and for what they find it useful are:
Spreadsheets: enable the manipulation, filter and sorting of data according to the specific
necessities;
Charts: for analysis, comparison and to show variation on time; and,
Maps: to access spatial information for design of strategies and planning.
The challenges encountered when obtaining, combining and analysing data were:
Accessibility: Excessive repetition of tasks to produce the data they need.
Poor data literacy: The workload of the data analysts is increased due the inability of data
requesters to find the information they need for themselves with the available tools.
Inconsistency of categorisation of data.
The readers of the reports produced by the participants do not understand the
information they receive.
Data protection and the necessity of anonymize data.
From this, a summary of the ideal features that the participants would like on a tool:
Visuals and raw data both available;
Information available for small areas;
Maps with indication of differences on areas using scales;
Comparison facilities by areas, GM and UK;
Reusable visuals that are simple and understandable; and,
D4.6 Pilots Evaluation Results – Third Round
Page 113 of 215
Easy to use user interface.
D4.6 Pilots Evaluation Results – Third Round
Page 114 of 215
4.1.4.4 Data Quality Evaluation
Table 37 – Data Quality Survey Results – Third Year
Data Quality
Attributes Results Overview
Discoverability
and Usability
1- App has links to the original data sets;
2- App has no searchable list of data sets used;
3- App allows users to download the data sets, including partial download of selected variables
and CubiQL endpoint is linked to the Trafford Data Store the app will source data from there;
4- App has multiple format download option;
5- App allows to user preview and interact with the data.
Granularity 1- There is no aggregate level of data, only raw level of data from internal data sets.
Intelligibility
1- Supporting documentation can be found at the same time as the data (e.g. right next to the
link for the data). All of the data derived from government sources so included comprehensive
metadata.
Trustworthiness
1- The data sets collected are well-known by developers, but there is neutral confidence on
how it was processed;
2- The data is 100% reliable, and is consistent with external sources.
Linkable to other
data
(5 Stars LOD)
1- The data sets are available in different formats to different audiences, from XLS to CSV. RDF
was created by OGI developers;
2- The RDF is not yet properly linked to other external data to provide context. Once the data is
converted to RDF and stored in the Trafford Data Store it will be 'linkable'.
15 Open Data
Principles
1- Data is complete;
2- Data is primary;
3- There is not issues about timely availability;
4- Data is accessible to the widest range of users (XLS, CSV and RDF)
5- Data is non-discriminatory because is freely available;
6- Data has license-free (CLS, CSV, RDF);
7- Data is permanent;
8- Data is safe to open;
9- Data is not 100% designed with public input (decide format and parts of data sets to
visualize and download).
D4.6 Pilots Evaluation Results – Third Round
Page 115 of 215
Table 38 – Data Quality Improvement Progress – Third Year
Data Quality
Attributes
Status
1st Year 2nd Year 3rd Year
Discoverability
and Usability
App in development, with
few functionalities and
few data sets.
App developed, still being
updated and without PARTIAL
expected functionalities asked
by users in the workshop.
Updated with expected
datasets and functionalities.
Granularity Diverse Data sets with low
quality and only raw level.
A task-force for data quality to
unify and enrich data sets was
made. Still room to improve.
Updated with expected level of
data granularity. Check datasets
quality section 4.1.4.5.3.
Intelligibility Low level of intelligibility,
with low level of context.
Improvements. High level of
intelligibility without proper
data context.
Updated with proper data
source context to end-users.
Check outcomes evaluation
section 4.1.4.5.
Trustworthiness
Poor data quality due
incomplete, not primary
and not linked data sets.
Data was not 100%
reliable.
Data is linked, but without
external context. Waiting for
stable version of OGI CubiQL API
to solve this issue.
Updated with OGI ICT toolkit
finished and data in linked
format.
Linkable to
other data
(5 Stars LOD)
Not following all the 5
stars LOD principles.
Following all the 5 Stars LOD
principles.
Following all the 5 Stars LOD
principles.
15 Open Data
Principles
Not following all the Open
Data Principles.
Following 9 of 9 OD principles
that matches OGI needs.
Following 9 of 9 OD principles
that matches OGI needs.
D4.6 Pilots Evaluation Results – Third Round
Page 116 of 215
Table 39 – Data Quality Improvement Progress – Third Year
Data Quality
Attributes Summary of Benefits Achieved Summary of Challenges Faced
Discoverability
and Usability
Trafford pilot improved the
discoverability and usability. Data
quality was improved and
visualizations were created.
Combining data quality and end-users surveys,
Trafford pilot can increase the number of data sets
and include new asked functionalities by end-users
after trial workshop in the second year.
Granularity The same level of granularity (raw)
and is accepted by users.
Pilot can consider to find new data sets that provides
the aggregate data instead of showing only raw level.
New functionalities can emerge from this action.
Intelligibility Data was linked and data cube
enabling data visualization.
Data is linked and pilot app has all the expected
functionalities for voisualization.
Trustworthiness
Still similar scenario of poor and
not reliable data sets. Focus of 2nd
year was given in other attributes.
The data is reliable, however there is still room to
improve the external connections.
Linkable to
other data
(5 Stars LOD)
The first year had data sets in
spreadsheets (PDF, XLS). The
second year increase from 2 stars
to 4 stars (RDF).
After stable version of OGI ICT Toolkit, the Trafford
Pilot App could link all the datasets and achieve the 5
stars LOD.
15 Open Data
Principles
The second year achieved 8 of 9
principles that should be followed
in Trafford OGI pilot.
Pilot is not following the design with public input
(users can select format and part of data to
download). The objective of this app is to provide an
efficient visualization of the data.
Beyond that, user can download all the data and
cleanse the data by themselves.
4.1.4.4.1 Datasets
Below the list of data sets:
https://github.com/traffordDataLab/projects/blob/master/opengovintelligence/apps/production/work%3Cnes
s/about.md
D4.6 Pilots Evaluation Results – Third Round
Page 117 of 215
4.1.4.5 Outcomes Evaluation
To evaluate the application from the perspective of end-users a questionnaire was distributed to the
end users of the Trafford Council Worklessness Application. The questionnaire that they distributed
is somewhat different to the questionnaire that is used for the other pilots, but allows for qualitative
views of the same themes. In the end the questionnaire had 50 valid respondents.
1. Linked Open Statistical Dataset (LOSD) Familiarity;
2. User Acceptance;
3. Datasets Quality; and,
4. Perceived Outcomes.
D4.6 Pilots Evaluation Results – Third Round
Page 118 of 215
4.1.4.5.1 Linked Open Statistical Data (LOSD) Familiarity
Figure 33 – LOSD Familiarity – Trafford Pilot
From Figure 33, we can assume the surveyed people in Trafford are majorly people without high
level of IT skills. 57% of people were aware about open data, and 37% of them pointed to be familiar
with LOSD technology, LOSD portals and the datasets presented in the pilot.
This result can be seen as an opportunity to improve the awareness about LOSD technology and
usage of new datasets, increasing the chances to bring value to users. The section 4.1.4.5.4 shows
the perceived outcomes such as increase of transparency, better decision-making and time spent
searching, probably due these opportunities identified in this section results about familiarity (new
technology, new datasets to users).
D4.6 Pilots Evaluation Results – Third Round
Page 119 of 215
4.1.4.5.2 User Acceptance
Figure 34 – User Acceptance – Trafford Pilot
The second section of the survey asked to users about the acceptance of the application. Checking
the results from the Figure 34 – User Acceptance – Trafford Pilot, we can assume major part of the
answers show us a positive reaction. The only negative reaction from users is the needed of high
level of technical knowledge to use the app. This makes sense, because Trafford is one of the most
complex application, in terms of methodology to present the data and combining different data.
Another questions that shows this assumption is the “not difficult to use”, which 25% believe that
app is difficult to use.
To sum up, the user acceptance of Trafford Pilot is above 90%, which means very positive reaction
and high level of acceptance of the Trafford Application.
D4.6 Pilots Evaluation Results – Third Round
Page 120 of 215
4.1.4.5.3 Datasets Quality
Figure 35 – Datasets Quality – Trafford Pilot
Datasets quality was tested in this survey. The first major complaint about data quality was the
completeness, with 37% of them pointing the incompleteness of datasets in the app. Second, 25% of
them suggesting that not all the required datasets were in the Trafford application. Others pointed
that data could be more accurate.
Beyond that, 88% of the users are satisfied with the data quality presented in the Trafford
Application. In accordance with the results of user acceptance, 96% of users believe that data has a
proper accessibility and in accordance with the results seen in the perceived outcomes, a reduced
time to find the proper data.
D4.6 Pilots Evaluation Results – Third Round
Page 121 of 215
4.1.4.5.4 Perceived Outcomes
Figure 36 – Perceived Outcomes – Trafford Pilot
Following the other sections, the perceived outcomes had majorly positive reactions. In summary,
users pointed out an acceptance of 82% of Trafford perceived The only major complain is the
needed of new functions to create more transparency (90%).
On the other hand, users are satisfied with the efficiency (84%), reducing time to search information
(86%), helping to make better decisions (88%), increase of transparency (90%) and better
interpretation of data (92%).
D4.6 Pilots Evaluation Results – Third Round
Page 122 of 215
4.1.4.6 Screenshots of Trafford Council Application
Figure below presents the screenshot of Trafford Council Application.
Figure 37 – Trafford Council Worklessness Application
D4.6 Pilots Evaluation Results – Third Round
Page 123 of 215
4.1.5 Pilot 5 – The Flemish Environment Agency (Belgium)
4.1.5.1 Pilot Description and Expectation
The Flemish Government will use OGI ICT Toolkit to enhance their environmental policy making in
terms of timely publication of the actual state of affairs related to environment, evaluations of the
permits policy, and develop tools to benchmark the pollution of companies to others working in the
same economical domain.
For a detailed description of Enterprise Lithuania and Lithuanian pilot, please check report D1.1 and
D4.1. A blog post at OGI Medium account was also published describing the Flemish Application:
https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-belgian-pilot-
1c52872cfe65 .
D4.6 Pilots Evaluation Results – Third Round
Page 124 of 215
4.1.5.1.1 Identified Pilot Problems in the First Year
Instead of workshops, the Flemish pilot made a series of interviews with stakeholders due the high
number of people involved. The drivers of the interviews were:
What is the link of your business with the environmental permits?
What kind of data from the permit would you like to have in a structured format?
There were two types of questions, business related and data related. It is presented on the table
below.
Table 40 - Questions made to the audience
Question
number Questions
Business Related Questions
1 Do you give permissions?
2 Do you advise about permits?
3 Do you have enforcement tasks?
4 Do you yourself manage a base registry and what is the link with permits?
5 What would be the added value of a basic registry of environmental permits for you
business?
Data-Related Questions
6 What administrative data do you want to have? (decision date, issuer, type of procedure)
7 What kind of data would like to have on an exploitation level (emissions, time series of
permitted and effectively emitted emissions, Environmental impact assessment reports)
8 What kind of spatial data would you like to have that is created during the permitting
process?
D4.6 Pilots Evaluation Results – Third Round
Page 125 of 215
Table 41 - Summary of Ideas and Solutions from participants
Problems Solutions
No central register of permits
Legislative action to see that all permitting authorities register
permits. Documents as well as structured data;
Building a basic register centrally to which data can be sent.
No publication of emission data in LOSD format
Publishing data via LOD technology;
Reuse of existing data sets, as example European Environment
Agency;
Adding new vocabularies on existing vocabularies.
No re-use of data gathered during permission
processes
Legislative action to see that all permitting authorities register
permits;
Documents as well as structured data building a basic register
centrally to which data can be sent.
No re-use of data from base registries
Re-use of data from base registries;
Starting program to publish base registries also via LOD
technology.
No base registry build-up
Legislative action to see that all permitting authorities register
permits;
Documents as well as structured data building a basic register
centrally to which data can be sent.
No possibility of linking permission data with
emission data
Publishing data via LOD technology;
Reuse of existing data sets, as example European Environment
Agency;
Adding new vocabularies on existing vocabularies.
No possibility of linking permission data with
enforcement data
No possibility of linking permission data with
data of new permissions for same exploitation
No possibility of benchmarking for legal persons Providing online analysis possibilities for example benchmarking
No possibility of data analysis Providing online analysis possibilities
No possibility of data-driven policy-making Providing online analysis possibilities
Data in general is locked in silo’s
Publishing data via LOD technology;
Reuse of existing data sets, as example European Environment
Agency;
Adding new vocabularies on existing vocabularies.
Data is not very accessible Publishing data as LOD under open data licence:
Providing analysis and reporting tools;
Starting LOD program to publish data in LOD way and as part of
process of the organisations. Not as extra work.
D4.6 Pilots Evaluation Results – Third Round
Page 126 of 215
Considering the issues and solutions purposed by participants on the Belgium pilot, we can point out
that this pilot is on a mature stage of technical aspects. For example, while data is not accessible
because IT departments silos and lack of legislation to cooperate, we can identify capacity building
with high level of skills due the demands such as desire to “link” and “data-driven service” to
“benchmark” individual cases on an “automated way”.
However, as Trafford pilot, the Flemish government is starting this project from the scratch if co-
creation framework is considered. The pilot partners reunited all the stakeholders and identified
that face a bigger problem. More data sets were identified and probably will be opened and linked
to enable the creation for functionalities and services demanded by all stakeholders. As example,
there is no way to create dashboards with real-time data sets without connecting and linking them
before.
The list of participants is below:
Administrations on the federal level
o Public health administration
Administrations on the Flemish level
o Department of environment nature and energy,
o Agency for forest and nature,
o The heritage agency,
o The environmental planning and permit department.
o Public waste agency of Flanders,
o Flemish energy agency,
o Flemish land Agency,
o Flemish agency for regulation of the energy market
Administrations on the province level
o PVV (Organisation representing the 5 provinces together with people representing
every individual province).
Administrations on the community level
Vereniging van Vlaamse Steden en Gemeenten (VVSG)
o Organisation representing the 308 communities together with people representing
the communities of Gent, Genk, Antwerp, Leuven and Kortrijk.
D4.6 Pilots Evaluation Results – Third Round
Page 127 of 215
4.1.5.1.2 Solutions for Pilot problems in Third Year
The problems identified were solved in the second year and considered as “done”. If not yet solved,
it was planned to be deployed in the third year and were considered as “in progress”.
Table 42 - Summary of Ideas and Solutions from participants – Third Year
Problems Solutions Actions Taken Progress
No central register of permits
Legislative action to see that all
permitting authorities register
permits. Documents as well as
structured data;
Building a basic register centrally to
which data can be sent.
This issue is still being
discussed internally. In Progress
No publication of emission data in
LOSD format
Publishing data via LOD technology;
Reuse of existing data sets, as
example European Environment
Agency;
Adding new vocabularies on
existing vocabularies.
The data is in Linked
Data format and
vocabularies were
created.
Done
No re-use of data gathered during
permission processes
Legislative action to see that all
permitting authorities register
permits;
Documents as well as structured
data building a basic register
centrally to which data can be sent.
This issue is still being
discussed internally. In Progress
No re-use of data from base
registries
Re-use of data from base registries;
Starting program to publish base
registries also via LOD technology.
This issue is still being
discussed internally. In Progress
No base registry build-up
Legislative action to see that all
permitting authorities register
permits;
Documents as well as structured
data building a basic register
centrally to which data can be sent.
This issue is still being
discussed internally. In Progress
No possibility of linking
permission data with emission
data Publishing data via LOD technology;
Reuse of existing data sets, as
example European Environment
Agency;
Adding new vocabularies on
existing vocabularies.
The tools allow the
pilot create the link In Progress
No possibility of linking
permission data with
enforcement data
This issue is still being
discussed internally. In Progress
No possibility of linking
permission data with data of new
permissions for same exploitation
This issue is still being
discussed internally. In Progress
No possibility of benchmarking for
legal persons
Providing online analysis
possibilities for example
The tools and
vocabularies will allow In Progress
D4.6 Pilots Evaluation Results – Third Round
Page 128 of 215
benchmarking the pilot overcome
this issue.
No possibility of data analysis Providing online analysis
possibilities
The stable version of
map visualisation and
interactive tables will
allow the analysis.
In Progress
No possibility of data-driven
policy-making
Providing online analysis
possibilities
The stable version of
map visualisation and
interactive tables will
allow the analysis and
data-driven policy-
making.
In Progress
Data in general is locked in silo’s
Publishing data via LOD technology;
Reuse of existing data sets, as
example European Environment
Agency;
Adding new vocabularies on
existing vocabularies.
This issue is still being
discussed internally. In Progress
Data is not very accessible
Publishing data as LOD under open
data licence:
Providing analysis and reporting
tools;
Starting LOD program to publish
data in LOD way and as part of
process of the organisations. Not as
extra work.
This issue is still being
discussed internally. In Progress
D4.6 Pilots Evaluation Results – Third Round
Page 129 of 215
4.1.5.2 Data Quality Evaluation
Table 43 – Data Quality Survey Results – Third Year
Data Quality
Attributes Results Overview
Discoverability and
Usability
1- App has no links to the original data sets;
2- App has no searchable list of data sets used;
3- App does not allow users to download the data sets, including partial download of
selected variables;
4- App does not have multiple format download option;
5- App allows to user preview and interact with the data.
Granularity 1- Both aggregate and raw level of data from internal
Intelligibility 1- There is no supporting documentation.
Trustworthiness 1- The data sets collected are well-known by developers, including how it was processed;
2- The data is 100% reliable, and is consistent with external sources.
Linkable to other
data
(5 Stars LOD)
1- The data sets are available in different formats to different audiences, from PDF, XLS to
CSV. RDF was created by OGI developers but not yet in stable version for end-users;
2- The RDF is not yet properly linked to other external data to provide context.
15 Open Data
Principles
1- Data has issues to be considered complete;
2- Data is primary, but aggregate level is also published;
3- There are issues about timely availability;
4- Data is not 100% accessible to the widest range of users (XLS, CSV and RDF)
5- Data is discriminatory because is not freely available;
6- Data has license-free (CLS, CSV, RDF);
7- Data is safe to open;
8- Data is not 100% designed with public input (decide format and parts of data sets to
visualize and download).
D4.6 Pilots Evaluation Results – Third Round
Page 130 of 215
Table 44 – Data Quality Improvement Progress – Third Year
Data Quality
Attributes Summary of Benefits Achieved Summary of Challenges Faced
Discoverability
and Usability
The Flemish application allows
users to preview and interact with
the data published in web
interactive maps and tables.
There is still no link to external data sources. Only
internal data sources have been linked. It is not ye
It is possible to download the data or part of this data
(slicing the data sets) as users’ desire.
Users can preview and interact with the data
(visualization functionality)
Granularity There is both raw and aggregate
data level in the application.
There is no aggregate level of data (e.g. average of
pollution). Data is in unit level (data from each
company, per type of pollution).
Intelligibility Data was linked and data cube
enabling data visualization.
Supporting document exist, however, is found
separately.
Trustworthiness
The data is reliable, reflecting the
reality. Developers know how it is
processed and is consistent.
There is still room to improve the reliability.
Linkable to
other data
(5 Stars LOD)
The data sets are in RDF format. Achieved the max level of 5 stars LOD.
15 Open Data
Principles
The third year achieved 8 of 9
principles that can be followed in
OGI pilot.
Pilot is not following the design with public input
(users can select format and part of data to
download) and only has unit level.
D4.6 Pilots Evaluation Results – Third Round
Page 131 of 215
4.1.5.2.1 Datasets
The data sets are already in RDF format and with metadata. The link to access data seta are at:
https://app.minbox.com/files/_fDIOp2
Figure 38 - Metadata of Belgium Pilot Data Set
D4.6 Pilots Evaluation Results – Third Round
Page 132 of 215
4.1.5.3 Outcomes Evaluation
To evaluate the application …
1. Linked Open Statistical Dataset (LOSD) Familiarity;
2. User Acceptance;
3. Datasets Quality; and,
4. Perceived Outcomes.
D4.6 Pilots Evaluation Results – Third Round
Page 133 of 215
4.1.5.3.1 Linked Open Statistical Data (LOSD) Familiarity
Figure 39 – LOSD Familiarity – Flemish Pilot
The results from Flemish pilot were not majorly positive. All of the 8 people surveyed (100%) pointed
that they were not familiar with Linked Open Statistical Data (LOSD). `Surprisingly, half of them knew
the datasets presented in the Flemish Application.
D4.6 Pilots Evaluation Results – Third Round
Page 134 of 215
4.1.5.3.2 User Acceptance
Figure 40- User Acceptance – Flemish Pilot
Major part of the dimensions surveyed resulted into positive reactions. The three mains issues found
by users were the high level of technical knowledge to use app. All users pointed that and makes
sense, considering the level of complexity of this specific datasets subject and also the functionalities
the applications has. Another problem was the experience of crashes, what could lead to the 13%
complaining about the application functionalities.
The other dimensions had positive reactions such as functions working properly, the interaction
application, usefulness, easy to use, integrated, app well designed, and, being not difficult to use.
Some of the answers are contradictory and deeper qualitative research must be done to discover
what happened specifically in this pilot.
D4.6 Pilots Evaluation Results – Third Round
Page 135 of 215
4.1.5.3.3 Datasets Quality
Figure 41 – Datasets Quality – Flemish Pilot
Figure 41 – Datasets Quality – Flemish Pilot shows that the main issue of Flemish Pilot App is the
completeness of data. Surveyed people believe the data is not complete, and probably influenced
13% of them point out that all the required datasets to monitor the Flemish environment were not
presented (or incomplete). It makes sense, because Flemish pilot is still under development and new
datasets will be linked in the future to let the application even more complete.
ON the other hand, it is clear the datasets provided had positive reaction. All of them believe that
the data is accessible, accurate and they reduced time to find the data. In summary, all of the 8
surveyed people think that the overall data quality is satisfactory.
D4.6 Pilots Evaluation Results – Third Round
Page 136 of 215
4.1.5.3.4 Perceived Outcomes
Figure 42 – Perceived Outcomes – Flemish Pilot
The positive reactions from surveyed people were better interpretation of data (100%), increase of
efficiency (100%), time reduction to search for data (100%), increase of transparency (100%). 88% of
the users pointed that can make better decision-making using the Flemish application and 75% of
them think the app helps to achieve their goals. On the other hand, they demand more functions to
enable even more transparency (88% of the surveyed people).
In general, 88% of the users believe that the Flemish Pilot app is acceptable.
D4.6 Pilots Evaluation Results – Third Round
Page 137 of 215
4.1.5.4 Screenshots of The Flemish Environment Agency Application
The next figure presents the screenshots of OGI ICT toolkit using data sets from the Pilot. The
platform is on a different address of other pilots:
Figure 43 – The Flemish Environment Agency Application
D4.6 Pilots Evaluation Results – Third Round
Page 138 of 215
4.1.6 Pilot 6 – Marine Institute (Ireland)
4.1.6.1 Pilot Description and Expectation
The Marine Institute pilot, in Galway (Ireland), aims to support three business cases scenarios:
1. Marine renewable energy
2. Maritime search and rescue; and,
3. Maritime tourism and leisure.
The function of the pilot will be to take non-linked tabular Comma-Separated Values (CSV) data
hosted by the Marine Institute and convert it to Linked Data formats using Data Cubes. Using Linked
Data Resource Description Framework (RDF) the new structured data management will enhance the
value of the marine data asset for the scenario purposes by structuring and enriching the data with
vocabularies and semantic value to aid the requirements of the scenarios.
The building of a Linked Data Dashboard to include data visualisations such as charts, maps and
widgets of marine Linked Data will support decision making scenarios for actors making a renewable
energy investment decision, a search and rescue expert in rescue and recovery and a maritime
sports enthusiast in their maritime sport.
For deep description of Marine Institute and Irish pilot, please check report D1.1 and D4.1. Also, a
blog post at OGI Medium account was published:
https://medium.com/opengovintelligence/opengovintelligence-pilot-showcase-the-marine-institute-
bde38cf89ac6 .
4.1.6.1.1 Identified Pilot Problems in the First Year
A number of significant barriers to co-creation of data-driven services in the Marine sector were
identified by stakeholders. These barriers can be divided into two higher level categories (Data
Infrastructure, and User Challenges), each of which contain a number of sub-categories. Examples of
barriers from these categories and sub-categories are provided below.
The initial options proposed by workshop participants opened the possibility space for the creative
thinking in the next phase of the workshop, which involved participants working with specific user
scenarios and generating key needs and requirements of stakeholders involved in co-creation of
services in the Marine sector.
These sceanrios involved hypothetical users including citizens, public administrators, data scientists,
engineers, and other stakeholders. The During this section of the workshop, participants highlighted
an extensive range of 1) Informaiton needs, and 2) Decision-making needs. Informaiton needs, refers
to types of information or data which are needed for co-creation in the Marine sector. In relation to
decision-making needs, participants were asked to generate a list of key tools, services, affordances
or other types of supports which would aid in decision-making.
The scenarios purposed to participants were:
D4.6 Pilots Evaluation Results – Third Round
Page 139 of 215
1. Engineer Ed is considering developing a wave energy converter and would like access to
real-time data on wind, waves and currents generated by multiple public organisations
through one web-based location in order to predict the performance of his device and to
monitor its safety once it is deployed in the environment. Ed needs to know when a high
energy wave event is happening in order to provide decision support to Technician Tony.
When Ed and Tony are in agreement that the environmental conditions are in danger of
causing damage to their ocean energy converter, they need to be able to shut it down in
order to protect it. In support of the data, it would be useful to Ed to have easy access to
supporting information (such as the Offshore Renewable Energy Development plan) from
the data access site. This is useful for Ed and Manager Mike when they are choosing
potential locations in which to deploy their ocean energy converters. Deputy David also uses
this supporting information in the portal when lobbying the minister to encourage central
government policy on marine renewable energy development.
2. Captain Cillian is a racing sailor looking to gain a competitive edge in his races on Galway
Bay. He’d like to be able to access tide predictions, wind data and current speeds in a format
which he can upload into his sailing computer so he can plan his races. On shore
Commodore Colm would like to receive the locations of each of the sailing vessels in the
races he organises so he can better plan safety operations associated with those races.
Public Administrator Patricia examines race activity data and service usage in order to plan
and inform policies in relation to boat racing activities.
3. Sergeant Steve is coordinating a search and rescue operation. He needs to know both the
current conditions in the waters around the coastline, where his teams can access the
shorefront and where an object which has been dropped into the water is likely to be since it
entered the sea. Garda Grainne is a member of Steve’s team. She should be able to return
information, such as geo-located photographs, to Steve so that he can be kept up to date of
the search team’s location and conditions. In addition to the public authorities, Volunteer
Val is a member of the public involved in searching the coastline. The volunteers should have
access to the same apps and much of the same data as the authorities, but some
information may not be available to the volunteers. Inspector Ian and Coastguard Chris
review the information collected by the app after each rescue to build up dataset which
allows them to develop local search and rescue policies.
4. Drone pilot David enjoys flying his device in coastal areas. He occasionally spots strange
patterns on the water surface and would like help in identifying what these are. He would
like to submit them to a web portal where an expert can tell him what he’s seeing. Biologist
Brenda logs in to the web portal to see if there is anything of interest to her. She sees one of
David’s photos from a flight earlier today. Brenda can use any identification she makes to
alert authorities of any potential harmful algal blooms. Other experts can confirm Brenda’s
identifications and engage in further analysis of other images uploaded on the system. These
expert identifications and alerts may be used alongside model outputs and processed
satellite images to inform policy on keeping open or closing aquaculture sites.
Participants generated a total of 40 barriers in relation to Data Infrastructure. The following sub-
categories emerged from the full Data Infrastructure set: Accessibility, Awareness and Engagement,
D4.6 Pilots Evaluation Results – Third Round
Page 140 of 215
Connectivity, Data Management, Lack of integration and collaboration, Policy Issues, Quality Issues,
and Service and Resources Issues. Table below provides a sample of barriers in each sub-category.
Table 45 - Sample Data Infrastructure Barriers
Sub-category Barrier
Accessibility
Issues
Lack of intuitive service design, such that training is not required
Lack of open access data – can everyone who wants to access the data get to it?
Awareness and
Engagement
Issues
Lack of knowledge by experts as to what data is available
Inadequate citizen data management expertise in people when collecting data on
the environment including knowledge of controlled vocabularies, metadata, GIS,
data enrichment value etc.
Connectivity
Issues
Loss of data/connectivity during e.g. storms, which is often the time when the
client wants to collect data
Failure of adequate internal/ broadband availability nationwide
Data
Management
Issues
Lack of focus on which data to prioritize and assign investment and resources to;
Lack of data sustainability, when projects end is the data still managed?
Is it accessible?
Lack of
integration/coll
aboration
Lack of integrated data infrastructure access;
Unnecessary duplication of data by different agencies
Policy Issues Legislative barriers and lack of legislation;
Conflict between the multitude of European directives on the marine
environment (e.g. MSFD, WFD, Shellfish, Habitats, EMFF) in terms of a
harmonised data standard for information describing the marine environment.
Quality Issues Lack of data quality control;
Lots of gaps in data, no guarantee of quality.
Service and
Resource Issues
Real-time data dimension;
No marine alert system for stakeholders.
Participants generated a total of 45 barriers in relation to User Challenges. The following sub-
categories emerged from the full User Challenges set: Awareness and Engagement, Collaboration,
Education and Training, Finance, Motivation, Needs Analysis, Policy Issues, Quality Issues, Resistance
and Conflict, and Service and Resource Issues. Table below provides a sample of barriers in each sub-
category.
D4.6 Pilots Evaluation Results – Third Round
Page 141 of 215
Table 46 - Sample User Challenges Barriers
Sub-category Barrier
Awareness and
Engagement
Issues
Lack of public awareness to the data systems available
Understanding value of data to society
Collaboration
Issues
Lack of opportunities for users to engage with data developers/
technologists (stakeholder involvement in development)
Tunnel vision e.g. focus on organizing an event not on surrounding issues
Education and
Training Issues
Lack of adequate user experience in the relevant technologies
Lack of adequate ongoing training
Finance Issues Lack of funding for the “processing” of the data
Lack of financial resources
Motivation
Issues
Lack of public drive to get government to change
Lack of public drive to get government to provide the financial resources
needed to ensure a robust system is in place
Needs Analysis
Issues
Going beyond mash-up is hard, need to identify services/ needs etc.
Lack of engagement with end users to develop targeted services
Policy Issues The silo effect, focus on the individual needs of a department, not looking
at the bigger picture
Lack of vision at a high level
Quality Issues • Lack of data quality standards
• Lack of quality control of data and data delivery
Resistance and
Conflict
Resistance to using anything other than the old tried and trusted methods
Conflict between the different stakeholder stalling progress
Service and
Resource Issues
Lack of will at state level to push/implement broadband network
Difficulty of creating a data collection method that is sustainable and
iterative
While these and other barriers highlighted many challenges, the expert group identified many
options which could help to overcome these barriers. In thinking about and generating options,
participants were asked to address three core stages of the co-creation process: Service Creation
(ideas), Service Engineering (requirements, design) and Service Management (delivery, evaluation).
Participants were asked to place their option beneath one or more of these headings, depending on
the nature of the option, and it’s relevance to overcoming barriers in one of more stages of co-
creation. Again, participants worked across the higher-level categories of Data Infrastructure and
User Challenges, which were divided into a set of sub-categories.
Participants generated 72 options for overcoming barriers to Data Infrastructure, 15 of which relate
to Service Creation, 32 of which relate to Service Engineering, and 25 of which relate Service
Management. In some cases, options were suggested by participants to be relevant to more than
D4.6 Pilots Evaluation Results – Third Round
Page 142 of 215
one stage, and thus are recorded as such in this report. Options within each of these three stages of
co-creation were further divided into sub-categories. Tables below provide a sample of options
within each stage.
Table 47 - Service Creation - Sample options for overcoming barriers
Category Option
Collaboration • Encourage government agencies to work on a common data
sharing framework
• Build up groups when starting and create multi-agencies - get-
togethers and get inputs
Education and
Training
• Training for public servants in relation to data protection
• Open data training for government
Finance • Focus finance away from policy towards data as the key value in
service development
• A data sharing unit to identify cost savings and efficiency across the
public service
Policy • Enact data sharing and governance legislation
• Build a long term sustainability plan into project applications
Services and
Resources
• A government level data storage service
• Develop a platform with plasticity to change in the future as user
needs mature
D4.6 Pilots Evaluation Results – Third Round
Page 143 of 215
Table 48 - Service Engineering - Sample options for overcoming barriers
Category Option
Accessibility • Formats - making datasets available - update data to new format
• A system that can be accessed by most appliances, smart phones, iPads,
radar chart plotters
Collaboration • Co-ordinated field of data sharing with authorities
• Key data “collectors” meet with a view to discuss and collaborate in future
to “standardize” data collection and management
Quality • Quality is a must so the intermediate and end user know the limits of the
data used to create products
• Use standardized date “flagging”/ annotation schemes (from e.g. UNESro) to
identify data quality
Services and Resources • A government level data storage service
• Develop a platform with plasticity to change in the future as user needs
mature
Tools • Gather weather information via weather stations and ODAS buoys
• Develop linked data transformation transformers and data mapping tools
Table 49 - Service Management - Options for overcoming barriers
Category Option
Collaboration • Access to servers/ hardware requires appropriate resources - collaborations
can aid this
• Build a system used by all (the team) entities e.g. a) input race data b) input
incidents data c) input road closure data
Education and Training • Citizen data management user guides and public sector domain data
collection training
• Training for public servants in relation to data protection
Policy • Ensure that each public body has a data management plan which is
adequately resourced
• Identification of and promotion of best practice service standards to break
data silos
Services and Resources • A central document is collated on existing resources/databases/services like
a systematic review as a start point
• Investigation of new IoT methods for data collection (e.g. sig fox, LoRa) to
minimise downtime
Tools • Back up sensors that can be used in the event of a breakdown at critical
times
• Create software to identify bad data and broken sensors
Participants generated 48 options for overcoming barriers to User Challenges, 20 of which relate to
Service Creation, 16 of which relate to Service Engineering, and 12 of which relate Service
Management. Again, in some cases, options were suggested by participants to be relevant to more
than one stage, and thus are recorded as such in the tables. Options within each of these three
stages of co-creation were further divided into sub-categories. Tables below provide a sample of
options within each sub-category.
D4.6 Pilots Evaluation Results – Third Round
Page 144 of 215
Table 50 - Service Creation - Options for overcoming barriers
Category Option
Awareness and
Engagement
• Technology road shows on the coder dojo model known as the
technology dojo for public
• Promote data availability/ service availability, create awareness
Education and Training • Provide ongoing training for all users of public data
• Educate government employees and public about what directives
are about and why they should care
Needs Analysis • Find out what exactly user needs + who are the potential users
• Avoid unnecessary cost by knowing or being able to access info on
what data is already available
Table 51 - Service Engineering - Options for overcoming barriers
Category Option
Awareness and
Engagement
• Technology road shows on the coder dojo model known as the
technology dojo for public
• Promote data availability/ service availability, create awareness
Education and Training • Induction courses when new employees start
• Publish guidelines on best practices for methodologies. Maybe
using non-technical language where possible
Needs Analysis • Run service design workshops including end-users
• Develop services/ apps from identified and documented user
requirements (not because we can!)
Policy • E.U. directive to encourage government drive
User-Friendly • Creates simple mobile apps
• Provide the user with an engaging, simple, and robust experience
to move them on to their services
D4.6 Pilots Evaluation Results – Third Round
Page 145 of 215
Table 52 - Service Management - Options for overcoming barriers
Category Option
Awareness and
Engagement
• Require a dissemination plan for all public projects
• Identify forums for promotion of data services which will reach a
wide audience
Collaboration • Bringing businesses in as partners
Education and
Training
• Training for people (possibly older people on using apps)
Infrastructure • Real time services fail - is there backup
Policy • State support to a single data structure
Finally, participants were asked to generate a set of co-creation contributions, again based on the
four scenarios. Participants were asked to list specific contributions that each actor or co-creator in
the scenarios could make, thereby highlighting opportunities for potential co-creation by means of a
pooling of resources, expertise, and ideas.
In total, participants generated 77 co-creation contributions, across the following categories: Data
provision, Expertise, Interpretation and Analysis, Modeling and Simulation, Networking and
Promotion, Policy Input, Software Input, and Visual Data. The co-creation contributions suggested by
partiicpants here, addressed all aspects of the co-creation stages discussed during the earlier options
stage. That is, the contributions in table below, which provides a sample of contributions from each
category, as well as reference to the specific scenario from which they were derived, contains
contributions which address various aspects of service creation (ideas), service engineering
(requirements, design), and service management (delivery, evaluation).
D4.6 Pilots Evaluation Results – Third Round
Page 146 of 215
Figure 44 - Percentage breakdown of co-creation contributions
Participants also engaged in discussion around the broader challenges of effective and fruitful co-
creation. It was highlighted that such complex scenarios, with multiple actors and contributions, may
require an incremental approach, beginning with a core and concrete starting point – a foundation
for additional actors and contributions to build upon.
This discussion then addressed the importance of validating and refining during development of co-
created services, with regular input from multiple actors and users. Given the core focus on co-
creation, this aspect was considered to be of great importance to participants.
Another point raised during the course of the discussion, consistent with some of the barriers
highlighted earlier, was the importance of accessibility and user-friendliness. It was suggested that
the platform on which the service is build must be easily accessible by citizens, for both contribution
of data and feedback.
Finally, and perhaps most crucially, the question of how to provide an environment that allows all of
the actors in a complex system, each with varying interests and levels of skills, and each providing
various kinds of information, to work together to co-create a service. This, it was agreed, is
something which will require deep thinking and consideration at each stage of the process.
D4.2 Evaluation Results – First Round
Page 147 of 215
Table 53 - Sample co-creation contributions
Co-creation contribution So that we can Category Scenario
Collect samples of water and shellfish to send to the
lab, upload private data, temperature etc.
Create maps and graphs Data Provision Drone Pilot David
Helicopter data and activity, vessel data & activity,
coastguard tool
Provide search support Data Provision Sergeant Steve
A specification for data service standards Develop an infrastructure to share data from different
creators in the app
Expertise Drone Pilot David
Support on information and data enrichment services Provide a service in which data are well described for
easy integration
Expertise Drone Pilot David
Expertise in identifying what is in the photographs Inform authorities of what is taking place in the
environment
Interpretation
and analysis
Drone Pilot David
Harmful algal bloom (HAB) analysis expertise and HABs
tool expertise
Conduct HAB's status evaluation, deliver status data to
aqua-culture and public, map reporting
Interpretation
and analysis
Drone Pilot David
Model outputs, specifically physical/ biochemistry
outputs
Provide video/cartoons of projections of water
movements, properties, and characteristics,
nowcast/forecast
Modeling and
simulation
Drone Pilot David
A numerical simulation of oil spills/ harmful algal
blooms
Provide evaluations of where the photographed things
will travel to
Modeling and
simulation
Drone Pilot David
Information in easy to understand language Provide a blog post with an overview of an
environmental issue using statistics
Networking and
Promotion
Drone Pilot David
D4.6 Pilots Evaluation Results – Third Round
Page 148 of 215
Activity, networking, promotion, lobbying Promote business and product, aid decision on potential
sites
Networking and
Promotion
Engineer Ed
Data analysis, expertise in policy Inform policy, access success, to inform data collection Policy input Captain Cillian
Influence, push for policy objectives, activity, provide
evidence
Consider other actors/interest groups, push for certain
policy objectives
Policy input Engineer Ed
A digital framework for awarding badges Develop a mechanism to provide general users with an
incentive to continue using the service
Software Input Drone Pilot David
A User Interface design Design an intuitive user interface hiding the underlying
service technology from the user
Software Input Drone Pilot David
Take photos/ describe visual photos Provide an alert system for risk levels (green, orange,
red)
Visual data Drone Pilot David
Use of my craft to assist with pictures/ videos as
required
Provide a data file of images over a defined period Visual data Drone Pilot David
This project has been funded with the support of the H2020 Programme of the European Union Copyright by the OpenGovIntelligence Consortium
.
4.1.6.1.2 Solutions for Pilot problems in Third Year
The problems identified were solved in the second year and considered as “done”. If not yet solved,
it was planned to be deployed in the third year and were considered as “in progress”.
Table 54 - Summary of Ideas and Solutions from participants – Third Year
Problems Solutions Actions Taken Progress
Accessibility
Issues
Lack of intuitive service design, such that
training is not required
Lack of open access data – can everyone who
wants to access the data get to it?
Data is being linked and will
be published for end-users
access.
Done in the
3rd Year.
Connectivity
Issues
Loss of data/connectivity during e.g. storms,
which is often the time when the client wants
to collect data
Failure of adequate internal/ broadband
availability nationwide
OGI ICT Toolkit can enable the
interoperability between data
and systems reducing this
issue.
Done in the
Second Year.
Quality Issues Lack of data quality control;
Lots of gaps in data, no guarantee of quality.
Data is linked following the 5
stars LOD.
Done in the
3rd Year.
Policy Issues
Legislative barriers and lack of
legislation;
Conflict between the multitude of
European directives on the marine
environment (e.g. MSFD, WFD,
Shellfish, Habitats, EMFF) in terms of a
harmonised data standard for
information describing the marine
environment.
This still have been internally
discussed.
In Progress
and out of
control of
OGI.
Education and
Training
Issues
Lack of adequate user experience in the
relevant technologies
Lack of adequate ongoing training
User experience was
improved and outcomes
section shows positive
reaction from end-users.
A video explaining how to use
was created and linked in the
app.
Done in the
3rd Year.
Finance Issues Lack of funding for the “processing” of
the data
Lack of financial resources
There is still no sustainable
business model for the LOSD.
Exploitation report D5.8
details the OGI plan for this
issue.
In Progress
and out of
control of OGI
Pilot.
D4.6 Pilots Evaluation Results – Third Round
Page 150 of 215
4.1.6.2 Pilot Co-Creation Framework Evaluation
As co-creation step for co-implementation, the Marine Institute started to employ online
vocabularies to describe its online data within existing ISO 19139 metadata content and aim to
deploy these same vocabularies in Linked Open Statistical Data (LOSD) within the
OpenGovIntelligence (OGI) Irish pilots. Vocabularies provide a list of standardised and consistent
terms over a broad range of disciplines (e.g. oceanography, geography and geology) that are of
relevance to describing things clearly for key stakeholders who wish to exploit marine LOSD through
the Irish pilots. Using standardised sets of terms (otherwise known as "controlled vocabularies") in
metadata and to label LOSD solves the problem of ambiguities associated withdata markup and
enables records to be interpreted as Linked Data by machines. This opens up data sets to a whole
world of possibilities for computer aided manipulation, distribution and long term reuse.
An example of how the Irish OGI pilots may benefit from the use of controlled vocabularies is in the
using of values taken from different data sets. For instance, one data set may have a column labelled
"Temperature of the water column" and another might have "water temperature" or even
"temperature". To the human eye, the similarity is obvious but a computer would not be able to
interpret these as the same thing unless all the possible options were hard coded into its software.
If data are marked up with the same terms, this problem is resolved. Sea temperature is an
important variable in understanding the real-time conditions of the marine environment which
affect scenarios such as search and rescue because the temperature of the water column indicates
potential survival time rates for an individual requiring immediate rescue from emergency services.
The Climate and Forecast vocabulary (P07) has a mailing list which can be used to request new
terms and new terms are added, modified and deprecated on a near monthly basis. The British
Oceanographic Data Centre facilities Co-Creation from the network of SeaDataNet distributed
marine partner consortium from more than 35 countries in managing this vocabulary.
D4.6 Pilots Evaluation Results – Third Round
Page 151 of 215
4.1.6.3 Data Quality Evaluation
Table 55 – Data Quality Survey Results – Third Year
Data Quality
Attributes Results Overview
Discoverability and
Usability
1- App has links to the original data sets;
2- App has no searchable list of data sets used;
3- App allows users to download the data sets, including partial download of selected
variables;
4- App has multiple format download option;
5- App allows to user preview and interact with the data.
Granularity 1- There has aggregate level of data and unite level.
Intelligibility 1- Supporting documentation can be found at the same time as the data.
Trustworthiness
1- The data sets collected are well-known by developers and technical partners has
confidence on how it was processed;
2- The data is 100% reliable, and is consistent with external sources.
Linkable to other
data
(5 Stars LOD)
1- The data sets are available in different formats to different audiences, from XLS to CSV.
RDF was created by OGI developers;
2- The RDF is not yet properly linked to other external data to provide context. Once the
data is converted to RDF and stored in the Trafford Data Store it will be 'linkable'.
15 Open Data
Principles
1- Data is complete;
2- Data is primary;
3- There is not issues about timely availability;
4- Data is accessible to the widest range of users (XLS, CSV and RDF)
5- Data is non-discriminatory because is freely available;
6- Data has license-free (CLS, CSV, RDF);
7- Data is permanent;
8- Data is safe to open;
9- Data is not 100% designed with public input (decide format and parts of data sets to
visualize and download).
D4.6 Pilots Evaluation Results – Third Round
Page 152 of 215
Table 56 – Data Quality Improvement Progress
Data Quality
Attributes
Status
1st Year 2nd Year 3rd Year
Discoverability
and Usability App in development
App developed, but not
completely ready. App allows
for access to data but doesn’t
allow downloading
Updated, there is download
and slicing option for end-users
of the Irish App.
Granularity
Diverse data sets of high
quality with non-aggregate
data
Diverse data sets of high
quality with non-aggregate
data
Updated, Four datasets were
combined resulting in 2
datasets linked in aggregate
format and unit level.
Intelligibility Good intelligibility with
documentation available.
Good intelligibility with
documentation available.
Linking of datasets could be
better
Updated, existence of
documentation found at the
same time as the data.
Trustworthiness
Data Quality is good, from
trusted source. Data validity
can be controlled.
Current scenario with some
improvements. Increase data
quality with data cubes linking
data sources however without
proper context from external
sources.
Updated. Technical partners
and end-users pointed via
survey the acceptance of
datasets quality.
Linkable to
other data
(5 Stars LOD)
Not following all the 5 stars
LOD principles, no own data
cubes
Not following all the 5 Stars
LOD principles.
Updated. Following the 5 stars
LOD.
15 Open Data
Principles
Following 8 of 9 OD
Principles
Following 8 of 8 OD principles
that matches OGI needs.
Updated. Following 8 of 9 OD
principles that matches OGI
needs.
D4.6 Pilots Evaluation Results – Third Round
Page 153 of 215
Table 57 – Data Quality Improvement Progress
Data Quality
Attributes Summary of Benefits Achieved Summary of Challenges Faced
Discoverability
and Usability
The discoverability and usability
was improved as the app was
developed further.
Data can be downloaded and combined with new
visual functionalities improved level of acceptance by
end-users (check outcomes section).
Granularity Same high level of granularity Both unit and aggregate level are presented in the
new functionalities delivered.
Intelligibility Data contains supporting
documentation
There is documentation, however, it is close to the
data.
Trustworthiness Data is still very trustworthy and
can be validated easily
Data is reliable and trusted. Outcomes sections
shows high acceptance from users and technical
partners.
Linkable to
other data
(5 Stars LOD)
Data is still available in non-
proprietary format
Since data is linked, achieved the max level of this
dimension.
15 Open Data
Principles
Data is in accordance with 8 of 15
Open Data principles There is no public input.
4.1.6.3.1 Data Sets
The list of data sets is presented below:
Irish Weather Buoy Network
Irish Wave Buoy Network
Irish National Tide Gauge Network
East Atlantic SWAN Wave Model
The data sets are already in RDF (LOSD format). The link to access data seta are at:
https://drive.google.com/open?id=0B0_fFuauifo0R2RLR0M1dThLMWs
D4.6 Pilots Evaluation Results – Third Round
Page 154 of 215
4.1.6.4 Outcomes Evaluation
To evaluate the application …
1. Linked Open Statistical Dataset (LOSD) Familiarity;
2. User Acceptance;
3. Datasets Quality; and,
4. Perceived Outcomes.
D4.6 Pilots Evaluation Results – Third Round
Page 155 of 215
4.1.6.4.1 Linked Open Statistical Data (LOSD) Familiarity
Figure 45 – LOSD Familiarity – Marine Institute Pilot
From the survey results, we can see that majority of Marine Institute App respondents stated that
they familiar with open data and open data portals, 83% and 73% respectively. Similar positive
reaction were found for familiarity with the app datasets. However, 65% of the respondents claimed
that they are not familiar with LOSD.
This result brings opportunity for the Irish pilot partner to introduce the LOSD to wider audience by
using the app so in the future more people can be aware about LOSD, thus create more benefits to
the society as seen in the perceived outcomes section.
D4.6 Pilots Evaluation Results – Third Round
Page 156 of 215
4.1.6.4.2 User Acceptance
Figure 46 – User Acceptance – Marine Institute Pilot
From the results, we should highlight that in the major part of the categories tested in this user
acceptance section, majority of the respondents gave positive reactions in terms of technical
functionality: the app provides a proper technical functionality (85%), functionality of the app
working properly (92%), clear and understandable interaction with the app (100%), functionality
well-integrated (94%), and the app provides adequate design (96%). However, 100% of the
respondents also stated that they experienced with a crash when using the app.
Regarding the easiness, 96% of the respondents stated that the app is easy to use; with less figure,
73% of the respondents stated the app is not difficult to operate. 96% of the respondents claimed
that they have sufficient skills to use the app even though 90% of the respondents said that it
requires high level technical knowledge.
From these section, improvement should be made in terms of fixing the crash, and make it less
technical to facilitate the use of the app for broader group of users.
D4.6 Pilots Evaluation Results – Third Round
Page 157 of 215
4.1.6.4.3 Datasets Quality
Figure 47 – Datasets Quality – Marine Institute Pilot
In terms of data quality, majority of the respondents claimed that the app already provided data
accessibility (98%), data accuracy (100%), all required datasets from co-creation process (87%), and
less time to find the data (95%). However, only 56% of the respondents stated that the data
provided by the app are incomplete, this can be an opportunity for the improvement.
Last, 90% of the respondents satisfy with the level of data quality provided by the Marine Institute
app.
D4.6 Pilots Evaluation Results – Third Round
Page 158 of 215
4.1.6.4.4 Perceived Outcomes
Figure 48 – Perceived Outcomes – Marine Institute Pilot
For the last section of the questionnaire, we were asking about the perceived outcomes of using the
app. For all of the potential outcomes, more than 90% of the respondents gave the positive reaction.
In addition, 90% of the respondents also asked more functions in the app to create even better
transparency.
D4.6 Pilots Evaluation Results – Third Round
Page 159 of 215
4.1.6.5 Screenshots of Marine Institute Application
The next figure presents the screenshot of Marine Institute Application:
Figure 49 - Average Bar chart of Irish National Tide Gauge Network data set
D4.6 Pilots Evaluation Results – Third Round
Page 160 of 215
4.1.7 Overall Results From Pilots’ Evaluation
In this section, we combine all the responses from 6 pilots, to see users’ perception about
applications provided by OGI project.
4.1.7.1 Overall Linked Open Statistical Data (LOSD) Familiarity
Figure 50 – Overall LOSD Familiarity
First, regarding familiarity of the technology, more people are familiar with open data, open data
portal and the datasets provided by all of the apps. On the other hand, less people are familiar with
LOSD.
D4.6 Pilots Evaluation Results – Third Round
Page 161 of 215
4.1.7.2 Overall User Acceptance
Figure 51 – Overall User Acceptance
From the overall user acceptance evaluation, there are two things should be highlighted: first, even
though the apps already provided good functions and design, majority of respondents still
experienced with a crash when using the apps; second, there is a contradiction regarding the
easiness to use the apps, 94% of the respondents claimed that the apps are easy to use but only 36%
stated that the apps are not difficult to use. Each pilot partners should take a deeper qualitative
investigation to identify solution for both issues.
D4.6 Pilots Evaluation Results – Third Round
Page 162 of 215
4.1.7.3 Overall Datasets Quality
Figure 52 –Overall Datasets Quality
From the results in data quality section, the main issues in data quality provided by the OGI apps are
regarding the data completeness and meeting the required datasets from co-creation process.
D4.6 Pilots Evaluation Results – Third Round
Page 163 of 215
4.1.7.4 Overall Perceived Outcomes
Figure 53 – Overall Perceived Outcomes
Majority of the respondents claimed that the OGI apps are useful and bring benefits, related to
better interpretation of the data, help them to make a better decision, help them in achieving their
goals, more efficient (time and cost), and increase transparency.
However, majority of the respondents asked to add more functions to create more transparency.
In terms of acceptance, 90% of the respondents claimed that they accept the app and 95% of the
respondents stated that their colleagues are also accept the apps.
D4.6 Pilots Evaluation Results – Third Round
Page 164 of 215
4.2 OGI ICT Toolkit Evaluation Results
This section shows the OGI ICT Toolkit Evaluation Results, including the evaluation of internal and
external evaluation. The internal evaluation contains:
1. OGI Data Quality;
2. OGI Architecture; and,
3. System Quality of Application.
The external evaluation includes the OGI ICT toolkit evaluation made external users and experts.
Figure 54 – Internal and External Evaluation Processes
D4.6 Pilots Evaluation Results – Third Round
Page 165 of 215
4.2.1 Internal Evaluation
4.2.1.1 Data Quality Evaluation
The LOSD OGI Data Quality evaluation was made within the technical partners in charge to design
and develop the six OGI pilots applications. The questionnaire is presented in the section 7.2,
containing 29 questions using a Likert scale of 5 level, from strongly disagree to strongly agree. This
survey evaluated the follow dimensions of LOSD OGI Data Quality:
1. Discoverability and Usability;
2. Granularity Section;
3. Intelligibility Section;
4. Trustworthiness Section;
5. Linkable to other data Section; and,
6. Open Data Principles Section.
D4.6 Pilots Evaluation Results – Third Round
Page 166 of 215
4.2.1.1.1 Discoverability and Usability
Figure 55 – Discoverability and Usability – Overall of All Pilots
From this discoverability and usability section, all of the pilots provided a pre-release version of pilot
app. 4 pilots have already provided links to data sources and searchable list of data sources. 5 pilots
have already provided option to download the data, option to download selected parts of the data
and allowed users to preview and interact with data. Last, 3 pilots have already provided option to
download in multiple formats.
D4.6 Pilots Evaluation Results – Third Round
Page 167 of 215
4.2.1.1.2 Granularity Section
Figure 56 – Granularity Section – Overall of All Pilots
The pilots’ apps provided different level of data granularity. All of the pilots provided at least
aggregated data. It means that the data is in statistical format but only provided as percentage or
average. On the other hand, some of the pilots also could provide the raw data (in individual unit
level), for example hourly data as unit level and daily data in average level.
D4.6 Pilots Evaluation Results – Third Round
Page 168 of 215
4.2.1.1.3 Intelligibility Section
Figure 57 – Intelligibility Section – Overall of All Pilots
Documentation let the users to understand in detail the datasets, providing context of the data and
how the data could be used. Most the datasets apps be equipped with the documentation.
However, in some datasets the documentation is provided separately, for example, by an external
provider (the owner of the data).
On the other hand, 1 pilot included the documentation in the visualization, and another pilot is still
without documentation.
D4.6 Pilots Evaluation Results – Third Round
Page 169 of 215
4.2.1.1.4 Trustworthiness Section
Figure 58 – Trustworthiness Section – Overall of All Pilots
Regarding the trustworthiness, there is 1 pilot which unsure how their data are collected, and 1
pilot which unsure how the data is processed.
In other categories all pilots stated that they already used trusted sources, and provided reliable and
consistency with external sources data.
D4.6 Pilots Evaluation Results – Third Round
Page 170 of 215
4.2.1.1.5 Linkable to other data Section
Figure 59 –Linkable to other Datasets Section – Overall of All Pilots
Related to the 5-star deployment scheme for LOD, 2 pilots already reached level 5 while the rest on
level 3.
D4.6 Pilots Evaluation Results – Third Round
Page 171 of 215
4.2.1.1.6 Open Data Principles Section
Figure 60 – Open Data Principles Section – Overall of All Pilots
Regarding open data principles, all of the pilots claimed that they already provide complete data,
primarily data, wide range of users data, non-discriminatory data, license-free data, permanent data,
safe to open data and designed with public input. While there is one pilot have not provided timely
data.
D4.6 Pilots Evaluation Results – Third Round
Page 172 of 215
4.2.1.2 OGI Architecture Evaluation
The LOSD OGI Architecture evaluation was made within the 7 technical partners in charge to design
and develop the six OGI pilots. The questionnaire is presented in the section 7.2, containing 29
questions using a Likert scale of 5 level, from strongly disagree to strongly agree. This survey
evaluated the follow dimensions of LOSD OGI Reference Architecture:
1. User Behaviour;
2. Perceived Usefulness;
3. Perceived Ease of Use;
4. Computer Self-efficacy, Perceptions of External Control, Computer Playfulness, Computer
Anxiety, Perceived Enjoyment, Objective Usability; and,
5. Result Demonstrability, Behavioural Intention, Use, Present Satisfaction, Future Desire.
D4.6 Pilots Evaluation Results – Third Round
Page 173 of 215
4.2.1.2.1 Use Behaviour
Figure 61 – Use Behaviour of LOSD OGI Architecture
From Figure 61, 6 out of 7 partners claimed that they referred to the LOSD OGI architecture when
designed the pilot apps.
D4.6 Pilots Evaluation Results – Third Round
Page 174 of 215
4.2.1.2.2 Perceived Usefulness
Figure 62 – Perceived Usefulness of LOSD OGI Architecture
In the 5 perceived usefulness parameters, 100% partners perceived that the LOSD OGI architecture
is useful in designing the pilot apps: increase productivity, help them to reach their objectives, help
them to be aware of data quality issue, help them to design and build flexible architecture and
provide better requirements.
5 out of 7 partners claimed that the LOSD OGI Architecture is useful to deal with data quality.
D4.6 Pilots Evaluation Results – Third Round
Page 175 of 215
4.2.1.2.3 Perceived Ease of Use
Figure 63 – Perceived Ease of Use of LOSD OGI Architecture
All of partners stated that the LOSD OGI architecture is clear and understandable, as well as help
them in designing linked-open data system. 6 out of 7 partners claimed that the LOSD OGI
architecture helps them to design LOD system with less cost.
D4.6 Pilots Evaluation Results – Third Round
Page 176 of 215
4.2.1.2.4 Computer Self-efficacy, Perceptions of External Control, Computer Playfulness, Computer Anxiety, Perceived Enjoyment, Objective Usability
Figure 64 – Computer Self-efficacy, Perceptions of External Control, Computer Playfulness, Computer Anxiety, Perceived Enjoyment, Objective Usability of LOSD OGI Architecture
This section addresses partners’ computer skills. All of the partners stated that they are fluent using
computers, do not have a problem working with any software, confident to design and develop their
own architecture, enjoy using a reference architecture and feel that the reference architecture
accelerates their work in achieving the goals.
In addition, 6 out of 7 partners stated that they have fun using computer, while 5 out of 7 claimed
they are confident in working with computers.
D4.6 Pilots Evaluation Results – Third Round
Page 177 of 215
4.2.1.2.5 Subjective Norm, Voluntariness, Image, Job Relevance, Output Quality
Figure 65 –Subjective Norm, Voluntariness, Image, Job Relevance, Output Quality of LOSD OGI Architecture
In terms of independency, all of the partners stated that their partners suggest them to use OGI
architecture, it is their choice to use OGI architecture to build the pilot app, and they have freedom
to choose to use or not to use the OGI architecture; moreover, there 5 partners stated that their
partners will think highly if they decide to use OGI architecture.
In terms of job relevance, 6 partners agreed that the use of OGI architecture is pertinent to their job-
related tasks, 6 partners stated that data quality has improved after using the OGI architecture and
all partners agreed that flexibility has improved after using OGI architecture.
D4.6 Pilots Evaluation Results – Third Round
Page 178 of 215
4.2.1.2.6 Result Demonstrability, Behavioral Intention, Use, Present Satisfaction, Future Desire
Figure 66 –Result Demonstrability, Behavioural Intention, Use, Present Satisfaction, Future Desire of LOSD OGI Architecture
In the last part, all of the partners claimed that they can explain the benefits of using the OGI
architecture, would like to use the OGI architecture, and satisfy with current OGI architecture. In
addition, only 3 out of 7 partners welcome a different OGI architecture.
D4.6 Pilots Evaluation Results – Third Round
Page 179 of 215
4.2.2 External Evaluation
Two external evaluations of OGI ICT Toolkit were made in the second year of project. The first in
Belgium, 16th March of 2017 and the second in 4th of September of 2017. The results are presented
below in the sections .
4.2.2.1 NTTS Workshop
The workshop was held in Brussels, in 16th March. The official web site for this conference is at:
https://ec.europa.eu/eurostat/cros/content/ntts-2017_en .
The workshop is described in the blog post at OGI Medium:
https://medium.com/opengovintelligence/hands-on-workshop-on-linked-open-statistical-data-lod-
7b9a8066f3e0 .
Figure 67 – NTTS Hands-on OGI ICT Toolkit Workshop
4.2.2.1.1 User Information
Most of the users who evaluated the OGI Toolkit in this workshop were working for some part of the
government or state-owned companies. Ages varied between 26 and 55, with only two respondents
being older and none being younger than this age group. They were highly educated and skilled
workers, who are however not very familiar with linked statistical data cubes. They seemingly were a
good representative set of end-users typical for most pilot applications, with a nice mix of managers,
statisticians, data analysts and administrative staff.
4.2.2.1.2 OGI Toolkit Evaluation
D4.6 Pilots Evaluation Results – Third Round
Page 180 of 215
The respondents were not familiar with any competitors to the OGI Toolkit at first sight. The one
exception judged the OGI Toolkit to be better than its competitors, but did not name any specific
competitive software or toolkit. Some attractive functionalities named were its combination of
regional data, its user friendliness and the ability to save (a portion of) useful data. The interface was
judged to be slightly unattractive though. What stands out here is that most respondents however
chose not to name any attractive or unattractive feature. The basic skills necessary to use the OGI
Toolkit were said to be database browsing, domain knowledge, SparQL skills and statistics. The
respondents named different kinds of settings to teach the OGI Toolkit, with a slight majority naming
MOOCs, but both text-based education in the form of a book or a classroom course were also
named.
In terms of the results of the OGI Toolkit, most respondents were overall fairly neutral. They
however thought the OGI Toolkit design was not yet adequate and were not happy with how clear
and understandable their interaction with the OGI Toolkit was. Respondents were unconvinced that
the OGI Toolkit led to either an increase in transparency, better output or that it was useful in their
job, although they also were not convinced that it did not have value in these areas whatsoever. The
same can be said for the acceptance of the OGI Toolkit by senior management, whether they would
encourage it to their colleagues or if it would be accepted by their teams.
The respondents overall feelings on the Toolkit can best be summarized as being neutral: they didn’t
feel the OGI Toolkit was amazing and would be a significant aid in their job, but they also didn’t
come out and say it was that bad. A more refined product that has more data could very much swing
the opinion more positive.
4.2.2.2 EGOV/IFIP Hands-on Workshop
The IFIP/EGOV is an academic conference in the area of electronic government. The conference was
held in Saint Petersburg, 4th of September. The official web site of this conference is at:
http://faculty.washington.edu/jscholl/ifip.working.group.8.5/website/conference/#/ .
4.2.2.2.1 User Information
The respondents in this workshop were for the overwhelming majority working in an academic
setting. They were aged 26-55, except for one respondent who was older. They were very highly
educated, with a majority of respondents attaining a Ph.D. and everyone else completing a Master’s
Degree. As a result about half were already familiar with linked statistical data cubes. They seem to
overall be mostly academic researchers and professors who will be working with the OGI Toolkit in
an academic setting. As such, they could both be end-users and co-developers of the OGI Toolkit and
its applications. All respondents were male.
4.2.2.2.2 OGI Toolkit Evaluation
The respondents named some competitors of the OGI Toolkit: R and IBM’s Watson were named
among others. The respondents who named competitors judged the OGI Toolkit to be better than its
D4.6 Pilots Evaluation Results – Third Round
Page 181 of 215
competitors at its assigned purpose. Some attractive functionalities of the OGI Toolkit were judged
to be its visualization, its usability and the relative ease of using statistical data. The user interface
was found to be one of the features that need improvement, as well as a lack of SPARQL queries and
descriptive statistics. The basic skills necessary to use the OGI Toolkit as identified by the
respondents were basic and advanced statistics, basic software engineering and database
management. MOOCs were overwhelmingly named as the most effective way to teach the OGI
Toolkit.
The results of the OGI Toolkit were positively judged by the respondents. They found for example
that the OGI Toolkit design was adequate. They found the OGI interactions to be clear and
understandable and thought that the OGI Toolkit led both to an increase of transparency and better
output and found it to be useful in their job. They did think that the OGI Toolkit requires a high
amount of knowledge, and despite their educated background were overall fairly neutral in judging
if they themselves were skilled enough to use the Toolkit, although there was some variation
amongst the respondents. They were also neutral when asked if the OGI Toolkit would be accepted
by their teams, despite them answering they would encourage the use of the OGI Toolkit to their
colleagues.
D4.6 Pilots Evaluation Results – Third Round
Page 182 of 215
4.2.2.3 Overall External Impression of OGI ICT Toolkit
The two workshops had very different respondents, which allows for a good overview and
comparison between these two sets. The existing functionalities of the OGI ICT Toolkit were overall
rated quite highly, although more functionalities need to be included. The user interface and SPARQL
queries need to be improved. Respondents need to be skilled in statistics and software engineering
to make full use of the API. The NTTS workshop participants were less content with the results from
the API than the participants of the EGOV/IFIP workshop and they disagreed very much on this
section, leading credence to the hypothesis that the OGI ICT Toolkit was still an academic project not
quite ready for non-academic end-users at the time of the workshop. Better functions and more
user friendliness could improve the transparency effects and overall output of the OGI ICT Toolkit.
D4.6 Pilots Evaluation Results – Third Round
Page 183 of 215
5 Conclusions
The evaluation has four areas: Co-creation, ICT Toolkit, Acceptance of ICT Toolkit and Outcomes. In
this report the evaluation methodology has been updated based on the insights gained from the first
and second year.
The co-creation was evaluated using the same framework used during first and second year, to
compare the evolution and changes made on pilots influenced by the participation of pilot’s
stakeholders. The reason is that most of the co-creation aspects already took place in the first couple
of years.
The ICT toolkit evaluation has been updated to include questions in the questionnaire to evaluate
the new tools which developed after the evaluation in the second year. Finally, the part about data
quality is new, given its importance.
The 6 pilots were evaluated by employing a user – questionnaire. Most users of 219 people surveyed
found the pilots benefits include the creation of transparency, reduction the administrative burden
by more efficient search of information and visualizing the results at a glance. We considered
positive reactions strongly agree, agree and neutral answers. Negative reactions were disagree and
strongly disagree. Below the descriptive statistics of Transparency, Decision-Making, Efficiency and
Better Data Interpretation:
The overall acceptance of the pilots’ apps was 90%;
92% pointed out an increase of Transparency after accessing the Pilots’ Apps;
The Pilots’ apps helped to 94% of surveyed people have better decision-making;
93% of people answered there is a better interpretation of Data;
For 92% of respondents, Pilots’ app reduced the time spent searching information;
94% of end-users identified a cost reduction when use OGI pilots’ apps; and,
There is an increase of efficiency to 91% of OGI Pilots’ users.
Developing the ICT-toolkits more time than expected, as this is a new field in which there are hardly
any examples. The development of the building block encountered the questions of which level of
granularity would be fine as this requires a trade-off between the flexiblity given to the developer
D4.6 Pilots Evaluation Results – Third Round
Page 184 of 215
and ease-of-use of manipulating data. User-friendliness and having right granularity is a trade-off
and developers have different requirements in this regard.
Co-creation remains challenging in government. Co-creation is bottom-up, whereas government
traditionally takes a more top-down approach by developing pilots and providing the services.
Realizing co-creation is not only a technological problem, but rather a cultural problem.
The end-users regard the pilots as successful as shown in the figure below. The pilots show that data
quality is a key aspect as garbage in is garbage out. Realizing high-data quality took often more time
than expected, whereas the OGI ICT-toolkit enables to develop the pilots within a short time frame.
They pilots help end-users to find information faster and in a more efficient way resulting in
transparency and better decision-making quality.
D4.6 Pilots Evaluation Results – Third Round
Page 185 of 215
6 References
Dodds, Leigh (2016). Exploring Open Data Quality. Open Data Institute Website
(https://theodi.org/blog/exploring-open-data-quality). Published in: 26/10/2016. Accessed in:
26/01/2018.
Barth, Michael (2012). API Evaluation: An Overview of API Evaluation Techniques. Michael Bart
Blog A portfolio Blog about Software Development (https://www.michaelbarth.net/). Paper
available at: https://www.michaelbarth.net/files/publications/api-evaluation.pdf. Published in 2012.
Accessed in: 28/01/2018.
OPENGOVDATA (2018). Eight principles of open government data. Available at: The annotated 8
Principles of Open Government Data Website (https://opengovdata.org/) from a meeting in 7th-8th
December 2007. Accessed in: 26/01/2018.
Bizer, C., Heath, T., & Berners-Lee, T. (2009). Linked data-the story so far. Semantic services,
interoperability and web applications: emerging concepts, 205-227.
Frank, M., & Walker, J. (2016). User Centred Methods for Measuring the Value of Open Data. The
Journal of Community Informatics, 12(2).
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-
7.
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability
scale. Intl. Journal of Human–Computer Interaction, 24(6), 574-594.
Joyce, E., & Kraut, R. E. (2006). Predicting continued participation in newsgroups. Journal of
Computer-Mediated Communication, 11(3), 723-747.
Glasow, P. A. (2005). Fundamentals of survey research methodology. Retrieved January, 18, 2013.
Levy, Y. and T. J. Ellis (2006). A systems approach to conduct an effective literature review in
support of information systems research. Informing Science: International Journal of an Emerging
Transdiscipline 9(1): 181-212.
Delbecq, Andre L., Andrew H. Van de Ven, and David H. Gustafson. Group techniques for program
planning: A guide to nominal group and Delphi processes. Scott Foresman, 1975.
20510:2010, ISO/IEC. (2010). Systems and Software Engineering—Systems and Software Product
Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models. Geneva,
International Organization for Standardization.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of
information technology.MIS quarterly: 319-340.
Davis, F. D., R. P. Bagozzi and P. R. Warshaw (1989). User acceptance of computer technology: a
comparison of two theoretical models. Management science 35(8): 982-1003.
D4.6 Pilots Evaluation Results – Third Round
Page 186 of 215
DeLone, W. H. and E. R. McLean (1992). Information systems success: The quest for the dependent
variable. Information systems research 3(1): 60-95.
Delone, W. H. and E. R. McLean (2003). The DeLone and McLean model of information systems
success: a ten-year update. Journal of management information systems 19(4): 9-30.
Venkatesh, V. and H. Bala (2008). Technology acceptance model 3 and a research agenda on
interventions. Decision sciences 39(2): 273-315.
Venkatesh, V. and F. D. Davis (2000). A theoretical extension of the technology acceptance model:
Four longitudinal field studies. Management science 46(2): 186-204.
Venkatesh, V., M. G. Morris, G. B. Davis and F. D. Davis (2003). User acceptance of information
technology: Toward a unified view. MIS quarterly: 425-478.
Ward, R. (2013). The application of technology acceptance and diffusion of innovation models in
healthcare informatics. Health Policy and Technology 2(4): 222-228.
Ward, S. J. (2014). The magical concept of transparency. Ethics for digital journalists: Emerging best
practices: 45-58.
Bovens, M. (2007). Analysing and assessing accountability: a conceptual framework. European law
journal 13(4): 447-468.
Coglianese, C. (2009). The transparency president? The Obama administration and open
government. Governance 22(4): 529-544.
Frank, M. and A. A. Oztoprak (2015). Concepts of Transparency: Open Data in UK Local Authorities.
Proceedings of the International Conference for E-Democracy and Open Government 2015
(CeDEM15).
Peixoto, T. (2013). The Uncertain Relationship between Open Data and Accountability: A Response
to Yu and Robinson's' The New Ambiguity of Open Government'.
Matheus, R. and M. Janssen (2013). Transparency of civil society websites: towards a model for
evaluation websites transparency. Proceedings of the 7th International Conference on Theory and
Practice of Electronic Governance, ACM.
Matheus, R. and M. Janssen (2015). Transparency Dimensions of Big and Open Linked Data. Open
and Big Data Management and Innovation. M. Janssen, M. Mäntymäki, J. Hidders et al., Springer
International Publishing. 9373: 236-246.
Matheus, R., M. M. Ribeiro and J. C. Vaz (2012). New perspectives for electronic government in
Brazil: The adoption of open government data in national and subnational governments of Brazil.
Proceedings of the 6th International Conference on Theory and Practice of Electronic Governance,
ACM.
Fox, J. (2007). The uncertain relationship between transparency and accountability. Development
in Practice 17(4-5): 663-671.
D4.6 Pilots Evaluation Results – Third Round
Page 187 of 215
Gallo, C., M. Giove, J. Millard, R. Kare and V. Thaarup (2014). Study on eGovernment and the
Reduction of Administrative Burden. Luxembourg, Luxembourg. doi 10: 42896.
Law, E. L.-C., V. Roto, M. Hassenzahl, A. P. Vermeeren and J. Kort (2009). Understanding, scoping
and defining user experience: a survey approach. Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems, ACM.
Hassenzahl, M. and N. Tractinsky (2006). User experience-a research agenda. Behaviour &
information technology 25(2): 91-97.
Forlizzi, J. and K. Battarbee (2004). Understanding experience in interactive systems. Proceedings of
the 5th conference on Designing interactive systems: processes, practices, methods, and techniques,
ACM.
Garrett, J. J. (2010). Elements of user experience, the: user-centered design for the web and
beyond, Pearson Education.
Jokela, T., N. Iivari, J. Matero and M. Karukka (2003). The standard of user-centered design and the
standard definition of usability: analyzing ISO 13407 against ISO 9241-11. Proceedings of the Latin
American conference on Human-computer interaction, ACM.
Travis, D. (2011). ISO 13407 is dead. Long live ISO 9241-210. London, UK: Userfocus Ltd., June 6.
Albert, W. and T. Tullis (2013). Measuring the user experience: collecting, analyzing, and presenting
usability metrics, Newnes.
D4.6 Pilots Evaluation Results – Third Round
Page 188 of 215
7 Annexes
7.1 End-User Questionnaires
7.1.1 Pilot 1 – The Greek Ministry of Administrative Reconstruction (Greece) End-User Questionnaire
OGI Evaluation Questionnaire MAREG Pilot App End-users – 2018
This questionnaire is aimed to be filled by End-Users who have used the applications developed within the pilots of H2020 OpenGovIntelligence (OGI) project. When answering the questions, the application developed using the OGI Toolkit, named here as MAREG Pilot App, should be taken into account. The following topics will be asked:
General background information; Data sets; and, Resulting Application (App developed using the OGI toolkit).
More Information about MARE Pilot APP (http://wapps.islab.uom.gr/CubeVisualizer/vehicles/)
A. General Background Information 1. How familiar are you with open data applications? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar
2. What is your role or position in terms of Vehicles Cube App? A: ____________________________________________________________________________
B. Acceptance of the Vehicles Cube App 3. The Vehicles Cube App provides all the functionalities I am interested in ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
4. All functions in the Vehicles Cube App work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
5. The Vehicles Cube App recovers well from crashes ( ) The Vehicles Cube App didn’t crash ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
6. The Vehicles Cube App helps me to achieve my goals ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
7. My interaction with the Vehicles Cube App was satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
8. The Vehicles Cube App is useful to me ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
9. My interaction with the Vehicles Cube App is clear and understandable
D4.6 Pilots Evaluation Results – Third Round
Page 189 of 215
( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
10. The Vehicles Cube App design is adequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
11. I have sufficient skills to use the Vehicles Cube App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
12. The Vehicles Cube App does not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
13. The Vehicles Cube App will be accepted by my peers ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
14. Using the Vehicles Cube App can be hard for the average user ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
15. The Vehicles Cube App is easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about acceptance? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
C. Vehicles Cube App Data Sets
16. The data in the Vehicles Cube App are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
17. The data in the Vehicles Cube App is inadequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
18. The data in the Vehicles Cube App are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
19. A lot of time is needed to find the information I am looking for regarding vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
20. I am able to find all the information I'm looking for when using the Vehicles Cube App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about data sets? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
D4.6 Pilots Evaluation Results – Third Round
Page 190 of 215
D. Vehicles Cube App results 21. The Vehicles Cube App has a clear visualization ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
22. The Vehicles Cube App helps to do my job about government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
23. The Vehicles Cube App results in an increase of transparency of government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
24. The Vehicles Cube App is too complex to acquire knowledge into government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
25. The Vehicles Cube App helps me make better decisions about government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
26. The Vehicles Cube App helps my understanding of the government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
27. The Vehicles Cube App will increase efficiency of government vehicles management ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
28. The Vehicles Cube App reduces time spent looking for information about government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
29. The Vehicles Cube App reduces the costs to search and decide about government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
30. More functions in the Vehicles Cube App are needed to create transparency of government vehicles ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about results? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
Thank you for your time to answer the questions!
D4.6 Pilots Evaluation Results – Third Round
Page 191 of 215
7.1.2 Pilot 2 – Enterprise Lithuania (Lithuania) End-User Questionnaire
OGI Evaluation Questionnaire
Lithuania Enterprise App End-users – 2018
This questionnaire is aimed to be filled by End-Users who have used the applications developed within the pilots of H2020 OpenGovIntelligence (OGI) project. When answering the questions, the application developed using the OGI Toolkit, named here as Lithuania Enterprise App (LE App), should be taken into account. The following topics will be asked:
General background information; Data sets; and, Resulting Application (App developed using the OGI toolkit).Ń
A text explaining more information about Lithuania Enterprise App (http://lithuania.opengovintelligence.eu/).
A. General Background Information 1. Please select your age: ( ) Less than 21 ( ) 21 – 40 ( ) 41 – 60 ( ) Over 60 ( ) Don’t want to share
2. How familiar are you with open data applications? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar
3. What is your role using the LE App? ( ) Investors ( ) Citizens ( ) Civil servants ( ) Entrepreneurs ( ) Other ______________________________________________________________________
B. Acceptance of the Lithuania Enterprise Pilot App 4. The LE App provides all the functionalities I am interested in ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
5. All functions in the LE App work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
6. The LE App is stable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
7. The LE App helps me to achieve my goals ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
8. My interaction with the LE App was satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
D4.6 Pilots Evaluation Results – Third Round
Page 192 of 215
9. The LE App is useful to me ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
10. My interaction with the LE App is clear and understandable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
11. The LE App design is adequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
12. I have sufficient skills to use the LE App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
13. The LE App does not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
14. The LE App will be accepted by my peers ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
15. Using the LE App can be hard for the average citizen ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
16. The LE App is easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about acceptance? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
C. Lithuania Enterprise App Data Sets 17. The data in the LE App are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
18. The data in the LE App is incomplete ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
19. The data in the LE App are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
D4.6 Pilots Evaluation Results – Third Round
Page 193 of 215
20. A lot of time is needed to find the right data in the LE App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
21. I am able to find all the data I am looking for when using the LE App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about data sets? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
D. E. Lithuania Enterprise Pilot Apps RESULTS
22. The LE App has a clear visualization ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
23. The LE App helps to create insights of Lithuanian business environment ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
24. The LE App results in an increase of transparency of Lithuanian business environment ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
25. The LE App is too complex to gain insight of Lithuanian business environment ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
26. The LE App helps me to make better decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
27. The LE App helps my understanding of the subject matter ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
28. The LE App will increase efficiency ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
29. The LE App reduces time spent looking for information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
30. The LE App reduces the costs to search and decide ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
31. More functions in the LE App are needed to create transparency of Lithuanian business environment
D4.6 Pilots Evaluation Results – Third Round
Page 194 of 215
( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about results? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
Thank you for your time to answer the questions!
D4.6 Pilots Evaluation Results – Third Round
Page 195 of 215
7.1.3 Pilot 3 – Tallinn Real Estate (Estonia) End-User Questionnaire
OGI Evaluation Questionnaire Tallinn Real Estate App End-users – 2017
This questionnaire is aimed to be filled by End-Users who have used the applications developed within the pilots of H2020 OpenGovIntelligence (OGI) project. When answering the questions, the application developed using the OGI Toolkit, named here as Tallinn Real Estate Pilot App, should be taken into account. The following topics will be asked:
General background information; Data sets; and, Resulting Application (App developed using the OGI toolkit).
More information about Tallinn Real Estate
(https://rnd-tut.shinyapps.io/Estonian_Pilot/)
A. General Background Information 1. What is your gender? ( ) Female ( ) Male ( ) Don’t want to share
2. Please select your age: ( ) Less than 21 ( ) 21 – 40 ( ) 41 – 60 ( ) Over 60 ( ) Don’t want to share
3. How familiar are you with open data applications? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar
4. What is your role using the Tallinn Real Estate App? ( ) Students ( ) Foreign employees ( ) Tourists ( ) Real Estate Agents ( ) Other ______________
B. Acceptance of the Tallinn Real Estate Pilot App 5. The Tallinn Real Estate App provides all the functionalities I’m interested in ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
6. All functions in the Tallinn Real Estate App work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
7. The Tallinn Real Estate App recovers well from crashes ( ) The Tallinn Real Estate App didn’t crash ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
8. The Tallinn Real Estate App helps me to achieve my goals ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
9. My interaction with the Tallinn Real Estate App user was satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
10. The Tallinn Real Estate App is useful to me ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
D4.6 Pilots Evaluation Results – Third Round
Page 196 of 215
11. My interaction with the Tallinn Real Estate App is clear and understandable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
12. The Tallinn Real Estate App design is adequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
13. I have sufficient skills to use the Tallinn Real Estate App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
14. The Tallinn Real Estate App does not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
15. The Tallinn Real Estate App will be accepted by my peers ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
16. Using the Tallinn Real Estate App can be hard for the average citizen ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
17. The Tallinn Real Estate App is easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about acceptance? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
C. Tallinn Real Estate Pilot App Data Sets 18. The data in the Tallinn Real Estate App are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
19. The data in the Tallinn Real Estate App is incomplete ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
20. The data in the Tallinn Real Estate App are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
21. A lot of time is needed to find the right data in the Tallinn Real Estate App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
22. I am able to find all the data I'm looking for when using the Tallinn Real Estate App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about data sets? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
D4.6 Pilots Evaluation Results – Third Round
Page 197 of 215
D. Tallinn Real Estate Pilot App RESULTS 23. The Tallinn Real Estate App has a clear visualization ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
24. The Tallinn Real Estate App helps to create insights ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
25. The Tallinn Real Estate App results in an increase of transparency ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
26. The Tallinn Real Estate App is too complex to gain insight ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
27. The Tallinn Real Estate App helps me to make better decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
28. The Tallinn Real Estate App helps my understanding of the subject matter ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
29. The Tallinn Real Estate App will increase efficiency ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
30. The Tallinn Real Estate App reduces time spent looking for information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
31. The Tallinn Real Estate App reduces the costs to search and decide ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
32. More functions in the Tallinn Real Estate App are needed to create transparency of Real Estate information in Tallinn ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about results? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
Thank you for your time to answer the questions!
D4.6 Pilots Evaluation Results – Third Round
Page 198 of 215
7.1.4 Pilot 4 – Trafford Council Worklessness (England) End-User Questionnaire
OGI Evaluation Questionnaire Trafford Worklessness Pilot Apps End-users – 2018
This questionnaire is aimed at end-users who have used the applications developed by the pilots involved in the H2020 OpenGovIntelligence (OGI) project. When answering the questions, the applications developed using the OGI Toolkit, named here as the Trafford Worklessness Pilot Apps, should be taken into account. A separate usability survey will be used for the individual applications. The apps are designed to enable Jobcentre Plus Managers and Local Authority leads to identify areas of need and inform service delivery related to worklessness. The following topics will be asked:
Your General background information; A Technical Evaluation of the Pilot developed using the OGI toolkit The Datasets available in the Pilot; and, Pilots Outcomes (e.g. transparency, administrative burden reduction, cost
reduction).
Trafford Worklessness Pilot Apps - http://www.trafforddatalab.io/opengovintelligence/
A. General Background Information 1. Which organisation do you work for? ( ) Department for Work and Pensions ( ) Borough / City Council ( ) Other: ___________________________________ 2. Which Local Authority do you work within? ( ) Bolton ( ) Bury ( ) Manchester ( ) Oldham ( ) Rochdale ( ) Salford ( ) Stockport ( ) Tameside ( ) Trafford ( ) Wigan ( ) Other: ___________________________________ 3. How long have you been working for your employer? ( ) Less than 6 months ( ) 6 to 12 months ( ) 1 to 3 years ( ) 3 to 6 years ( ) 6 years or more 4. What is your employee status? ( ) Managerial ( ) Non-managerial 5. How comfortable do you feel using the Internet? ( ) Very uncomfortable ( ) Somewhat uncomfortable ( ) Neither comfortable nor uncomfortable ( ) Somewhat comfortable ( ) Very comfortable 6. How familiar are you with open data? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar 7. How familiar are you with Linked Open Statistical Data (LOSD)?
D4.6 Pilots Evaluation Results – Third Round
Page 199 of 215
( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar 8. How familiar are you with downloading data from open data applications like nomis and Stat-
Xplore? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar
B. Trafford Worklessness Pilot Apps Datasets 9. The data in the Trafford Worklessness Pilot Apps are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 10. The Trafford Worklessness Pilot Apps contain all of the datasets that I require ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 11. The data in the Trafford Worklessness Pilot Apps are incomplete ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 12. The data in the Trafford Worklessness Pilot Apps are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 13. A lot of time is needed to find the right data in the Trafford Worklessness Pilot Apps ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 14. I am satisfied with the datasets quality provided by the Trafford Worklessness Pilot Apps ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree Do you want to explain your answers or comment on anything not covered about data sets? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
C. Acceptance of the Trafford Worklessness Pilot Apps 15. The Trafford Worklessness Pilot Apps provide all the functionalities I am interested in ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 16. All functions in the Trafford Worklessness Pilot Apps work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 17. I experienced a crash while using the Trafford Worklessness Pilot Apps ( ) The Trafford Worklessness Pilot Apps didn’t crash ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 18. The Trafford Worklessness Pilot Apps help me in my work ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 19. My interaction with the Trafford Worklessness Pilot Apps is satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 20. The Trafford Worklessness Pilot Apps are useful to me
D4.6 Pilots Evaluation Results – Third Round
Page 200 of 215
( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 21. The Trafford Worklessness Pilot Apps are easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 22. The Trafford Worklessness Pilot Apps have an adequate design ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 23. I have sufficient skills to use the Trafford Worklessness Pilot Apps ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 24. The Trafford Worklessness Pilot Apps do not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 25. The Trafford Worklessness Pilot Apps are accepted by my peers ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 26. The Trafford Worklessness Pilot Apps are unnecessarily complex ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 27. I found the various functions in the Trafford Worklessness Pilot Apps are well integrated ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree 28. I want to use the Trafford Worklessness Pilot Apps frequently ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree Do you want to explain your answers or comment on anything not covered about acceptance? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
D. Trafford Worklessness Pilot Apps Outcomes Evaluation 29. The visualization provided by the Trafford Worklessness Pilot Apps makes better
interpretation of data ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
30. The Trafford Worklessness Pilot Apps help me to make better decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
31. The Trafford Worklessness Pilot Apps increase efficiency of my work ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
32. The Trafford Worklessness Pilot Apps reduce time spent looking for information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
33. The Trafford Worklessness Pilot Apps reduce costs to find information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
D4.6 Pilots Evaluation Results – Third Round
Page 201 of 215
34. More functions in the Trafford Worklessness Pilot Apps are needed to create transparency to
support decision-making ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
What functionality would you like to see added to the applications to add transparency? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Finally, do you want to explain your answers or comment on anything not covered about results? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
Many thanks for taking the time to complete this questionnaire!
If you have any questions about the survey please email [email protected]
D4.6 Pilots Evaluation Results – Third Round
Page 202 of 215
7.1.5 Pilot 5 – The Flemish Environment Agency (Belgium) End-User Questionnaire
OGI Evaluation Questionnaire Flemish Government App End-users – 2017
This questionnaire is aimed to be filled by End-Users who have used the applications developed within the pilots of H2020 OpenGovIntelligence (OGI) project. When answering the questions, the application developed using the OGI Toolkit, named here as Flemish Government Pilot App, should be taken into account. The following topics will be asked:
General background information; Data sets; and, Resulting Application (App developed using the OGI toolkit).
B. General Background Information 1. Please select your age: ( ) Less than 21 ( ) 21 – 40 ( ) 41 – 60 ( ) Over 60 ( ) Don’t want to share
2. How familiar are you with open data applications? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar
D. Acceptance of the Flemish Government Pilot App 3. The Flemish Government App provides all the functionalities I’m interested in ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
4. All functions in the Flemish Government App work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
5. The Flemish Government App recovers well from crashes ( ) The Flemish Government App didn’t crash ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
6. The Flemish Government App helps me to achieve my goals ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
7. My interaction with the Flemish Government App was satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
8. The Flemish Government App is useful to me ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
9. My interaction with the Flemish Government App is clear and understandable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
10. The Flemish Government App design is adequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
11. I have sufficient skills to use the Flemish Government App
D4.6 Pilots Evaluation Results – Third Round
Page 203 of 215
( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
12. The Flemish Government App does not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
13. The Flemish Government App will be accepted by my peers ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
14. Using the Flemish Government App can be hard for the average citizen ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
15. The Flemish Government App is easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about acceptance? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
C. Flemish Government Pilot App Data Sets
16. The data in the Flemish Government App are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
17. The data in the Flemish Government App is incomplete ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
18. The data in the Flemish Government App are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
19. A lot of time is needed to find the right data in the Flemish Government App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
20. I am able to find all the data I'm looking for when using the Flemish Government App ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about data sets? ______________________________________________________________________________ ______________________________________________________________________________
D4.6 Pilots Evaluation Results – Third Round
Page 204 of 215
______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
D. Marine Institute Pilot App results 21. The Flemish Government App has a clear visualization ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
22. The Flemish Government App helps to create insights about air pollution in Flanders ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
23. The Flemish Government App results in an increase of transparency of air pollution in Flanders ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
24. The Flemish Government App is too complex to gain insight ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
25. The Flemish Government App helps me to make better decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
26. The Flemish Government App helps my understanding of the subject matter ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
27. The Flemish Government App will increase efficiency ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
28. The Flemish Government App reduces time spent looking for information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
29. The Flemish Government App reduces the costs to search and decide ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
30. More functions in the Flemish Government App are needed to create transparency of air pollution in Flanders ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about results? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
D4.6 Pilots Evaluation Results – Third Round
Page 205 of 215
______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
Thank you for your time to answer the questions!
D4.6 Pilots Evaluation Results – Third Round
Page 206 of 215
7.1.6 Pilot 6 – Marine Institute (Ireland) End-User Questionnaire
OGI Evaluation Questionnaire Marine Institute Pilot End-users – 2018
This questionnaire is aimed to be filled by End-Users who have used the Pilot applications developed within the H2020 OpenGovIntelligence (OGI) project. When answering the questions, the Pilot application developed using the OGI Toolkit, named here as Marine Institute Pilot, should be taken into account. The following topics will be asked:
General background information; Data sets; and, Resulting Pilot (Pilot developed using the OGI toolkit).
A. General Background Information 1. What is your organisational role using the Marine Institute Pilot? ( ) Search and rescue professional ( ) Search and rescue volunteer ( ) Marine Energy Developer ( ) Marine Energy Consultant ( ) Planning Consultant ( ) Planner ( ) Private Citizen ( ) Public Servant ( ) Civil Servant ( ) Tourism Agent ( ) Tourism business owner ( ) Sailor ( ) Volunteer ( ) IT Professional Other: ________________________________________________
2. For what purpose might you use the Marine Institute Pilot Dashboard? ( ) Recreational ( ) Work ( ) Both
3. How familiar are you with data dashboards? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar
4. How familiar are you with marine data availability suitable to your needs? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar
5. How familiar are you with Linked Open Statistical Data? ( ) Not at all familiar ( ) Slightly ( ) Somewhat ( ) Moderately ( ) Extremely familiar
B. Marine Institute Pilot Evaluation 6. The Marine Institute Pilot provides all the functionalities I require ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
7. All functions in the Marine Institute Pilot work properly ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
8. The Marine Institute Pilot system is stable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
9. The Marine Institute Pilot supports my requirements ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
D4.6 Pilots Evaluation Results – Third Round
Page 207 of 215
10. My interaction with the Marine Institute Pilot was satisfying ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
11. The Marine Institute Pilot is useful to me ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
12. My interaction with the Marine Institute Pilot is clear and understandable ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
13. The Marine Institute Pilot design is adequate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
14. I have sufficient skills to use the Marine Institute Pilot ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
15. The Marine Institute Pilot does not require high level technical knowledge ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
16. I will share the The Marine Institute Pilot with colleagues ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
17. Using the Marine Institute Pilot can be hard for the average citizen ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
18. The Marine Institute Pilot is easy to use ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Have you any comments and or recommendations regarding the evaluation of this pilot? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
C. Marine Institute Pilot Data 19. The data in the Marine Institute Pilot are accessible ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
20. The data in the Marine Institute Pilot is complete ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
21. The data in the Marine Institute Pilot are accurate ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
22. The data in the Marine Institute Pilot are consistent ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
23. A lot of time is needed to find the right data in the Marine Institute Pilot
D4.6 Pilots Evaluation Results – Third Round
Page 208 of 215
( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
24. I am able to find all the data I'm looking for when using the Marine Institute Pilot ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Are there any particular datasets you are aware of that would prove useful in the pilot evaluated? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________
E. Marine Institute Pilot 25. The Marine Institute Pilot has a clear visualisation ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
26. The Marine Institute Pilot helps to support emergency response operations ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
27. The Marine Institute Pilot helps to support marine renewable energy investment decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
28. The Marine Institute Pilot helps to support planning a sailing event ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
29. The Marine Institute Pilot helps me to make better decisions ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
30. The Marine Institute Pilot helps my understanding of ocean conditions relevant to emergency
response, marine energy and marine event planning ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
31. The Marine Institute Pilot will increase efficiency ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
32. The Marine Institute Pilot reduces time spent looking for information ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
33. The Marine Institute Pilot reduces the costs to search and decide ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
34. More functions in the Marine Institute Pilot are needed to create transparency to support
marine decision-making ( ) Strongly Agree ( ) Agree ( ) Neutral ( ) Disagree ( ) Strongly Disagree
Do you want to explain your answers or comment on anything not covered about results? ______________________________________________________________________________ ______________________________________________________________________________
D4.6 Pilots Evaluation Results – Third Round
Page 209 of 215
______________________________________________________________________________
Thank you for your time in completing this evaluation questionnaire.
D4.6 Pilots Evaluation Results – Third Round
Page 210 of 215
7.2 Data Quality Questionnaire
Table 58 – Pilots’ Data Quality Questionnaire
Data Quality
Section # Question Answer
A- Discoverability
and Usability
1 There is a (pre-release) version of an
application already:
( ) Yes ( ) No
If No, please skip to section B- Granularity.
2 The application contains links to the data
sources ( ) Yes ( ) No
3 The application contains a searchable list
of data sources that are used ( ) Yes ( ) No
4 The application contains an option to
download the data ( ) Yes ( ) No
5 The application contains an option to
download the data in multiple formats ( ) Yes ( ) No
6 The application contains an option to
download selected parts of the data ( ) Yes ( ) No
7 The application allows the user to
preview and interact with the data ( ) Yes ( ) No
B- Granularity
8 Your data includes only aggregated data
(e.g. national averages)? ( ) Yes ( ) No
9
Your data includes individual unit level
data, but without generic class data
(data on a small, transactional level)
( ) Yes ( ) No
10
Your data includes generic class data
(The data also includes information
about different subsets within)
( ) Yes ( ) No
C- Intelligibility
11 Is there existing supporting
documentation for the data you used?
( ) Supporting documentation exists, but has to be
found separately
( ) Supporting documentation can be found at the
same time as the data (e.g. right next to the link for
the data)
( ) Supporting documentation can be immediately
accessed from within the data set, but it is not
context sensitive
( ) Supporting documentation can be immediately
accessed from within the data set, and is context
sensitive.
If there was no supporting
documentation, how did you deal with
these datasets?
Open question.
D-
Trustworthiness
12 I know how the data I used was collected ( ) Strongly Agree
( ) Agree
( ) Neutral
13 I know how the data I used was
processed
14 The data I used was published by a
D4.6 Pilots Evaluation Results – Third Round
Page 211 of 215
trusted source ( ) Disagree
( ) Strongly Disagree 15 The data I used is realistic
16 The data I used is consistent with
external sources
E- Linkable to
other data
(5 Stars LOD)
17 The data I used was available in a
structured format (e.g. xls) ( ) Strongly Agree
( ) Agree
( ) Neutral
( ) Disagree
( ) Strongly Disagree
18 The data I used was available in a non-
proprietary open format (e.g. csv)
19 The data I used contained URLs to
denote external things (e.g. RDF)
20 The data I used linked to other data to
provide context
F- 15 Open Data
Principles
21
The data is complete (all data that is not
subject to valid privacy, security or
privilege reasons is opened)
( ) Strongly Agree
( ) Agree
( ) Neutral
( ) Disagree
( ) Strongly Disagree
22 The data is primary (collected at the
source, not in aggregate form)
23
The data is made available as quickly as
necessary to preserve value for the end-
users
24
The data is accessible (for the widest
range of users; for the widest range of
purposes)
25 The data is non-discriminatory (available
to anyone, no registration needed)
26 The data is license-free (not subject to
copyright concerns etc.)
27 The data is permanent (made available
at a stable location indefinitely)
28 The data is safe to open (doesn’t contain
executable content)
29
The data is designed with public input
(the user decides how he can access the
data)
D4.6 Pilots Evaluation Results – Third Round
Page 212 of 215
7.3 OGI Architecture Questionnaire
OGI Linked Open Statistical Data (LOSD) Architecture Questionnaire Evaluation – Developer Questionnaire – Third Year 2018/2019
This questionnaires is aimed to be filled by Developers at various aspects of the applications developed within the toolkit provided by H2020 OpenGovIntelligence (OGI) project. When answering the questions the application developed using the OGI Linked Open Statistical Data (LOSD) Architecture should be taken into account. The following topics will be asked:
User Behaviour; Perceived Usefulness; and, OGI Architecture characteristics aiming to improve Data Quality and Systems Flexibility.
A. User Behaviour 1. I referred to the LOSD OGI Reference Architecture to design my own architecture.
1 2 3 4 5 Almost Never [ ] [ ] [ ] [ ] [ ] Almost Always
B. Perceived Usefulness
2. Using the LOSD OGI Reference Architecture increases my architecture building work related productivity and effectiveness.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
3. The LOSD OGI Reference Architecture helps me to attain my objectives with the data.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
4. The LOSD OGI Reference Architecture is useful to deal with data quality issue in my work.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
5. The LOSD OGI Reference Architecture helps me to become aware of data quality issues.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
6. The LOSD OGI Reference Architecture helps me to design and build a flexible architecture.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
7. The LOSD OGI Reference Architecture helps me to elicit better requirements.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
D4.6 Pilots Evaluation Results – Third Round
Page 213 of 215
C. Perceived Ease of Use 8. I find the LOSD OGI Reference Architecture clear and understandable.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
9. I find that the LOSD OGI Reference Architecture helped me to accelerate architecting
linked open data system. 1 2 3 4 5
Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
10. I find the LOSD OGI Reference Architecture helped me to architect linked open data system with less cost (resources).
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
D. Computer Self-efficacy, Perceptions of External Control, Computer Playfulness, Computer
Anxiety, Perceived Enjoyment, Objective Usability 11. I am fluent in the use of a computer.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
12. I can figure out almost any software program (application) with a minimum of effort.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
13. I am confident in my ability to design and develop my own architecture using the
reference architecture. 1 2 3 4 5
Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
14. I enjoy to be creative and have fun when using computers. 1 2 3 4 5
Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
15. I get dysfunctional nervous when working with a computer. 1 2 3 4 5
Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
16. It feels enjoyable using the reference architecture. 1 2 3 4 5
Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
17. The use of the reference architecture accelerates me attaining my data objectives. 1 2 3 4 5
Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
D4.6 Pilots Evaluation Results – Third Round
Page 214 of 215
E. Subjective Norm, Voluntariness, Image, Job Relevance, Output Quality 18. My business partners think I should use the LOSD OGI Reference Architecture.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
19. It is my choice whether I use the LOSD OGI Reference Architecture to build my own
architecture. 1 2 3 4 5
Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
20. I am given the freedom to choose whether or not I use the LOSD OGI Reference Architecture.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
21. My business friends will think highly of me if I use the LOSD OGI Reference Architecture.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
22. The use of the LOSD OGI Reference Architecture is pertinent to my job-related tasks.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
23. I consider data quality is improved after using the LOSD OGI Reference Architecture.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
24. I consider flexibility is improved from using the LOSD OGI Reference Architecture.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
F. Result Demonstrability, Behavioural Intention, Use, Present Satisfaction, Future Desire
25. I believe I would have no problem explaining to someone else the benefits of using the LOSD OGI Reference Architecture.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
26. I intend to make good use of the LOSD OGI Reference Architecture.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
27. On average, what is the percentage do you spend on using/referring the LOSD OGI
Reference Architecture during architecting?. 1 2 3 4 5
Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
28. I am satisfied with the present LOSD OGI Reference Architecture.
D4.6 Pilots Evaluation Results – Third Round
Page 215 of 215
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree
29. I would welcome a different LOSD OGI Reference Architecture.
1 2 3 4 5 Strongly Disagree [ ] [ ] [ ] [ ] [ ] Strongly Agree