brussels,18. march 2010 1rural wings ip rural wings final review meeting usability evaluation thomas...
TRANSCRIPT
Brussels,18. March 2010 1 RURAL WINGS IP
RURAL WINGS Final Review Meeting
Usability Evaluation
Thomas Köhler, Nina KahnwaldMedia Center, Dresden Univ. of Technology
Brussels, 18. March 2010 2 RURAL WINGS IP
structure
1. Introduction2. Usability Questionnaire
• Procedure• Response rates • Results
- Personal data, Infrastructure, Usage Profiles- Usability of Rural Wings infrastructure, training and
support- Usability of Rural Wings applications
3. Usability monitoring during user training Procedure and Results
4. Heuristic Evaluation Procedure and Results
5. Summary
Brussels, 18. March 2010 3 RURAL WINGS IP
procedure
Overall usability evaluation activities:
• Usability Assessment through Questionnaires – after a few months of usage (11/2007-2/2008, 12/2008-2/2009 and 10/2009-12/2009)
• Usability Testing through monitoring of questions and problems – during implementation and training sessions (6/2007-12/2009)
• Usability heuristic Evaluation of Rural Wings CAP – prototype stage (8/2007) and implementation stage (12/2009)
• Technical Evaluation – continued monitoring during test runs (11/07- ongoing)
Brussels, 18. March 2010 4 RURAL WINGS IP
usability evaluation questionnaire
Usability questionnaire Structure of the questionnaire (based on „Questionnaire for User Interface Satisfaction“
[QUIS]): Personal data Infrastructure Usage profile Performance/Usability of technical infrastructure Usability of CAP Usability of applications
Scales are applicable to all user groups identifiedin the user needs analysis (WP3).
Brussels, 18. March 2010 5 RURAL WINGS IP
results
Response rates
data entry that was completed until 20th of December 2009
analysis of all questionnaires where users specified their pilot site
questionnaire was completed by 159 end users from 90 pilot sites
Brussels, 18. March 2010 6 RURAL WINGS IP
response rates I
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final Run
Phase B
RW applications used and rated
(red=test run, black=final run A,
green=final run B)
All sites 52 133 159
Armenia 0 0 0
Cyprus 2 5 8
CYP01 (Λεμύθου / Lemýthou) 1 - 1 webTV, TeacherNet
CYP02 (Κρήτου Τέρρα / Krī Rtou
Térra) 1 - 1 Health Training
CYP03 (Καμπιά / Kampiá) - 1 1 YouRA
CYP04 (Παραμύθα / Paramýtha) - 1 1
YouRA, D-Space
agroweb, webTV, YouRA, D-Space,
TeacherNet, NEMED, HealthTraining
CYP05 (Ασγάτα / Asgáta) - 1 1 webTV, YouRA
CYP06 (Μαρώνι / Marōni) - 1 1 YouRA
CYP07 (Άγιος Γεώργιος / Ágios
Geōrgios) - - 1
Xplora, YouRA, UNITE, Connect,
TeacherNet,
CYP08 (Κάμπος / Kámpos) - 1 1
YouRA, D-Space
agroweb, webTV, YouRA, D-Space,
TeacherNet, NEMED, AgroTD,
HealthTraining
Brussels, 18. March 2010 7 RURAL WINGS IP
response rates II
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final Run
Phase B
RW applications used and rated
(red=test run, black=final run A,
green=final run B)
Estonia 3 12 7
EST01 (Ruhnu) 2 1 1 webTV, Health Training
EST02 (Piirissaare) 1 1 1 -
EST03 (Pamma) - 2 1
webTV, Health Training
HealthTraining
EST04 (Nasva) - 2 1
Health Training
agroweb, webTV, Xplora, NEMED,
HealthTraining
EST05 (Oitme) - 2 1
Health Training
Health Training
EST06 (Panga) - 3 1
Health Training, Xplora
Health Training
EST07 (Võhma) - 1 1 Health Training
Brussels, 18. March 2010 8 RURAL WINGS IP
response rates III
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final Run
Phase B
RW applications used and rated
(red=test run, black=final run A,
green=final run B)
France 7 19 13
FRA01 (INSEAD) 5 1
Agroweb, RCCM, MEDSKY
RCCM, MEDSKY
FRA02 (Martinique) 6 7 2
webTV, Xplora, UNITE, RCCM, MEDSKY
webTV, RCCM, MEDSKY
RCCM, MEDSKY
FRA03 (Manso) 1 1 3
D-Space, Teacher net, MEDSKY, RCCM,
MEDSKY
RCCM, MEDSKY
FRA04 (CC2F) - 1 2 RCCM, MEDSKY
FRA05 (Puscaghia) - 2 1 webTV, MEDSKY
FRA06 (La Grande Paroisse) - 3 2 RCCM, MEDSKY
FRA08 (Letia) - - 2 -
Brussels, 18. March 2010
response rates IV
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final Run
Phase B
RW applications used and rated
(red=test run, black=final run A,
green=final run B)
Greece 11 41 32
GRE01 (Σάλακος (Sálakos) 1 1 4
YouRA, D-Space, CONNECT, YouRA
webTV, YouRA, D-Space, TeacherNet,
NEMED, AgroTD, HealthTraining, RCCM
GRE02 (Αιγιάλη / Aigiálī) 1 2 1 YouRA, D-Space, CONNECT
GRE03 (Πυλές / Pylés) 1 1 1
NEMED, AgroTeleDiag.
webTV, YouRA, NEMED
webTV, YouRA, TeacherNet
GRE04 (Κλειστός / Kleistós) - 2 1
YouRA
YouRA
GRE05 (Βαλτεσίνικο / Valtesíniko) 1 3 1
YouRa, Teacher net
Xplora, YouRA
webTV, Xplora, TeacherNet
GRE06 (Άγιος Νικόλαος Βοιών / Ágios
Nikólaos Voiōn) 2 1 2 VEMUS, YouRA
GRE07 (Μεστά / Mestá) 1 1 1
webTV, VEMUS
webTV, VEMUS
webTV
GRE08 (Γεράκι / Geráki) 3 1 1
agroweb, webTV, Health Training
AgroTeleDiag
webTV
GRE09 (Νυμφαίο / Nymfaío) - 2 1
WebTV
webTV
GRE10 (Άγιος Λαυρέντιος / Ágios
Lavréntios) - 1 1
MEDSKY
RCCM
Brussels, 18. March 2010 10 RURAL WINGS IP
response rates V
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final Run
Phase B
RW applications used and rated
(red=test run, black=final run A,
green=final run B)
Greece GRE11 (Λέχοβο / Léchovo) - - 1 WebTV, Xplora, YouRA, TeacherNet
GRE12 (Ψαρά / Psará) - 1 1 YouRA, NEMED
GRE13 (Γλαύκη / Glávkī) - 1 1
ExperiNet
Xplora, ExperiNet
GRE14 (Οργάνη / Orgánī) - 1 1
D-Space
Xplora, D-Space, ExperiNet
GRE15 (Ορεστιάδα (Orestiáda) - 1 1
ExperiNet, TeacherNet
ExperiNet
GRE16 (Φλομοχώρι / Flomochōri) - 1 1 WebTV, NEMED
GRE17 (Κορυστά / Korystá) - 1 2 ExperiNet, EUDOXOS
GRE18 (Ανώπολη / Anōpolī) - 1 1
YouRA
YouRA
GRE19 (Σίσες / Síses) - 1 1
webTV, Health Training
NEMED
GRE20 (Γκούρα / Gkoúra) 1 2 webTV
GRE21 (Κομοτηνή / Komotīnī R) - 1 1
YouRa, ExperiNet, CONNECT
Xplora
Brussels, 18. March 2010
response rates VI
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final Run
Phase B
RW applications used and rated
(red=test run, black=final run A,
green=final run B)
Greece GRE22 (Αργύρι / Argýri) - 2 - YouRa, Teacher Net, NEMED
GRE23 (Βουνιχώρα / Vounichōra) - 7 1 Health Training
GRE24 (Καλλιφώνι / Kallifōni) - 1 1
YouRA
YouRA, D-Space, ExperiNet
GRE25 (Βερτίσκος / Vertískos) - 1 1 -
GRE26 (Λυσσαρέα / Lyssaréa) - 2 1
agroWeb, Xplora, ExperiNet, Teacher Net
Xplora
GRE27 (Άγιος Δημήτριος / Ágios
Dīmī Rtrios) - 3 1 NEMED
Hungary - - 2
HUN03 (N.N.) - - 2 YouRA
Israel - 1 59
ISR01 (חורה / حورة / Ḥūra) - - 5 agroweb, VEMUS, webTV
ISR02 (Fourier " פורייהفورييه/ ") - - 8 Agroweb, webTV
Brussels, 18. March 2010
response rates VII
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final Run
Phase B
RW applications used and rated
(red=test run, black=final run A,
green=final run B)
Israel ISR03 (اتد / Atid) - - 10 VEMUS, D-Space, NEMED
ISR04 (ابو تلول / Abū Tlūl) - - 11 VEMUS, Xplora, YouRA, TeacherNet
ISR05 ( ' / بئر هداجביר הדאג / Bir Hadāğ) - - 7 agroweb, webTV, UNITE
ISR06 (بئر مشاش / Bir Mišāš) - - 18
agroweb, VEMUS, webTV, Xplora, YouRA, D-
Space, ExperiNet, TeacherNet
ISR07 (ابو قويدر / Abū Qwaydar) - 1 0 -
Poland 2 17 12
POL01 (Babiogórski PN, Zawoja) 1 1 1 webTV
POL02 (Polana) 1 2 1
D-Space, Teacher net, VEMUS
Xplora
POL03 (Wiśniowa) - 2 1
Xplora, UNITE, D-Space, ExperiNet,
CONNECT, AgroTeleDiag
Xplora
POL04 (OAUJ, Mt. Lubomir) 2 1 D-Space
POL05 (Rokiciny) 1 1
webTV, Teacher Net
webTV, Xplora, YouRA
POL06 (Skawa) - 2 2 RCCM
Brussels, 18. March 2010
response rates VIII
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final Run
Phase B
RW applications used and rated (red=test run,
black=final run A, green=final run B)
POL07 (Nowy Łupków) - 1 1
D-Space, teacher Net, Health training
webTV
POL08 (Kęty) - 6 2
VEMUS, webTV, Xplora, YouRA, CONNECT
agroweb, Xplora, YouRA, TeacherNet
POL09 (Myczkowce) - - 1 WebTV, AgroTD
POL10 (Harkabuz) - - 1 D-Space
Romania 8 19 15
ROM01 (Dezna) 8 2 1
VEMUS, webTV, Xplora, YouRA, UNITE, D-Space, ExperiNet,
CONNECT, Teacher net, NEMED, AgroTeleDiag., Health
Training, RCCM
agroWeb, VEMUS, Xplora, YouRA, UNITE, D-Space,
ExperiNet, CONNECT, Teacher net, NEMED
VEMUS, YouRA, NEMED
ROM02 (Piatra Arsă) 1 1 Health Training
ROM03 (Cozieni) - 1 1
agroweb, VEMUS, webTV, Xplora, YouRA, UNITE, D-Space,
ExperiNet, CONNECT, Teacher net, NEMED
VEMUS, YouRA, NEMED
ROM04 (Golu Grabicina) - 1 1
VEMUS, webTV, Xplora, D-Space
NEMED
ROM05 (Arefu) - 1 1
Teacher Net
NEMED
ROM06 (Brebu) - 3 1
VEMUS, ExperiNet, CONNECT, Teacher net, NEMED, Health
Training
NEMED
Brussels, 18. March 2010 14 RURAL WINGS IP
response rates IX
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final
Run
Phase B
RW applications used and rated (red=test run,
black=final run A, green=final run B)
Romania
(cont.) ROM07 (Măguri-Răcătău) - 1 1
Xplora, UNITE, D-Space, ExperiNet, NEMED, AgroTeleDiag
NEMED
ROM08 (Dăbâca) - 1 1
Xplora, CONNECT, TeacherNet
NEMED
ROM09 (Orlat) - 1 2
agroweb, UNITE, CONNECT
VEMUS, NEMED
ROM10 (Rod) 1 1
VEMUS, TeacherNet, NEMED
NEMED
ROM11 (Deal) 1 1 VEMUS, NEMED
ROM12 (Nadăş) - 1 1
agroweb, VEMUS, webTV, Xplora, D-Space, NEMED,
MEDSKY
NEMED
ROM13 (Tilişca) - 1 1
agroweb, VEMUS, webTV, Xplora, YouRA, D-Space,
NEMED, AgroTD
NEMED
ROM14 (Bucureşti) - 3 1
VEMUS, YouRA, UNITE, Teacher Net, NEMED, Health
Training, MEDSKY
VEMUS; YouRA, NEMED, Health Training
Brussels, 18. March 2010 15 RURAL WINGS IP
response rates X
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final Run
Phase B
RW applications used and rated
(red=test run, black=final run A,
green=final run B)
Spain SPA05 (Rellinars) - 2 1
YouRA
YouRA
SPA06 (Laredo) - 1 - YouRA, webTV
SPA07 (Santa Comba) - 1 1 YouRA, webTV
SPA08 (Vilaverd) 1 -
YouRA, webTV, D-Space
webTV, YouRA
SPA09 (Reboreda) 2 -
YouRA, webTV
YouRA, D-Space
SPA10 (Blancafort) 2 1 YouRA, webTV
Ramallosa 7 - -
NEMED, AgroTeleDiag., agroweb, webTV,
Xplora, YouRA
Sant Sernip - 1 - webTV, YouRa, NEMED
Sweden 1 3 3
SWE01 (Tarfala) 1 - 1 MEDSKY
SWE02 (Lomträsk) - 2 1 webTV, MEDSKY
SWE03 (Avaviken) - 1 1 -
Brussels, 18. March 2010
response rates XI
Country Pilot site n
Test Run
n
Final Run
Phase A
n
Final Run
Phase B
RW applications used and rated
(red=test run, black=final run A,
green=final run B)
Spain 9 14 8
SPA01 (Teo-Campos) 1 1 1
NEMED, AgroTeleDiag, webTV, YouRA,
NEMED
webTV, YouRA, NEMED, EUDOXOS
SPA02 (Prats) 1 - 1
NEMED, AgroTeleDiag.
WebTV, YouRA
SPA03 (Bamonde) - 2 1
YouRA
YouRA
SPA04 (Arcos de Furcos) - 1 1 YouRA
UK 10 2 0
Bewholme 4 2 - webTV, CONNECT
Cilcennin 5 - - webTV, CONNECT, D-Space
Biggar 1 - - -
Brussels, 18. March 2010 17 RURAL WINGS IP
personal data I
Demographic aspects:
Gender: 66 female, 86 male participants
Age: majority (69%) is between 25 and 49 years
old.
Professions:
Other: i.e. civil servant, school consultant,
journalist or librarian
Brussels, 18. March 2010 19 RURAL WINGS IP
infrastructure
Infrastructure characteristics:75% of the RW users use MS Internet Explorer as webbrowser, Firefox is used by 22% of the users. Netscape and Opera are used by only 1,3%.Nearly all responding RW users (96%) have MS windows installed as operating system. Five (3%) use a Linux system and no one uses a Macintosh.
Connection to the RW internet access:via Sat modem: 36,5% via local area network: 32,1%via indoor wireless network connection: 14,5%via outdoor wireless network: 6,9%
web browser IE Netscape Firefox Opera Safari
% overall 75.2% 1.3% 22.2% 1.3% 0.0%
operating
system Win Win Vista WinXP Win2000 Win98 WinNT Linux Mac N/A
users 153 29 118 6 0 0 5 0 2
in % 96.2 18.02 74.2 3.8 0.0 0.0 3.1 0.0 1.3
Brussels, 18. March 2010 21 RURAL WINGS IP
usage profiles II
Services used:1. web browsing (88,7%)2. e-mail (74,8%)3. RW applications and tools (41,5%)4. Instant messaging (31,4%) 5. software updates (28,9%)
Websites visited (3 most visited):1. search engines2. news-sites 3. online-mailing4. RW-CAP5. weather forecast
Brussels, 18. March 2010 22 RURAL WINGS IP
usage profiles III
CAP use: Duration per week User Location
Brussels, 18. March 2010 23 RURAL WINGS IP
usability of infrastructure I
All categories are rated as excellent or good by at least 39%
Highest satisfaction can be seen in the area of assistance ratings which every fifth user (19.5% and 22%) judged as excellent. The initial installation was rated good or excellent by 69% of the users.
Deficits or problems seem to exist with the availability of the service (rated “poor” by 17.6% of the end users) and the reliability of the satellite terminal (rated “poor” by 7.5%).
Also free text answers and comments refer to low bandwidth and unreliable connections with repeated failures.
Brussels, 18. March 2010 25 RURAL WINGS IP
usability of infrastructure III
Summary usability ratings „poor“ (part I)
Brussels, 18. March 2010 28 RURAL WINGS IP
usability RW-CAP I
RW applications are often used independently from the CAP: Data from Google Analytics has shown that slightly more than 50%
of the users did access these RW applications directly. The following evaluation results thus reflect the experience of the remaining up to 50% users.
Some users perhaps don‘t see the added value of the CAP: „Do not see a strong need to always enter through the portal when you already know where you want to reach out.”
RW-CAP ratings overall impression was similar to the interim evaluation:
A critical topic was the missing localization and translation of RW-CAP contents, as stated by several users in the free text comments: “The CAP is in English, because of that we cannot judge its relevancy.” (Spain) / „Not available in French.“ (France) / „The main problem in using the site is the language barrier.” (Poland)
However this has been solved by the beginning of 2010 with a new Greek + French version. Also national partners may insert translations by themselves easily if needed.
Brussels, 18. March 2010 30 RURAL WINGS IP
usability RW applications I
15 different RW applications were used and rated during the final runs.
39% of the participants in the survey had not used any RW application when answering the questionnaire (32% in final runs Phase A evaluation).
On the whole the applications were rated positively. less positive rating than during the last evaluation: Agroweb (15
users), VEMUS (14 users), WebTV (30 users), Xplora (29 users), D-Space (13 users) and Teacher eTraining (13 users)
lost slightly: YouRA (27 users), UNITE (2 users) and ExperiNet (8 users) rated as positive as the last time: NEMED (25 users), AgroTeleDiag (3
users), the Health Training (12 users), RCCM (9 users) and MEDSKY (6 users)
See Annex A of this presentation for detailed usability ratings for each application.
Brussels, 18. March 2010 31 RURAL WINGS IP
usability monitoring
Principles of RW usability monitoring User problems and questions can be indicative for aspects of the technical infrastructure, the
manuals, the central access point or the rural wings platforms and applications that are not designed intuitively.
It can thus be used to identify possible usability problems and areas of improvement during the training of the end users.
A structured template for the collection of user questions and problems arising during the training of the end users was provided to all NCs.
Data collection realized after February 2009; in final test runs feedback only from Spain.
Brussels, 18. March 2010 32 RURAL WINGS IP
results usability monitoring
Emerging topics:
Language was not a problem as localized sites are available;
PC-skills become an issues;
RW applications and CAP (problems with D-Space and CAP-Structure);
RW Infrastructure (Problems with network - suggested usage of a faster browser did improve the performance somewhat but reportedly not enough).
Manuals: 1
Tools: 6
Infrastructure: 1
Central Access Point: 3
Other: 0
Overall questions/
problems 7
Brussels, 18. March 2010 33 RURAL WINGS IP
heuristic evaluation
Design of heuristic evaluation Expert evaluation of the RW Platform by usability experts of TUD:
checklist of recognized usability principles (heuristic) will be used to identify possible usability problems (see Schweibenz/Thissen 2003 and Nielsen/Mack 1994).
identified usability problems and possible recommendations will be aggregated and communicated to FORTHnet and all NCs by TUD.
Expert evaluation was conducted twice: at prototype stage (2007) and end of the project (2009) after improvements were made.
Results overall better impression than during 1st usability evaluation:
lucid, structured, not overloaded. but information offered and options could be structured and
grouped better and adapted to the needs of the users often available in English only, even the help function (!)
language specific (sub) sites should be developed for Further details of the heuristic evaluation cp. annex A
Brussels, 18. March 2010 34 RURAL WINGS IP
Conlcusions I
Language and localisation Language barrier had been identified as one of the most crucial
issues in previous evaluations.
RW Infrastructure RW services are mostly used during the week and earlier in the
day, while the usage during the weekend has increased as compared to the last evaluation.
Apart from language barriers, problems with network performance and wifi coverage most dominant in user feedbacks.
Availability of the RW internet service has been rated as “poor” by 17.6% of the end users (compared to 11% during the last evaluation phase).
Negative experiences with bandwidth reliability did again retard local activities, hinder training sessions or even prevent proper use of RW services most of the time.
Brussels, 18. March 2010 35 RURAL WINGS IP
Conlcusions II
RW-CAP Development of CAP-usage:
in test run evaluation nearly all participants had used RW-CAP, in the final runs 18% stated “not applicable”;
only 29% of the participants compared to 40% in the test runs did name it as one of the three most visited websites.
Language barriers and navigation problems remain (monitoring and heuristic evaluation).
Overall feedback less critical than in the test runs (higher satisfaction or lower relevance of CAP?).
Suggested improvements: Single log-in for all applications via CAP which would provide an
added value; Community features (have been implemented roughly but not
promoted).
Brussels, 18. March 2010 36 RURAL WINGS IP
Conlcusions III
RW Applications 16 different RW applications were used and rated – predominantly
positive, albeit slightly lower than during final phase A. Although 39% of the participants in the survey had not used any RW
applications when answering the questionnaire. Free text comments division between users who found the applications
“very good”, “marvellous” or “motivating” and those who, for various reasons, didn’t manage to get some of the applications working at all.
RW training and support The overall rating of the RW installation services and the RW support was
very positive. About a fifth of the participating users did rate it as “excellent”.
Suggestion for successive projects: to ensure sustainability meta-competencies should be imparted (i.e.
problem-solving strategies in case of computer problems) to enable users to help themselves outside of training sessions.
Supported by community features in RW CAP to allow end-users to share best practice with other pilot sites and support each other when problems occur.
Brussels, 18. March 2010 37 RURAL WINGS IP
Conlcusions IV
Positive feedback
despite all the criticism that an evaluation has to reflect we can conclude the usability evaluation of the Rural Wings Project with positive comments of users on the RW-project:
“Thanks to this project we were able to contact other different realities and to share experiences and knowledge gained with them .”
“It is an extraordinary help for rural localities such as ours.”
“It's good to feel myself as a part of international project.”
“It is a great project that allows the educational rural communities to have access to Technologies of Information and Communication.”
Brussels, 18. March 2010 38 RURAL WINGS IP
Thank you for your attention.
Prof. Dr. Thomas Köhler Technische Universität Dresden
Media Centre http://mz.tu-dresden.de Phone: +49-(0)351-463-32772 Fax: -463-34963
Brussels, 18. March 2010
Further details heuristic evaluation I
Issues of 2007 that still remain• In a second phase of the evaluation process the issues that
were identified during the first evaluation of the CAP website in 2007 were reassessed.
General problems• website suggests depth, but there merely are link lists
still triggering wrong expectations
• scope has to be made clearer still a problem, e.g. labels, country/user group specific adaptations
• navigation seems oversized in relation to available content• information cannot be accessed quickly enough
overall structure ok, but still no breadcrumb-type of navigation
57 RURAL WINGS IP
Brussels, 18. March 2010
Further details heuristic evaluation II
Missing features• „help“-page/function should be added and links to it on every page
ok, but probably still more help needed by ordinary users, English only(!)
• FAQs should be added and updated on a regular basis not implemented, but less serious
Structure• structure is not clear/consistent
issue remains, s.a.• navigation and paths should match!
paths seem to have been eliminated - not the preferred solution Design / Layout• layout will have to be improved (i.e. home>All rural
Wings>learning@work) issue remains, s.a.
58 RURAL WINGS IP
Brussels, 18. March 2010
Further details heuristic evaluation III
Broken Links / missing attributes corrected, but new ones occur – cf. Annex B of evaluation report
Other Features that should be improved• external and internal links have to be distinguishable
serious issue still remains, only solved in sitemap
• final URL should be international (.org or .net) and easier to remember than http://ruralwings.rd.forthnet.gr again considered a good idea, but not implemented
• wording should be target-group-specific (i.e. tools for students) very good idea, not implemented
59 RURAL WINGS IP
Brussels, 18. March 2010 61 RURAL WINGS IP
usability evaluation
Usability Evaluation
Usability of user interfaces is one narrow concern of the overall system acceptability. Figure 1 depicts Nielsen’s Taxonomy of System Acceptability, showing the relative placement of usability (Nielsen 1993, p.25).
Brussels, 18. March 2010 62 RURAL WINGS IP
evaluation focus
What will be evaluated?
utility vs. usability: utility is the question of whether the functionality of the
system in principle can do what is needed, usability is the question of how well users can use that
functionality.
Thus, the usability evaluation of the rural wings project will not investigate, if the services offered are suitable to meet the user needs, but how well end users and content providers can use the provided interface.
Brussels, 18. March 2010 63 RURAL WINGS IP
research questions I
Evaluation focus: usability of the RW “common access point” (CAP) and
RW platforms and services. This includes Internet access services (hybrid:
satellite-WiFi system), documentation and training materials and installation and maintenance procedures.
Central research questions are:
1. How well can users access and use the functionalities of RW CAP and the applications?
2. How is system usability of RW CAP and applications rated by end users in terms of the most important criteria of system usability (Learnability, Efficiency, Memorability, Errors,Satisfaction)?
Brussels, 18. March 2010 64 RURAL WINGS IP
research questions II
3. How satisfied are end-users with the Rural Wings Services and Infrastructure, i.e.
the installation services, the availability of the Rural Wings internet service, the reliability of the WiFi client equipment, the reliability of the Satellite terminal equipment, the technical support for the satellite and WIFI
infrastructure, training and introduction to the rural wings infrastructure,
platform and applications?
4. What questions, problems and needs arise during the training of the end users (i.e. are additional trainings required for specified pilot sites and/or contexts? Do specific target groups have special needs in terms of support, content or competencies?)?
Brussels, 18. March 2010 65 RURAL WINGS IP
coaching study
B.2.2 Coaching Method
The Coaching Method (Mack and Burdett 1992) includes explicit interaction between the test subject and the coach.
During a coaching study, the test user is allowed to ask any system-related question of an expert coach.
Coaching thus focuses on the novice user and is aimed at discovering the information needs of such users in order to provide better training and documentation, as well as possibly redesigning the interface to avoid the need for questions (Nielsen 1993).
The coaching method can relatively easy and without large expenses be integrated into the workshops that will be held to train the users and/or multiplicators (Task 6B.2) and can in addition serve to evaluate the provided support material (Task 6A.4).