brussels,18. march 2010 1rural wings ip rural wings final review meeting usability evaluation thomas...

65
Brussels,18. March 2010 1 RURAL WINGS IP RURAL WINGS Final Review Meeting Usability Evaluation Thomas Köhler, Nina Kahnwald Media Center, Dresden Univ. of Technology

Upload: hayley-wires

Post on 14-Dec-2015

220 views

Category:

Documents


2 download

TRANSCRIPT

Brussels,18. March 2010 1 RURAL WINGS IP

RURAL WINGS Final Review Meeting

Usability Evaluation

Thomas Köhler, Nina KahnwaldMedia Center, Dresden Univ. of Technology

Brussels, 18. March 2010 2 RURAL WINGS IP

structure

1. Introduction2. Usability Questionnaire

• Procedure• Response rates • Results

- Personal data, Infrastructure, Usage Profiles- Usability of Rural Wings infrastructure, training and

support- Usability of Rural Wings applications

3. Usability monitoring during user training Procedure and Results

4. Heuristic Evaluation Procedure and Results

5. Summary

Brussels, 18. March 2010 3 RURAL WINGS IP

procedure

Overall usability evaluation activities:

• Usability Assessment through Questionnaires – after a few months of usage (11/2007-2/2008, 12/2008-2/2009 and 10/2009-12/2009)

• Usability Testing through monitoring of questions and problems – during implementation and training sessions (6/2007-12/2009)

• Usability heuristic Evaluation of Rural Wings CAP – prototype stage (8/2007) and implementation stage (12/2009)

• Technical Evaluation – continued monitoring during test runs (11/07- ongoing)

Brussels, 18. March 2010 4 RURAL WINGS IP

usability evaluation questionnaire

Usability questionnaire Structure of the questionnaire (based on „Questionnaire for User Interface Satisfaction“

[QUIS]): Personal data Infrastructure Usage profile Performance/Usability of technical infrastructure Usability of CAP Usability of applications

Scales are applicable to all user groups identifiedin the user needs analysis (WP3).

Brussels, 18. March 2010 5 RURAL WINGS IP

results

Response rates

data entry that was completed until 20th of December 2009

analysis of all questionnaires where users specified their pilot site

questionnaire was completed by 159 end users from 90 pilot sites

Brussels, 18. March 2010 6 RURAL WINGS IP

response rates I

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final Run

Phase B

RW applications used and rated

(red=test run, black=final run A,

green=final run B)

All sites 52 133 159

Armenia 0 0 0

Cyprus 2 5 8

CYP01 (Λεμύθου / Lemýthou) 1 - 1 webTV, TeacherNet

CYP02 (Κρήτου Τέρρα / Krī Rtou

Térra) 1 - 1 Health Training

CYP03 (Καμπιά / Kampiá) - 1 1 YouRA

CYP04 (Παραμύθα / Paramýtha) - 1 1

YouRA, D-Space

agroweb, webTV, YouRA, D-Space,

TeacherNet, NEMED, HealthTraining

CYP05 (Ασγάτα / Asgáta) - 1 1 webTV, YouRA

CYP06 (Μαρώνι / Marōni) - 1 1 YouRA

CYP07 (Άγιος Γεώργιος / Ágios

Geōrgios) - - 1

Xplora, YouRA, UNITE, Connect,

TeacherNet,

CYP08 (Κάμπος / Kámpos) - 1 1

YouRA, D-Space

agroweb, webTV, YouRA, D-Space,

TeacherNet, NEMED, AgroTD,

HealthTraining

Brussels, 18. March 2010 7 RURAL WINGS IP

response rates II

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final Run

Phase B

RW applications used and rated

(red=test run, black=final run A,

green=final run B)

Estonia 3 12 7

EST01 (Ruhnu) 2 1 1 webTV, Health Training

EST02 (Piirissaare) 1 1 1 -

EST03 (Pamma) - 2 1

webTV, Health Training

HealthTraining

EST04 (Nasva) - 2 1

Health Training

agroweb, webTV, Xplora, NEMED,

HealthTraining

EST05 (Oitme) - 2 1

Health Training

Health Training

EST06 (Panga) - 3 1

Health Training, Xplora

Health Training

EST07 (Võhma) - 1 1 Health Training

Brussels, 18. March 2010 8 RURAL WINGS IP

response rates III

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final Run

Phase B

RW applications used and rated

(red=test run, black=final run A,

green=final run B)

France 7 19 13

FRA01 (INSEAD) 5 1

Agroweb, RCCM, MEDSKY

RCCM, MEDSKY

FRA02 (Martinique) 6 7 2

webTV, Xplora, UNITE, RCCM, MEDSKY

webTV, RCCM, MEDSKY

RCCM, MEDSKY

FRA03 (Manso) 1 1 3

D-Space, Teacher net, MEDSKY, RCCM,

MEDSKY

RCCM, MEDSKY

FRA04 (CC2F) - 1 2 RCCM, MEDSKY

FRA05 (Puscaghia) - 2 1 webTV, MEDSKY

FRA06 (La Grande Paroisse) - 3 2 RCCM, MEDSKY

FRA08 (Letia) - - 2 -

Brussels, 18. March 2010

response rates IV

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final Run

Phase B

RW applications used and rated

(red=test run, black=final run A,

green=final run B)

Greece 11 41 32

GRE01 (Σάλακος (Sálakos) 1 1 4

YouRA, D-Space, CONNECT, YouRA

webTV, YouRA, D-Space, TeacherNet,

NEMED, AgroTD, HealthTraining, RCCM

GRE02 (Αιγιάλη / Aigiálī) 1 2 1 YouRA, D-Space, CONNECT

GRE03 (Πυλές / Pylés) 1 1 1

NEMED, AgroTeleDiag.

webTV, YouRA, NEMED

webTV, YouRA, TeacherNet

GRE04 (Κλειστός / Kleistós) - 2 1

YouRA

YouRA

GRE05 (Βαλτεσίνικο / Valtesíniko) 1 3 1

YouRa, Teacher net

Xplora, YouRA

webTV, Xplora, TeacherNet

GRE06 (Άγιος Νικόλαος Βοιών / Ágios

Nikólaos Voiōn) 2 1 2 VEMUS, YouRA

GRE07 (Μεστά / Mestá) 1 1 1

webTV, VEMUS

webTV, VEMUS

webTV

GRE08 (Γεράκι / Geráki) 3 1 1

agroweb, webTV, Health Training

AgroTeleDiag

webTV

GRE09 (Νυμφαίο / Nymfaío) - 2 1

WebTV

webTV

GRE10 (Άγιος Λαυρέντιος / Ágios

Lavréntios) - 1 1

MEDSKY

RCCM

Brussels, 18. March 2010 10 RURAL WINGS IP

response rates V

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final Run

Phase B

RW applications used and rated

(red=test run, black=final run A,

green=final run B)

Greece GRE11 (Λέχοβο / Léchovo) - - 1 WebTV, Xplora, YouRA, TeacherNet

GRE12 (Ψαρά / Psará) - 1 1 YouRA, NEMED

GRE13 (Γλαύκη / Glávkī) - 1 1

ExperiNet

Xplora, ExperiNet

GRE14 (Οργάνη / Orgánī) - 1 1

D-Space

Xplora, D-Space, ExperiNet

GRE15 (Ορεστιάδα (Orestiáda) - 1 1

ExperiNet, TeacherNet

ExperiNet

GRE16 (Φλομοχώρι / Flomochōri) - 1 1 WebTV, NEMED

GRE17 (Κορυστά / Korystá) - 1 2 ExperiNet, EUDOXOS

GRE18 (Ανώπολη / Anōpolī) - 1 1

YouRA

YouRA

GRE19 (Σίσες / Síses) - 1 1

webTV, Health Training

NEMED

GRE20 (Γκούρα / Gkoúra) 1 2 webTV

GRE21 (Κομοτηνή / Komotīnī R) - 1 1

YouRa, ExperiNet, CONNECT

Xplora

Brussels, 18. March 2010

response rates VI

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final Run

Phase B

RW applications used and rated

(red=test run, black=final run A,

green=final run B)

Greece GRE22 (Αργύρι / Argýri) - 2 - YouRa, Teacher Net, NEMED

GRE23 (Βουνιχώρα / Vounichōra) - 7 1 Health Training

GRE24 (Καλλιφώνι / Kallifōni) - 1 1

YouRA

YouRA, D-Space, ExperiNet

GRE25 (Βερτίσκος / Vertískos) - 1 1 -

GRE26 (Λυσσαρέα / Lyssaréa) - 2 1

agroWeb, Xplora, ExperiNet, Teacher Net

Xplora

GRE27 (Άγιος Δημήτριος / Ágios

Dīmī Rtrios) - 3 1 NEMED

Hungary - - 2

HUN03 (N.N.) - - 2 YouRA

Israel - 1 59

ISR01 (חורה / حورة / Ḥūra) - - 5 agroweb, VEMUS, webTV

ISR02 (Fourier " פורייהفورييه/ ") - - 8 Agroweb, webTV

Brussels, 18. March 2010

response rates VII

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final Run

Phase B

RW applications used and rated

(red=test run, black=final run A,

green=final run B)

Israel ISR03 (اتد / Atid) - - 10 VEMUS, D-Space, NEMED

ISR04 (ابو تلول / Abū Tlūl) - - 11 VEMUS, Xplora, YouRA, TeacherNet

ISR05 ( ' / بئر هداجביר הדאג / Bir Hadāğ) - - 7 agroweb, webTV, UNITE

ISR06 (بئر مشاش / Bir Mišāš) - - 18

agroweb, VEMUS, webTV, Xplora, YouRA, D-

Space, ExperiNet, TeacherNet

ISR07 (ابو قويدر / Abū Qwaydar) - 1 0 -

Poland 2 17 12

POL01 (Babiogórski PN, Zawoja) 1 1 1 webTV

POL02 (Polana) 1 2 1

D-Space, Teacher net, VEMUS

Xplora

POL03 (Wiśniowa) - 2 1

Xplora, UNITE, D-Space, ExperiNet,

CONNECT, AgroTeleDiag

Xplora

POL04 (OAUJ, Mt. Lubomir) 2 1 D-Space

POL05 (Rokiciny) 1 1

webTV, Teacher Net

webTV, Xplora, YouRA

POL06 (Skawa) - 2 2 RCCM

Brussels, 18. March 2010

response rates VIII

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final Run

Phase B

RW applications used and rated (red=test run,

black=final run A, green=final run B)

POL07 (Nowy Łupków) - 1 1

D-Space, teacher Net, Health training

webTV

POL08 (Kęty) - 6 2

VEMUS, webTV, Xplora, YouRA, CONNECT

agroweb, Xplora, YouRA, TeacherNet

POL09 (Myczkowce) - - 1 WebTV, AgroTD

POL10 (Harkabuz) - - 1 D-Space

Romania 8 19 15

ROM01 (Dezna) 8 2 1

VEMUS, webTV, Xplora, YouRA, UNITE, D-Space, ExperiNet,

CONNECT, Teacher net, NEMED, AgroTeleDiag., Health

Training, RCCM

agroWeb, VEMUS, Xplora, YouRA, UNITE, D-Space,

ExperiNet, CONNECT, Teacher net, NEMED

VEMUS, YouRA, NEMED

ROM02 (Piatra Arsă) 1 1 Health Training

ROM03 (Cozieni) - 1 1

agroweb, VEMUS, webTV, Xplora, YouRA, UNITE, D-Space,

ExperiNet, CONNECT, Teacher net, NEMED

VEMUS, YouRA, NEMED

ROM04 (Golu Grabicina) - 1 1

VEMUS, webTV, Xplora, D-Space

NEMED

ROM05 (Arefu) - 1 1

Teacher Net

NEMED

ROM06 (Brebu) - 3 1

VEMUS, ExperiNet, CONNECT, Teacher net, NEMED, Health

Training

NEMED

Brussels, 18. March 2010 14 RURAL WINGS IP

response rates IX

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final

Run

Phase B

RW applications used and rated (red=test run,

black=final run A, green=final run B)

Romania

(cont.) ROM07 (Măguri-Răcătău) - 1 1

Xplora, UNITE, D-Space, ExperiNet, NEMED, AgroTeleDiag

NEMED

ROM08 (Dăbâca) - 1 1

Xplora, CONNECT, TeacherNet

NEMED

ROM09 (Orlat) - 1 2

agroweb, UNITE, CONNECT

VEMUS, NEMED

ROM10 (Rod) 1 1

VEMUS, TeacherNet, NEMED

NEMED

ROM11 (Deal) 1 1 VEMUS, NEMED

ROM12 (Nadăş) - 1 1

agroweb, VEMUS, webTV, Xplora, D-Space, NEMED,

MEDSKY

NEMED

ROM13 (Tilişca) - 1 1

agroweb, VEMUS, webTV, Xplora, YouRA, D-Space,

NEMED, AgroTD

NEMED

ROM14 (Bucureşti) - 3 1

VEMUS, YouRA, UNITE, Teacher Net, NEMED, Health

Training, MEDSKY

VEMUS; YouRA, NEMED, Health Training

Brussels, 18. March 2010 15 RURAL WINGS IP

response rates X

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final Run

Phase B

RW applications used and rated

(red=test run, black=final run A,

green=final run B)

Spain SPA05 (Rellinars) - 2 1

YouRA

YouRA

SPA06 (Laredo) - 1 - YouRA, webTV

SPA07 (Santa Comba) - 1 1 YouRA, webTV

SPA08 (Vilaverd) 1 -

YouRA, webTV, D-Space

webTV, YouRA

SPA09 (Reboreda) 2 -

YouRA, webTV

YouRA, D-Space

SPA10 (Blancafort) 2 1 YouRA, webTV

Ramallosa 7 - -

NEMED, AgroTeleDiag., agroweb, webTV,

Xplora, YouRA

Sant Sernip - 1 - webTV, YouRa, NEMED

Sweden 1 3 3

SWE01 (Tarfala) 1 - 1 MEDSKY

SWE02 (Lomträsk) - 2 1 webTV, MEDSKY

SWE03 (Avaviken) - 1 1 -

Brussels, 18. March 2010

response rates XI

Country Pilot site n

Test Run

n

Final Run

Phase A

n

Final Run

Phase B

RW applications used and rated

(red=test run, black=final run A,

green=final run B)

Spain 9 14 8

SPA01 (Teo-Campos) 1 1 1

NEMED, AgroTeleDiag, webTV, YouRA,

NEMED

webTV, YouRA, NEMED, EUDOXOS

SPA02 (Prats) 1 - 1

NEMED, AgroTeleDiag.

WebTV, YouRA

SPA03 (Bamonde) - 2 1

YouRA

YouRA

SPA04 (Arcos de Furcos) - 1 1 YouRA

UK 10 2 0

Bewholme 4 2 - webTV, CONNECT

Cilcennin 5 - - webTV, CONNECT, D-Space

Biggar 1 - - -

Brussels, 18. March 2010 17 RURAL WINGS IP

personal data I

Demographic aspects:

Gender: 66 female, 86 male participants

Age: majority (69%) is between 25 and 49 years

old.

Professions:

Other: i.e. civil servant, school consultant,

journalist or librarian

Brussels, 18. March 2010 18 RURAL WINGS IP

personal data II

Brussels, 18. March 2010 19 RURAL WINGS IP

infrastructure

Infrastructure characteristics:75% of the RW users use MS Internet Explorer as webbrowser, Firefox is used by 22% of the users. Netscape and Opera are used by only 1,3%.Nearly all responding RW users (96%) have MS windows installed as operating system. Five (3%) use a Linux system and no one uses a Macintosh.

Connection to the RW internet access:via Sat modem: 36,5% via local area network: 32,1%via indoor wireless network connection: 14,5%via outdoor wireless network: 6,9%

web browser IE Netscape Firefox Opera Safari

% overall 75.2% 1.3% 22.2% 1.3% 0.0%

operating

system Win Win Vista WinXP Win2000 Win98 WinNT Linux Mac N/A

users 153 29 118 6 0 0 5 0 2

in % 96.2 18.02 74.2 3.8 0.0 0.0 3.1 0.0 1.3

Brussels, 18. March 2010 20 RURAL WINGS IP

usage profiles I

Brussels, 18. March 2010 21 RURAL WINGS IP

usage profiles II

Services used:1. web browsing (88,7%)2. e-mail (74,8%)3. RW applications and tools (41,5%)4. Instant messaging (31,4%) 5. software updates (28,9%)

Websites visited (3 most visited):1. search engines2. news-sites 3. online-mailing4. RW-CAP5. weather forecast

Brussels, 18. March 2010 22 RURAL WINGS IP

usage profiles III

CAP use: Duration per week User Location

Brussels, 18. March 2010 23 RURAL WINGS IP

usability of infrastructure I

All categories are rated as excellent or good by at least 39%

Highest satisfaction can be seen in the area of assistance ratings which every fifth user (19.5% and 22%) judged as excellent. The initial installation was rated good or excellent by 69% of the users.

Deficits or problems seem to exist with the availability of the service (rated “poor” by 17.6% of the end users) and the reliability of the satellite terminal (rated “poor” by 7.5%).

Also free text answers and comments refer to low bandwidth and unreliable connections with repeated failures.

Brussels, 18. March 2010 24 RURAL WINGS IP

usability of infrastructure II

Brussels, 18. March 2010 25 RURAL WINGS IP

usability of infrastructure III

Summary usability ratings „poor“ (part I)

Brussels, 18. March 2010

usability of infrastructure IV

Summary usability ratings „poor“ (part II)

Brussels, 18. March 2010 27 RURAL WINGS IP

usability infrastructure V

Brussels, 18. March 2010 28 RURAL WINGS IP

usability RW-CAP I

RW applications are often used independently from the CAP: Data from Google Analytics has shown that slightly more than 50%

of the users did access these RW applications directly. The following evaluation results thus reflect the experience of the remaining up to 50% users.

Some users perhaps don‘t see the added value of the CAP: „Do not see a strong need to always enter through the portal when you already know where you want to reach out.”

RW-CAP ratings overall impression was similar to the interim evaluation:

A critical topic was the missing localization and translation of RW-CAP contents, as stated by several users in the free text comments: “The CAP is in English, because of that we cannot judge its relevancy.” (Spain) / „Not available in French.“ (France) / „The main problem in using the site is the language barrier.” (Poland)

However this has been solved by the beginning of 2010 with a new Greek + French version. Also national partners may insert translations by themselves easily if needed.

Brussels, 18. March 2010 29 RURAL WINGS IP

usability RW-CAP II

Brussels, 18. March 2010 30 RURAL WINGS IP

usability RW applications I

15 different RW applications were used and rated during the final runs.

39% of the participants in the survey had not used any RW application when answering the questionnaire (32% in final runs Phase A evaluation).

On the whole the applications were rated positively. less positive rating than during the last evaluation: Agroweb (15

users), VEMUS (14 users), WebTV (30 users), Xplora (29 users), D-Space (13 users) and Teacher eTraining (13 users)

lost slightly: YouRA (27 users), UNITE (2 users) and ExperiNet (8 users) rated as positive as the last time: NEMED (25 users), AgroTeleDiag (3

users), the Health Training (12 users), RCCM (9 users) and MEDSKY (6 users)

See Annex A of this presentation for detailed usability ratings for each application.

Brussels, 18. March 2010 31 RURAL WINGS IP

usability monitoring

Principles of RW usability monitoring User problems and questions can be indicative for aspects of the technical infrastructure, the

manuals, the central access point or the rural wings platforms and applications that are not designed intuitively.

It can thus be used to identify possible usability problems and areas of improvement during the training of the end users.

A structured template for the collection of user questions and problems arising during the training of the end users was provided to all NCs.

Data collection realized after February 2009; in final test runs feedback only from Spain.

Brussels, 18. March 2010 32 RURAL WINGS IP

results usability monitoring

Emerging topics:

Language was not a problem as localized sites are available;

PC-skills become an issues;

RW applications and CAP (problems with D-Space and CAP-Structure);

RW Infrastructure (Problems with network - suggested usage of a faster browser did improve the performance somewhat but reportedly not enough).

Manuals: 1

Tools: 6

Infrastructure: 1

Central Access Point: 3

Other: 0

Overall questions/

problems 7

Brussels, 18. March 2010 33 RURAL WINGS IP

heuristic evaluation

Design of heuristic evaluation Expert evaluation of the RW Platform by usability experts of TUD:

checklist of recognized usability principles (heuristic) will be used to identify possible usability problems (see Schweibenz/Thissen 2003 and Nielsen/Mack 1994).

identified usability problems and possible recommendations will be aggregated and communicated to FORTHnet and all NCs by TUD.

Expert evaluation was conducted twice: at prototype stage (2007) and end of the project (2009) after improvements were made.

Results overall better impression than during 1st usability evaluation:

lucid, structured, not overloaded. but information offered and options could be structured and

grouped better and adapted to the needs of the users often available in English only, even the help function (!)

language specific (sub) sites should be developed for Further details of the heuristic evaluation cp. annex A

Brussels, 18. March 2010 34 RURAL WINGS IP

Conlcusions I

Language and localisation Language barrier had been identified as one of the most crucial

issues in previous evaluations.

RW Infrastructure RW services are mostly used during the week and earlier in the

day, while the usage during the weekend has increased as compared to the last evaluation.

Apart from language barriers, problems with network performance and wifi coverage most dominant in user feedbacks.

Availability of the RW internet service has been rated as “poor” by 17.6% of the end users (compared to 11% during the last evaluation phase).

Negative experiences with bandwidth reliability did again retard local activities, hinder training sessions or even prevent proper use of RW services most of the time.

Brussels, 18. March 2010 35 RURAL WINGS IP

Conlcusions II

RW-CAP Development of CAP-usage:

in test run evaluation nearly all participants had used RW-CAP, in the final runs 18% stated “not applicable”;

only 29% of the participants compared to 40% in the test runs did name it as one of the three most visited websites.

Language barriers and navigation problems remain (monitoring and heuristic evaluation).

Overall feedback less critical than in the test runs (higher satisfaction or lower relevance of CAP?).

Suggested improvements: Single log-in for all applications via CAP which would provide an

added value; Community features (have been implemented roughly but not

promoted).

Brussels, 18. March 2010 36 RURAL WINGS IP

Conlcusions III

RW Applications 16 different RW applications were used and rated – predominantly

positive, albeit slightly lower than during final phase A. Although 39% of the participants in the survey had not used any RW

applications when answering the questionnaire. Free text comments division between users who found the applications

“very good”, “marvellous” or “motivating” and those who, for various reasons, didn’t manage to get some of the applications working at all.

RW training and support The overall rating of the RW installation services and the RW support was

very positive. About a fifth of the participating users did rate it as “excellent”.

Suggestion for successive projects: to ensure sustainability meta-competencies should be imparted (i.e.

problem-solving strategies in case of computer problems) to enable users to help themselves outside of training sessions.

Supported by community features in RW CAP to allow end-users to share best practice with other pilot sites and support each other when problems occur.

Brussels, 18. March 2010 37 RURAL WINGS IP

Conlcusions IV

Positive feedback

despite all the criticism that an evaluation has to reflect we can conclude the usability evaluation of the Rural Wings Project with positive comments of users on the RW-project:

“Thanks to this project we were able to contact other different realities and to share experiences and knowledge gained with them .”

“It is an extraordinary help for rural localities such as ours.”

“It's good to feel myself as a part of international project.”

“It is a great project that allows the educational rural communities to have access to Technologies of Information and Communication.”

Brussels, 18. March 2010 38 RURAL WINGS IP

Thank you for your attention.

Prof. Dr. Thomas Köhler Technische Universität Dresden

Media Centre http://mz.tu-dresden.de Phone: +49-(0)351-463-32772 Fax: -463-34963

Brussels, 18. March 2010

ANNEX A

Annex Ausability ratings for RW applications

Brussels, 18. March 2010 40 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 41 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 42 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 43 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 44 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 45 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 46 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 47 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 48 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 49 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 50 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010 51 RURAL WINGS IP

usability RW applications

Brussels, 18. March 2010

usability RW applications

Brussels, 18. March 2010

usability RW applications

Brussels, 18. March 2010

usability RW applications

Brussels, 18. March 2010

usability RW applications

Brussels, 18. March 2010

usability RW applications

Brussels, 18. March 2010

Further details heuristic evaluation I

Issues of 2007 that still remain• In a second phase of the evaluation process the issues that

were identified during the first evaluation of the CAP website in 2007 were reassessed.

 General problems• website suggests depth, but there merely are link lists

still triggering wrong expectations

• scope has to be made clearer still a problem, e.g. labels, country/user group specific adaptations

• navigation seems oversized in relation to available content• information cannot be accessed quickly enough

overall structure ok, but still no breadcrumb-type of navigation 

57 RURAL WINGS IP

Brussels, 18. March 2010

Further details heuristic evaluation II

Missing features• „help“-page/function should be added and links to it on every page

ok, but probably still more help needed by ordinary users, English only(!)

• FAQs should be added and updated on a regular basis not implemented, but less serious

 Structure• structure is not clear/consistent

issue remains, s.a.• navigation and paths should match!

paths seem to have been eliminated - not the preferred solution Design / Layout• layout will have to be improved (i.e. home>All rural

Wings>learning@work) issue remains, s.a.

 58 RURAL WINGS IP

Brussels, 18. March 2010

Further details heuristic evaluation III

Broken Links / missing attributes corrected, but new ones occur – cf. Annex B of evaluation report

 Other Features that should be improved• external and internal links have to be distinguishable

serious issue still remains, only solved in sitemap

• final URL should be international (.org or .net) and easier to remember than http://ruralwings.rd.forthnet.gr again considered a good idea, but not implemented

• wording should be target-group-specific (i.e. tools for students) very good idea, not implemented

59 RURAL WINGS IP

Brussels, 18. March 2010

ANNEX B

Annex BAdditional material

Brussels, 18. March 2010 61 RURAL WINGS IP

usability evaluation

Usability Evaluation

Usability of user interfaces is one narrow concern of the overall system acceptability. Figure 1 depicts Nielsen’s Taxonomy of System Acceptability, showing the relative placement of usability (Nielsen 1993, p.25).

Brussels, 18. March 2010 62 RURAL WINGS IP

evaluation focus

What will be evaluated?

utility vs. usability: utility is the question of whether the functionality of the

system in principle can do what is needed, usability is the question of how well users can use that

functionality.

Thus, the usability evaluation of the rural wings project will not investigate, if the services offered are suitable to meet the user needs, but how well end users and content providers can use the provided interface.

Brussels, 18. March 2010 63 RURAL WINGS IP

research questions I

Evaluation focus: usability of the RW “common access point” (CAP) and

RW platforms and services. This includes Internet access services (hybrid:

satellite-WiFi system), documentation and training materials and installation and maintenance procedures.

Central research questions are:

1. How well can users access and use the functionalities of RW CAP and the applications?

2. How is system usability of RW CAP and applications rated by end users in terms of the most important criteria of system usability (Learnability, Efficiency, Memorability, Errors,Satisfaction)?

Brussels, 18. March 2010 64 RURAL WINGS IP

research questions II

3. How satisfied are end-users with the Rural Wings Services and Infrastructure, i.e.

the installation services, the availability of the Rural Wings internet service, the reliability of the WiFi client equipment, the reliability of the Satellite terminal equipment, the technical support for the satellite and WIFI

infrastructure, training and introduction to the rural wings infrastructure,

platform and applications?

4. What questions, problems and needs arise during the training of the end users (i.e. are additional trainings required for specified pilot sites and/or contexts? Do specific target groups have special needs in terms of support, content or competencies?)?

Brussels, 18. March 2010 65 RURAL WINGS IP

coaching study

B.2.2 Coaching Method

The Coaching Method (Mack and Burdett 1992) includes explicit interaction between the test subject and the coach.

During a coaching study, the test user is allowed to ask any system-related question of an expert coach.

Coaching thus focuses on the novice user and is aimed at discovering the information needs of such users in order to provide better training and documentation, as well as possibly redesigning the interface to avoid the need for questions (Nielsen 1993).

The coaching method can relatively easy and without large expenses be integrated into the workshops that will be held to train the users and/or multiplicators (Task 6B.2) and can in addition serve to evaluate the provided support material (Task 6A.4).