dissemination and evaluation
DESCRIPTION
Strategy for evaluating and communicate the results of the AtGentive projectTRANSCRIPT
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Dissemination and Evaluation
Claudia Roda, Damien Clauzel, Joona Laukkanen
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Dissemination: Atgentive book
• Latest version has been distributed– Need feedback on proposal (5 pages to
read!)– Participation:
• I am listing as committed: Cantoche, OBU, Ontdeknet, and AUP
• Informed but not committed: INSEAD, UTA (I have no description of contribution)
• I had no feedback from: CELN, STC • Could you please check your bios
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Dissemination: Atgentive Book
• Timeline:– First version of each chapter: 15 February
2008– Feedback from review: 15 March 2008– Final formatted version of each chapter 15
April 2008– Publication: ??
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Evaluation
• Objective: enable the assessment of the value of the conceptual framework and reasoning module in contexts that are not represented in the pilots
• Should this go under strategic evaluation?
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Evaluation
• Conceptual Framework - Demonstrate that the conceptual framework may hold in the context of very different implementations and applications.
• Reasoning module - New version of the test centre: – Visualize the reasoning of the reasoning module in
applications such as Ontdeknet– Show how the reasoning module can be used also in less
structured, non-task-based applications, and in the context of several (rather than one) applications.
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Evaluation: Conceptual Framework
• We look at the context restoration issue. • Virtual multi-desktop interface as an implementation of
context-restoration facility • We hope to demonstrate that context restore will
enhance both performance and user satisfaction• Hypotheses (Virtual multi-desktop interface vs Control):
– Increased perception of productivity and pleasure– Tasks are completed faster and better– Tasks are resumed faster and more easily– Fewer interface management actions and fewer mistakes on
interface management actions
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Evaluation: Conceptual Framework
• Experiment:– Subjects work for 20 minutes on three fully described tasks
(spreadsheet based, translation, video). – They are interrupted regularly by requests to work at a
different task, they are required to satisfy these requests– Control group (standard window interface), experimental
group (virtual multi-desktop)• Measures:
– Task completion level– Task quality level (# of errors)– Task resumption times– Reported evaluation
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Task 1: Spreadsheet
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Task 2: Translation
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Task 3: Video
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Interruption
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Resume
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Evaluation: Reasoning Module
• Reasoning module - New version of the test centre: – Visualize the reasoning of the reasoning
module in applications such as Ontdeknet– Show how the reasoning module can be
used also in less structured, non-task-based applications, and in the context of several (rather than one) applications.
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
The test center
• Tool for experimenting with the RM• The test center was not very intuitive• Desire to experiment with the future
vision of the RM (still working on)
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Desktop
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Desktop
• Minimal applications on a desktop– application provides task
model, sends events– interact with the application
instead of creating events
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Desktop
• Interventions
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
User and task properties editable
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Reasoning module activity
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Utilities for creating environment events: New information available, idle input
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
EXAMPLE
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
user has finished intro and is notified about next task (learning goals)..
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
user starts learning goals (interventions disappear)..
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
user starts the wrong task, gets interventions suggesting correct subtask. Math task is also suggested as possible alternative task ..
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
The applications
• Pretty much like real applications– Started from desktop icons..
• Known task model• For now just one application
(AtGentSchool)• Displays interventions allowing..
– to start suggested task– In the future to provide feedback on the
suggestions
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
The applications
• Pretty much like real applications– Started from desktop icons..
• Known task model• For now just one application
(AtGentSchool)• Displays interventions allowing..
– to start suggested task– In the future to provide feedback on the
suggestions
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
The future vision
• Several applications– Some applications provide their task model– Applications may be very different from
what we have seen so far• Task management side bar
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Future test center
AtGentive General Meeting, Amsterdam 22-23 Oct 2007
Future test center