a balance of the 2 nd edition stefano chessa evaal 2012 - evaluating aal systems through competitive...
TRANSCRIPT
A BALANCE OF THE 2ND EDITIONSTEFANO CHESSA
EvAAL 2012 - Evaluating AAL Systems through Competitive
Benchmarking
1
EvAAL 2012 - final workshop 24/09/2012
Motivations
24/09/2012EvAAL 2012 - final workshop
2
Ambient Assisted Living (AAL) seeks for technical solutions for the improvement of the quality of life of elders and disabled.
EvAAL aims at giving a pulse to research in AAL: by identifying research challenges relevant to AAL
The long term research challenge of EvAAL is the evaluation of complex platforms for AAL
Short term research challenges are the evaluation of components/services of AAL systems
by making available to researchers the datasets obtained during the competition;
by contributing to the construction of a community of people interested in AAL.
EvAAL : a brief history and current status
24/09/2012EvAAL 2012 - final workshop
3
EvAAL is a result of an EU project (universAAL)It was conceived in early 2010In 2011 the first edition
Only one track on Indoor Localization and Tracking for AAL
Relatively easy to formalize as a competition
In 2012 the second edition An additional track on Activity Recognition for AAL Growing community
Already planning the 2013 edition With (hopefully) another track
The EvAAL 2012 team
24/09/2012EvAAL 2012 - final workshop
4
Steering board (9 members) Italy (3), Spain (3) Germany (2) Denmark (1)
Program Committee of the Indoor Localization and Tracking track 19 members from EU, USA and Australia
The local and evaluation team
The EvAAL 2012 team
24/09/2012EvAAL 2012 - final workshop
5
Program Committee of the Activity Recognition track 16 members from EU, USA and Japan
(17 members from Spain and Italy)
Local and evaluation teams
Competitors selection process
24/09/2012EvAAL 2012 - final workshop
6
Indoor Localization: 9 submissions Each submission was reviewed by 3 TPC members 1 rejected as not suitable for AAL 1 admitted as guest 7 admitted as competitors
Activity Recognition: 6 submissions Each submission was reviewed by 2 TPC members 1 rejected as not suitable for AAL 1 withdrew 4 admitted as competitors
Indoor Localization TrackiLocPlus (Germany/Switzerland)
CPS Group @ Utah (USA)
CAR (Spain)
LOCOSmotion (Germany)
Grupo TAIS (Spain)
OwlPS (France)
Smart-Condo (Canada)
Lambda4 (Germany) - GUEST
7
24/09/2012EvAAL 2012 - final workshop
Activity Recognition Track
SenseCam (Ireland)
AMEVA (Spain)
Chiba University (Japan)
Carnegie Mellon University (USA)
8
24/09/2012EvAAL 2012 - final workshop
The Localization Benchmark
24/09/2012EvAAL 2012 - final workshop
9
The benchmark consists of a set of tests, each of which contributes to assessment of the scores for the competing system. Phase 1: Area of interest test Phase 2: Locate the actor Phase 3: The disturber
Each localization system need to produce localization data with a frequency of 1 data every half a second.
The paths followed by the person were the same for each test, and it was not disclosed to competitors before the competition.
We also provide some context information Switch Cyclette
Area of Interest
24/09/2012EvAAL 2012 - final workshop
10
Phase 1: Locate a person inside an Area of Interest (AoI). The AoI in a typically
AAL scenario could be inside a specific room (bathroom, bedroom), in front of a kitchen etc.
Each system was requested to identify: 6 big AoI 4 hierarchical small AoI.
The actor moved along predefined paths staying inside each AoI for at least seconds (227 steps)
Starting Point
End Poin
t
The Actor
24/09/2012EvAAL 2012 - final workshop
11
Phase 2: A person that moves inside the Living Lab had to be located and tracked. Only the person to be
localized was inside the Living Lab.
The first path was 54 steps length, the second path 94.
All the paths were characterized by 3 waiting points, i.e. the actor stayed in the same position for 5 seconds.
The disturber
24/09/2012EvAAL 2012 - final workshop
12
Phase 3: Was organized as the former phase, but the competing artefacts were evaluated in the presence of another person who moves inside the Living Lab. Only the actor had to be localized by
the competing artefacts There was no requirement of
localizing the disturber Only the actor has carried equipment
provided by the competitor The actor started moving at least 5
seconds before the disturber Both the actor and the disturber have
generated contextual events but the first such event was generated by the actor
When the disturber generated an event, the actor was at least 2 meters away.
The first path was 94 steps length (while the disturber path was 54), the second path 80 (while the disturber path was 94).
The reference localization system
24/09/2012EvAAL 2012 - final workshop
13
The reference localization system is essential to measure the accuracy of the competing systems. Was composed by predetermined coordinates of the paths
followed by the actor during the competition The Living Lab’s floor was covered with marks (with
different color to distinguish the right and left foot) that show where the actor had to step on.
In order to facilitate the installation and removal of the paths the marks were put on a piece of wood.
The synchronization between the steps and the evaluation tool was guaranteed by a digital metronome that indicated the right cadence (one beep one step). In this way we guaranteed that the actor repeated the
same paths at the same speed for every competitor.
Evaluation criteria
24/09/2012EvAAL 2012 - final workshop
14
Accuracy [weight 0.25] This is the main purpose of a localization system The score is attributed in a range of errors between 0
and 4 meters An average error above 4 meters would mean that the
system does not really know in which room the user is…
Availability [weight 0.2] …nobody wants a system that does not responds…
Evaluation criteria
24/09/2012EvAAL 2012 - final workshop
15
Installation complexity [weight 0.15] Measure the invasiveness of the system installation in
the home of the final user Think whether if you would be happy to install these
systems in your home…
Interoperability with AAL systems [weight 0.15] How easy is to use your system from the point of view
of AAL applications programmers
Evaluation criteria
24/09/2012EvAAL 2012 - final workshop
16
User acceptance [weight 0.25] Would you use your system in your daily life? More advanced and better engineered systems where
advantaged against research prototypes… … but we considered also your system in the
perspective of an engineering
Financing of EvAAL
24/09/2012EvAAL 2012 - final workshop
17
SPONSORS
ASUS and Texas Instruments contributed with prizes for winners
GIRAFF Plus and universAAL contributed with free registrations to AAL Forum for competitors
18
24/09/2012EvAAL 2012 - final workshop