UBICOMP ‘01

A Hybrid Evaluation Approach for Ubiquitous Computing Environments

Mark Burnett, Chris Rainsford DSTO, Australia

Types of EvaluationTypical

CharacteristicsQuantitative Qualitative

Target Problem Well defined, agreed metrics.

Loosely defined, new, unknown metrics

Pros Well defined outcomes, easy to compare candidates

Allows exploration, encourages diversity, human subjectivity allowed.

Cons Narrow focus, does not accommodate human subjectivity.

Not well defined, hard to compare candidates, inconclusive.

Evaluation Outcomes

Improved performance Discovery of new approaches, identification of strong points

TREC

What’s the equivalent of ROBOCUP for a ubicomp environment?

PKDD

A Hybrid Evaluation Framework

Environment:

•    Ubiquitous/pervasive

   Invisible

    Connected

   Context-aware

suggests hybrid of qualitative and quantitative approaches

How to evaluate a ubiquitous computing environment?

• choose a well-defined environment, for example a smart room

• choose some well-defined tasks for a hybrid evaluation

What to evaluate in a ubiquitous computing environment?

• software infrastructure and services in support of a task

• participants can build their own smart room (devices+infrastructure+s/w)