conga – the context gallery - etd.dtu.dketd.dtu.dk/thesis/224649/ep08_107.pdf · conga – the...

Conga – The Context Gallery

Using Context Awareness for Personal Content

Management on Mobile Devices

Brian Munck Andersen

Kongens Lyngby 2008

Technical University of Denmark

Department of Informatics and Mathematical Modeling

Building 371, 2nd Floor, DK-2800 Kongens Lyngby, Denmark

Phone +45 45253351, Fax +45 45882673

[email protected]

www.imm.dtu.dk

Abstract

In this thesis, I will present my suggestion of using context awareness to ease

personal content management on mobile computing devices. Different

approaches to content management exist, with the use of folder hierarchies

being the predominant solution in use today. Other approaches used are tagging,

and annotating content using faceted metadata. Common for all these

approaches is the tediousness inherent in managing content manually, as well as

the user interface problems present in using them on certain types of mobile

computing devices, due to the limitations in user interaction options.

To lower the impact of these problems, I suggest automatically gathering context

information and use it to semi-automatically annotate generated content. This

context information can then be used to provide the user of the context aware

personal content management system with different search and filtering options,

easing the task of organizing content.

Following a theoretical introduction to the fields of personal content

management, mobile computing, and context awareness, I describe different

suggestions for the design of such a system, focusing on the possibilities present

today. During the course of work on this thesis, a prototype application has been

constructed. This prototype is named Conga, which is short for Context Gallery,

and allows users to take pictures and manage these based on gathered context

information. Conga is intended as a proof of concept, demonstrating the

ii

usefulness of context information for personal content management. The findings

made in working with this prototype are discussed, and it is found that the

possibilities of gathering context information and using it to ease personal

content management are available today, as demonstrated by the Conga

prototype, but there are also certain limitations and considerations to keep in

mind when constructing a context aware personal content management system.

Resumé

I dette eksamensprojekt for civilingeniørstudiet vil jeg præsentere mit forslag til

brug af kontekst-opmærksomhed for at lette personlig datahåndtering på mobile

enheder. Der findes forskellige tilgange til datahåndtering, med brugen af mappe-

hierarkier som den mest fremtrædende løsning i dag. Andre tilgange er tagging,

og beskrivelsen af data ved brug af facet-opdelt metadata. Fælles for disse

tilgange er den trivielle opgave bundet til manuel datahåndtering, så vel som

problemerne med brugergrænsefladen på visse typer mobile enheder grundet

begrænsninger i mulighederne for brugerinteraktion.

For at mindske betydningen af disse problemer, foreslår jeg automatisk

indsamling af kontekst-information, og brug af denne information til semi-

automatisk at annotere genereret data. Denne kontekst-information kan derefter

bruges til at tilbyde brugeren af det kontekst-opmærksomme personlige

datahåndterings-system forskellige søge- og filtreringsmuligheder, der kan lette

opgaven med at organisere data.

Efter en teoretisk introduktion til områderne personlig datahåndtering, mobile

computing og kontekst-opmærksomhed, beskriver jeg forskellige forslag til

designet af et sådant system, med fokus på mulighederne der findes i dag. Under

arbejdet med dette speciale har jeg konstrueret en prototype applikation.

Prototypen er døbt Conga, hvilket er kort for Context Gallery, og den lader

brugeren tage billeder, og håndtere disse baseret på den indsamlede kontekst-

iv

information. Conga er ment som et proof of concept, der demonstrerer

brugbarheden af kontekst-information ved personlig datahåndtering. Fundene

gjort under arbejdet med prototypen diskuteres, og det ses, at mulighederne for

indsamling af kontekst-information samt brugen af dette til at lette personlig

datahåndtering, er til stede i dag, men der er også visse begrænsninger og

overvejelser at gøre sig ved konstruktion af et kontekst-opmærksomt personligt

datahåndterings-system.

Preface

This thesis was prepared at the Department of Informatics and Mathematical

Modeling (IMM) at the Technical University of Denmark (DTU) in partial

fulfillment of the requirements for acquiring the degree of Master of Science in

Engineering (M.Sc.Eng).

This M.Sc.Eng thesis is the result of work carried out in the period from February

2008 to October 2008, with a workload of 30 ECTS credits.

Lyngby, October 2008

Brian Munck Andersen

Acknowledgements

I would like to thank a number of people who have assisted me in writing this

thesis.

First and foremost I would like to thank my supervisor Assistant Professor Jakob

Eg Larsen for always providing insightful feedback, sharing his knowledge, and for

our long discussions of my work, and the topic in general, which have inspired me

greatly.

I would also like to thank my friends and colleagues, with whom I have discussed

my work at numerous occasions, always yielding useful results and new insights

into the topics I have worked with.

Finally, I would like to thank my girlfriend Marianna and my family, for moral

support and for coping with my absence while working on this thesis.

Contents

ABSTRACT I

RESUMÉ III

PREFACE V

ACKNOWLEDGEMENTS VII

CONTENTS IX

LIST OF FIGURES XV

LIST OF TABLES XVII

1 INTRODUCTION 1

1.1 A Suggestion for a Different Content Management Scheme 2

1.2 The Possibilities Already Exist 4

x Contents

1.3 Report Overview 4

2 PERSONAL CONTENT MANAGEMENT 7

2.1 Growing Amount of Personal Content 7

2.2 Traditional Content Management 9

2.2.1 Digital Personal Content Management 10

2.2.2 The Problem with Categorization 11

2.3 Tagging 12

2.3.1 The Use of Tagging Today 12

2.3.2 The Problems with Tagging 14

2.4 Faceted Metadata 15

2.5 Searching for Content 17

2.5.1 Searching for Contents 17

2.5.2 The Dimensions of Memory 19

2.6 Summary 19

3 MOBILE COMPUTING 23

3.1 The Spread of Mobile Computing Devices 23

3.2 Different Mobile Computing Devices 24

3.2.1 Mobile Phones 25

3.2.2 PDAs and Smart Phones 25

3.2.3 Ultraportable Computers 26

3.2.4 Laptop Computers 26

3.3 Personal Content Management on Mobile Devices 27

3.3.1 The Limitations in Mobile Computing Devices 27

3.4 Summary 28

Contents xi

4 CONTEXT AWARENESS 31

4.1 History and Definition 31

4.2 Context Aware Systems Today 34

4.3 Inferring Context from Sensors 35

4.3.1 Inferring Context from the Sensors Available Today 36

4.4 Using Context Awareness for Personal Content Management 36

4.4.1 Going Beyond Specific Types of Content 38

4.5 Problems with Context Awareness 39

4.6 Summary 41

5 DESIGN 43

5.1 Different Design Possibilities 44

5.1.1 Stand-Alone Application Creating Content 44

5.1.2 Stand-Alone Application for Content Management 45

5.1.3 Stand-Alone Content Management Application Integrated With the

Operating System 45

5.1.4 Building Context Awareness into the Operating System 46

5.1.5 Extending the Existing Content Management Scheme with Context

Awareness 48

5.2 Gathering Context Information 50

5.3 Using the Gathered Context Information 52

5.4 Considerations and Challenges 54

5.4.1 Saving Context with Content 54

5.4.2 Limitations in Mobile Devices 55

5.4.3 Faulty Assumptions and Incomplete Context Information 56

5.4.4 The Vocabulary Problem 57

xii Contents

5.5 Summary 57

6 IMPLEMENTATION 59

6.1 The Aim of the Prototype 59

6.2 Building the Prototype 60

6.2.1 Modules Used 61

6.2.2 Sensors Used 61


6.3.1 Translating the Gathered Context Information 62

6.4 Saving All Context Information 65

6.4.1 Saving Context Information in a Database 67

6.5 Structure of the Prototype Implementation 69

6.6 The User Interface 70


6.7.1 Simple Search 72

6.7.2 Advanced Search 73

6.8 Problems, Limitations and Result 74

6.8.1 Using the Sensors in Pys60 74

6.8.2 Pys60 75

6.8.3 The Implications 75

6.8.4 Result of the Prototype Implementation 76

6.9 Summary 78

7 TEST 81

7.1 Context Information Gathering on the Prototype 81

7.2 Finding Content Using Search 84

Contents xiii

7.3 The Ideal User Test 85

7.4 An Actual User Test 86

7.4.1 Observations 88

7.4.2 Analyzing the Search Terms 90

7.4.3 Usability 91

7.5 Summary 91

8 DISCUSSION 93

8.1 The Prototype 93



8.4 Summary 97

9 RELATED WORK 99

9.1 MMM and MMM2 99

9.2 MediAssist 100

9.3 MobTAG 101

9.4 Summary 101

10 CONCLUSION 103

10.1 Contributions 104

11 BIBLIOGRAPHY 105

A INSTALLING CONGA 113

xiv Contents

B SETTING UP CONGA 115

C USING CONGA 117

D CONTENTS OF THE ENCLOSED CD 125

E CODE OVERVIEW 127

List of Figures

FIGURE 1 - FUSING THE DISCIPLINES OF CONTEXT AWARE SYSTEMS, PERSONAL CONTENT

MANAGEMENT, AND MOBILE COMPUTING, TO CREATE A CONTEXT AWARE PERSONAL

CONTENT MANAGEMENT SCHEME TO BE USED ON MOBILE COMPUTING DEVICES. .......................... 3

FIGURE 2 – THE TYPICAL USE OF A TRADITIONAL FOLDER HIERARCHY, WITH MULTIPLE LAYERS OF

FOLDERS DIVIDING CONTENT INTO DESCRIPTIVE CATEGORIES AND SUB-CATEGORIES (76) .............. 10

FIGURE 3 – A TAG CLOUD CONSTRUCTED USING TAGS FROM FLICKR. THIS IS USED TO DISPLAY THE

POPULARITY OF DIFFERENT TAGS ON THE FLICKR WEB SITE, PROVIDING VISITORS WITH AN

OVERVIEW OF OFTEN USED TAGS AS WELL AS NAVIGATION OPTIONS BY CLICKING ON LINKS (13) ..... 13

FIGURE 4 – THE FACETED METADATA SYSTEM USED IN SEARCHING FOR WINE BASED ON THE DIFFERENT

FACETS ON WINE.COM (18). THE DIFFERENT FACETS CAN BE USED TO SELECT SPECIFIC PRICES,

TYPES, REGIONS AND PROFESSIONAL RATINGS, FILTERING THE RESULTS BASED ON THESE ............... 15

FIGURE 5 – NOKIA 3500 AND NOKIA 6500 CLASSIC (35). TWO TYPICAL SIMPLE MOBILE PHONES

FOUND ON THE MARKET TODAY ......................................................................................... 25

FIGURE 6 – NOKIA N78 (35) AND APPLE IPHONE 3G (36). TWO EXAMPLES OF MODERN HIGHLY

CAPABLE SMART PHONES ................................................................................................. 25

FIGURE 7 – ASUS EEE PC 2G SURF (38), THE FIRST VERY POPULAR LOW-COST ULTRAPORTABLE

COMPUTER ................................................................................................................... 26

FIGURE 8 – ACER ASPIRE ONE (39), ACERS NEW LOW-COST ULTRAPORTABLE COMPUTER ....................... 26

FIGURE 9 – DELL XPS M1330 (40), AN EXAMPLE OF A MORE TRADITIONAL LAPTOP .............................. 26

FIGURE 10 - STAND-ALONE APPLICATION CREATING AND MANAGING ITS OWN CONTENT. CONTEXT

INFORMATION IS SAVED AND MANAGED BY THE APPLICATION .................................................. 44

FIGURE 11 - STAND-ALONE APPLICATION USED FOR MANAGING ALL CONTENT. THE GATHERED CONTEXT

INFORMATION IS SAVED AND MANAGED BY THE APPLICATION .................................................. 45

xvi List of Figures

FIGURE 12 - STAND-ALONE CONTENT MANAGEMENT APPLICATION INTEGRATED WITH THE OPERATING

SYSTEM. THE GATHERED CONTEXT INFORMATION IS SAVED AND MANAGED BY THE APPLICATION ... 45

FIGURE 13 - BUILDING CONTEXT AWARE PERSONAL CONTENT MANAGEMENT INTO THE OPERATING

SYSTEM. THE GATHERED CONTEXT INFORMATION IS SAVED AND MANAGED BY THE OPERATING

SYSTEM, WHICH USES IS AS THE BASIS FOR THE ENTIRE CONTENT MANAGEMENT SCHEME ............. 47

FIGURE 14 - BUILDING CONTEXT AWARE PERSONAL CONTENT MANAGEMENT INTO THE OPERATING

SYSTEM. THE GATHERED CONTEXT INFORMATION IS SAVED AND MANAGED BY THE OPERATING

SYSTEM, WHICH USES IT TO EXPAND THE SEARCHING AND FILTERING OPTIONS OF THE PRESENT

CONTENT MANAGEMENT SCHEME ..................................................................................... 49

FIGURE 15 - THE TRANSLATION OF THE SOLAR TIMES OF DAY INTO CONTEXT INFORMATION. THE TIME

OF DAY AS SEEN ON THE TIMELINE IS TRANSLATED INTO THE GIVEN DESCRIPTIVE NAME SHOWN

IN THE BOXES AT THE BOTTOM .......................................................................................... 64

FIGURE 16 - DATABASE DIAGRAM FOR THE DATABASE CONSTRUCTED TO HOLD THE GATHERED CONTEXT

INFORMATION. THE TABLES ARE CONSTRUCTED BASED ON THE FACETS DESCRIBED IN SECTION

4.4.1, WITH A SEPARATE TABLE CONTAINING INFORMATION ABOUT THE GATHERED CONTENT ...... 68

FIGURE 17 - THE LAYERED STRUCTURE OF CLASSES IN THE PROTOTYPE IMPLEMENTATION. THE CLASSES

ARE DIVIDED INTO A TYPICAL 3 LAYER ARCHITECTURE, WITH A USER INTERFACE LAYER, A

FUNCTION LAYER, AND A DATABASE LAYER .......................................................................... 69

FIGURE 18 - THE TWO MAIN PARTS OF THE USER INTERFACE, NAMELY THE CAMERA AND THE GALLERY,

AND THE OPTIONS THESE PROVIDE. WHEN CONGA IS STARTED, THE CAMERA PART IS INITIALLY

ACTIVE, ALLOWING THE USER TO TAKE PICTURES OR OPEN THE GALLERY FROM THE MENU. ........... 70

FIGURE 19 - PERFORMING A SINGLE SEARCH USING MULTIPLE SEARCH TERMS SEPARATED BY COMMA ....... 72

FIGURE 20 - PERFORMING AN ADVANCED SEARCH USING THE ADVANCED SEARCH FORM. THIS ALLOWS

A SEARCH BASED ON CHOSEN DATES, TIMES, TEMPERATURES, HUMIDITY, TIME ZONE, AND A

LARGE NUMBER OF TEXT INPUT FIELDS ............................................................................... 73

FIGURE 21 - A PICTURE TAKEN USING CONGA, DISPLAYED IN FULL SCREEN MODE, AND A DISPLAY OF

THE CONTEXT INFORMATION GATHERED AND SAVED TOGETHER WITH IT .................................... 78

List of Tables

TABLE 1 - AN OVERVIEW OF ALL THE GATHERED CONTEXT INFORMATION USED TO ANNOTATE A TAKEN

PICTURE. THE INFORMATION, GROUPED INTO THE FACETS GENERATED IN SECTION 384.4.1, IS

GATHERED USING THE NOTED SENSOR, OR TRANSLATED USING THE WEB SERVICE BASED ON THE

FOUND GPS COORDINATE. ................................................................................................. 66

TABLE 2 - THE SUMMARIZED RESULTS OF A SIMPLE TEST OF CONTEXT GATHERING. THE CONTEXT

INFORMATION GATHERED FOR A PICTURE TAKEN INDOORS AND ONE TAKEN OUTDOORS IS

COMPARED TO THE EXPECTED CONTEXT INFORMATION. ............................................................ 84

TABLE 3 – THE RESULTS OF THE USER TEST. THE PICTURE NUMBER, SEARCH TERM, NUMBER OF FOUND

PICTURES, AND THE RESULT OF THE SEARCH IS NOTED, WITH (ADV) DENOTING A SEARCH USING

THE ADVANCED SEARCH FORM ............................................................................................. 88

TABLE 4 – A COMPARISON BETWEEN DIFFERENT RELATED PROJECTS BASED ON SELECTED MAIN POINTS

OF THE WORK PERFORMED ................................................................................................ 102

xviii List of Tables

CHAPTER 1

1 Introduction

As people create and store an ever increasing amount of data, especially personal

content such as images, videos, documents, bookmarks, audio recordings, and

emails, the challenge of managing, organizing, and finding this content is growing

rapidly.

Most people tend to remember some information about the context they were in

when creating a given piece of content, such as the location they created it in,

who was there when it was created, the day of the week, or even arbitrary

information such as the weather at the time of creation. Using context

information for personal content management is, in a sense, already the norm for

pictures and videos, which are often categorized by their context, such as a

person, place, or event depicted in these, but in this thesis, I will argue that this

context information could help describe all types of content and ease the process

of finding it at a later point in time.

2 Introduction

1.1 A Suggestion for a Different Content Management Scheme

In the last few years, a new trend has been emerging for picture management,

namely geo-tagging. Geo-tagging is simply to add location information, usually a

GPS coordinate, to a picture, thus allowing people to search for, and browse,

their pictures based on the location in which it was taken.

I suggest taking this simple context based categorization one step further, by

automatically adding all available context information to all created content, thus

allowing users to find their content based on the context in which it was created.

In a stationary computing environment, changes in context are minor, but with

computing devices today becoming increasingly mobile, much more context

information is available which can help describe the content being created. This

can form the basis for a personal content management scheme, with content

being organized not only based on its contents, but on the context in which it was

created.

Such a described personal content management scheme would fuse the

disciplines of context awareness, mobile computing, and personal content

management, and utilize the advantages found in these disciplines to create an

alternative personal content management scheme, very suitable for modern

mobile computing.

A Suggestion for a Different Content Management Scheme 3

Figure 1 - Fusing the disciplines of Context Aware Systems, Personal Content Management, and

Mobile Computing, to create a context aware personal content management scheme to be used

on mobile computing devices.

Automatically adding context information introduces another significant

advantage, namely the partial removal of the tedious task of content annotation.

Since all available context information can be added automatically, the user does

not have to spend time annotating his files. Even though no time has to be spent,

it is still very important to enable users to add more information, and to edit the

already created metadata, to cater to different user types, and to allow users to

correct wrongly assumed context information.

Context

Aware

Systems

Personal

Content

Management

Mobile

Computing

4 Introduction

1.2 The Possibilities Already Exist I choose to focus my work in this thesis on the possibilities already existing today,

and not on a thought up example requiring technology which is not already

available. Even though lots of progress is made in context awareness research

focusing on placing multiple sensors on a user, this is not present in everyday life

today. What is present is a great variety of highly capable mobile computing

devices, with lots of context information at hand, which has not yet been used in

a greater scheme. My work will therefore focus on using this available context

information in a constructive way, and examining how using it can aid us in our

daily personal content management.

With computer processing power constantly increasing, and Moore’s law still

holding true today, I have chosen not to focus on implementing advanced search

algorithms and optimizations for reducing the performance requirements of the

personal content management scheme. This is of course an important aspect of

creating a commercially available successful personal content management

scheme, but falls outside my focus for this thesis.

I will focus my work in this thesis on the possibilities already existing today, and

have chosen deliberately not to focus on privacy considerations, performance

optimization, and partly the inherent vocabulary problem in my suggestion, as

these fields alone contain enough material for a thesis. Instead, I focus on

describing my idea, designing a prototype to test the usability of adding inferred

context information to context, describing my findings in working with the Conga

prototype, and presenting my suggestions for further work in this field.

1.3 Report Overview In this thesis, I will first give a theoretical introduction to the fields of personal

content management, mobile computing, and context awareness. I will also

introduce my suggestion of fusing these disciplines in the creation of a context

aware personal content management scheme.

Report Overview 5

Next, I present my ideas for the design of a system using the created scheme, and

discuss the considerations to be made in the creation of such a system.

Following the design suggestions, I describe my work on implementing the Conga

prototype serving as a proof of concept of my suggested context aware personal

content management scheme. Working with the prototype, and testing it, has

yielded a number of interesting findings, which are discussed next.

Finally, I introduce possibilities for future work on this topic, as well as some

related work being done in the field, and conclude on my work and findings of

this thesis.

6 Introduction

CHAPTER 2

2 Personal Content Management

In this chapter, I will give an insight into the growing amounts of personal

content, followed by an introduction to traditional content management. I will

then introduce two other content management schemes, namely tagging and the

use of faceted metadata. Following the introduction to these different content

management schemes, I will describe new advances in search technology. Finally

a summary will relate these three schemes, and discuss their usage.

2.1 Growing Amount of Personal Content In a study carried out by the School of Information Management and Systems at

the University of California at Berkeley (1), a low estimate of the amount of data

created in 2002 is set at 3.4 Exabyte. This data consists of both professional

information, ranging from cinema and TV to corporate documents, and personal

content. 12% of this information is stored on harddisks, and I assume this

percentage to be increasing, as harddisks are becoming more common for

especially multimedia storage, in for example video cameras and TV recorders

such as TiVos. I also assume the percentage of information stored on flash

memory, which was around 0.3% in 2002, to be increasing, as digital cameras and

camera phones, as well as most digital portable music players, store information


in flash memory. More than 3.5% of the information created in 2002 was

photographs recorded on film. As digital cameras are taking over from film

cameras, this information will also be stored in magnetic media. Studies show

that people tend to take a lot more pictures with digital cameras, a trend that

might also be seen with digital video cameras, which will also influence the

amount of personal information created (2).

The amount of newly stored information grew at about 30% a year between 1999

and 2002, and this trend has most likely continued with the spread of computers,

digital cameras and camera phones, and digital video cameras. This effect is

already being seen with users of camera phones reporting, that they take more

humorous and ad-hoc pictures with their camera always at hand (3).

As mentioned by Fitzgibbon and Reiter (4), it is now possible to store every digital

photograph one takes, and this is not only true for photographs, but for all digital

personal content. Storage space is likely to keep growing together with the

amount of personal content, which, along with the new possibilities brought

forward by the digital revolution, enables us to document and save almost all

aspects of our daily lives, creating a so-called Memex which was first envisioned

by Vannevar Bush in 1945 (5). Creating a Memex, or digital diary/memory, is the

focus of the works of many researchers. As an example of a work in progress,

Gemmell, Bell, and Lueder from Microsoft Research have been working on the

MyLifeBits system, which is directly inspired by Bells vision (6). This type of

application will likely be commercialized and introduced to the public within the

next decade, and will result in a massive growth of personal content to manage.

Traditional Content Management 9

2.2 Traditional Content Management Managing analogue personal content, such as pictures in print, video cassettes,

and books, has mostly been done by categorization. Each piece of content is put

into a category, for example a photo album or box containing pictures from a

vacation, a video collection sorted by recording date, or an alphabetical sorting

scheme for books. Given the fact of physical objects existing in only one place at a

time, it is impossible to have objects in several categories at once, for example

having the same picture in two photo albums, one for a specific vacation and one

for sunsets, except if a copy is made, which is very impractical. This has led to

elaborate categorization schemes being constructed for libraries and other large

collections, to aid in finding sought after content.

In 1983, Malone analyzed how a group of office workers organized information

on their desks and in their offices (7). In his findings, he grouped his interviewees

into two groups, namely “messy” and “neat”, with “messy” office workers having

many piles of documents, magazines, books, and notes spread out on their desk

and around the office. “Neat” workers, on the other hand, had a much more

organized desk and office, with established workflows and specific stacks of

documents for specific tasks. The “messy” group does not necessarily represent a

group of people who work less efficiently, and have lost control over their tasks,

they merely have a different preference when it comes to organization.

Malone found, that an important aspect of desk organization is reminding. The

interviewed workers would leave documents visible, to serve as reminders of

work to be done, which leads him to suggest that this reminding function should

be built into computer-based systems. Furthermore, Malone finds that there is

both a mechanical and cognitive difficulty in categorizing documents. The

mechanical difficulty consists of the work required to create labeled folders,

binders, and so on, and the cognitive difficulty is inherent in the problem of

creating appropriate categories for filing and easy retrieval.

Malone lists a number of “implications for designing computer-based information

systems”. These include “multiple classification”, where the computer system

would allow a document to be easily put in several categories, “deferred


classification”, where files can be categorized by storing them in some physical

location (much like a pile of papers on a desk), and “automatic classification”,

where the computer system automatically classifies information as much as

possible.

2.2.1 Digital Personal Content Management The categorization approach to content management has been brought into the

world of digital content management as well, and has resulted in the traditional

folder hierarchy which most people rely on to organize their information today.

As examples of this, we see most email programs using folders for organization,

and operating systems letting users sort files into folders as well.

The digital folders suffer from the same

limitation as their analogue counterpart,

namely that a file can only exist in one folder

at a time, unless copies are made. This leads

to different categorization schemes being

invented, which try to aid management of

files, but also require quite a bit of caretaking,

especially as the amount of content grows.

This is in line with the findings made by

Malone. Most operating systems allow the

users to place documents and folders on their

digital desktop, enabling them to keep

important documents “at hand”, thus

following Malone’s suggestion of being able

to keep documents visible as reminders.

The aforementioned increase in personal

content can easily lead to so called

“information overload”, which is most

commonly defined as “an excessive volume

Figure 2 – The typical use of a traditional

folder hierarchy, with multiple layers of

folders dividing content into descriptive

categories and sub-categories (76)

Traditional Content Management 11

of information”, and “difficulty or impossibility of managing it” according to

Farhoomand and Drury (8). The problem of information overload underlines the

need for efficient and useful content management, and therefore much research

has been carried out in this field.

Henderson has analyzed the way a group of knowledge workers store their digital

files (9). She finds that the participants in her study have very different patterns

of organization, with one participant being a classic “non-filer”, with a high

number of files per folder and a shallow hierarchy, who relies more on search to

locate his files, and another participant being a typical “frequent filer”, who stays

organized and uses the folder hierarchy to locate needed documents. These are

two typical approaches to personal content management, demonstrating how

people use different methods to organize their files. The findings are completely

in line with Malone’s findings 21 years earlier, and show that we have adopted

the same organizational schemes for our digital content that we use to manage

physical documents.

2.2.2 The Problem with Categorization As described by Malone, categorization can lead to cognitive difficulties. This is

further discussed by Sinha (10), who describes how the categorization required to

organize information in folders can be problematic, since only one of potentially

many describing categories for a given piece of information has to be chosen as

the filing category. This can lead to insecurity as to which category is most fitting,

since it has to be both the most descriptive category, and also be the one we are

most likely to search when looking for this information in the future.

Furthermore, we have to consider the overall categorization scheme at the time

of categorization, and worry about the correctness of this. Finally, the expensive

nature of the time needed to change categories at a later time means, that the

initial categorization is very important. The author has coined the phrase ”post

activation analysis paralysis” to describe the state of fear of making the wrong

decision in categorization, and that the information at hand will therefore be lost

forever.


The before mentioned tradeoff between the time spent searching for files and

the time spent organizing them, together with the problem of categorization

leading to “post activation analysis paralysis”, underlines a significant limitation

inherent in the classical folder hierarchy. The limitations found in the folder

approach can significantly hinder efficient file management, or at the least make

it a tedious task.

2.3 Tagging With the inherent problems found in categorization in a folder hierarchy, another

approach to information management is becoming increasingly popular, namely

tagging of content. Tagging means adding metadata information to content, with

the metadata information containing any type of describing information, such as

a description of the content, or information about a genre, author, situation,

quality of the content, and so on. Tagging can, simply put, be described as adding

labels to content, and there are no restrictions on the number of labels, and the

information these contain. Using this approach, files can simply be organized into

folders based on e.g. their creation date, but using the tags it is possible to search

for content, and group it into a number of categories across folders, by doing

simple text searches.

According to Sinha, tagging “taps into an existing cognitive process without

adding much cognitive cost”, which is why many users find it enjoyable and easy.

By tagging content, all associated categories can be assigned as metadata, which

eliminates the decision process of choosing the right category, thereby removing

the “post activation analysis paralysis” stage (10). Tagging therefore enables

people to do “multiple classification”, as desired by Malone (7), which is not

possible in a traditional folder hierarchy.

2.3.1 The Use of Tagging Today Tagging has become very popular on the Internet, where websites such as Flickr,

del.icio.us, Digg, and numerous news-sites have embraced tagging to help

Tagging 13

navigation and cross-referencing across their content. Websites like Flickr and

del.icio.us have gained a lot of popularity due to the open user-based tagging

scheme, which lets users organize content using tags to build non-hierarchical

bottom-up classification systems, also known as folksonomies (11). Folksonomies

are the result of many users tagging content, but tagging can also be

implemented for personal content management, and is becoming popular for file

management as well, with especially photo management software, such as

Picasa, Photoshop, and Adobe Lightroom, leading the way. Tagging is also slowly

being built into operating systems, such as Windows Vista, which contains native

support for tagging files, and searching these tags.

Apart from being a way to facilitate the searching of content, tagging is becoming

a social phenomenon online, where users see tagging as a way of expressing

themselves and providing a personal description of the content they tag (12).

Tags are also being used to give a more graphical representation of the focus of a

given site, such as a blog, by displaying the tags used on the site in a so called “tag

cloud”. A tag cloud contains the tags used on the particular website, and displays

them varying in size according to how much the different tags are used. This

representation of tags can give readers a quick overview of the main contents of

a website, as well as provide a tool to find content tagged using these tags.

Figure 3 – A tag cloud constructed using tags from Flickr. This is used to display the popularity of

different tags on the Flickr web site, providing visitors with an overview of often used tags as well

as navigation options by clicking on links (13)


2.3.2 The Problems with Tagging In traditional categorization schemes, similar content is grouped into fixed

categories, which are constructed to describe the content as precisely as possible,

without having different categories for synonyms of the same content. Since

tagging is based on the user entering metadata, it does not have the same

protection against synonyms being used to describe similar content (e.g. cat,

kitten, kittens, feline), which can lead to only some content being found in a

search using a single synonym. This problem arises due to the lack of a fixed set of

terms without synonyms, also known as a controlled vocabulary, which can be

used for tagging. Unfortunately, a controlled vocabulary can also limit the

precision with which content can be described. This is a serious limitation to

tagging of personal content.

According to Ames and Naaman (12), supporters of folksonomies claim that this is

a design choice, rather than a weakness, when speaking of folksonomies, and

with many users tagging the same content, more synonyms will likely be added as

tags, and the use of popular synonyms can be encouraged by providing lists of

“related terms”. This somehow minimizes the vocabulary problem in

folksonomies, but does not provide a good solution to the vocabulary problem in

tagging of personal content.

Another problem with tagging is the tedious work required to assign the

necessary tags to a collection of personal content, to make it searchable. As

mentioned by Ames and Naaman, the perceived benefits of annotation, which

are vaguely-defined and at some indeterminate time in the future, do not

overcome the investment, even with the most advanced annotation systems.

They further believe that people are more inclined to tag their content, when

they are given the right incentives and affordances for annotation (12).

Faceted Metadata 15

2.4 Faceted Metadata A third approach to content management, which is in part similar to both

categorization and tagging, is the use of faceted metadata. The idea behind

faceted metadata is to identify a finite number of properties of the content to

describe, so called facets, into which the descriptive tags are divided. The facets

can be seen as different axes along which content can be categorized, with each

facet containing a number of categories, which can in turn be hierarchical. This

enables searching along the different facets in any

order, to reduce the number of search results until a

desired result is found. This type of content

management was first introduced by S.R.

Ranganathan for use in library classification, and his

work is the foundation for the use of faceted

metadata (14).

An often used example, see for example Merholz (15)

and Doyle (16), is the use of facets on Wine.com (17),

which lets you search for wine along facets such as

wine type, region, winery, and price. Each search

result can then be further narrowed down by selecting

more facets, until the user ends up with a satisfying

selection to choose from. If no satisfying selection is

found, the user can choose to deselect certain facets,

thereby viewing all wines containing the other

selected facets, or in other words broaden the search

again. Searching in this manner is therefore an

iterative process, continuously allowing the user to

modify the search until a desired result is found.

Faceted metadata can be used to classify many kinds

of content, but requires the construction of the facets

along which to classify the content. Another good

example of using faceted metadata is seen in the

Figure 4 – The faceted

metadata system used in

searching for wine based on

the different facets on

Wine.com (18). The different

facets can be used to select

specific prices, types, regions

and professional ratings,

filtering the results based on

these


image search interface constructed by Yee, Swearingen, Li, and Hearst (18), which

was used in a usability study where 32 art history students explored a collection

of 35.000 fine arts images. The faceted metadata structure used in this study,

which I find to be a very concise and useful definition of faceted metadata, is

summed up as follows:

• The metadata may be faceted, that is, composed of

orthogonal sets of categories. For example, in the

domain of fine arts images, possible facets might be

themes (military, religious, etc.), artist names, time

periods, media (etching, woodblock, ceramic, etc.),

geographical locations, and so on.

• The metadata (or an individual facet) may be flat (“by

Pablo Picasso”) or hierarchical (“located in Vienna >

Austria > Europe”).

• The metadata (or an individual facet) may be single-

valued or multi-valued. That is, the data may allow at

most one value to be assigned to an item(“measures

36 cm tall”) or it may allow multiple values to be

assigned to an item (“used oil paint, ink, and

watercolor”).

The results of the user study showed, that once the users grew accustomed with

the new search interface, many found it to be more useful and inspiring,

compared to a normal textual search implementation.

The downside of faceted metadata is the necessity of defining the facets, which

makes it hard to use for arbitrary content, but a very potent tool for searching

content which can be classified using a defined set of facets.

Searching for Content 17

2.5 Searching for Content No matter the content management scheme, the time needed to find sought

after information depends greatly on the care taken in categorizing, tagging, or

describing content carefully.

Barreau and Nardi (19)summarize two studies of file organization in a folder

hierarchy, describing how the users in these studies overwhelmingly preferred

finding their files using location-based search, meaning manually browsing folders

for the files based on a guess of their location in the folder hierarchy. The

effectiveness of this approach most probably depends on the type of user, with a

“non-filer” using more time to find files, but less time to organize them, and a

“frequent filer” finding his files faster, but spending more time organizing them.

This again shows how different users have different approaches to organization,

and how a good content organization scheme needs to support different types of

users.

Malone also describes how retrieving information can be hindered by the user

only being able to search using one retrieval key at a time, e.g. the title of a

folder. Often information is specified by using more than one identifier at a time,

e.g. “a message from M.A. Smith, last week, about the meeting in Palo Alto”.

Retrieval of documents is often further hindered, since documents are often filed

under one identifier, e.g. an author, but sought after using other identifiers, e.g. a

topic or point in time. Tagging or using faceted metadata can help with this task,

by allowing the user to search using several identifiers at once, but this still

requires content to be sufficiently annotated, which is not always the case

according to Ames and Naaman (12).

2.5.1 Searching for Contents A different approach to finding files is by performing a search of the file contents.

This is easy for files with textual contents, and is becoming increasingly popular

on computers, using e.g. Google Desktop (20) or Windows Desktop Search, the

latter being built directly into Windows Vista (21). This kind of search has many


limitations, as it can only search textual contents as well as filenames of files, and

is therefore useless for multimedia data. As a result, a lot of research is being

done in the fields of multimedia searching, as seen for example in the challenges

listed by Fitzgibbon and Reiter (4).

Recently, several services for music search have been introduced, which can aid

users in their organization of their music libraries, by adding metadata to the

songs, and find the music they are looking for. As an example, Sony Ericsson has a

range of mobile phones with TrackID search, which lets you search for metadata

about the piece of music you are currently listening to on the built-in radio, MP3

player, or even the music in your current surroundings (see Sony Ericsson's

presentation on their website for a demonstration (22)). This search is based on

an analysis of the music, with a lookup in the commercial Gracenote music

database, which also powers products from Apple, Philips, and even a new

Cadillac (23). An open source alternative to Gracenote can be found in

MusicBrainz (24), which is a “community music metadatabase”. Textual searches

in the database are possible, but MusicBrainz can also be used by different MP3

tagging applications, such as their own Picard, which analyses songs and retrieves

metadata for them from MusicBrainz, which enables textual searches of this

metadata at a later time. A similar service is available online from Midomi (25),

where users can search for metadata about songs by humming or singing them,

and having the website find similar songs. The songs found on Midomi are not

only the original recordings, but also recordings made by a large user community,

which use the website for sharing their talent with the community. The metadata

found for songs will usually be stored in the ID3tag of MP3 files, or similar for

other formats, which is a good example of faceted metadata in use in file

organization today.

Other advances in multimedia content search are using image recognition and

image analysis for image search. As computer processing power is constantly

increasing, and image recognition and analysis algorithms are being perfected

and used in new ways, we are seeing new ways of searching for images, as for

example the “Multicolr Search Lab”, which lets users search a selection of images

from Flickr based on the colors found in the images (26), or “Celebrity Collage”,

Summary 19

where users can upload a portrait and find the celebrities who they look like the

most (27). The techniques behind this type of search can very well be

implemented in applications for personal image management, and could change

the way people search their images.

2.5.2 The Dimensions of Memory The use of different identifiers for information, as mentioned by Malone (7) and

as seen above, is interesting since it demonstrates how people tend to remember

and describe information by a number of different factors. These dimensions of

memory are also described by Eg Larsen (28), and include:

• Spatial – The location of information or a place it was

created

• Temporal – When information was used or stored

• Identification such as a name or a number – For

example a case number or name

• Classification – The category or file in which the

information was categorized

• Content – The contents of a file

These different dimensions all serve to describe information, and since people

tend to remember at least some of this information about a given piece of

content, facilitating information search using all of these dimensions would

greatly ease the finding of content.

2.6 Summary Traditional personal content management, in the form of categorization, suffers

from a number of limitations, and coupled with the growing amount of personal

content, this is likely to cause information overload for both experienced and new

computer users, unless better personal content management schemes are

implemented.


In this chapter, I have introduced two alternate ways of handling personal

content, namely tagging and the use of faceted metadata, which are both

possible solutions to the problems of categorization, but also include their own

limitations.

Tagging of content is a tedious task, which is often not performed due to the

perceived lack of benefits of doing so. Furthermore, the vocabulary problem of

tagging can easily render it useless, or at least much less beneficial, as a content

management scheme. To solve the vocabulary problem, faceted metadata could

be considered for personal content management, but the facets suitable for

describing different types of content vary greatly. In addition, faceted metadata

suffers from the same problem of tedious annotation as found in tagging.

The approach best suited for personal content management depends on the

scenario in which it will be used:

• Categorization is best for content that can strictly be

divided into single categories, which is not often the

case for personal content.

• Tagging is the most open and diverse solution to

managing all sorts of content.

• Faceted metadata offers an approach somewhat in

the middle between the two extremes of

categorization and tagging.

New advances in search technology has made searching for certain types of files

easier, and can aid in the management of these, but searching specifically for the

contents of files is only a tool that can aid in certain types of content search, not a

solution to content management problems.

The classical “frequent filer” and “non-filer” user types mentioned by Henderson

(9) and Malone (7) both need to be taken into consideration when designing a

new personal content management scheme. In a tagging scheme, as well as one

using faceted metadata, these different kinds of users would most likely manage

their content in quite different manners, as is the case for categorization. The

Summary 21

“frequent filer” would spend time tagging, or adding metadata to, his content,

which would enable him to find it quickly. The “non-filer” would spend much less

time adding tags or metadata, but would in return spend a substantial amount of

time finding needed content again. Both workflows need to be possible, and the

result of both types of management needs to be acceptable to the user.

CHAPTER 3

3 Mobile Computing

In this chapter, I will present the spread of mobile computing devices today, and

quickly sum up the different types of mobile devices available. I will then describe

the current personal content management scheme found in mobile devices, and

how this has certain limitations.

3.1 The Spread of Mobile Computing Devices Mobile phones are by far the most wide-spread mobile computing devices in the

world, with more than one billion mobile phones being sold in 2007 alone (29).

According to Gartner, the simple mobile phones are most popular in emerging

markets such as China and India, while mature markets such as Japan and

Western Europe tend to show a higher interest in more feature-laden mobile

phones, such as high-end phones and smart phones.

Ultraportable laptops are predicted to spread rapidly in developing markets over

the next years, especially due to the very low-cost education PCs (30). Low-cost

ultraportables have also seen a great interest in established computer markets,

especially following the introduction of the Eee PC from ASUS, which has had

remarkably high sales figures, selling more than 350.000 units in the first half year

24 Mobile Computing

from the release date, and projected sales of 3 to 5 million units in 2008 (31). The

success of the Eee PC has had a great effect on many laptop manufacturers, with

MSI, ECS, Dell, HP, Acer, and Medion all having introduced competitors to the Eee

PC throughout 2008 (32).

Traditional laptops are also spreading rapidly, with their sales exceeding the sales

of desktop computers in Denmark the last two years in a row (33). This trend is

also expected to hold true for the United States in 2008 (34), which can be

interpreted as a general indication, that more and more new PCs sold in the

established PC markets will be mobile.

The sales figures and new emerging markets all point to a quickly increasing

spread in mobile computing devices of all kinds, which is a trend that is likely to

continue for many years to come.

3.2 Different Mobile Computing Devices The mobile computing field is not dominated by a single type of device, but is

comprised of a great variety of different devices, differing in size, price,

functionality, and last but not least specifications. Mobile computing devices of

today offer many possibilities, but also present many challenges in personal

content management. The different kinds of devices can roughly be grouped into

the following categories.

Different Mobile Computing Devices 25

3.2.1 Mobile Phones This category consists of simple mobile phones,

whose main function is telephony. The

functionality of the devices in this group is ever

evolving, with almost all new mobile phones

having a camera, mp3 player, color screen, sound

recorder, calendar, Bluetooth connectivity, and

internet browsing option. Except for the calendar,

all of this functionality was only seen in high-end

mobile phones few years ago, and more and

more functions from high-end phones are

gradually trickling down into the cheaper phones.

Most simple mobile phones have very limited

computing power, and only some allow the

running of third party applications.

Figure 5 – Nokia 3500 and Nokia

6500 Classic (35). Two typical

simple mobile phones found on the

market today

3.2.2 PDAs and Smart Phones

Figure 6 – Nokia N78 (35) and Apple iPhone

3G (36). Two examples of modern highly

capable smart phones

In this category we find the traditional

middle-way between a mobile phone and

a laptop, namely the PDA, and its

successor, the smart phone. Smart phones

are, simply put, high-end mobile phones,

with a lot of functionality such as Wi-Fi and

3G connectivity, GPS navigation, higher

quality camera, document editor, higher

quality screen, and multimedia playback.

Smart phones are constructed to be a

complete personal multimedia device, and

not just a telephone with a few extra

options. Most smart phones run small

operating systems such as Symbian or

Windows Mobile, which provides more

options for running third party

applications.

26 Mobile Computing

3.2.3 Ultraportable Computers In the gap between smart phones and normal

laptops, a quickly growing market for

ultraportable computers is forming. Ultraportable

laptops have been available for many years, but

at a very high cost, which has kept the market

reserved for business consumers mostly.

Recent projects aimed at constructing a very low

price, simple, and robust laptop for developing

countries, such as the OLPC project (37), has

created a market for low-cost ultraportable

computers, not only in developing countries, but

also in the established computer markets, where

the first low-cost ultraportable computer, the

ASUS Eee PC, has had great success (31).

Ultraportable computers are generally less

powerful than traditional laptops, but often run

normal PC operating systems and programs. Most

ultraportable laptops have Wi-Fi connectivity, and

many models also come with Bluetooth, and

some even 3G connectivity and GPS built in.

Figure 7 – Asus Eee PC 2G Surf

(38), the first very popular low-

cost ultraportable computer

Figure 8 – Acer Aspire One (39),

Acers new low-cost ultraportable

computer

3.2.4 Laptop Computers

Figure 9 – Dell XPS m1330 (40), an

example of a more traditional

laptop

Traditional laptop computers, or notebooks, are

the most powerful, but also largest, heaviest, and

often most expensive, mobile computing devices,

ranging from sub-notebooks with screen sizes of

around 12 inches to large desktop replacement

laptops with screen sizes passing 20 inches.

Modern laptops generally come equipped with Wi-

Fi and Bluetooth connectivity, and a few have also

been introduced with integrated 3G and WiMAX

connectivity.

Personal Content Management on Mobile Devices 27

3.3 Personal Content Management on Mobile Devices All types of mobile computing devices are gaining increasing amounts of features,

allowing the users to create a growing variety of personal content which needs to

be managed. Today, the norm for personal content management on most types

of mobile units is the traditional folder hierarchy, with its inherent content

management problems as underlined in Section 2.2.2. Using folder hierarchies for

content management on laptops and ultraportable computers is an obvious

choice, since these devices usually run the same operating systems as their

desktop counterparts. The adoption of folder hierarchies on mobile and smart

phones is likely due to the wish to follow computing norms, even though this is

not necessarily the most usable solution for smaller mobile devices.

One exception to the use of folder hierarchies is the content organization of many

portable music players, most notably the iPod, which allows selecting music

based on artist, album, song title, genre, and so forth. This selection is based on

the faceted metadata found in the ID3 tags embedded in MP3 files, and allows

the user a much more versatile approach to organizing his music and selecting the

songs to play.

3.3.1 The Limitations in Mobile Computing Devices One aspect of mobile computing, which greatly differs from desktop computing,

is the different limitations in user interaction options. Different types of mobile

devices each have their own limitations and advantages, concerning both the

graphical user interface and the input options, which make the design of

applications and operating systems a challenge. Dunlop and Brewster (41) list

these challenges, and especially the challenge of “Designing for limited

input/output facilities” has an impact on personal content management on

mobile computing devices.

The most limited devices are the normal mobile phones. These usually have a

small, low-resolution screen and a 12-button keypad along with a few selection

buttons. Smart phones usually feature a larger screen, though still smaller than 4

28 Mobile Computing

inches, with higher resolution, and many smart phones feature touch-screens or

full keyboards for better interaction options. Ultraportable computers normally

feature either a touch-screen or a full keyboard, some feature both, and are also

often equipped with a track-pad or other pointing device replacing a mouse. The

largest device, the laptop, features a full keyboard, a track-pad or similar mouse

replacement, and a large screen, making this the device that traditionally has the

most user interaction options. These limited input/output options can make the

task of creating folders, arranging files, and generally maintaining a folder

hierarchy tedious, and often quite hard, since especially smaller devices lack the

input options, as well as overview, needed to efficiently manage folders and files.

Due to these limitations in user interaction, using faceted metadata or tagging for

personal content management on mobile computing devices could easily prove

useful.

Another important limitation in mobile computing is the processing power, with

the smaller units often having a very limited processing power compared to

desktop computers and laptops. The larger the device, the larger processing

power is usually found. The processing power can create limitations in the

searching and filtering options of personal content management systems on

mobile devices, but with processing power in new mobile devices constantly

increasing, this is a disappearing problem.

3.4 Summary Mobile computing devices are seeing a rapid spread, as well as a great evolution

in functionality, which is very unlikely to stop in the years to come.

With much more content being created on mobile devices, and storage space in

them increasing, the problem of efficient personal content management is

increasing as well. Together with the growing amounts of content, the limitations

inherent in many mobile devices further underline the need for a good personal

content management scheme on mobile devices.

Summary 29

Together with the constant increase in storage space, all types of mobile devices

are getting an increasing amount of sensors built in, with GPS and accelerometers

being the biggest trend at the moment. Cheaper devices, like standard mobile

phones, are becoming equipped with built in sensors which were only found in

high end devices a few years ago, and the higher end devices are getting a whole

range of new sensors, such as digital compasses and accelerometers, built in. All

these sensors provide many new options for application development on mobile

devices.

30 Mobile Computing

CHAPTER 4

4 Context Awareness

As described in the previous chapters, personal content management is

increasingly taking place on mobile computing devices, with the limitations of

categorization, combined with limited input/output options of mobile devices,

creating a growing need for a better personal content management scheme. In

this chapter, I will describe how using the sensors found in mobile devices can aid

in facilitating easier content management by using context awareness to

automatically collect metadata for created content.

I will begin with a brief history and definition of context awareness, followed by a

discussion of inferring context information from sensors, and how this can be

done in mobile devices. Finally, I will describe how context awareness can be

used for personal content management, and which problems exist related to

context awareness.

4.1 History and Definition The notion of context awareness was firstly introduced by Schilit et al. in 1994

(42), and is most often defined as a sub-discipline of ubiquitous computing

research. The field of context aware computing focuses on making computing

32 Context Awareness

devices aware of their surroundings, and the user’s current context, so they can

adapt to the situation and needs of the user. Since Schilit et al. coined the term a

lot of work has been put into this discipline, on both the theoretical and practical

level.

According to Dey, Schilit’s definition of context as “location, identities of nearby

people and objects, and changes to those objects” (quoted freely from Dey), can

be hard to use in practice. Other definitions based on simple synonyms for

context, such as “referring to context as the environment or situation”, can be

equally hard to use, while other definitions are too restrictive. This has led Dey to

create one of the most widely used definitions of context, which is the following

(43):

”Context is any information that can be used to

characterize the situation of an entity. An entity is a

person, place, or object that is considered relevant to

the interaction between a user and an application,

including the user and applications themselves.”

This definition of context is not restrictive concerning the types of information

that can be used, but is at the same time specific enough to actually be usable in

practice. After defining context, Dey also created his own definition of context,

which the following:

”A system is context-aware if it uses context to

provide relevant information and/or services to the

user, where relevancy depends on the user’s task.”

Again, this is an open, but at the same time quite precise, definition, which is also

usable in practice. These definitions have been modified by many authors, but

still remain a basic foundation for work in the field of context awareness.

Zimmermann, Lorenz and Oppermann believe Dey's definition of context is too

general, and should be encapsulated in a formal and an operational part (44).

Their definition of context is therefore:

History and Definition 33

”Context is any information that can be used to

characterize the situation of an entity. Elements for

the description of this context information fall into

five categories: individuality, activity, location, time,

and relations. The activity predominantly determines

the relevancy of context elements in specific

situations, and the location and time primarily drive

the creation of relations between entities and enable

the exchange of context information among entities.”

They divide context information into five fundamental categories, namely:

• Individuality: Gives access to contextual information

about the entity the context is bound to. This can be

anything that can be observed about an entity,

typically its state. Individuality is clustered into four

entity types, natural, human, artificial, and group.

• Activity: Describes the activities the entity is involved

in, and answers the question: “What does the entity

want to achieve and how?”

• Location: Describes the location both quantitatively,

using GPS-coordinates and the likes, and qualitatively,

such as room, street, country, and so forth.

• Time: Handles anything related to time, e.g. time

zone, time of day, and time intervals.

• Relations: Covers the relations an entity has with

other entities, such as people, things, devices,

services, or information. Like individuality, relations

information is clustered into sub categories, namely

social, functional, and compositional.


In my eyes, the context definitions of Zimmermann, Lorenz, and Oppermann are

very usable and thoroughly worked through, as they gather context into distinct

categories, suitable to be used as facets in a faceted metadata scheme.

Furthermore, they contain similarities to the memory dimensions discussed in

Section 2.5.2. I also find that they provide a better overview of possible context

information when working with context awareness.

4.2 Context Aware Systems Today Although we have yet to see commercially available applications and devices that

adapt themselves heavily to the user’s needs depending on his current context,

we are already surrounded by context aware devices, which make smaller

adaptations to the user’s context.

A good example of context awareness is the ABS braking system, which is found

in most cars today. ABS brakes adapt the braking force applied when braking

heavily, to prevent the tires from skidding uncontrollably, thereby shortening the

braking distance, especially in bad weather conditions. This is done by sensing if

the tire is skidding, and reacting to this information, making it a simple context

aware system.

Some smart phones, and other electronic devices featuring displays, have built-in

light sensors, which regulate the brightness level of the display depending on the

ambient light. This is done to ensure visibility of the screen, as well as conserve

battery power, and is another good example of simple context awareness. Many

of the same devices, especially smart phones and digital cameras, have built-in

accelerometers, allowing them to sense the movement of the device. This

information is mostly used to rotate the graphical user interface on the display,

depending on the orientation of the device, but a few devices have also used the

accelerometers for user interaction, for example the Sony Ericsson W910i with

shake control, allowing the user to change songs by shaking the mobile phone

(45), and the iPhone, allowing developers to use the accelerometers for user

interaction in applications (46).

Inferring Context from Sensors 35

These simple context aware applications can greatly enhance the user experience

of a system, making applications and devices much more usable and user friendly.

With more and more sensors being built into devices, the amount of context

aware applications is likely to greatly increase in the years to come.

4.3 Inferring Context from Sensors With the definitions of context and context awareness in place, the next question

is how to infer the current context of a device or application. Some research

concerning context awareness focuses mainly on location, since a lot of

information can be inferred from the current location, and a user’s needs often

depend on it. A good example of an application depending heavily on location is a

tourist guide system, which displays information to the tourist about the

attractions in his vicinity, but for many other applications, the context depends on

many other factors than location alone.

Context aware applications not only focused on location usually utilize several,

and other types of, sensors to infer the current context, as described by Schmidt,

Beigl, and Gellersen (47). Their work focuses on the use of a whole variety of

sensors built into a device to infer context. The discussed sensors include

optical/vision, audio, motion, location, bio-sensors, and specialized sensors,

which can all, using simple sensor-technology, help to infer a great deal of high-

level context information. Later work by the same authors describes different

applications made using simple sensors, demonstrating how these have been

used to infer context information on a higher level. (48).

Other research concerning context awareness describes placing a number of

sensors all over a user’s body, for example by integrating them in clothing, to

infer the current context of the user. This type of work is still very experimental,

and outside the focus of this thesis, but great progress is made in inferring

context using simple sensors, and we might start seeing results of this in the near

future.


4.3.1 Inferring Context from the Sensors Available Today With the increasing amount of embedded sensors in mobile computing devices,

much more context information is becoming available for use in applications. A

typical smart phone like the Nokia N82 is equipped with a light sensor,

microphone, accelerometer, Bluetooth, Wi-Fi, and even GPS, which can all be

used to gather context information. Likewise, ultraportable and regular laptops

include a growing amount of sensors such as an accelerometer, Bluetooth, Wi-Fi,

3G connectivity, WiMAX, light sensor, microphone, and some even GPS, which

can help in inferring context.

Using this type of low-level sensors to infer context is well described by Schmidt,

Beigl, and Gellersen (47), but their work focuses on inferring very specific context

information from these sensors, whereas I wish to use it for personal content

management, by simply adding as much context information to the created

content as possible, to enable the user to search for this at a later time.

4.4 Using Context Awareness for Personal Content Management

A lot of work has already been done in exploring the possibilities of adding

context information to content created on mobile computing devices. Much of

this work has focused on adding content information to specific types of content,

usually multimedia content such as images and videos, and how this can facilitate

easier management of the created content.

A good example of this type of work in progress is the MMM framework, which

focuses on metadata annotation of digital images taken with mobile phones, and

the sharing of these images online. The added metadata is comprised of

information about where and when a picture is taken, and who is in it. Metadata

is added automatically, to reduce the user effort in annotation (3).

Using Context Awareness for Personal Content Management 37

Another interesting application of context aware personal photo collection

management is the MediAssist system, which offers context-based photo

indexing, based on the following methods as described by the authors (49):

• Time of capture is augmented with time-based fields:

year, month, day of month, day of week, and hour,

enabling queries such as, for example, ‘all photos

taken in the summer, in the weekend’.

• Latitude/longitude co-ordinates are converted into

place-names using a publicly available gazetteer.

• Standard astronomical algorithms calculate the light

status of all images (i.e. day/night/dusk/dawn).

• Each photo is annotated with weather data from the

nearest weather station when the photo was taken.

• Indoor/outdoor classification is inferred from digital

camera metadata, such as the ambient light levels.

• We also segment personal photo collections into

‘Events’ collections by detecting large temporal gaps

between consecutively captured photos.

We can already begin to see context information being added to images today,

with so called geo-tagging, meaning the addition of location information to

images, becoming increasingly popular. This trend has especially caught on

online, with websites such as Flickr offering users the possibility to view images

based on location (50). The use of geo-tagging is likely to become very popular

with users using the built-in geo-tagging features of the new iPhone 3G (51) or

the geo-tagging software on its way from Nokia (52).


4.4.1 Going Beyond Specific Types of Content My suggestion is to take geo-tagging, and research like the MediAssist system and

the MMM framework, one step further, and use gathered context information as

metadata for all types of content. As described in Section 2.5.2, people identify

content by an arbitrary amount of information (7), and remember it along several

dimensions (28). Therefore, I believe that context information is useful for

annotating any type of content.

The available context information that can be gathered from an advanced mobile

computing device today is a perfect candidate for use in a faceted metadata

scheme, in which I have constructed the following facets, inspired heavily by the

work of Zimmermann, Lorenz and Oppermann (44). These facets have also been

chosen to follow the dimensions of memory as described in Section 2.5.2 (28):

• Ambience: Describes the near surroundings, with

information such as weather, temperature, ambient

light, and noise level.

• Activity: Describes the activities the device is, and has

just been, involved in. This can be information

inferred from movement patterns and device

settings, but also information taken from a current

calendar event.

• Location: Describes the location quantitatively, using

GPS coordinates and cell ID, and qualitatively,

containing country, state, city, area, street, and

nearby attractions.

• Time: Includes all time information, such as year,

month, day, hour, and minute, but also descriptive

terms such as weekend, holiday, time zone, and time

of day (day/night/dusk/dawn).

• Relations: Handles relations to both connected and

available Wi-Fi networks and Bluetooth devices, thus

ultimately identifying locations and networks using

Problems with Context Awareness 39

Wi-Fi, and people close by using their Bluetooth

devices.

• Comments: Contains any tags added by the user, not

suitable in the other facets. This can be comments

and descriptions for the content, which is not directly

related to the context in which it was created.

I believe that adding as much of this context information as possible to any type

of content will ease personal content management. Automatically annotating

content also follows the suggestion made by Malone concerning “automatic

classification”. Always adding all context information to content makes sense, as

users can relate this type of information to all types of content. For example, a

travelling businessman could search for all documents he created on his trip to

Tokyo last year, or when he was connected to his company’s wireless network, a

photographer can find all pictures taken in the rain, and a musician can find all

songs created with a certain person by his side.

I have included the “comments” facet, which is a very open descriptive facet, to

allow users to add comments and other types of information. This is to embrace

the social incentive and advantages of tagging content, and can help users to

further describe and manage their content.

4.5 Problems with Context Awareness By using sensors to infer context information, there is a chance of faulty

measurements and wrong assumptions about context entering the system.

Context aware applications which make wrong assumptions about context, and

act upon these, can be of great annoyance to the user, and can turn the benefit of

context awareness into a disadvantage of the system. One solution to this

problem is to display the uncertainty of the current context assumption, letting

the user choose the correct context, instead of making the application completely

reliant on correct assumptions.


Antifakos, Schwaninger, and Schiele (53) have evaluated the effects of displaying

uncertainty through a number of experiments, and found that displaying

uncertainty information resulted in a substantial increase in hit rates in the tasks

performed in the experiment. Users also reported that when uncertainty is

displayed, it is easier to understand what the system is doing and how well it is

doing it. These are both testaments to the higher usability of an application if the

uncertainty of context measurements is displayed. The authors mention that the

tradeoff between the cognitive load, which displaying uncertainty information

causes, and the added value that it provides should be considered when

designing an application.

Constructing context aware applications can also lead to a few pitfalls, as

described by Cheverest, Davies, Mitchell, and Efstratiou, who describe the

following three possible problems (54):

1. The system might not reach a stable state, if both the

user and system attempt to adapt to the current

context.

2. Due to the trade-off between prescription and

freedom, the system could prevent the user from

achieving a wanted task.

3. The user must trust the agent performing the

adaptation, in order to react appropriately.

The authors have worked on a context aware tourist guide system named GUIDE,

and have learnt some lessons from this work, causing them to suggest a number

of strategies for building context aware applications. In essence, a context aware

application needs dependable sensing technology, to avoid erroneously

presuming the users goals, and a flexible agent acting on the user’s behalf, to

enable users to override the adaptation strategy.

Summary 41

4.6 Summary In this chapter, I have introduced the history and most common definition of

context awareness, as well as a derivate of this definition. Following this, I have

given examples of a few context aware systems in use today. I have described the

possibilities of inferring context from sensors, and how mobile computing devices

can infer context from the built-in sensors available today.

Next, I have discussed using context awareness for personal content

management, and shown examples of using it for certain types of content. I have

then presented my suggestion for a personal content management scheme, in

which context awareness is used to gather context information to be saved with

all types of content generated in the system. This context information can then

be used for finding content based on the context in which it was originally

created. I argue that saving all possible context information for all created

content makes sense, since we remember information about created content

along several dimensions, the so called dimensions of memory (28).

Automatically gathering and saving context lives up to Malone’s suggestion of

“automatic classification” (7) and makes much more versatile searching possible,

while removing the task of annotating content, which we have seen is perceived

as being tedious and therefore often not done (12).

Ending this chapter, I have listed a few possible problems of using context

awareness in a system.

CHAPTER 5

5 Design

With the suggestion for a context aware personal content management scheme

presented, and the theory behind it outlined, it is now time to look into the

design of such a system.

In this chapter, I will introduce and discuss my suggestions for different designs of

a context aware personal content management scheme. I will begin by describing

five different design possibilities, ranging from a simple stand-alone application to

a full implementation built into the operating system, and discuss the advantages

and problems with these possibilities along the way. Afterwards, I illustrate how

the different sensors typically found in mobile computing devices today can be

used to infer context information, and how this information can be interpreted.

Next, I give three suggestions for using the inferred context information for

content management, and lastly I discuss some of the considerations and

challenges found in implementing a context aware personal content management

system.

44 Design

5.1 Different Design Possibilities Different possibilities exist for designing a context aware personal content

management scheme, and the option most suitable for a given system depends

on the preferences of the user and the developer of the scheme. Here, I have

listed a number of different options, which differ greatly in the functionality

offered to the user, and also in their integration with the operating system of the

device on which they run.

5.1.1 Stand-Alone Application Creating Content One type of stand-

alone application could

be an application made

for creating content,

which annotates

created content with

gathered context

information when it is

saved. Such an application provides the user with many, and more diverse,

options for managing and finding content, but is limited to managing content

created using this application only. Since only content created by this application

is annotated with context information, the user does not benefit from the

advantages of being able to search all content using the same techniques.

Such an application can be practical for managing single types of content, but its

usability to doing only that. Therefore, the general personal content management

scheme of the system on which this application is running is not improved by this

application.

Figure 10 - Stand-alone application creating and managing its own

content. Context information is saved and managed by the

application

Operating System

App 3 App 2 App 1

Content 1 Content 2 Content 3

Context

Different Design Possibilities 45

5.1.2 Stand-Alone Application for Content Management Another option would

be to create a stand-

alone application,

simply used for

managing content

saved on the operating

system on which it is

installed. Such an

application would

need to be installed by the user, and started every time it is needed. It can

provide the user with new options for managing and searching for his content,

but it relies on the user to find and select content to annotate. Since content is

already created when the user chooses to annotate it, user interaction is also

required to ensure that correct context information is saved.

Saving context information at a later time is a dilemma, since at least some

context information is bound to be lost, and having the user input context

information is likely to lead to the user not annotating much content, since the

perceived benefit of this is low, as described by Ames and Naarman (12). This in

turn severely limits the usability of this application.

5.1.3 Stand-Alone Content Management Application Integrated With the Operating System

A better option than a

completely stand-

alone application

would be to create an

application that

integrates with the

operating system.

Such an application

Figure 11 - Stand-alone application used for managing all content.

The gathered context information is saved and managed by the

application

Figure 12 - Stand-alone content management application

integrated with the operating system. The gathered context

information is saved and managed by the application

Content 2

Operating System

App 3 App 2 App 1

Content 1 Content 3

Context

Operating System

App 3 App 2 App 1


Context

46 Design

could integrate with the operating system much in the same manner as an

antivirus program does, starting up together with the operating system, and

monitoring all content being created.

Whenever new content is created, the application can annotate this content with

any available context information, allowing the user to use the application for

managing, and searching for, all his created content at a later time.

Such a scheme enables the user to use all his usual programs for creating content,

and automatically have this content annotated with available context

information, thus fulfilling the suggestion by Malone concerning “automatic

classification” (7).

Furthermore, an application integrated with the operating system, could enable

other applications to take advantage of the context information gathered for

created content. As an example, a calendar application could allow searching for

files created within certain time spans, or an application managing contacts could

provide searching for content related to certain people saved as contacts.

A downside to having a stand-alone application is that the user has to install the

application to benefit from these features, which removes the option of

integrating the context aware personal content management options deeply into

other applications made for the current operating system, since they should also

function without this application installed.

5.1.4 Building Context Awareness into the Operating System The fourth option would be to completely build context aware personal content

management into the operating system, and use it in place of the traditional

personal content management scheme, namely the folder hierarchy. In this

manner, users would not have to install any third party applications, or make any

choices between applications, to be able to benefit from the options found in

using context awareness for personal content management. The operating

system simply keeps track of all content being created, and annotates any new


content with all available context information. All annotation therefore happens

completely without user interaction, and, like in the suggestion for a stand-alone

integrated application, therefore follows Malone’s suggestion concerning

“automatic classification” (7).

Having the operating

system handle content

annotation also

enables all

applications created

for the specific

operating system to

utilize saved context

information for

information search, and cross-referencing of information, as described for the

operating system integrated application. With context awareness directly

integrated into the operating system, the user also always has the search and

filtering options provided by saving all context information at hand.

Like for the integrated application, this provides many options of integrating

applications, allowing context to be linked to for example contacts or calendar

events saved in other applications. This can provide a very fluent approach to

finding content, lowering the amount of user input needed for content search.

A content management scheme based on context information is able to elude

content being organized into folders, by displaying a folder hierarchy based on

the annotated context information. Content can be shown as organized into a

folder hierarchy with for example Europe, Denmark, and Copenhagen as nested

folders. The user can select which context information to use as the basis for this

“virtual” folder hierarchy, allowing for a representation of content that he is more

accustomed to.

A disadvantage of basing the content management scheme solely on context

information and these “virtual” folders is that users will need to embrace the new

Operating System

App 3 App 2 App 1


Context

Figure 13 - Building context aware personal content management

into the operating system. The gathered context information is

saved and managed by the operating system, which uses is as the

basis for the entire content management scheme

48 Design

scheme and adapt their usual workflows to this new way of organizing content.

Many users might also find this more abstract way of handling content hard to

understand and get used to, feeling that they lose control over the organization

of their content, since this approach is not related to the typical way of organizing

physical objects into groups and piles. This might in turn lead to “information

overload” due to the lack of overview of content saved on the system (8).

Basing the content management scheme solely on the context information is

more reminiscent of the organization in a database, where many different views

of the same data are possible, but with users not used to this way of handling

data, many will probably find this scheme more incomprehensible and annoying,

than actually useful, while others will easily adapt and find this scheme much

more efficient.

Finding all content based on filtering and searching also makes casual browsing of

for example pictures in a folder harder, since no folders exist. It is easy to find

specific pictures that are sought after, but browsing through a picture collection is

suddenly not possible in the same way as just looking through a folder. All files

that for some reason have incomplete context information, for example due to

the lack of any usable location information at the time of annotation, will also be

harder to find, and might easily “disappear” in the system, which is not as likely to

happen in a traditional file system with a folder hierarchy where all files are

always “visible”.

5.1.5 Extending the Existing Content Management Scheme with Context Awareness

Building the context aware personal content management scheme into the

operating system as an extension to the already implemented personal content

management scheme is another option. In this manner, the context aware

personal content management scheme is not meant to completely replace the

traditional folder hierarchy, but to extend it by providing new features for finding

and organizing content.


With context awareness

built directly into the

operating system, all

content being

generated can

automatically be

annotated, still following

Malone’s suggestion for

automatic

classification(7), without

any user interaction needed. In this way, all available context information will be

saved, and the users are free of the tedious work of annotating their content.

Furthermore, the scheme does not need to be installed as an application, and is

always present in the operating system, so the user does not need to make any

decisions about which, if any, context aware personal content management

system to install. All applications made for the operating system can also take

advantage of the context metadata, as it is always present in the operating

system.

Integrating the context aware personal content management scheme into the

operating system without replacing the folder hierarchy makes the adaptation to

this scheme a lot easier for the users, since they do not have to adapt to the more

abstract concept of storing content in “one big pile” and organizing it solely based

on the context in which it was created.

This allows users to organize their content into folder hierarchies, with “neat”

users likely to build elaborate hierarchies, and “messy” users more likely to have

shallow hierarchies as described by Malone (7) and Henderson (9), but still

provides them with the full range of searching and filtering options inherent in a

faceted metadata scheme. Furthermore, the casual browsing of folders is still

possible, as opposed to the suggestion of completely replacing folder hierarchies

with a faceted metadata scheme.

Operating System

App 3 App 2 App 1


Context

Figure 14 - Building context aware personal content management

into the operating system. The gathered context information is

saved and managed by the operating system, which uses it to

expand the searching and filtering options of the present content

management scheme

50 Design

In short, this suggestion contains all the advantages of a context aware personal

content management scheme as presented in the previously discussed designs,

but is free from the disadvantages discussed.

5.2 Gathering Context Information To provide the user of a context aware personal content management system

with as many search and filtering options as possible, precise and elaborate

context information has to be gathered. The more context information is saved,

the more different search and filtering options can be provided.

To gather as much context information as possible, all sensors on the given

mobile computing device should be used. Below, I have listed the sensors and

built in functions most often found in mobile computing devices today, and

described the context information these can help infer:

• Light sensor – Can be used for determining the

ambient light level. This can indicate if the device is

located indoors or outdoors, especially if combined

with information about the time of day.

• Microphone – Used for measuring the ambient

sound level. This indicates if the mobile device is in a

crowded and noisy area, or in a more peaceful

location.

• GPS – Determines the GPS coordinate of the current

location of the mobile device. This is a good source

for a lot of location information, if combined with

some sort of translation of the coordinate into more

humanly understandable place names.

• Wireless LAN – Can be used to find wireless networks

in the vicinity of the mobile device. These can help

indicate the location and the surroundings of the

mobile device.

Gathering Context Information 51

• Bluetooth – Used for scanning for Bluetooth devices

near the mobile device. Bluetooth devices can help

infer people and services in close vicinity to the

device.

• Accelerometer – Indicates how much, and in which

directions, the mobile device is moving and being

tilted and rotated. This can give an indication of the

activity of the mobile device user, e.g. if he is running

or sitting still.

• Digital compass – Determines the geographical

direction in which the device is facing.

• Cell ID – Determines which mobile telephone

network cell the mobile device is connected to. This

ID can help in determining the location of the mobile

device.

• Camera – The camera can be used to take a picture,

which can then be analyzed for information about

ambient light levels, surroundings and people nearby.

This requires a lot of image processing, but many

advances are being made in this field.

• Clock – Can be used to get the current date and time,

which is very essential context information.

• Calendar – Many mobile devices have a built in

calendar, which can be used to determine if any

events are entered in the calendar at the time when

context information is gathered.

• Device profile – The current device profile of for

example a mobile or smart phone can give an

indication of the activity and location of the user.

• Running programs – Other currently running

programs can give an indication of the activity of the

user.

52 Design

As seen by this description, context information suitable for all of my suggested

facets, except for the comments facet which is intended for comments entered

by the user, can be gathered using these simple sensors and built in functions.

Much data gathered by the sensors is on a relatively low level, which in itself is

not necessarily enough to offer a good description of the current context.

Schmidt, Beigl, and Gellersen (47) have explored how to use simple sensors for

deriving complex context information, for example how to use a light sensor to

derive if the mobile device is currently inside or outside based on the sensor data

and an analysis of its pattern. This sort of analysis can provide much more useful

context information for certain types of sensor input.

The MediAssist system (49) uses an online translation of the latitude and

longitude coordinates from a GPS sensor, to infer more meaningful location

information, which is also a good way of providing much more descriptive and

intuitive context information. In this manner, much data gathered from the

sensors could be translated into more humanly understandable information, such

as a country, state, and city name instead of merely a GPS coordinate. Much

other work is currently taking place concerning how to infer context from sensor

information, and this is likely to yield results which can greatly improve context

inference from simple sensors.

5.3 Using the Gathered Context Information One could argue, that some facets of the gathered context information make

much more sense for certain content types, and almost no sense for others. As an

example, a reading from the digital compass could be useful for a picture,

describing in which direction the picture was taken, but hardly makes any sense

for a written document. I believe that all context information should always be

saved, regardless the content being created, as the rapidly increasing storage

space on all mobile computing devices allows for this without much

consideration, and advances in context inference could prove this data to be

useful at a later time.

Using the Gathered Context Information 53

Once context information has been gathered and saved as metadata for created

content, it can be used to filter and search for content in a wide variety of ways:

• Simple searches can be performed by having the user

input text in a textbox, and using this to search

through all available context metadata for any

matches. This is a quick, but not very precise, way to

search for content based on a keyword match.

• If more precise searches are required, advanced

search modes could be designed to allow the user to

enter keywords to search in specific fields of the

context information gathered, so the user can for

example search for content based specifically on

where, when, and in the company of who it was

created.

• Alternatively, a filtering scheme in the same manner

as the use of facets on wine.com (17), as described in

Section 2.4, could be used, based on the facets

constructed in Section 4.4.1. This would allow the

user to organize his content by filtering out content

based on the gathered context information, instead

of performing a search trying to match specific terms.

• Other search types could for example be based on

the current location, allowing the user to find all

content created in the same location, or the current

situation, allowing the user easy access to content

created in the same situation. This could for example

be used when the user is at his office and needs all

content he has created here, or to find all content

created with another person nearby, when the same

situation is valid again.

Numerous other search and filtering options exist, as the context information can

be used in very diverse ways. Different applications can each incorporate their

54 Design

own search methods, and as described earlier, they can link information they

contain, such as locations, contact persons, or calendar events, to content

searches.

5.4 Considerations and Challenges A number of considerations have to be made, and challenges met, when

designing a context aware personal content management scheme, to ensure that

the scheme is usable and concise, and lives up to the expectations of privacy.

5.4.1 Saving Context with Content As content is shared between users, and moved to other devices for editing, use,

and storage, it is important that the context metadata follows the content. This

implies that context metadata should not be stored solely in a database, or other

means of storage disconnected from the content, on the device on which the

content is created, but should be embedded in the content, or at least

transferred together with the content.

This presents a challenge of storing the context with the content, as a uniform

method of doing this for all content types is needed. A few file formats today

already contain metadata, such as the author, title, and technical specifications of

the content, but this varies greatly from file type to file type. To successfully be

able to share content between devices, and have context information follow the

content, standards for embedding context have to be developed.

A possible alternative would be to make sure that context information is always

transferred along with content, but this is only possible if the context aware

personal content management scheme is embedded in the operating system, and

third party applications are forced to follow this policy.

To ensure good performance in searches and filtering operations, it is most

practical to also store context information in a central index or database, instead

Considerations and Challenges 55

of relying on searching through context information embedded in all files each

time a search or filtering operation is performed. This presents another challenge

as such an index has to be synchronized with the context information stored in

the individual pieces of content. A solution is to enforce a policy, dictating that

every time any context information is edited or any new content is added,

context information is updated in both the content and index before any other

actions are taken, which in turn requires the context aware personal content

management scheme to be built into the operating system to ensure this.

Transferring context with content not only presents a challenge in embedding

context metadata in various content types, but also introduces a very important

privacy concern. As content is shared among users, context metadata which could

be considered private is shared along with the content. This could, for example,

be information about who was in the vicinity when a photo was taken, or where a

certain document was created. A system for setting up filters for removing

“personal context information” when content is shared with other people could

help solve this issue, and could allow users to choose what context information to

share, and with who. Furthermore, users should be able to choose which context

information to share on a “per content” basis. The privacy considerations are

essential to ensure the usability and success of a new personal content

management scheme, without removing privacy from its users.

5.4.2 Limitations in Mobile Devices The previously mentioned limitations found in the user interface of mobile

computing devices can severely limit the usability of a personal content

management scheme. Therefore, these limitations have to be taken into

consideration already in the design phase.

To make a context aware personal content management scheme usable on a

wide variety of mobile computing devices, the input method and graphical user

interface for using such a scheme should be adapted to the individual operating

system and mobile device, and present good interaction options, even with very

56 Design

limited screen sizes and input options. As an example, using facets for filtering

content, as described in Section 2.4, might be very useful on a mobile computing

device with a larger screen, where overview is not a great concern, but once the

system is implemented on a small mobile device with very limited interaction

options, such a filtering scheme might end up being almost useless due to the

lack of overview.

Performing searches requiring lots of user input is also very impractical on devices

without full keyboards, and can easily end up being a very tedious task for the

user. On such devices, integrating the content management scheme with other

applications, such as a contact manager or calendar, could provide the user with

better interaction options, requiring less input to the system.

5.4.3 Faulty Assumptions and Incomplete Context Information The possible pitfall of context aware systems reacting to faulty assumptions, as

described in Section 4.5, should be considered for the construction of any context

aware system. Since my suggestion for a context aware personal content

management scheme does not directly adapt to context changes, but merely

saves context information for created content, the system will always be in a

stable state. Saving faulty assumed context information can none the less cause

annoyance for the user, and ultimately render the system much less usable. To

avoid this pitfall, or at least to largely reduce its implications, the system should

be designed in such a way, that the user is able to correct any mistakes in the

context assumption, and thereby ensure that correct context information is saved

with the content.

Incomplete context information, such as missing location information due to the

lack of GPS signals, should likewise be updatable by the user at a later time, to

allow users to maintain correct context information for more useful content

management.

Summary 57

5.4.4 The Vocabulary Problem One remaining problem in the suggested system is the lack of a controlled

vocabulary in the “comments” facet, as well as the remaining facets if the user is

allowed to edit information in these. One possible solution to this problem could

be to suggest values for user entered information, based on the information

already saved in the system. This would heighten the possibility of a user

choosing the same tag for annotation of similar content every time, thus limiting

the impact of the lack of a controlled vocabulary. An example of such a system

can be found in the nTag system (55).

Context information inferred by the system should be safe from the vocabulary

problem, since the values of context metadata are chosen in a similar manner,

and from the same sources, every time. Giving the user the option to edit this

information introduces the possibility of synonyms being used, but this is an issue

of weighting user control versus system integrity.

5.5 Summary Different design possibilities for a context aware personal content management

scheme have been discussed in this chapter, with my suggestion for the best

result being to build context awareness into the operating system, and not use it

to replace the folder hierarchy, but to provide much more elaborate options for

content searching and filtering. I have described how using simple sensors often

found in mobile computing devices today can be used to infer context, and how

this data can be analyzed and translated into information more humanly

understandable.

Following this description, I have given a few examples of how the context

annotated content can be searched and filtered, followed by a discussion of some

of the considerations and challenges that come with implementing a context

aware personal content management scheme. These considerations and

challenges are important to keep in mind to design a successful commercial

product, as failure to do so could easily result in a system which is not very

58 Design

usable, or which violates the users’ expectations for privacy, resulting in an

unsuccessful system.

CHAPTER 6

6 Implementation

Based on my suggestions for designing a context aware personal content

management system, I have implemented a prototype application named Conga,

which is short for Context Gallery. In this chapter, I describe the aim of this

prototype, and how it is constructed. I then describe how context information is

gathered and saved, and how searching for content using this context

information has been implemented. Finally, I describe some of the problems and

limitations I met in working on the prototype.

6.1 The Aim of the Prototype To examine how context information can be gathered, and how this information

can be used for personal content management in practice, I have implemented a

prototype application which runs on a smart phone. This prototype is meant to be

a proof of concept, demonstrating my suggestion of context aware personal

content management, and also as a study of context gathering in practice, and

how useful this information is for managing content.

I have based Conga on my first suggestion for a context aware personal content

management application, namely the stand-alone application which creates

60 Implementation

content and annotates this with context information. I have chosen this

suggestion since it is the simplest proof of concept, which still demonstrates the

usability of context for content management. I have focused my implementation

on creating a useable application, which gathers context information, annotates

created content with this, and allows the user to retrieve content based on the

same information. I have not focused on performance and the creation of a fully

functional, bug-free implementation. Neither have I focused on the

considerations and challenges listed in Chapter 5, as these are important to keep

in mind when building a full implementation, but less important for simply

demonstrating the use of context information for content management.

My prototype application takes pictures and annotates these with gathered

context information. The user is able to view already taken pictures in a gallery,

and can find pictures using a few simple search methods to search through the

annotated context information. The reason for using pictures as my sample

content is, that pictures themselves contain a lot of information, and all my

suggested context information to be saved can be descriptive of pictures.

Ambience, activity, location, time, relations, and comments all make sense as

descriptive information about a picture, and therefore pictures are good for

examining the usability of context information.

6.2 Building the Prototype The prototype is implemented in Python for S60 (56), and runs on Nokia S60 3

rd

Edition Feature Pack 1 devices with Python for S60 3rd

Edition Feature Pack 1

version 1.44 installed. For the development and test of the prototype, I have used

a Nokia N82 and Nokia N95 smart phone. The reason for choosing Python for S60

(from here on called Pys60) for my implementation is, that it is an excellent

prototype tool, which allows the developer to quickly implement a lot of

functionality using built-in functions in the API. Furthermore, Pys60 is able to

access most of the sensors found in the smart phones used, so these can be used

for gathering context information.

Building the Prototype 61

6.2.1 Modules Used I have used the following modules to be able to use more of the sensors found in

the Nokia smart phones, which helps me gather more context information:

• LightBlue – Version 0.3.3 3rd

Edition (57). This module

is used to scan for nearby Bluetooth devices, and

return all needed information about the found

devices.

• BlueS - Version 1.0.0 3rd

Edition (58). Since the

LightBlue module depends on the Bluetooth module

being switched on, and this is not possible to do with

commands found in Pys60, I use this module to turn

on Bluetooth on the phone.

• Wlantools – 3rd

Edition (59). This module is used to

scan for available wireless networks and return all

needed information about the found networks.

Using these modules has enabled me to gather more extensive context

information, and has simplified the process of scanning for wireless networks and

Bluetooth devices. How to use the various modules was sufficiently described on

their respective websites, which helped me get them working without any bigger

problems.

6.2.2 Sensors Used In my implementation, I make use of the following sensors and built-in functions

of the smart phone for context gathering:

• GPS – Used for getting location information.

• Wireless LAN – Used to find wireless networks in the

vicinity of the mobile device.

• Bluetooth – Used for scanning for Bluetooth devices

near the mobile device. These can be used to infer

people and services in vicinity of the mobile device.

62 Implementation

• Cell ID – Determines which mobile telephone

network cell the mobile device is connected to.

• Clock – Used to get the current date and time, which

is very essential context information.

• Calendar – Used to determine if any events are

entered in the calendar at the time when context

information is gathered.

These sensors are either accessed directly through the built-in functions in Pys60

or by using the modules described in Section 6.2.1.

Compared to the list of possible sensors in Section 5.2, this list of used sensors is

quite a bit shorter due to several factors. The accelerometer is not used, since

interpreting data from this would require the construction of an algorithm to

produce useable data. This is a big task, and outside the focus for this thesis. The

Nokia smart phones I have used do not have built in digital compasses, and while

they do contain the rest of the sensors and built-in functions mentioned in

Section 5.2, the ones I haven’t used have been left out because they are simply

not available for use in Pys60.

6.3 Gathering Context Information The prototype is implemented with a separate class handling the gathering of

context information. This class spawns a few threads that enable it to

continuously receive GPS data when available, and to scan for Bluetooth devices

available with one minute intervals. Whenever context information needs to be

saved, this class is used to gather the available information from all sensors,

translate parts of the information, and return it in a standard Python dictionary.

6.3.1 Translating the Gathered Context Information If a GPS signal is present at time of context gathering, the GPS coordinate is used

to look up much more descriptive information online, using a small web service I


have made as part of the prototype implementation. This web service is

implemented in ASPX using C# on the Microsoft .Net 2.0 platform.

The idea for this translation is inspired by the use of an online gazetteer in the

MediAssist system (49). The implemented web service for translation uses a few

of the web services offered by geonames.org (60) to translate the given GPS

coordinate into much more information. The information I gather from

geonames.org is found by utilizing the following web services:

• http://ws.geonames.org/findNearbyPlaceNa

me - This web service is used to look up extensive

information about the given location based on the

GPS coordinate (a process also known as reverse

geocoding).

• http://ws.geonames.org/neighbourhood -

This web service is used to look up the name of the

current neighborhood. This only works for larger

cities in the United States.

• http://ws.geonames.org/findNearByWeather

XML - This web service is used to look up current

weather information from the nearest weather

station for the given GPS coordinate.

Apart from looking up information using the web services of geonames.org, I use

a standard astronomical algorithm to calculate the solar time of day based on the

given location and time. This algorithm is published by NOAA – the American

National Oceanic and Atmospheric Administration (61) and ported to C# by Pete

Gray (62) (The code itself is found in (63)). Based on the result of this algorithm, I

save the solar time of day as one of the following values:

64 Implementation

I have set a tolerance of 30 minutes on either side of the solar times of day,

meaning that from half an hour before sunset, to half an hour after, the result will

be “Sunset”, and so forth. This is done to provide more useful data, as the solar

times of day are not only interesting in the precise minute they are occurring, but

also a bit before and after.

The single GPS coordinate, along with the current time, is used to translate this

simple information into a lot of information, which is much more humanly

understandable than just a coordinate. The result of the translation consists of

the following:

• Location information – Based on the given GPS

coordinate I look up the continent, country, country

code (for example DK or US), state, province, city, and

if the location is in the United States, also the

neighborhood.

• Weather information – Also based on the GPS

coordinate, I look up information from the closest

weather station, and save the weather station name,

Before Sunrise

Sunrise

Morning Solar

Noon

Afternoon

Sunset

After Sunset

Solar Noon Midnight Sunset Sunrise Midnight

Figure 15 - The translation of the solar times of day into context information. The time of day as

seen on the timeline is translated into the given descriptive name shown in the boxes at the

bottom

Saving All Context Information 65

temperature, dew point, humidity, cloud cover,

weather condition, wind direction, wind speed, and

air pressure in hectopascal.

• Time information – Using the GPS coordinate and

time, I calculate to solar time of day and return this

along with the current time zone.

6.4 Saving All Context Information The result of the online translation of the GPS coordinate and time stamp is

combined with the gathered context information from the other sensors, and

results in the following context information being saved in the constructed facets

as metadata for a taken picture:

Context

Information

Sensor Description

Ambience

Weather station Translation The name of the weather station used for

gathering the current weather information

Temperature Translation The temperature in degrees Celsius

Dew point Translation The dew point in degrees Celsius

Humidity Translation The humidity in percent

Cloud cover Translation A text string describing the cloud cover

Weather

condition

Translation A text string describing the weather

condition

Pressure

in hectopascal

Translation The air pressure in hectopascal

Wind direction Translation The wind direction in degrees from North

Wind speed Translation The wind speed

Activity

Movement Accelero-

meter

A number indicating historic movement

measured by the accelerometer. I have not

implemented this since getting useful data

for this would be a big project in its own

66 Implementation

Calendar event Calendar The calendar events entered in the phones

calendar for the current day

Location

Latitude GPS The latitude from the GPS sensor

Longitude GPS The longitude from the GPS sensor

Altitude GPS The altitude from the GPS sensor

GPS time stamp Clock The time GPS data was gathered

Cell ID Cell tower The ID of the mobile telephone cell the

phone is connected to

City Translation The current city

State Translation The current state

Province Translation The current province

Country Translation The current country

Country code Translation The current country code

Continent Translation The current continent

Time

Timestamp Clock A timestamp describing when the context

information was gathered

Weekday Clock The full name of the current weekday

Month Clock The full name of the current month

Week Clock The current week number

Time of week Clock A text string describing the time of week.

Saved as “Workday” if the day is Monday to

Friday, otherwise saved as “Weekend”

Time of day Translation The calculated solar time of day

Time zone Translation The current time zone

Relations

Wlans Wlan All currently found Wlans are saved with

name and MAC address

Bluetooth

devices

Bluetooth All currently found Bluetooth devices are

saved with name and address

Table 1 - An overview of all the gathered context information used to annotate a taken picture.

The information, grouped into the facets generated in Section 384.4.1, is gathered using the

noted sensor, or translated using the web service based on the found GPS coordinate.

Since many people tend to name their Bluetooth devices and wireless networks

using very non-descriptive names, this information is not always suitable for

Saving All Context Information 67

searching for content in a context aware personal content management scheme.

To make information about wireless networks and Bluetooth devices more

useable, I have implemented an option for the user to enter aliases for the

already saved wireless networks and Bluetooth devices, which allows him to add

more meaningful names than the standard name found.

6.4.1 Saving Context Information in a Database All the gathered context information for a taken picture is saved in an SQL

database on the smart phone. The database has been constructed using a

number of tables, with one table per facet in the faceted metadata constructed in

Section 4.4.1. To save the “Relations” facet, two further tables have been added,

namely “BluetoothDevices” and “Wlans”, which contain information about the

found Bluetooth devices and wireless networks. These tables are made so each

found Bluetooth device and wireless network is only saved once, easing the

saving of aliases for these.

Apart from the tables containing facets, an “Images” table is made, which

contains an ID and the filename for each saved picture. The ID is used as a foreign

key in all the facet tables, linking saved context information to a specific picture.

68 Implementation

Figure 16 - Database diagram for the database constructed to hold the gathered context

information. The tables are constructed based on the facets described in Section 4.4.1, with a

separate table containing information about the gathered content

Saving all context information in a database gives me the advantage of being able

to perform quick searches in all context information, and the ease of constructing

more advanced searches using the features of SQL. I have not focused my work

on saving context information with the generated content, since the challenges

and considerations of doing so, as described in Section 5.4.1, is a whole topic of

its own.

Structure of the Prototype Implementation 69

6.5 Structure of the Prototype Implementation I have constructed the prototype implementation using a number of classes, with

the aim of splitting up functionality and enabling higher reusability of the

different classes. Conga is structured in a classical 3-layer design, with a user

interface layer, a function layer, and a database layer.

This design is chosen, to allow the reuse of the function and database layer for

other implementations of a context aware personal content management

application, by simply building a new user interface layer on top. Very few

changes would have to be made to the lower layers, to reuse them in a new

application creating and managing completely different types of content.

The database layer can easily be extended to implement other search types and

more saved context information, by simply editing the existing functions to allow

Figure 17 - The layered structure of classes in the prototype implementation. The classes are

divided into a typical 3 layer architecture, with a user interface layer, a function layer, and a

database layer

IO and Database Layer

User Interface Layer

Function Layer

ContextFetcher

FuncLayer

BluetoothAliasForm

AdvancedSearchForm

Conga

WlanAliasForm

ImageSaver

DB

IODBLayer

70

for more types of context information, and adding new functions with new search

and filtering methods.

The functions layer is likewise easy to extend, to allow for the gathering of more

context information using other sensors and built

which the application will run.

6.6 The User Interface The user interface is constructed with usability and ease of

Camera Start Conga

Take a picture

Quit

Open gallery

Return to camera

Here the user can

take pictures, open

the gallery, and quit

Conga.

Figure 18 - The two main parts of the user interface

the options these provide. When Conga is started, the camera part is initially active, allowing

the user to take pictures or open the gallery from the menu.

Implementation

for more types of context information, and adding new functions with new search

likewise easy to extend, to allow for the gathering of more

context information using other sensors and built-in functions of the device on

The user interface is constructed with usability and ease of navigation in mind.

Gallery Start Conga

Browse pictures

lery

Return to camera

Here the user can search

for pictures, see context

info, add comments, add

Wlan and Bluetooth

aliases, return to the

camera, and quit Conga.

The two main parts of the user interface, namely the camera and the gallery, and

. When Conga is started, the camera part is initially active, allowing

the user to take pictures or open the gallery from the menu.

Searching for Content 71

I have tried to construct a user interface that gives a good overview, and allows

for use of the application with little user interaction needed.

The main navigation in the application is done using a menu. This menu is

constructed to present the user with options where they are logically available,

with the application having two main functions, namely taking pictures, and

viewing taken pictures in a gallery.

When the application starts up, the camera is ready for use, since having to start

the camera manually would be a great hinder in taking snap-shots of current

events. The user is now able to take the pictures he desires, with context

information being saved automatically for each picture taken.

When the user chooses to open the gallery to view his already taken pictures, a

new set of options becomes available. The user is now able to search for content

using different search methods, as well as browsing through the pictures found as

a result of these searches. Pictures can also be viewed in full screen, comments

can be added to pictures, and context information gathered for a picture can be

displayed. Furthermore, the user is able to enter aliases for already saved

Bluetooth devices and wireless networks.

6.7 Searching for Content The different search types I have implemented, to allow the user to search

through pictures already taken, are two types of simple search, and an advanced

search option. As suggested in Section 5.3, numerous different possibilities exist

for searching and filtering content, but I have chosen to implement these, as I

believe they give a good idea about the options found in using context

information for content management, and are also well suited for use on a device

with limited user interaction options, as well as a limited display size.

72

6.7.1 Simple Search The user is able to perform simple searches, where

keywords can be entered in a textbox. This text box approach is inspired by the

simple approach to searching used by Google

a simple “and” search, and a simple “or” search, which returns pictures matching

all the entered keywords and pictures matching any of the entered keywords

respectively.

The keywords entered are used to search all

text strings saved in the context information

database, and very elaborate searches are thus

possible using this simple search function. As

an example, a simple “and” search for

“Denmark, Thursday, Frederiksberg, Brian”

would return all pictures taken in Denmark, in

Frederiksberg, on a Thursday with a Bluetooth

device or wireless network with the alias

“Brian” entered. A similar “or” search would

return all images taken in Denmark, in

Frederiksberg, or on a Thursday, or with a

comment or device with the name or alias

“Brian” close by.

Using the simple search function, very specific

searches can be performed, as for example a

single “and” search for “Denmark, Thursday, Frederiksberg,

few clouds, birthday” would only return all pictures matching all these values,

which would most probably describe a birthday party in Frederiksberg, on a

Thursday, close to a wireless network named “home wireless”, with Brian close by

and weather with only a few clouds.

The simple search is very easy to use, and users are already somewhat

accustomed to the Google-style search using a single textbox. The downside is the

amount of user input required in typing the search terms. This could be solved by

Implementation

The user is able to perform simple searches, where comma separated search

can be entered in a textbox. This text box approach is inspired by the

used by Google (64). The user can choose between

a simple “and” search, and a simple “or” search, which returns pictures matching

all the entered keywords and pictures matching any of the entered keywords

keywords entered are used to search all

saved in the context information

, and very elaborate searches are thus

possible using this simple search function. As

search for

an”

would return all pictures taken in Denmark, in

Frederiksberg, on a Thursday with a Bluetooth

device or wireless network with the alias

“Brian” entered. A similar “or” search would

return all images taken in Denmark, in

or with a

name or alias

Using the simple search function, very specific

searches can be performed, as for example a

“Denmark, Thursday, Frederiksberg, home wireless, Brian,

rthday” would only return all pictures matching all these values,

which would most probably describe a birthday party in Frederiksberg, on a

Thursday, close to a wireless network named “home wireless”, with Brian close by

The simple search is very easy to use, and users are already somewhat

style search using a single textbox. The downside is the

amount of user input required in typing the search terms. This could be solved by

Figure 19 - Performing a single search

using multiple search terms

separated by comma

Searching for Content

providing some sort of dictionary which suggests search terms based on words

already saved in the context information index. An approach to using such a

system could be adopted from for example the nTag system

6.7.2 Advanced Search To allow for more specific searches than can

be performed using the simple search

possibilities, I have implemented an advanced

search option. Performing advanced searches

are done using a form with 20 fields that can

be used to search in specific parts of the saved

context information. Due to undocumented

limitations in Pys60, forms can only contain up

to 20 fields, which is why I have limited the

advanced search form in this manner.

Using the advanced search, it is possible to

perform searches for pictures taken whe

temperature was between a given minimum

and maximum temperature, the humidity was

over a given threshold, the date between

given dates, and the time between given

times. Furthermore, keywords can be entered

for most of the textual context informatio

entered, with each of the textual searches being performed using an “or” search.

The result from every line in the form is then gathered

to yield search results only fulfilling all entered requirements. Any field left with

its default value will not be used in the search performed.

With the advanced search, it is thus possible to for example search for all pictures

taken at sunset, with the temperature between 10 and 20 degrees Celsius,

73

of dictionary which suggests search terms based on words

already saved in the context information index. An approach to using such a

system could be adopted from for example the nTag system (55).

more specific searches than can

using the simple search

, I have implemented an advanced

Performing advanced searches

are done using a form with 20 fields that can

be used to search in specific parts of the saved

Due to undocumented

limitations in Pys60, forms can only contain up

to 20 fields, which is why I have limited the

Using the advanced search, it is possible to

perform searches for pictures taken where the

temperature was between a given minimum

and maximum temperature, the humidity was

over a given threshold, the date between

given dates, and the time between given

times. Furthermore, keywords can be entered

for most of the textual context information

entered, with each of the textual searches being performed using an “or” search.

The result from every line in the form is then gathered in an “and” search fashion,

to yield search results only fulfilling all entered requirements. Any field left with

default value will not be used in the search performed.

With the advanced search, it is thus possible to for example search for all pictures

taken at sunset, with the temperature between 10 and 20 degrees Celsius,

Figure 20 - Performing an advanced

search using the advanced search form.

This allows a search based on chosen

dates, times, temperatures, humidity,

time zone, and a large number of text

input fields

74 Implementation

humidity over 80 percent, in London, Copenhagen, Paris, or Rome, between the

first of January 2007 and the 15 of June 2009.

Such advanced searches require more user interaction, but yield much more

precise results which can aid the user in finding images based on very specific

terms. Still, advanced searches are most probably seldom used, since simple

searches can be used to perform very elaborate searches. Another downside with

the advanced search is the lack of overview in the advanced search form due to

the limited screen size of the smart phones.

6.8 Problems, Limitations and Result During my work on the prototype, I have met a few limitations which have had an

impact on how the prototype works, and which context information can be

gathered.

6.8.1 Using the Sensors in Pys60 Only some of the sensors found in the Nokia S60 3

rd Edition Feature Pack 1

devices can be used for context gathering in Pys60, which limits the amount of

context information that can be inferred. I have used all available sensors, except

for the accelerometer, but more sensors are present in the Nokia N82 and Nokia

N95 telephones I have worked with. Both smart phones include a light sensor and

of course a microphone, but neither are available through pys60. These sensors

could have been used to give an indication of the ambient light and noise levels,

but unfortunately this is not possible. The amount of sensors available to Pys60 is

likely to grow, with newer versions of the Symbian S60 operating system coming

out, and Pys60 being updated often.

Problems, Limitations and Result 75

6.8.2 Pys60 Some of the standard features in Python have not been implemented in Pys60,

which can sometimes result in problems when trying to implement features that

you expect are easy to make. One such limitation is the absence of sets in Pys60,

which makes the “and” and “or” search functions a bit harder to implement, since

no set operations like union and intersection are available.

Another problem I faced was getting the “like” keyword working in SQL. “Like”

allows searches with wildcards, so the searched for keyword does not have to be

a full match. This would have been very useful in providing better search options,

but despite many different approaches, I failed to get it working. Searches have to

be performed with full text strings matching the saved context information for

this reason.

A third limitation is the calendar implementation. There is no simple way to

extract entered calendar events for the current time only. It is only possible to

extract all calendar events for the given day, rendering the calendar function less

precise. It is easy to imagine several calendar events being entered for a specific

day, and with no straightforward method of extracting only the one valid at the

given moment, this data is less useful than it could have been.

6.8.3 The Implications The limitation in the amount of sensors available in Pys60 has had an impact on

the amount of context information that can be gathered in Conga. Being able to

use the light sensor and microphone could have given interesting and useful data,

and the use of such sensors would be very valuable in a full implementation. The

ambient light and sound level could easily be very descriptive of the ambience in

which content is created, and therefore makes for good context information.

With further analysis of this data, much more context information could be

extracted, but that falls outside the aim of this thesis.

The problems with getting the “like” keyword working in the database

implementation has meant that all searches can only be performed with full text

76 Implementation

string as search terms. I have implemented a workaround for this in the search

for calendar events, just to experiment with it, but implementing this workaround

for searching every single string in the saved context information would be much

more computing intensive than using the native function in SQL. Unfortunately,

limiting text searches to only using full search terms lowers the user-friendliness

of the application.

The limitation in the calendar implementation has caused me to save all calendar

events valid for the given day in the context information. I have chosen this

approach, to still be able to search for calendar information, but this also causes

pictures taken at one event in a day to be indexed together with all other

calendar events of the day.

Lacking sets in Pys60 has not had a big impact on the implementation, since I

have programmed a simple workaround for this, which performs quite well. From

a performance viewpoint, sets are a desirable programming feature that would

be valuable in a full implementation. For the prototype, the lack of it was more of

a simple annoyance during development, but has had no bigger impact on

usability of the system.

6.8.4 Result of the Prototype Implementation Conga has been implemented as described, and works as I had hoped. The

gathering of context functions well, although a bit slow at times, but serves the

purpose of the prototype. There are still a few bugs left in Conga, which I have

not paid much attention to, since they do not have a big influence of the main

functionality of the system. These bugs are:

• Freezing at start-up – The application will sometimes

freeze at start-up, when the user has to select an

internet access point. The reason for this is unclear,

but a restart of the phone usually solves the problem.

• Sporadic failure to save pictures – In a few cases, the

prototype has taken a picture, but failed to save it

Problems, Limitations and Result 77

properly. An inspection of the log file I generate

suggests that the problem stems from issues with

saving data in the database. This can be caused by

trying to save data which is not encoded correctly,

but since it happens very rarely, I have left this bug

unsolved. Furthermore, the application keeps

running, so a new picture can simply be taken.

• Failure to show all context information – If a large

amount of context information is gathered, the

option to show context information for a selected

picture in the gallery will fail to show all context

information. I have left the implementation like this,

since most context information can be seen, and for

the purpose of the prototype, this is sufficient.

• Camera white balance – Even though I have set all

options of the camera module in Pys60 to automatic,

the camera still has major issues with taking pictures

in sunlight. The normal camera application on the

smart phones function well, but using my prototype,

all pictures taken with even a moderate amount of

sunlight are very overexposed, usually resulting in

almost completely white images. The reason for this

remains a mystery, and I have left this bug alone,

since it has no impact on the demonstration of using

context information for content management.

Apart from these bugs, Conga works well. Context information is gathered and

saved when content is created, content can be found by searching for the context

information, and using context information for content management is well

presented by it.

78

Figure 21 - A picture taken using Conga, displayed in full screen mode

display of the context information gathered and saved together with it

As already described, I have not focused my work on performance questions.

Despite this, it is nice to see that the prototype works at a useable speed.

picture and saving context information for it usually takes less than

and searching for pictures in the gallery takes

search types and one to twenty keywords, with

of 40.

In the next chapter, I will go deeper into the

these tests.

6.9 Summary In this chapter, I have described my work with implementing a prototype

Conga as a proof of concept of gathering and using context information

content management. Conga is implemented in Python for S60 and runs on Nokia

Implementation

, displayed in full screen mode, and a

gathered and saved together with it

escribed, I have not focused my work on performance questions.

Despite this, it is nice to see that the prototype works at a useable speed. Taking a

for it usually takes less than 20 seconds,

s in the gallery takes between 1 and 7 seconds using all

keywords, with a current total amount of pictures

the testing of Conga, and the results of

is chapter, I have described my work with implementing a prototype named

as a proof of concept of gathering and using context information for

is implemented in Python for S60 and runs on Nokia

Summary 79

S60 3rd

edition Feature Pack 1 devices, such as Nokia N82 and Nokia N95 smart

phones.

Conga gathers context information using the built in sensors and functions of the

smart phone. Afterwards, the current GPS coordinate and time is used in an

online translation, which returns much more usable context information

describing the current weather, location, and time of day. All this context

information is saved in a database on the smart phone, after which the user is

able to search for created content using both simple and advanced search

features.

I have presented an overview of my design of the prototype application, as well

as the structure of the user interface and database, and described why I have

chosen to build the application in this manner. During the development of the

prototype, I encountered a few limitations and problems, which are also

described in this chapter.

80 Implementation

CHAPTER 7

7 Test

In this chapter, I will describe the tests I have performed using the constructed

prototype, and the results of these. First I will focus on the context information

gathered, followed by using this context information for content search. I will

then describe the ideal test to be performed using this prototype, followed by the

actual user test I have carried out. Finally I will reflect on the results of the user

test, and summarize my findings.

7.1 Context Information Gathering on the Prototype I have begun my testing of Conga by verifying that context information is in fact

gathered, and that the gathered context information is accurate. The tests have

taken place at various locations in Denmark, as I have had the smart phone with

me almost all the time while working on this thesis. I have tried capturing images

on different trains, as well as in various inside and outside locations, with good

results. I have taken around 70 pictures while testing different versions of the

prototype, and have taken 40 with the final version of the prototype at the time

of writing.

82 Test

I have found that if the application is used outdoors, and given 30 seconds to a

minute to ensure that a GPS location has been found, context information is

gathered and translated correctly. The location information gathered from

GeoNames (60) has been highly accurate, and the weather information is

accurate as well. The translated time information also makes good sense, so all in

all, the translated context information seems accurate and meaningful.

If used indoors, the GPS sensor is unfortunately unable to get a signal, unless the

smart phone is held very close to a window. This limits the amount of context

information gathered quite a bit, but is to be expected when relying solely on GPS

for positioning.

If Conga is left running for a minute or two, nearby Bluetooth devices and

wireless networks are usually found, but this seems to be a bit more sporadic. I

have had many cases where known nearby Bluetooth devices and wireless

networks have not been detected, unfortunately leaving this information less

reliable than I had hoped.

All other context information gathered works as I had expected, and apart from

the few bugs mentioned in Section 0, Conga works well.

The table below sums up a single test, listing the expected as well as the actual

context information gathered for a picture taken outdoors and one taken indoors

at the same day and location, two minutes apart. The Conga application was

started up and left for 2 minutes to get a GPS position and scan for Bluetooth

devices and wireless networks. The expected information is based on a weather

report from the website of the Danish Meteorological Institute (65) as well as

assumptions about the context.

The nearby Bluetooth device is another mobile phone left close by for the test,

and the wireless network expected is a network usually available at the test

location. No expectations were entered for the Cell ID, since this is not

information a user would often have expectations about, and the state and

province have been left empty, since states and provinces are not used in

Denmark.

Context Information Gathering on the Prototype 83

Context

Information

Expected Outdoors Indoors

Ambience

Weather

station

Kastrup Koebenhavn/Kastrup

Temperature 10.4 11

Dew point 9 9

Humidity 90 87

Cloud cover Clouds Few clouds

Weather

condition

Rain Light rain

Pressure

in hectopascal

1012 1011

Wind

direction

235 230

Wind speed 8 10

Activity

Movement

Calendar

event

Test event Test event Test event

Location

Latitude 55.681819 55.68172

Longitude 12.549262 12.54917

Altitude 20-22 19.5

GPS time

stamp

“Time of capture” “Time of capture” “Time of

capture”

Cell ID 13591 13591

City Frederiksberg Frederiksberg

State Capital region

Province None

Country Denmark Denmark

Country code DK DK

Continent Europe Europe

Time

Timestamp “Time of capture” “Time of capture” “Time of

capture”

Weekday Saturday Saturday Saturday

Month October October October

84 Test

Week 41 41 41

Time of week Weekend Weekend Weekend

Time of day After sunset After sunset

Time zone GMT+1 GMT+1

Relations

Wlans Default Default, Zyxel,

Home-wavelan

network, Skovlund,

dhihome

Default, Home-

wavelan-

network, Dlink

Bluetooth

devices

Brians telefon Brians telefon Brians telefon

Table 2 - The summarized results of a simple test of context gathering. The context information

gathered for a picture taken indoors and one taken outdoors is compared to the expected context

information.

As seen in Table 2, the gathered context information for the picture taken

outdoors lives up to the expectations. The weather information is almost

precisely what was expected, and is more than accurate enough to be useable,

and the location information even includes a descriptive word for the state. The

picture taken indoors clearly suffers from the lack of a GPS position, as all

information gathered based on this is missing, underlining the lack of much

context information when the GPS location is absent.

7.2 Finding Content Using Search After taking an amount of pictures in different locations and situations, I have

tried searching for them using various context information gathered. I have found

that the implementation of the simple and advanced search features works as

expected, allowing the user to search for all text strings using the two simple

search functions and for much more context information using the advanced

search form.

The results of testing the simple search functions have always been as expected,

returning pictures matching all the keywords entered and any of the keywords

The Ideal User Test 85

entered using ‘and’ and ‘or’ search respectively. I still find the single search

functions to be quite powerful, since multiple terms can easily be combined to

allow for quite specific searches.

Using the advanced search has also yielded the results I expected, and all the

different search fields seem to work as they are supposed to. As expected, the

advanced search allows for making much more specific searches, and especially

the to and from dates and times can be used to quickly filter out lots of pictures.

An advantage of having all search results shown in the gallery is, that as soon as

the search results are narrowed down to twelve pictures or less, the easy

overview provided by the gallery lets the user find the picture quickly. This is

practical, since even with hundreds of pictures, narrowing the result set down to

around 12 pictures does not take extensive searching, as long as some context

information is remembered for the picture sought after.

7.3 The Ideal User Test An ideal test of the prototype, and of using context information for personal

content management in general, would be to have a select number of users use it

for an extended period of time, such as half a year or more. During this period,

they should be instructed to use the system on a regular basis, creating a large

amount of content, for example by taking at least a picture a day on average. At

the time of the test, a number of pictures should be picked out, and the users of

the system would be given the task of finding these pictures based on the context

information they remember about it.

Such a test is interesting, since it tests the system in four ways:

• User friendliness – The users’ use of the menus can

be monitored, and they can be asked to explain how

they go about using the prototype in general. This

can give an insight into flaws and inconvenient design

issues of the user interface.

86 Test

• The remembrance of context information – The

users should explain which search terms they use in

searching for a given picture, and why. This can give

an indication about which context information users

of the system tend to remember, suggesting which

types of context information are most important to

the users of the prototype.

• The search methods used – Monitoring which search

methods the users favor using gives an insight into

where focus on further development of search

features should be put. It might also underline any

flaws that exist in the current implementation.

• The hit and miss rate – The amount of pictures

returned with the performed searches should be

noted, as well as the success in finding the correct

picture. This can show if users are actually able to

find their pictures using the remembered context

information, and also if the saved context

information is elaborate, as well as accurate, enough

for the searches performed.

Such a test is most useful with a large content base to search from, as well as a

long enough time span, allowing for the users to forget some of the information

about given pictures. It is not very hard to search through a small collection of

pictures taken recently, since much context information is remembered about the

taken pictures, and with up to twelve pictures being presented on screen at a

time, not much filtering has to be performed to be able to find a given picture in a

small collection.

7.4 An Actual User Test To have Conga tested by others than myself, I have equipped a friend with a

Nokia N95 running the prototype, and had him use it for 12 days. Even though the

An Actual User Test 87

time span is very short, and the test user has only taken 23 pictures spread out

over the test period, I have performed the test described above. I have done this

to see if any indications about the four mentioned test areas could be found

despite of the limited time span. The test was carried out by me selecting 11

pictures the user had taken, after which he had to find these in Conga by using

the various search methods and the context information he remembered. I noted

down the search terms used, the amount of pictures found at each search, and

the result. The test results are summarized in Table 3.

Picture Search term Found Result

1 Thursday 2 Easy to find

2 Frederiksberg 0 Picture not found

2 (Adv) From time = 18.00.00 2 Picture not found


3 Lyngby 0 Picture not found

3 Norgesvej22 0 Picture not found

3 Weekend 10 Picture was number 2 in the list

4 Weekend 10 Was in same result as picture 3

5 Lyngby 0 Picture not found

5 (Adv) From date 04/10 to

date 04/10

9 Picture was in the list

6 Esther 0 Picture not found

6 Saturday 10 Picture was second to last

7 Gitte 0 Picture not found

7 Monday 2 Picture was number 1 in the list

8 Dtu 0 Picture not found

8 Dtu wireless 0 Picture not found

8 Tuesday 4 Picture was number 4 in the list

9 Brian 1 Not the right picture

9 David 6 Picture not found


9 (Adv) From time = 17.00.00 8 Picture was number 6 in the list

10 Sara 0 Picture not found

88 Test

10 Friday 2 Picture not found

10 Saturday 10 Picture was last in the list

11 Jacob 1 Picture was the only result

Table 3 – The results of the user test. The picture number, search term, number of found pictures,

and the result of the search is noted, with (Adv) denoting a search using the advanced search

form

7.4.1 Observations A number of interesting observations were made while performing the test,

describing how a normal user interacts with the prototype, which context

information was predominantly used to perform searches, and which context

information was most useful:

• Picture 2 was never found, since context information

for it apparently failed to save due to the bug

described in Section 6.8.4.

• Picture 3 and 8 were both attempted found using the

name of a wireless network known to be close by, but

neither had this context information saved. This

further indicated problems with relying on

information about wireless networks close by.

• Picture 5 was attempted found using the search term

“Lyngby”. The picture did in fact have location

information saved, but the city name of “Brede” had

been saved instead. Brede is the correct name for the

location, but since Brede is a very small area, the user

expected the system to have saved Lyngby, which is

the name of the area in which Brede lies. This

highlights an interesting problem with the translated

context information, since the user does not always

An Actual User Test 89

know how precise he should expect this data to be.

One solution could be to add more translated

location information, so both search terms could be

used.

• Picture 8 was sought after using the search term

“Tuesday”. This was merely a guess, since the user

did not remember much context information for that

particular picture. This highlights how the user can

easily resort to guessing for context information,

using this to narrow down the number of results, and

then using the gallery to visually find the picture.

• Picture 9 is a picture of a person named David, but

searching for David returned six pictures taken with

another person named David close by. Apparently his

Bluetooth device is named David, and therefore

these pictures were found. This demonstrates the

need for more precise naming in certain situations,

where a user could for example enter the alias

“David A” for one Bluetooth device, and “David B” for

another, allowing him to search for pictures using

more specific names of people. One can easily

imagine Henderson’s “frequent filer” (9) spending

time adding comments to his pictures, and making

sure that aliases are added for different peoples

Bluetooth devices and wireless networks, easing the

search for such a picture. A “non-filer” is more likely

to resort to simple searches like the one performed

by the user, and visually looking for the sought after

picture using the gallery.

90 Test

• Picture 10 was taken at a party starting Friday

evening, but since the picture was taken after

midnight, the day was saved as Saturday, which

initially confused the user. This is a good example of

the user remembering context information which

does not initially help him find the picture. More

reliable location and relations information could ease

the finding of such a picture, or an advanced search

could be used searching for content created at events

taking place overnight.

• Picture 11 was easily found since “Jacob” had been

added as a comment at an earlier time. This

demonstrates the power of adding comments to

pictures, improving the hit rate of the search terms.

7.4.2 Analyzing the Search Terms Looking at the search terms used in the searches for content, a trend is emerging,

namely that the first keyword used is usually either a location or the name of a

person in the picture. This pattern underlines the importance of gathering

elaborate and correct location information, as well as the high usability of

information saved in the relations facet.

Most pictures taken in the test were taken indoors, and were therefore lacking

location information. In a discussion of the overall usability of the system taking

place after the initial user test, the user described how he sees this lack of

location information as a big limiting factor of the system.

We also discussed the problem of relying on people having Bluetooth activated

on their mobile devices, since this is seldom the case. In an ideal situation, where

Bluetooth can be expected to be turned on at all times, the user does agree that

this is very useful context information.

Summary 91

When searching using location or relations information failed, the user resorted

to searching by the name of the day the picture was taken. He described this as

being easy since there weren’t too many results, and said he would probably use

a more specific approach if the amount of content had been much higher. This is

the problem with such a limited test, and the reason a larger scale test would

yield more interesting results.

7.4.3 Usability In our discussion about the prototype, the test user had a few remarks about the

general usability of the system. One remark was concerning the editing of aliases

for Bluetooth devices and wireless networks. The user felt that it wasn’t entirely

logical that you can always choose from all saved devices, even though you chose

the menu while a picture is selected. His initial expectation was that he would be

able to add aliases to the devices saved when the chosen picture was taken, and

not be able to choose from all devices ever saved. This is a very good point that

should be addressed in future work.

Another annoyance to the test user was the need to enter the precise text string

saved in the system when performing searches. This is already a known problem,

due to the problems with getting the “like” keyword working in the SQL

implementation, and a solution needs to be found in any future work on the

system.

The test user was quite impressed with the usefulness of being able to find all

pictures taken at a family birthday by simply searching for “David”, which was

another person attending the event. He had named his Bluetooth device “David”,

making this search possible without having to enter any comments or aliases.

7.5 Summary The performed tests verify that the constructed prototype is able to gather

context information and use it to annotate created content. Searching for created

92 Test

content using the annotated context information also yields the expected results,

demonstrating the use of the various search types.

Unfortunately not all context information can be gathered as reliably as one could

wish for, which is especially seen in the sporadic nature of the saved Bluetooth

devices and wireless networks. Furthermore, the dependence of GPS signals for

location information usually hinders the collection of good location information

for all content created indoors, which is a big limiting factor when searching for

this content.

The performed user test, although limited due to the available time for testing,

has resulted in some interesting observations. These observations are especially

concerned with which context information is used in searching for content, and

how the lack of always available location and relations information sometimes

results in the need for performing several searches before the desired content is

found. Other observations give good pointers to the areas needing focus in future

work.

Despite these problems, Conga functions well, and is a good demonstration of

gathering context information by using the already available sensors in a mobile

computing device, and using this context information for personal content

management.

To further investigate the usability of context awareness for personal content

management, as well as the user friendliness of the prototype, the described

ideal user test should be performed. This would offer better insights into how

different users use the system to manage larger amounts of content, and would

provide more indications for the directions in which future work should be carried

out.

CHAPTER 8

8 Discussion

Many possibilities exist for continuing work on both the prototype and the

suggested context aware personal content management system in general. In this

chapter, I will first discuss a number of ideas for future work on the user

friendliness of the prototype. Following this, I will describe my suggestions for

how to gather more elaborate and reliable context information, and then propose

a different approach to content search. Lastly, I summarize the suggestions

presented in this chapter.

8.1 The Prototype Based on the test results, a number of improvements to Conga are desirable,

which could make it more user friendly. Apart from removing the few found bugs,

performance of the prototype could be taken into consideration. To decrease the

waiting time experienced when taking a picture, a separate thread in the program

could be used to gather context information and save the picture, leaving the

user free to take more pictures instead of waiting for the first one to be saved. To

further increase performance, the code could be optimized in various ways,

which I have not focused on in this thesis.

94 Discussion

A function for better viewing, and possibly also editing, the saved context

information could allow the user to keep context information for his content up

to date. Editing context information, and entering search terms, could also greatly

benefit from a system aiding the user in choosing keywords to enter, based on

already existing terms, like the nTag (55) and mobTAG (66) systems.

The editing of aliases for wireless networks and Bluetooth devices should also be

made more logical, allowing the user to choose from devices saved for a selected

picture, all devices saved, and devices that can currently be detected. In this

manner, the user can more easily edit the aliases for devices captured together

with a selected picture, instead of having to run through a long list of devices.

Letting the user enter aliases for all devices currently detectable lets the user save

devices in the system, before these devices have been saved as context

information for a particular picture, also heightening user friendliness.

8.2 Gathering Context Information For both the prototype and any other implementation of context awareness for

personal content management, more options should be considered for the

gathering of elaborate, descriptive, and humanly understandable context

information. As described in Section 5.2, as many sensors as possible should be

used to gather as much context information as possible. The data gathered from

these sensors could be analyzed using various techniques, as described by

Schmidt, Beigl, and Gellersen (47), which can yield better and more descriptive

results.

As Conga has demonstrated, the lack of location information and all the

information gathered based on the GPS coordinate, leaves a big part of the often

remembered context information missing. Using alternative methods to gather

location information, together with GPS, should be investigated. An obvious

approach would be to use mobile phone tower information and information

about the wireless networks available to get location information. This approach

is already used on the iPhone (51) and other mobile devices, and while location


information gathered from mobile phone towers and wireless networks available

is less precise than GPS, it is often more than precise enough to yield useful

location information. Since the accuracy of location information using cell phone

towers varies from a few hundred meters and up to several kilometers, this

information cannot be used to infer information about the current street, or

maybe even neighborhood, but it can still be used to gather information about

the current city, region, state and country, just as Conga does today, and as

opposed to using GPS, getting the location information based on cell phone

towers also works indoors.

Another technique, which should be investigated further, is the translation of

gathered context information into more elaborate and descriptive information. I

have experimented with this in the prototype, translating a simple GPS

coordinate and time stamp into much more descriptive and humanly

understandable context information. This can be taken a step further, by for

example adding a list of attractions nearby, which could be coupled with data

from a digital compass to infer which attraction the mobile device is facing. Other

very useful information could be street names, postal codes, any known name of

the location such as a theater, shopping centre, or airport name, and other

descriptive location information. Finally, information about known events, such

as concerts and festivals taking place at the given location and time, could be

gathered and saved. All of this translated context information could prove very

useful for the user when managing his annotated content.

Expanding the use of aliases to also include locations could prove useful. Such a

system could allow the user to choose a location, either on a map, or from a

previously saved piece of content, or use his current location, and enter an alias

for this. The alias could possibly be entered together with a tolerance of the

precision of this location. This approach would allow the user to for example

enter “Home”, “School”, and “Work” locations, and be able to find his content

based on this at a later time.

96 Discussion

8.3 Searching for Content To draw full use of the gathered context information, many different types of

filtering and search can be implemented. I have described some possibilities in

Section 5.3, and working with the prototype has inspired another type of search

that should be further worked on, namely a sort of similarity search. This search

could work by allowing the user to use a piece of content as a starting point, and

find other content created under chosen similar circumstances as the current

content. For example, it could allow the user to find a picture, and use this picture

to find other pictures taken at the same day, event, location, with the same

people, or with the same comments.

A system could be created, allowing the user to use the annotated context

information as sort of links, starting new searches based in this information. The

user should be able to choose how much context information has to be similar to

the currently selected piece of content, thereby setting a sort of tolerance.

Letting the user search his content in this way could ease user interaction

considerably, by requiring less typing of keywords, and allows the user to browse

his content in many useful ways. It is also a valuable tool in organizing content in

interesting ways.

Another possibility for searching content could be to allow for searches which

also return “near matches”. For example, a search for “Friday” could also return

content created a few hours after midnight on Saturday, and a search for

Copenhagen could also include content created in nearby suburbs. This could

help the user find content mentally associated with certain context information,

which was actually created in a different, but almost identical, situation. Such a

search scenario was observed in the user test, where the user tried finding a

picture taken at a party Friday evening, but since the actual picture was taken

after midnight, it was saved with Saturday as the day instead. Such a “near

match” search options should be further considered, to provide a better hit rate

for searches.

Summary 97

8.4 Summary I have listed a number of possibilities for future work on both the prototype and

the context information gathering system, as well as a new approach to searching

for created content. The suggestions for Conga are aimed at making it more user

friendly, and are primarily based on the results of the tests performed. The

suggestions for the gathering of context information mainly focus on the addition

of even more usable information, thereby further expanding the search options

possible. Finally, the suggestion for a different approach to content search is

founded in the desire to lower the amount of required user input, and offering

more different ways to browse and find content. These suggestions are also

mainly based on experiences gained from working with the prototype.

Many of the presented suggestions require large amounts of research on their

own, and represent interesting ways to expand the usability of using context

awareness for personal content management.

A full system implementation could benefit from these suggestions, as they would

heighten the usability of the system. Alongside with these improvements, any

future work on an implementation aimed at everyday use, and not just as a

research tool, should incorporate the considerations listed in Section 5.4.

98 Discussion

CHAPTER 9

9 Related Work

During the work on this thesis, I have encountered other projects which share

similarities to the work I have done. In this chapter, I will describe the projects I

find most similar to mine, and have drawn inspiration from.

9.1 MMM and MMM2 The MMM (67) and MMM2 (68) systems let mobile phone users take pictures,

which can then be annotated by manually entering tags at the time of capture.

The pictures are then uploaded to a web server, together with automatically

gathered context information which has been gathered when the picture was

taken. This context information consists of “all automatically available

information at the point of capture (time, spatial location, phone user, etc.)” (67).

The web server analyzes the uploaded picture and the connected metadata, and

annotates the picture with more information, based on the metadata of pictures

uploaded with similar context information. The suggested metadata is stored in a

faceted metadata structure, with facets such as Person, Location, Object, and

Activity.

100 Related Work

Users accept the suggested metadata using the mobile phones web browser, and

are now able to share the online image with friends saved in the MMM/MMM2

system.

9.2 MediAssist The MediAssist system (49) focuses on efficient searching of personal photo

collections by using context metadata, such as time and location, content

analysis, such as face detection, and semi-automatic annotation to annotate the

pictures, allowing for better search possibilities. All pictures used in the system

have had time and location information added, which is then translated into a

wide range of information as quoted directly from (49):

• Time of capture is augmented to include a number of

time-based fields: year, month, day of month, day of

week and hour, enabling queries such as, for example,

‘all photos taken in the summer, at the weekend’.

• Latitude/longitude co-ordinates are converted into

placenames using a publicly available gazetteer.

• Standard astronomical algorithms calculate the light

status of all images (i.e. day/night/dusk/dawn).

• Each photo is annotated with weather data from the

nearest weather station when the photo was taken.

• Indoor/outdoor classification is inferred from digital

camera metadata, such as the ambient light levels.

• We also segment personal photo collections into

‘Events’ collections by detecting large temporal gaps

between consecutively captured photos.

Summary 101

Furthermore, large building objects and faces in the pictures are detected, and

identities are suggested for the found faces.

The MediAssist system also lets users edit the semi-automatic annotations,

allowing them to add more information, and correct wrong assumptions.

9.3 MobTAG The mobTAG system aims to ease the annotation and searching of personal photo

collections on mobile devices, by implementing an approach to tagging based on

a mapping of tags to the numerical keys of the mobile device. This allows the user

to quickly annotate taken pictures with tags, without having to manually enter

them every time.

Using the same mapping technique, the user is able to search for his photos,

selecting between different search strategies.

The mobTAG system is thus not focused on context awareness, but attempts to

ease the management and use of photo collections by taking the limited user

interaction options of mobile phones and smart phones into account.

9.4 Summary The mentioned systems share similarities with the work done in this thesis, but all

have different focuses than my general suggestion.

The MMM and MMM2 systems share similarities with this thesis when it comes

to the gathering of context information, and saving of this in a faceted hierarchy,

but here the similarities end. MMM and MMM2 do not run solely on the mobile

device, but use the mobile device more as a client in a client-server architecture.

MMM and MMM2 depend on an external web server for most context inference,

annotation, and photo sharing, with all content being stored and managed on the

server. They are therefore not solutions for personal content management on a

mobile device.

102 Related Work

The MediAssist system has inspired me with the translation of simple context

information to much more elaborate information, and using this for managing

content. Other interesting techniques, such a content analysis, are being used in

the MediAssist system, but these are very focused on pictures as the only type of

content, whereas I suggest using context information for a general personal

content management system. Furthermore, the pictures used in the MediAssist

system are not annotated with context information at the time of capture, but at

a later time.

The MobTAG system contains good ideas for easing user interaction with a

system running on mobile and smart phones. These ideas could be used in

improving my prototype, and should be taken into account for designing user

friendly applications for mobile devices with fewer options for user interaction.

Since the Conga prototype is solely created as a proof of concept, it has been

limited to using pictures as the content type to create and manage. Despite this,

the general suggestion of this thesis is to use the same context aware personal

content management approach to all types of content, and this is different from

the presented related work.

In the table below, I have compared the mentioned systems with the work

presented in this thesis on a number of key elements.

MMM MediAssist MobTAG Conga

Context gathering X X - X

Context translation - X - X

Automatic annotation X X - X

Content management - X X X

User friendliness - X X X

Content analysis - X - -

Sharing content X - - -

All types of content - - - (X)

Table 4 – A comparison between different related projects based on selected main points of the

work performed

CHAPTER 10

10 Conclusion

The work presented in this thesis has partly been concerned with constructing a

scheme for using context awareness to ease personal content management, and

partly with designing such a system and testing it by constructing a prototype

named Conga. Working with the prototype, and performing a user test, has

highlighted a few issues to take into consideration when implementing such a

system. It has also demonstrated that constructing such a system is indeed

possible using the sensors and technology already found in high end smart

phones today, and clearly confirmed the usability of context information for

personal content management.

If the suggested context aware personal content management system is

implemented, taking the considerations mentioned in Section 5.4 as well as the

test results in Chapter 7 into consideration, it will provide users with new content

management options not found in the used personal content management

schemes of today. Different possibilities for implementing such a system exist, as

described in Section 5.1. My preference would be to have context awareness built

directly into the operating system, and use the gathered context information to

provide much more elaborate search and filtering options on current file systems,

but not completely replace these.

104 Conclusion

Such a system could greatly ease the tedious work of content management by

supplying “automatic classification” (7), and by allowing further addition and

editing of saved context information, it would cater to both “messy” and “neat”

users, as described by Malone (7).

10.1 Contributions I consider the following points to be contributions of this thesis:

• Using context awareness for personal content management – Most

work reviewed during the work on this thesis has focused solely on the

management of specific types of content, often multimedia. I have here

presented my suggestions for a general use of context information to

easy the management of all types of content, with no specific focus, as I

believe context information is meaningful for any type of content.

• Facets for saving context information – The facets constructed in Section

4.4.1 are designed to contain all gathered context information to be used

in a personal content management scheme, and allow for various search

and filtering options.

• Design suggestions – I have laid out five design suggestions for context

aware content management systems, ranging from a stand-alone

application managing its own content, to a system fully built into the

operating system. These design suggestions can easily form the basis of

further work in this field.

• Prototype implementation – The constructed prototype can without

difficulty be used as a basis for further work, with the database and most

classes in the implementation being completely reusable in another

project.

• Proof of concept – By constructing and testing the prototype, I have

proven that a great deal of context information can be gathered on

mobile computing devices, using the built in sensors and functions

already available today. I have also demonstrated the usability of context

information for personal content management

CHAPTER 11

11 Bibliography

1. Lyman, P and Varian, HR. How Much Information? [Online] 2003.

http://www.sims.berkeley.edu/research/projects/how-much-info-2003.

2. Wire, Business. InfoTrends Study Reveals that Digital Camera Owners Expect

Digital Photography to Replace Film in 4 years. BNet Business Network. [Online]

February 21, 2000. [Cited: April 10, 2008.]

http://findarticles.com/p/articles/mi_m0EIN/is_/ai_59580803.

3. Photo Annotation on a Camera Phone. Wilhelm, Anita, et al. Vienna, Austria :

CHI, 2004. CHI '04 extended abstracts on Human factors in computing systems.

pp. 1403-1406.

4. Fitzgibbon, Andrew and Reiter, Ehud. "Memories for life" Managing

information over a human lifetime. [book auth.] T. Hoare & R. Milner (Eds.).

Grand Challenges in Computing Research. Swindon : British Computing Society,

2003, pp. 13-16.

5. Bush, Vannevar. As We May Think. The Atlantic. 176, July 1945, 1, pp. 101-108.

106 Bibliography

6. MyLifeBits: A Personal Database for Everything. Gemmell, Jim, Bell, Gordon

and Lueder, Roger. 1, New York, NY, USA : ACM, January 2006, Communications

of the ACM, Vol. 49, pp. 88-95.

7. Malone, Thomas W. How Do People Organize Their Desks? Implications for the

Design of Office Information Systems. ACM Transactions on Information Systems

(TOIS). 1983, Vol. 1, 1, pp. 99-112.

8. Farhoomand, Ali F. and Drury, Don H. Managerial Information Overload.

Communications of the ACM. 45, October 2002, 10, pp. 127-131.

9. Henderson, Sarah. Personal Digital Document Management. Lecture Notes in

Computer Science. Berlin, Heidelberg : Springer-Verlag, 2004, Vol. 3101, pp. 651-

655.

10. Sinha, Rashmi. A cognitive analysis of tagging (or how the lower cognitive cost

of tagging makes it popular). Rashmi Sinha - thoughts on technology, design &

cognition. [Online] September 27, 2005. [Cited: Februar 22, 2008.]

http://rashmisinha.com/2005/09/27/a-cognitive-analysis-of-tagging/.

11. Kroski, Ellyssa. The Hive Mind: Folksonomies and User-Based Tagging.

InfoTangle. [Online] December 7, 2005. [Cited: March 31, 2008.]

http://infotangle.blogsome.com/2005/12/07/the-hive-mind-folksonomies-and-

user-based-tagging/.

12. Why We Tag: Motivations for Annotation in Mobile and Online Media. Ames,

Morgan and Naaman, Mor. San Jose, California, USA : ACM, 2007. Proceedings of

the SIGCHI conference on Human factors in computing systems. pp. 971 - 980.

13. Godwin-Jones, Robert. LLT Vol 10 Num 2: EMERGING TECHNOLOGIES Tag

Clouds in the Blogosphere: Electronic Literacy and Social Networking. Language

Learning & Technology. [Online] May 2006. [Cited: April 12, 2008.]

http://llt.msu.edu/vol10num2/emerging/default.html (picture:

http://llt.msu.edu/vol10num2/emerging/figure1.gif).

Bibliography 107

14. Ranganathan, S. R. Elements of library classification. New York : Asia

Publishing House, 1962.

15. Merholz, Peter. Innovation in Classification. peterme.com. [Online]

September 23, 2001. [Cited: April 1, 2008.]

http://www.peterme.com/archives/00000063.html.

16. Doyle, Bob. Facets in your Future. EContentmag.com. [Online] June 8, 2004.

[Cited: March 26, 2008.] http://www.econtentmag.com/?ArticleID=6707.

17. Wine.com. Wine.com. [Online] [Cited: April 20, 2008.] http://www.wine.com.

18. Faceted Metadata for Image Search and Browsing. Yee, Ka-Ping, et al. Ft.

Lauderdale, Florida, USA : ACM, 2003. Proceedings of the SIGCHI conference on

Human factors in computing systems. pp. 401-408.

19. Barreau, Deborah and Nardi, Bonnie A. Finding and Reminding: File

Organization from the Desktop. ACM SIGCHI Bulletin. July, 1995, Vol. 27, 3, pp. 39

- 43.

20. Google Desktop. Google Desktop. [Online] Google. [Cited: May 18, 2008.]

http://desktop.google.com/.

21. Windows Search. [Online] Microsoft. [Cited: May 18, 2008.]

http://www.microsoft.com/windows/products/winfamily/desktopsearch/default.

mspx.

22. Sony Ericsson - Track ID. [Online] Sony Ericsson. [Cited: May 18, 2008.]

http://www.sonyericsson.com/product/trackid/.

23. Gracenote. Gracenote. [Online] [Cited: May 18, 2008.]

http://www.gracenote.com/.

24. MusicBrainz. [Online] MetaBrainz Foundation. [Cited: May 18, 2008.]

http://musicbrainz.org/.

108 Bibliography

25. Midomi. Midomi. [Online] Midomi. [Cited: May 18, 2008.]

http://www.midomi.com/.

26. Inc., Idée. Multicolr Search Lab. Idée Inc. [Online] Idée Inc, 2008. [Cited:

September 15, 2008.] http://labs.ideeinc.com/multicolr#colors=000000,842db8;.

27. Ltd., MyHeritage. Celebrity Collage. MyHeritage. [Online] MyHeritage Ltd.,

2008. [Cited: September 17, 2008.] http://www.myheritage.com/celebrity-

collage.

28. Eg Larsen, Jakob. NEXUS - A Unified Approach to Personal Information

Management in Interactive Systems. Lyngby, Denmark : Technical University of

Denmark, 2005.

29. Gartner. Gartner Says Worldwide Mobile Phone Sales Increased 16 Per Cent

in 2007. Gartner. [Online] February 27, 2008. [Cited: May 15, 2008.]

http://www.gartner.com/it/page.jsp?id=612207.

30. —. Gartner Says Ultralow-Cost Mobile PCs To Drive Growth for Education PC

Segment in Emerging Markets. Gartner. [Online] Gartner, October 2, 2007. [Cited:

May 16, 2008.] http://www.gartner.com/it/page.jsp?id=528107.

31. ASUS. ASUS Eee PC is America’s Most Wanted Christmas Gift. ASUS | Eee PC.

[Online] November 21, 2007. [Cited: May 15, 2008.]

http://eeepc.asus.com/global/news11212007.htm.

32. Portal, UMPC. The complete UMPC comparison, details and specifications

website. UMPC Portal. [Online] [Cited: May 20, 2008.]

http://www.umpcportal.com/products/.

33. Elsparefonden. Rekordsalg af bærbare computere. Elsparefonden. [Online]

Elsparefonden, January 31, 2008. [Cited: May 15, 2008.]

http://www.elsparefonden.dk/aktuelt/nyt-fra-elsparefonden/rekordsalg-af-

baerbare.

Bibliography 109

34. Hruska, Joel. 2008 could be the year laptop sales eclipse desktops in US. Ars

Technica. [Online] Ars Technica, January 3, 2008. [Cited: May 16, 2008.]

http://arstechnica.com/news.ars/post/20080103-2008-could-be-the-year-laptop-

sales-eclipse-desktops-in-us.html.

35. Nokia. Nokia Mediearkiv. Nokia. [Online] [Cited: May 1, 2008.]

http://www.billedarkiv.nokia.dk/Nokia/loginAnonymous.do?lang=dk.

36. Apple. Apple - PR - Products - iPhone. Apple. [Online] [Cited: May 1, 2008.]

http://www.apple.com/pr/products/iphone/iphone.html.

37. Child, One Laptop per. One Laptop per Child. One Laptop per Child. [Online]

One Laptop per Child. [Cited: May 15, 2008.] http://www.laptop.org/laptop/.

38. Computer, ASUSTeK. Eee PC 2G Surf. ASUSTeK Computer Inc. [Online] [Cited:

May 3, 2008.]

http://www.asus.com/products.aspx?l1=24&l2=164&l3=0&l4=0&model=2007&m

odelmenu=1 (picture:

http://www.asus.com/999/images/products/1907/EeePC4G-2.jpg).

39. AspireOneUser.com. Acer Aspire One Images | AspireOneUser.com - Acer

Aspire One User Resource Forum & Blog . AspireOneUser.com. [Online] [Cited:

May 2, 2008.] http://www.aspireoneuser.com/aspire-one-images/ (picture:

http://www.aspireoneuser.com/wp-

content/uploads/2008/06/acer_one_white_blue_closed.jpg).

40. Dell. Dell. [Online] [Cited: May 2, 2008.]

http://www.dell.com/downloads/global/corporate/imagebank/notebooks/xps_m

1330_open.jpg.

41. The Challenge of Mobile Devices for Human Computer Interaction. Dunlop,

Mark and Brewster, Stephen. 4, London, UK : Springer-Verlag, September 2002,

Personal and Ubiquitous Computing, Vol. 6, pp. 235-236.

110 Bibliography

42. Context-Aware Computing Applications. Schilit, Bill N., Adams, Norman and

Want, Roy. Palo Alto, CA, USA : IEEE, 1994. Proceedings of the Workshop on

Mobile Computing Systems and Applications. pp. 85-90.

43. Understanding and Using Context. Dey, Anind K. 1, London, UK : Springer-

Verlag, February 2001, Personal and Ubiquitous Computing, Vol. 5, pp. 4-7.

44. Zimmermann, Andreas, Lorenz, Andreas and Oppermann, Reinhard. An

Operational Definition of Context. Modeling and Using Context. Berlin,

Heidelberg : Springer-Verlag, 2007, pp. 558-571.

45. Ericsson, Sony. Sony Ericsson - Mobile Phones - Overview - W910i. Sony

Ericsson. [Online] [Cited: June 3, 2008.]

http://www.sonyericsson.com/cws/products/mobilephones/overview/w910i?cc=

gb&lc=en.

46. Inc., Apple. Apple - iPhone - Features - Accelerometer. Apple.com. [Online]

Apple Inc. [Cited: Jube 10, 2008.]

http://www.apple.com/iphone/features/accelerometer.html.

47. There is more to Context than Location. Schmidt, Albrecht, Beigl, Michael and

Gellersen, Hans-W. 6, Karlsruhe : Elsevier, December 1999, Computers &

Graphics Journal, Vol. 23, pp. 893-902.

48. Multi-Sensor Context-Awareness in Mobile Devices and Smart Artifacts.

Gellersen, Hans W., Schmidt, Albrecht and Beigl, Michael. 5, Hingham, MA, USA :

Kluwer Academic Publishers, October 2002, Mobile Networks and Applications,

Vol. 7, pp. 341-351.

49. Using Text Search for Personal Photo Collections with the MediAssist System.

O'Hare, Neil, et al. Seoul, Korea : ACM, 2007. Proceedings of the 2007 ACM

symposium on Applied computing. pp. 880-881.

50. Flickr. Flickr: Explore everyone's photos on a Map. Flickr.com. [Online] Flickr.

[Cited: June 3rd, 2008.] http://www.flickr.com/map.

Bibliography 111

51. Apple. Apple- iPhone - Features - GPS. Apple.com. [Online] Apple Inc. [Cited:

June 14, 2008.] http://www.apple.com/iphone/features/gps.html.

52. Nokia. Nokia - Location Tagger. Nokia Beta Labs. [Online] Nokia, May 21,

2008. [Cited: June 4, 2008.] http://www.nokia.com/betalabs/locationtagger.

53. Antifakos, Stavros, Schwaninger, Adrian and Schiele, Bernt. Evaluating the

Effects of Displaying Uncertainty in Context-Aware Applications. [book auth.] N.

Davies et al. UbiComp 2004. Heidelberg : Springer-Verlag Berlin, 2004, pp. 54-69.

54. Using Context as a Crystal Ball: Rewards and Pitfalls. Cheverst, Keith, et al. 1,

London, UK : Springer-Verlag, February 2001, Personal and Ubiquitous

Computing, Vol. 5, pp. 8-11.

55. Hu, Qi. nTag - an image management system on camera-euipped

smartphones. Kongens Lyngby : IMM, DTU, 2008.

56. Pys60. Pys60. [Online] [Cited: April 18, 2008.]

http://sourceforge.net/projects/pys60.

57. Lam, Bea. LightBlue: a cross-platform Python Bluetooth API. [Online] 2008.

[Cited: May 10, 2008.] http://lightblue.sourceforge.net/.

58. BlueS. BlueS. [Online] November 27, 2007. [Cited: May 20, 2008.]

http://sourceforge.net/project/showfiles.php?group_id=132176&package_id=25

3746.

59. Berger, Christophe. Christophe Berger: Py S 60. Christophe Berger. [Online]

2008. [Cited: May 15, 2008.] http://chris.berger.cx/PyS60/PyS60.

60. GeoNames. GeoNames WebServices Overview. GeoNames. [Online]

GeoNames. [Cited: April 03, 2008.] http://www.geonames.org/export/ws-

overview.html.

61. NOAA. Solar Calculation Details. NOAA. [Online] NOAA. [Cited: April 20, 2008.]

http://www.srrb.noaa.gov/highlights/sunrise/calcdetails.html.

112 Bibliography

62. Gray, Pete. Pete Gray's Astronomical Page. Pete Gray's Astronomical Page.

[Online] August 26, 2007. [Cited: April 25, 2008.]

http://home.comcast.net/~pete.gray/Astronomy/index.htm.

63. —. AstronomyClock v1.1. Pete Gray's Astronomical Page. [Online] August 26,

2007. [Cited: April 25, 2008.]

http://home.comcast.net/~pete.gray/Astronomy/AstronomyClock1.1.zip.

64. Google. Google. [Online] Google Inc. [Cited: June 10, 2008.]

http://www.google.com.

65. DMI. DMI - vejrobservationer. DMI. [Online] DMI. [Cited: October 18, 2008.]

http://www.dmi.dk/dmi/index/danmark/vejrobservationer/vejrobservationer-

land.htm.

66. Jensen, Søren Svane. mobTag - Mobile Tagging. Kongens Lyngby : IMM, DTU,

2007.

67. Mobile Media Metadata for Mobile Imaging. Davis, Marc and Sarvas, Risto.

New York, NY : IEEE Press, 2004. Proceedings of ICME 2004 (Taipei, Taiwan, June

27-30, 2004).

68. MMM2: Mobile Media Metadata for Media Sharing. Davis, Marc, et al. New

York, NY, USA : ACM, 2005. Proceedings of the 13th annual ACM international

conference on Multimedia. pp. 267-268.

69. Greer, Matt. Matt Greer Photography: Image File Management. Matt Greer

Photography. [Online] December 14, 2006. [Cited: April 10, 2008.]

http://mgreerphoto.blogspot.com/2006/12/image-file-management.html

(picture: http://bp3.blogger.com/_QvUXNiUo8UQ/RYH8-

SVTu8I/AAAAAAAAAI8/7RzpgKyR8jI/s1600/DriveProperties.jpg).

APPENDIX A

A Installing Conga

To be able to run Conga on a Nokia S60 device, a few requirements need to be

met:

• The device should run S60 3rd

edition FP1

• Python for S60 (Pys60) version 1.44 should be

installed

• The Python Script Shell version 1.44 needs to be

signed and installed

With these requirements met, a few additional programs have to be signed and

installed:

• Blues for 3rd

edition version 1.00, to enable Conga to

activate Bluetooth

• LightBlue for 3rd

edition version 0.3.2, to enable

Conga to scan for Bluetooth devices

• WlanTools for Pys60 1.44 3rd

edition, to enable Conga

to scan for Wlans

After installing these programs, the following directories should be created:

114 Installing Conga

• E:\Python

• E:\Conga

• E:\Conga\Images

• E:\Conga\Images\Thumbs

Finally, the source code for Conga (2 files and a directory named Conga) should

be placed in E:\Python

APPENDIX B

To setup Conga, start the Python script

shell and run the script “CongaSetup.py”,

which will open the CongaSetup

application

B Setting up Conga

script

shell and run the script “CongaSetup.py”,

which will open the CongaSetup

116

From the menu, first select “Create

settings file”. This will save the settings

file for Conga in E:\Conga.

Secondly, choose “Create DB” from the

menu, which will create and save the

database for Conga in E:\Conga

After following these two steps, quit

CongaSetup.

Conga is now installed, setup, and ready

to run.

Setting up Conga

APPENDIX C

To start Conga, start the Python script

shell and run “Conga.py”.

When Conga runs, it will first require a

selection of internet connection, which

has to be available for Conga to be able

to properly fetch context for the pictures

taken.

C Using Conga

To start Conga, start the Python script

When Conga runs, it will first require a

selection of internet connection, which

le

pictures

118

After selecting an internet connection,

Conga will automatically start the

camera, and display a viewfinder.

To take a picture, simply press the select-

button on the phones keyboard.

The captured picture will be shown while

it is saved.

Conga will display a message when the

picture is saved, and the viewfinder will

become active again, to allow for more

pictures being taken.

To open the gallery, and search for

pictures already taken, select “Open

gallery” from the menu.

The menu also allows the camera to be

stopped, and Conga to be closed.

Using Conga

Using Conga

When the gallery is first opened, no

pictures are shown.

To search for pictures, and access the

other functions of Conga, simply open

the menu and select the desired action.

From the menu a selection of options are

available.

To return to the camera function to

capture more pictures, simply select

“Return to camera”.

To quit Conga, select “Quit” at the

bottom of the menu.

119

When the gallery is first opened, no

To search for pictures, and access the

other functions of Conga, simply open

From the menu a selection of options are

To return to the camera function to

capture more pictures, simply select

To quit Conga, select “Quit” at the

120

To search for pictures, three options are

available:

- Simple ‘and’ search

- Simple ‘or’ search

- Advanced search

The two simple search modes allow you

to enter one or more keywords,

separated by comma, which will then be

used to search for pictures.

All the keywords entered are matched

against all text strings saved in the

context for a picture.

Using the ‘and’ search, all pictures

containing all keywords will be shown,

while using the ‘or’ search, all pictures

containing any of the keywords will be

shown.

In the advanced search, search terms can

be entered for most context information,

enabling you to do more precise

searches.

Some fields in the advanced search form

are text fields. Here one or more comma

separated keywords can be entered, and

all matches are found using an ‘or’ search

on these keywords.

Any field left with its default value will

not be used in the search, and all fields

with an entered value will be used to

search in an ‘and’ type search.

As an example, a search with Continent

Using Conga

Using Conga

set to “Europe”, “Denmark, London,

Tokyo” entered in the City field, the

temperature set to be between 10 and

20 degrees, and the Time zone set to

GMT+1 will return all images taken in

Denmark with a temperature of between

10 and 20 degrees, since Tokyo is not in

Europe, and London is not in the GMT+1

time zone.

After a search has been performed, the

resulting pictures are shown in a gallery.

This gallery can be browsed using the up,

down, left, and right arrow keys of the

phone, which will move the green

selection square.

If a gallery consists of more images than

can be shown on the screen, it is possible

to scroll one line at a time by pressing

down when the selection square is at the

bottom of the screen, and up when the

selection square is at the top.

121

n,

Tokyo” entered in the City field, the

temperature set to be between 10 and

set to

GMT+1 will return all images taken in

Denmark with a temperature of between

10 and 20 degrees, since Tokyo is not in

t in the GMT+1

After a search has been performed, the

resulting pictures are shown in a gallery.

This gallery can be browsed using the up,

down, left, and right arrow keys of the

phone, which will move the green

lery consists of more images than

can be shown on the screen, it is possible

to scroll one line at a time by pressing

down when the selection square is at the

bottom of the screen, and up when the

122

To view a picture in full screen, simply

select it with the green selection square,

and press the select button on the phone

keyboard. This will load the full screen

picture.

To return to the gallery, simply press the

selection button again.

To show the context information saved

for a given picture, select the picture in

the gallery using the green selection

square and choose “See context info”

from the menu. Alternatively, open the

picture in full screen, and select the same

menu point.

This will display the gathered context

information for the selected picture.

To return to the previous view, simply

press the select button on the phone

keyboard.

Using Conga

Using Conga

To enter comments for a picture, select it

in the gallery view using the green

selection square, and choose “Add

comments” from the menu.

Alternatively, the same menu point can

be selected from a full screen picture

view.

This will open an input box, allowing you

to enter comments separated by comma,

similarly to the simple search box.

All entered comments can thereafter be

used to search for pictures using both a

simple or advanced search.

To enter aliases for the currently saved

Bluetooth devices, select “Set BT alias”

from the menu.

This will open a window with a

dropdown box allowing the selection of

one of the currently saved Bluetooth

devices. All devices have their current

alias shown in parenthesis in front of

their name, with empty parenthesis

indicating that no alias has been saved.

After selecting a Bluetooth device to

enter an alias for, simply enter the alias

in the “Alias” textbox and press OK.

The alias is now saved, and can be used

to search for pictures taken with the

current Bluetooth device nearby.

123

To enter comments for a picture, select it

in the gallery view using the green

selection square, and choose “Add

m the menu.

Alternatively, the same menu point can

be selected from a full screen picture

This will open an input box, allowing you

to enter comments separated by comma,

All entered comments can thereafter be

d to search for pictures using both a

To enter aliases for the currently saved

Bluetooth devices, select “Set BT alias”

This will open a window with a

dropdown box allowing the selection of

y saved Bluetooth

devices. All devices have their current

alias shown in parenthesis in front of

their name, with empty parenthesis

After selecting a Bluetooth device to

enter an alias for, simply enter the alias

The alias is now saved, and can be used

to search for pictures taken with the

124

Similar to the Bluetooth alias function, it

is possible to enter aliases for already

saved Wlans by selecting “Set Wlan alias”

from the menu.

In the opened window, a Wlan can be

selected from the dropdown box, and an

alias can be entered in the textbox.

The alias is saved by pressing OK, after

which it can be used to search for

pictures taken in the vicinity of the given

Wlan.

Using Conga

APPENDIX D

D Contents of the Enclosed CD

The CD attached to this report contains this thesis, the Python code for Conga,

and the ASP.Net code for the web service in the following directories:

• Thesis\ - Contains this thesis

• Conga\Prerequisites\ - Contains the Symbian

packages to be installed before installing Conga, as

described in Appendix A.

• Conga\Code\ - Contains the Python code for Conga,

to be installed on a Nokia Series 60 3rd

Edition

Feature Pack 1 smart phone, as described in

Appendix A

• Web service\ - Contains the code for the web service

used for translation of context information.

126 Contents of the Enclosed CD

APPENDIX E

E Code Overview

The following is an overview of the code developed for Conga:

• User interface layer

� Conga.py – The main code for Conga and the

graphical user interface layer. This contains

most of the functionality used for showing

the graphical user interface.

� Conga\advancedSearchForm.py – Used for

displaying the advanced search form.

� Conga\bluetoothAliasForm.py – Used to

display the form for entering aliases for

Bluetooth devices.

� Conga\wlanAliasForm.py – Used to display

the form for entering aliases for wireless

networks.

• Function layer

� Conga\FuncLayer.py – The main code for the

function layer. This contains the code called

by the user interface layer.

128 Code Overview

� Conga\contextFetcher.py – Used to gather

all context information. When this code is

initialized it begins GPS positioning, and

starts scanning for Bluetooth devices nearby.

• Database layer

� Conga\IODBLayer.py – The main code for

the database layer. This contains the code

called by the function layer.

� Conga\db.py – Contains all database

functionality, allowing for saving and

searching for context information in the

database.

� Conga\ImageSaver.py – Used to save the

taken picture, as well as a generated

thumbnail, to a file.

� Conga\SystemTools.py – Used to save and

load the settings file

• Other

� Conga\LogFile.py – Used to provide logging

capabilities in the various code files.

� Conga\PicoIO.py – A small IO class, used by

several classes for saving files in a given

folder.

� CongaSetup.py – Used to setup Conga as

described in Appendix B.

conga – the context gallery - etd.dtu.dketd.dtu.dk/thesis/224649/ep08_107.pdf · conga – the...

Documents