Context-Aware Labels: Generation and Verification

Seng W. Loke

Department of Computer Science and Computer Engineering, La Trobe University, Australia

Email: s.loke@latrobe.edu.au

Abstract-Systems to date have labels which are assigned by

a person, e.g., tagging an object or a place with a keyword or

phrase. Given some entities already labelled, a formal mecha­

nism of generating labels using spatial context (detectable by

sensors) is useful, not only to create new labellings without

manual effort, but also that the labels can be used as queries

to verify spatial properties of collections of objects.

I. INTRODUCTION

There are numerous applications for labelling the world

ranging from tourist guides, memory aids, advertising, warn­

ings to helping the blind [5]. What of the world do we label?

The three ontological categories of people, things and places

seem a reasonable answer, and labelling associates a label or

tag to people, things or places, in order to describe aspects

of the entity's contents, properties, and identify (uniquely)

the entity within a space of other entities. Most systems

to date have labels which are assigned by a person, e.g.,

mobile geo-tagging a note, object with GPS coordinates,

or tagging a place with a keyword or phrase, such as [3],

[2].' Our view is that, given some entities already labelled,

meaningful labels can not only be provided by a person but

that certain types of labels and compositions of labels can be

inferred or generated based on common spatial relationships

among labelled entities, that is, the labels are generated using

context information of the entities being labelled. We require

that there is a formally described language for labels and that

these spatial relationships can be detectable via sensors (with

some data processing). Roughly speaking, we aim towards

a label generation system2 II which takes detectable spatial

relationships or context information among entities and an

initial set of labellings of entities, and yields new labels, Le.

II: spatial contextsxinitial labellings --+ new labellings

1 A long list of spatial annotation projects is at

http://www.elasticspace.comJ2004/06/spatial-annotation; http://layar.eu

provides augmented reality style viewing of labels of the world; similar to

Layar is the Sekai Camera http://vator.tv/news!show/2009-02-19-time-to­

start-tagging-the-physical-world 20ur use of the word "system" here refers to formal system as opposed

to a software system.

978-1-4244-5328-3/10/$26.00 ©201O IEEE 776

We contend that new meaningful labellings can be generated

automatically by exploiting the spatial context information

about the entities.

II. A CONTEXT-AwARE LABEL GENERATION SYSTEM

This section gives an example of a formal system II for

generating context-aware labels, comprising four ingredi­

ents: • labels l, which are syntactically well-formed labels

given in EBNF. One could invent a domain-specific set of labels. But an example of a small generic label language is as follows:

L ::=E I L $L I collection (L) I collection.e• I pcollection (L) I pcollection.e• I container (0) I container.e• I near L I lin I [inside

lin ::= in Q I lin < in Q

Iinside ::= inside 0 I [inside < inside 0 E::=PI Q I 0

where E denotes a string, which is a label for an entity,

and can either be a label for a person P, a place Q, or

an object 0 (thing), and E9 denotes the joining of two

alternative labels for an entity (Le., an object, people or

place), Le., SI E9 S2 is comparable to two labels given

to the same entity (similar to two different tags for a

video on youtube). The other labels are derived from

the relative spatial relationships between the entities, as

we detail below.

• entities e, which are in three ontological categories,

people, places and things, to which tags or labels are

associated.

• labellings p:e:l, where an entity e is labelled l given

by an agent p (person or software), or more generally,

A --+ p:e:l expressing that e has label l assigned by p

if assumptions A holds. Note that we might leave out

p in discussions, when it is not important to take that

into account or it is obvious from context; when the

labelling is not by a person but by a system generating

the label, we may use s instead. We denote the labelling

of a collection of entities e1, ... , en with label l as

follows: el, e2, " . , en : l. It is of course possible, that

the same entity can have multiple different labels, i.e.,

e:h and e:h , and h i:- b • labelling rules for inferring labels based on known

labels, as we illustrate below.

Given our example language of labels above, we pro­

vide labelling rules in the Plotkin style [4] written in the

form pre;iS�S where the conclusion holds whenever the con uswn'

premises hold. The rules are used to infer new labellings

from known labellings, and so, could be read as infer the

conclusion, given the premises, and then the conclusion

can be used as premises in rules to derive further conclu­

sions. In the rules below, we assume that there are sensors

to detect the following spatial relationships between entities,

represented abstractly via three Boolean functions: (1) the

"proximity" function, proximity { el , ... , en}, is true if and

only if the entities e1, ... , en are close enough together,

where "enough" is predefined; (2) the "inside" function or

(inverse of) containment, e inside f, is true if and only if the

entity e is inside f, where e and f are things (some object),

i.e., they are not people and not places, assuming that this

is possible in the physical world; and (3) the "located in"

function, e in f, is true if and only if the entity e is located

in q, where q is a place.

The distinction between inside and in is drawn in order to

differentiate two types of containment relationships typical

in computing, when objects are within some container, and

when people or objects are located at or in some place (using

a positioning technology).

1) composition: the joining of two labels into a more

complex (and informative) label: e:�':I;EIl7�12, provided

it and h are not incompatible (e.g., with respect to a

dictionary of antonyms3). We define EB to be commu­

tative and associative. Also, to eliminate compositions

that don't add information, EB is idempotent, i.e., e:lEBl

= e:l, and stronger still, we have that e:it EB l2 = e:it

when l2 is a substring of it. This also eliminates non­

terminating productions that keep applying this rule

(i.e., we discard longer strings that we know do not

carry more information than a shorter version).

2) uniform collection: referring to a collection of entities .

'1 lIb 11 d' e,:l A ... A e,,:l SImI ar y a e e . et, ... ,e,,:collection(l) 3) heterogeneous collection: referring to a collection

f . .

·th d·fti t 1 b l ' e,:l, A ... A e":l,, o entItles WI 1 eren a e s. el , ... ,e,,:collectionbet '

where it, ... , In are not all the same.

3http://thesaurus.reference.coml

777

4) uniform collection by proximity: referring to a collec­

tion of entities similarly labelled and located close to

h th . el:l A ... A e,,:l A proximity el,···,e" eac 0 er. et, ... ,e,,:pcollection I 5) heterogeneous collection by proximity: referring to a

collection of entities with different labels located close

h th . el:h A ... A e":l,, A proximity{el, ... ,e,,} to eac 0 er. et, ... ,e,,:pcollectionbet ' where it, ... , In are not all the same.

6) uniform container: a label for a container

of objects which are similarly labelled: el:l A ... A e,,:l A Irf iE l, ... ,n ,ei inside I

I:container(l 7) heterogeneous container: a label for a container of

objects with different labels

el : h /\ ... /\ en: In /\ Irf i E {I, . . . ,n},ei inside en+l en+l : containeznet

where it, ... , In are not all the same.

8) near: a labelling of an object based on what the object . . proximity{e,f} A I:l IS near. e:near I

9) in: a labelling of an entity based on where the entity

is located: e i:Jn � f:l, where f must be a place.

10) nested in: to model the transitivity of in, we can

generate labels that capture the nested locations: e:li,,<in h A e:in 12 A I:ll A g:12 A (f in g) where f e:I;" < in h < in b '

and 9 are places, and lin may be empty.

11) inside: a labelling of an object based on what other

ob;ect it is contained in: e ins�de .� Al

f:l , where f must � eanSl e be a container object for e.

12) nested inside: to model the transitivity of inside, we can generate labels that capture the nested containers,

e : [inside < inside'1 A e: inside '2 A J : 'I A g: '2 A (I inside g) e : [inside < inside l1 < inside l2

where f and 9 are objects, 9 can contain f, and linside

may be empty.

Note that we can easily generalize rule (1) by replacing

e with a collection of entities denoted by el, ... , en, in

order to obtain a rule for composing a label for collections.

Similarly, we can generalize the other rules. Also, we define

the operator is "<" to be not commutative but associative.

Using the rules above, we can derive labels of entities or

collections of entities.

Example 1.: Given three pairs of shoes, with labellings

01 : shoe, 02 : shoe and 03 : shoe, using rule (2), we derive

the labelling 01, 02, 03:collection (shoe) , expressing a

shoe collection.

Example 2.: Given four chairs near my table

t : my table and within 1.5 meters of each other, labelled

Cl : chair, C2 : chair, C3 : chair and C4 : chair,

using rule (4), we first derive a label for the collection

C1, C2, C3, c4:pcollection (chair). Using rule (8), we

also derive the labels Cl : near my table, C2 : near my table,

C3 : near my table, and C4 : near my table. After applying

rule (8) in this way, we can also apply (4) to obtain the

labelling Cl l C2, C3, c4:pcollection (near my table) ,

expressing a collection of objects close enough to each

other near my table. Then, applying the generalized rule

(1), we obtain the more informative labelling:

Cl l C2, C3, C4: pcollection (near my table) EEl

pcollection (chair)

expressing a collection of chairs near my table, in close

proximity to each other. The idea is that this more complex

label has a derivation, through the application of rules, as

summarised below:

1) from Cl : chair, C2 : chair, C3 : chair and C4 : chair,

and proximity{cl , C2, C3, C4}, by rule (4), infer

Cl l C2, C3, c4:pcollection (chair)

2) from Cl : chair, C2 : chair, C3 : chair and C4 : chair,

and proximity{Ci, t}, by rule (8), infer

Cl : near my table, C2 : near my table, C3 :

near my table, and C4 : near my table,

after which by applying rule (4), infer

Cl l C2, C3, c4:pcollection (near my table)

3) then using results from steps (1) and (2) above, ap­

plying rule (1), we derive the labelling

Cl l C2, C3, C4: pcollection (near my table) EEl

pcollection (chair)

In this example, rule (5) also applies to the collec­

tion of chairs and the table, and we obtain the label

t, Cl l C2, C3, c4:pcollectionhet, which labels the fact that

the four chairs and the table forms a heterogeneous collec­

tion of objects in close proximity with each other. Hence,

assuming sensors capable of detecting proximities among

objects and an initial labelling, we derive a set of new

labellings.

Example 3.: Given a collection of bottles, bl l ... , bk,

for some k, and the labellings of them as bottles bi:bottle,

and that each bottle is within a crate G, i.e., bi inside G,

for each i, a labelling of G is generated via rule (6):

G:container (bottle) .

Example 4.: Similar to Example 2 above, we can

have a set of (four, say) chairs near Table number 9 in a

foodcourt in the shopping mall called "Chadstone", which

is in the suburb Oakleigh in Melbourne. Different labellings

from the same starting point are sometimes possible,

depending on which rules we apply first. For example,

labeUing 1: A labelling of these chairs can be generated

as in Example 2 for each i, Ci:near table9, and

ci:chair. but in addition, we can apply rule (9) repeatedly

knowing the locations of chairs with respect to the

foodcourt, Chadstone, and Oakleigh, obtaining, for each

i, Ci:in foodcourt,Ci:in chadstone.Ci:in oakleigh, and

778

ci:in melbourne. But observing the transitive containment

relationships, and applying rule (10) repeatedly, we

have e;:in foodcourt < in chadstone < in oakleigh <

in melbourne. And then rule(l) gives us, for each i, the

labelling: Ci:(chair EEl near table9 EEl (in foodcourt <

in chadstone < in oakleigh < in melbourne».

Then, with these labellings and proximity{ Cl l C2, C3, cd,

by rule (4), we infer the labelling:

Cl,C2,C3,C4:pcollection (chair EEl near table9 ffi

(in foodcourt < in chadstone < in oakleigh <

in melbourne» , which labels objects Cl l C2, C3 and C4 as

a collection of chairs near table 9 located in a food court

in Chadstone in Oakleigh in Melbourne, starting with the

labellings ci:chair, for i E {I, 2, 3, 4}.

labelling 2: From ci:chair, we apply rule (4), to get

Cl l C2, C3, c4:pcollection (chair)

and from ci:near table9, we apply rule (4), to get

Cl , C2, C3, c4:pcollection (near my table)

and from the above in relationships, we apply rule (4),

to get e;:pcollection (in foodcourt < in chadstone <

in oakleigh < in melbourne).

Now, by applying rule (1) twice, we get an alternative

labelling: Cl, C2, C3, C4:pcollection (chair)

ffi pcollection (near table9)

ffi pcollection (in foodcourt < in chadstone <

in oakleigh < in melbourne) , which describes the same

four chairs as a composition of three pcollection labels,

as three different kinds of close proximity collections, a

collection of objects which are chairs, a collection of

objects near table9, and a collection of objects in the food

court in Chadstone, in Oakleigh, in Melbourne. We term

a collection of objects in certain spatial relationships with

each other as a configuration. We see that if the spatial

relationships among a collection of objects change, then

the labelling must be updated or regenerated to reflect the

new configuration. The question of equivalence comes into

play, in what way are the two labellings equivalent? Just

because they are derived from the same starting points does

not mean they are equivalent, but we have the following

two theorems.

Theorem 1. Given the initial labellings ei : Ii' for

all i E {I, ... , n} and j E {I, ... , k}, for some

configuration, using the above rules, we can always derive

the labellings el , ... , en : collection (II EEl ... EEllk) and

el l ... , en: collection(ltl EEl ... EElcollection(lk).

Proof. The proof is by inspecting

the derivation tree for both labellings: el : h 1\ . . . 1\ el : lk [ ( )]

en : h 1\ . . . 1\ en : lk ----,-----'=----"-Ir 1 1\ . . . 1\ [r(l )]

el : h $ ... $lk en : h $ ... $lk [r(2)] el, . . . , en : collection (/1 $ ... $lk)

el ,11 /\ . . . /\ en ,11 [r(2)J /\ . . . /\ el' lk

/\ . . . /\ e� 'lk [r(2)J is near the table, and if the CiS are close enough to each and ell" " en: collection(ll) ell" " en: collect1on(lk)

[r(l)] • • • el," " en' collection(ll) Ell ··· Ell collection (lk)

. other as determmed by prox2m2ty. Below, we show the where r(l) and r(2) are rules (1) and (2) respectlVely, and

left branch of a proof "tree" from applying the rules in seeing that both derivations have the same initial labellings.

order to validate the label. Note that the "leaves" of the D

tree are queries to sensors (as encapsulated in the calls to Theorem 2. Similar to Theorem 1 but for pcollection

the proximity function). There is also a call to proximity when the configuration is such that proximity{ e1, ... , en}

on the right branch (not shown) of the proof, and so, query holds. Proof. Similar to Theorem 1. D

results can be cached for repeated use. To achieve convergence in this case, we can define prox{cl, t}

[raJ /\ . . . /\ prox{c4, t} [raJ /\ prox{cl" '" C4}

collection (ll EB ... EB In) and collect ion (i1) EB Cl 'near my table c4 'near my table [r4J /\ ...

ci • . . . ,en: pcollection (near m.ytable) [rll, ... EB collection (in) to be equal, but there seems to Cl," " C4 'pcollection(near my table) Ell pcollection(chair)

where prox is the proximity function. be no obvious reason for this. However, the two labels If all the spatial relationships are satisfied, the label is have slightly different shades of meaning, and so different

valid for the given collection of objects, Le. this configura­descriptions of the same objects could be valuable.

Other Label Generation Systems. There could be other

label generation systems with a different label set from L

defined in EBNF above, including other spatial relationships

such as above, below, adjacent, and regions such as "in the

kitchen" and "on chair" (e.g., in [1]). Apart from people,

places and things, GPS coordinates could be added, e.g., a

set of GPS coordinates may form a polygon and be labelled

so, 91, 92, 93, 94:square, or if the set of GPS coordinates

along a road can be labelled as a path.

Personalizing Labels. It must be noted that each person

my build up his/her own set of labels. For example, from

Example 2 earlier, one could, via a system, label the four

chairs as follows: C1, C2, C3, c4:mychairs, or one could

state that mychairs = pcollection (near my table) EB

pcollection (chair), thereby making mychairs a short

form of the lengthier label.

III. LABEL VERIFICATION

Given a system II as given above, instead of using

the system to generate new labels, in a forward chaining

style, the same system could be used, in a backward

chaining style to verify if a label expression holds

for a given collection of (one or more) entities. For

example, given a collection of objects C1, C2, C3, C4,

and a table t:mytable, we might put this query to a

system: Cl, C2, C3, c4:pcollection (near my table) EEl

pcollection (chair)?

which effectively asks if the collection of objects is

near my table, whether they are chairs and whether they

are in close proximity to each other. The evaluation

of this query is then by using rules as above, working

from conclusion to premises, e.g., using rule(l) and then

rule(4), and then having to finally establish if, for each

i, Ci:near my table (Le., proximity { Ci, t} by rule(8»,

ci:chair, and proximity{ C1, C2, C3, C4} holds. In other

words, we need to query sensors to determine if each Ci

779

tion is a model for the label.

IV. CONCLUSION AND FUTURE WORK

We contend that a formal mechanism of generating labels

(based on a formal labelling language) using spatial context

(detectable by sensors) is useful. With some familiarity

with the labelling language, the labels from the proposed

language are readable, but syntactic sugar can also be

employed. Future work will (i) explore more powerful

labelling languages with their own set of rules, as well

as labelling languages for specific applications, and (ii)

implement our model on a mobile device connecting to

infrastructure sensors, and explore the efficiency of label

generation and verification.

REFERENCES

[1] J. Hois, F. Dylla, and M. Bhatt. Qualitative spatial and

terminological reasoning in the ambient domain. In Spatial and

Temporal Reasoning for Ambient Intelligence Systems, COSIT

2009 Workshop Proceedings, pages 32-43, 2009.

[2] Y.S. Maarek, N. Marmasse, Y. Navon, and V. Soroka. Tagging

the physical world. In Collaborative Web Tagging Workshop

at WWW2006, Edinburgh, 2006.

[3] P. P ersson, F. Espinoza, P. Fagerberg, A. Sandin, and R. Coster.

Geonotes: a location-based information system for public

spaces. In K. hook, D. Benyon, A.J. Munro, D. Diaper, and

C. Sanger, editors, Designing information spaces: the social

navigation approach, pages 151-173, London, UK, 2003. Springer-Verlag.

[4] G. D. P lotkin. A Structural Approach to Operational Seman­

tics. Technical Report DAIMI FN-19, University of Aarhus,

1981.

[5] M. Tomitsch, R. Schlogl, T. Grechenig, C. Wimmer, and

T. KOltringer. Accessible real-world tagging through audio­

tactile location markers. In NordiCHI 'DB: Proceedings of the

5th Nordic conference on Human-computer interaction, pages

551-554, New York, NY, USA, 2008. ACM.