semantic metadata to support device interaction in smart environments

Semantic Metadata to Support Device Interaction in Smart Environments 1|Simon Mayer

http://people.inf.ethz.ch/mayersi

Simon Mayer and Gianin Basler, ETH Zurich

Fourth International Workshop on the Web of Things (WoT 2013)

September 9th 2013, Zurich, Switzerland

Semantic Metadata to Support Device

Interaction in Smart Environments



MyselfSimon Mayer, PhD Student at ETH Zurich, WoT since 2009

Realize the full potential of smart environments

by integrating functionality across services

I know the

temperature!

What’s the

temperature?

Is it going to rain

today?

Do I need to

be watered?

I can find song

files for you!

I want to listen to

“Yellow Submarine”

I can play song files!

Realize the full potential of smart environments

by integrating functionality across services

Enabling services to talk to each other and form mashups

Enabling people to talk to service mashups



Heterogeneous services from different providers should

be dynamically composable

Services don’t know anything about how to interact with other

services when they are created (i.e., no “standards”)

Interaction metadata discoverable on the fly!

Flexibility

System should work in the user’s home, office, hotels, etc.

Challenges

Simon’s Office

- Contains 2 services

Think about a smart environment…

I have a song title,

“Yellow Submarine”.

I’d like to play that song.

Soodle S.A

Songs Search

Machine

Method: GET

Input: Song Title

Output: MP3 URL

Sapple, Inc.

Media Player

Method: PUT

Input: MP3 URL

Output: Plays song

Not possible! No Link!

I can do

that!

:-(

The songs search machine doesn’t provide any

hypermedia controls that link to the media playerNor any information that helps the client find out that it could play a song using the services

I can find that

out, actuallyYES. I GOT IT!

Our goal is to enable it for this guy!



Crowd-sourced Service Integration

Web Linking, “Globalized” HATEOAS

Semantics-based Service Integration

Web Linking + Functional metadata

Service Integration: “Self-aware” Things?

Techniques for Semantic Service Integration




Web Linking, “Globalized” HATEOAS

Semantics-based Service Integration

Web Linking + Functional metadata

Service Integration: “Self-aware” Things?

Techniques for Semantic Service Integration



Rely on developers or users to publish links

Semantic Service Integration


Simon’s Office


Soodle S.A

Songs Search

Machine

Method: GET

Input: Song Title

Output: MP3 URL

Sapple Inc.

Media Player

Method: PUT

Input: MP3 URL

Output: Will play music

I know that you can

connect these services!

Let me show you!




Approach: Service Marketplaces

S. Mayer, D. Karam. A Computational Marketplace for the Web of Things, 2012

Mashup developers find links manually, but publish their

mashups in a “WoT Marketplace”

Clients can ask the marketplace for linking information

These links make sense, because they are used in a mashup!

Clients only need to choose which link to follow

We called these «Computational» and «WoT»

marketplaces…




Service Marketplaces

Marketplace

Soodle S.A

Songs Search

Machine

Method: GET

Input: Song Title

Output: MP3 URL

Sapple Inc.

Media Player

Method: PUT

Input: MP3 URL


Node A

Node B

Node C

Node D

Node E

Marketplace

Node A

Node B

Node C

Node D

Node E

I know that you can connect

the services at A and E!

Let me show you!

POST wot_marketplace.org/nodes/A?link=nodes/E&name=play_song

202 Created

Marketplace

Node A

Node B

Node C

Node D

Node E

Now, others can use

that information!

play_song

GET wot_marketplace.org/nodes/A

“service” : “www.songsearch.org”

“forward_paths” :

“play_song” : “nodes/E”

I can follow that

link now, too!

Issue 1: Availability and reliability of the crowd?

Issue 2: Clients are compiled against the link names (e.g., “play_song”)!

This didn’t enable fully automatic service usage!

Marketplace

Node A

Node B

Node C

Node D

Node Eplay_song

Can we automate the publishing of links and the traversal itself?

Simon’s Office


Soodle S.A

Songs Search

Machine

Method: GET

Input: Song Title

Output: MP3 URL

Sapple Inc.

Media Player

Method: PUT

Input: MP3 URL




Idea: Semantic metadata to find potential links!


Functional Semantic Metadata

Simon’s Office


Soodle S.A

Songs Search

Machine

Method: GET

Input: Song Title

Output: MP3 URL

Sapple Inc.

Media Player

Method: PUT

Input: MP3 URL


R.Verborgh et al. RESTdesc - A Functionality-Centered Approach to Semantic Service Description and Composition, 2012

S. Mayer and G. Basler. Embedded Semantic Metadata to Support Device Interaction in Smart Environments, 2013



Simon’s Office


Soodle S.A

Songs Search

Machine

Method: GET

Input: Song Title

Output: MP3 URL

Sapple Inc.

Media Player

Method: PUT

Input: MP3 URL


Based on what I know

about these services, there

could be a link here!



Embedding of semantic functional

service descriptions

How? Connect that

information to its API

What?Describe what a

service does

Based on what I know

about these services, …

Use of a reasoning engine to

process these descriptions



Usability as one major constraint

It should not be too hard to create service descriptions

It should be easy to create user goals

Pragmatism as another

Make use of previous research in semantic technologies

Use existing ontologies. Avoid creating new ones!

Make it work for real use cases

Format: Notation3 (user-friendly RDF with inference rules)



The title of the resource people.org/person1/, is “Person 1”, and it knows

another resource, people.org/person2/

@prefix dc: <http://purl.org/dc/elements/1.1/>.

@prefix foaf: <http://xmlns.com/foaf/0.1/>.

<people.org/person1>

dc:title "Person 1";

foaf:knows "people.org/person2/".

No

tati

on

3Brief N3 Primer: Statements

Brief N3 Primer: Rules

𝑹𝒖𝒍𝒆: 𝑃𝑟𝑒𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 → 𝑃𝑜𝑠𝑡𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛

“people.org/person1/ knows people.org/person2/ ”

implies

“people.org/person2/ knows people.org/person1/ ”

If one person knows another person, this second person also knows the first. (“foaf:knows is symmetric”)

No

tati

on

3Brief N3 Primer: Rules



Precondition.

=>

Postcondition.

is shorthand for=> log:implies


No

tati

on




?x a foaf:Person;

foaf:knows ?y.

=>

Postcondition.



Superb tutorial at n3.restdesc.org (by Ruben Verborgh)

http://n3.restdesc.org/

No

tati

on




?x a foaf:Person;

foaf:knows ?y.

=>

?y foaf:knows ?x.



Superb tutorial at n3.restdesc.org (by Ruben Verborgh)

http://n3.restdesc.org/



Simon’s Office


Soodle S.A

Songs Search

Machine

Method: GET

Input: Song Title

Output: MP3 URL

Sapple Inc.

Media Player

Method: PUT

Input: MP3 URL


Now: Semantic metadata for the Songs Search Machine





Precondition: “What facts does a meaningful execution of the service require?”

We have a Song Title

Postcondition: “What additional facts hold after an execution of the service?”

We have the URL of a song with that title

Soodle S.A

Songs Search

Machine

Method: GET

Input: Song Title

Output: MP3 URL



No

tati

on

3The Songs Search Machine


Precondition.

=>

Postcondition.

If we know a song title, we also know the URL of a song with that title.

No

tati

on



?songTitle a dc:title.

=>

Postcondition.



No

tati

on




=>

?songURL a dc:Song;

dc:title ?songTitle.

Store this as ssm_rule.n3

“Yellow Submarine” a dc:title.

“Yellow Submarine” a dc:Title.

?songURL a dc:Song;

dc:title “Yellow Submarine”.

[+ proof]

ssm_rule.n3

But I still don’t know

how to call that

service!!

How to include the request? Basically two options…

𝑹𝒖𝒍𝒆: 𝑃𝑟𝑒𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 ∧ 𝑅𝑒𝑞𝑢𝑒𝑠𝑡 → 𝑃𝑜𝑠𝑡𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛

𝑹𝒖𝒍𝒆: 𝑃𝑟𝑒𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 → 𝑃𝑜𝑠𝑡𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 ∧ 𝑅𝑒𝑞𝑢𝑒𝑠𝑡

This works, but is impractical (knowledge about all requests required from the start…)

It also messes up the quantification!

“Given the Preconditions, all GET requests give the Postcondition”

This works too, and it is sound!“Given the Preconditions, there exists a GET request that gives the Postcondition”

R.Verborgh et al. RESTdesc - A Functionality-Centered Approach to Semantic Service Description and Composition, 2012

The second option

sounds better to me!

No

tati

on




=>

?songURL a dc:Song;


Request.

If we know a song title, we also know the URL of a song with that title. Plus, we

know about an HTTP request.

No

tati

on



@prefix http: <http://www.w3.org/2011/http#>.


=>

?songURL a dc:Song;


Request.



No

tati

on



@prefix http: <http://www.w3.org/2011/http#>.


=>

?songURL a dc:Song;


_:request http:methodName “GET”;

http:requestURI (<www.ssm.org?title=>?songTitle);

http:resp [ http:body ?songURL ].

Store this as ssm_rule.n3 We’ll refer to it as a “RESTdesc” description




“Yellow Submarine” a dc:Title.

?songURL a dc:Song;

dc:title “Yellow Submarine”.

_:request http:methodName “GET”;

http:requestURI (<www.ssm.org?title=>?songTitle);

http:resp [ http:body ?songURL ].

[+ proof]

ssm_rule.n3



Now, we only need to link that to the service!Multiple ways to do that…

We use HTTP OPTIONS

“This method allows the client to determine the options and/or requirements

associated with a resource, or the capabilities of a server, without implying a

resource action or initiating a resource retrieval.” (RFC 2616)

OPTIONS www.ssm.org

Link: <ssm_rule.n3>; rel=“describedBy“

And the Link Header (RFC 5988) to return the description

Soodle S.A

Songs Search

Machine

ssm_rule.n3

http://www.w3.org/Protocols/rfc2616/

http://tools.ietf.org/html/rfc5988

How do I say that I

want a song URL?


Goal: ?song a dc:Song; hasTitle “Yellow Submarine”.

A proof that is parsed to an internal HTTP request description format

Send a GET request to www.ssm.org?title=songTitle and substitute “songTitle” with

“Yellow Submarine”. Then, unpack the response. Its body contains the song.

GET ?title=YellowSubmarine

www.songs.org/yellowsub.mp3

ssm_rule.n3

Soodle S.A

Songs Search

Machine

ssm_rule.n3

http://www.ssm.org/?title=songTitle

And how do I play “Yellow

Submarine” now??



Missing: Semantic metadata for the Media Player!

Simon’s Office


Soodle S.A

Songs Search

Machine

ssm_rule.n3

Sapple Inc.

Media Player

Method: PUT

Input: MP3 URL






Perfect!

Simon’s Office


Soodle S.A

Songs Search

Machine

ssm_rule.n3

Sapple Inc.

Media Player

mp_rule.n3



I want to play “Yellow

Submarine” now!!

Then formulate your goal.

But be precise!

?state a st:State;

log:includes

:SimonsOffice myOnt:musicPlaying [ dc:title “Yellow Submarine” ]

I want Simon’s office to be in a state

where “Yellow Submarine” is playing.


Client Goal

A proof that is parsed to an internal HTTP request description format


“Yellow Submarine”. Then, unpack the response. Its body contains the songURL. Take this

variable and send a PUT to www.mp.org?song=songURL

ssm_rule.n3

Execute the requests!

mp_rule.n3


http://www.mp.org/?song=songURL


“Yellow Submarine”. Then, unpack the response. Its body contains the songURL. Take this

variable and send a PUT to www.mp.org?song=songURL

Sapple Inc.

Media Player

mp_rule.n3

GET ?title=YellowSubmarine


Soodle S.A

Songs Search

Machine

ssm_rule.n3

PUT ?song=www.songs.org/yellowsub.mp3


Yay!


http://www.mp.org/?song=songURL

Sounds complicated. Could you

sum up what I have to do?

1. Create your goal and inputs

2. Find a reasoner

3. Ask the reasoner how the goal is reachable

4. Parse the proof to extract the requests

5. Execute the requests

I don’t believe this!

Let’s see it!

http://192.168.0.21:8000/proof?

q=http://people.inf.ethz.ch/mayersi/testdescriptions/n3goals

/user-goal-music.n3&

r=http://people.inf.ethz.ch/mayersi/testdescriptions/n3goals

/user-input.n3

user-input.n3 contains “Yellow Submarine” a dc:title.”

Reasoner has access to ssm_rule.n3, mp_rule.n3, and a states ontology.

Do Proof!

Oh, it actually works…

But I can’t understand a thing!

Do Proof!

http://192.168.0.21:8000/proofHumanReadable?

q=http://people.inf.ethz.ch/mayersi/testdescriptions/n3goals

/user-goal-music.n3&

r=http://people.inf.ethz.ch/mayersi/testdescriptions/n3goals

/user-input.n3

user-input.n3 contains “Yellow Submarine” a dc:title.”

Reasoner has access to ssm_rule.n3, mp_rule.n3, and a states ontology.

But isn’t it too hard

to create the goal, for

a client like me?

That’s tricky. But we’re working on it and have come up with some good

ideas… one of those ideas would even allow for integration with Clickscript !

And you know how to use that, right?

Now I get it!

Hm. So I should

just state my goals.

And I can find and

use services, too!



Localization + stream transmitter+receivers in all rooms

First version took 10s to switch songs (mostly loading .n3 files… described in the paper)

Cut that by actively discovering services and pre-loading descriptions

This is in the paper…

Prototype: A music stream that follows you!



Our Goal



User defines preferences(temperature, preferred song, etc.)

User agent negotiates with the user’s environment to set

these preferences

User gets feedback regarding success/failure

Our Goal

Thank you!

Simon Mayer and Gianin Basler: Semantic Metadata to Support Device Interaction in

Smart Environments. Proceedings of the 4th International Workshop on the Web of Things

(WoT 2013). Zurich, Switzerland, September 2013



Credits

Nadine Inhelder, Gianin Basler

Ruben Verborgh

http://andrewbleakley.com

http://www.webmarchand.com

http://www.patentspostgrant.com/

http://www.orkin.com

http://sweetclipart.com

http://la-matrice.org

http://learn.randolph.k12.ma.us

http://theparisreview.org

http://bestclipartblog.com

http://freepik.com

http://http://www.rugdots.com

http://www.productdose.com

http://westwoodblinds.co.uk

http://www.ambientdevices.com

http://www.pats.ua.ac.be

http://www.vesternet.com

http://www.rovingnetworks.com

http://www.qualitymechanicalservices.com

http://chantalteakettle.org

http://blog.eeye.com

http://cartographersguild.com

http://haverford.edu

http://clker.com

http://stickers-addict.fr

http://en.wikipedia.org

Image Sources

http://andrewbleakley.com/bigcommerce-developer/

http://www.webmarchand.com/

http://www.patentspostgrant.com/

http://www.orkin.com/

http://sweetclipart.com/

http://la-matrice.org/

http://learn.randolph.k12.ma.us/

http://theparisreview.org/

http://bestclipartblog.com/

http://freepik.com/

http://http/www.rugdots.com

http://www.productdose.com/

http://westwoodblinds.co.uk/news/95/hooray-you-did-it

http://www.ambientdevices.com/products/umbrella.html

http://www.pats.ua.ac.be/content/images/sunspot.jpg

http://www.vesternet.com/images/products/large/Plogg.jpg

http://www.rovingnetworks.com/

http://www.qualitymechanicalservices.com/images/ac-1.jpg

http://chantalteakettle.org/

http://blog.eeye.com/wp-content/uploads/2011/09/saupload_happy_robot.jpg

http://cartographersguild.com/

http://haverford.edu/

http://clker.com/

http://stickers-addict.fr/

http://en.wikipedia.org/

semantic metadata to support device interaction in smart environments

Technology