![Page 1: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/1.jpg)
Semantic Knowledge and Privacy in
the Physical WebPRAJIT KUMAR DAS, ABHAY KASHYAP,
GURPREET SINGH, CYNTHIA MATUSZEK, TIM FININ, ANUPAM JOSHI
UMBC ebiquity and IRAL Labs
![Page 2: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/2.jpg)
Motivation
Our goal is to provide contextually aware information, using the IoT, that is privacy preserving and ubiquitously helpful
Image courtesy Batman Wikia
CARLTON
Slide 2 of 44
![Page 3: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/3.jpg)
IoT by Volume
Slide 3 of 44
![Page 4: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/4.jpg)
IoT by Domain
Slide 4 of 44
![Page 5: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/5.jpg)
IoT by Privacy Concerns
Slide 5 of 44
![Page 6: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/6.jpg)
Salient features CARLTON: A context-aware, NL question-
answer BOT Context derived from the Physical Web
(IoT) Under development, prototype system Simple NLP using tools like Stanford
CoreNLP Mobile app and Kiosk for front-end ABAC privacy model, Privacy rules using
SWRL Hierarchical context ontology Optional authentication for UMBC people
Slide 6 of 44
![Page 7: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/7.jpg)
Concretization of IoT
Small, quick seamless
interactions with
physical objects and
locations with your
device
Physical web: What?
Slide 7 of 44
![Page 8: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/8.jpg)
Physical web: What? Everything is a tap
away
Slide 8 of 44
![Page 9: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/9.jpg)
Physical web: What? See what’s useful
around you
Slide 9 of 44
![Page 10: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/10.jpg)
Physical web: What? Any object or place
can broadcast
content
Slide 10 of 44
![Page 11: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/11.jpg)
Physical web: How?Three main techniques
Nearby Connections
Slide 11 of 44
![Page 12: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/12.jpg)
Physical web: How?Three main techniques
Nearby Connections
Nearby Notifications
Slide 12 of 44
![Page 13: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/13.jpg)
Physical web: How?Three main techniques
Nearby Connections
Nearby Notifications
Nearby Messages
Slide 13 of 44
![Page 14: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/14.jpg)
System Overview
Slide 14 of 44
![Page 15: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/15.jpg)
System Overview
Slide 15 of 44
![Page 16: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/16.jpg)
System Overview
Slide 16 of 44
![Page 17: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/17.jpg)
Who is Tim Finin?
System Overview
Slide 17 of 44
![Page 18: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/18.jpg)
“Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
Who is Tim Finin?
System Overview
Slide 18 of 44
![Page 19: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/19.jpg)
“Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
SPARQL query generatorContext
Who is Tim Finin?
System Overview
Slide 19 of 44
![Page 20: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/20.jpg)
“Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
SPARQL query generatorContext
Who is Tim Finin?
Inference EngineOntologyKnowledge base
System Overview
Slide 20 of 44
![Page 21: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/21.jpg)
“Tim Finin”: Person Entity type
“Who”: WH query type
Text to Semi-Structured Text
Intent
SPARQL query generatorContext
Who is Tim Finin?
Inference EngineResponse: JSON{“text”: “He’s a Professor in the
Computer Science department!”}
OntologyKnowledge base
System Overview
Slide 21 of 44
![Page 22: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/22.jpg)
Example query
Slide 22 of 44
![Page 23: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/23.jpg)
Is this room booked from
2PM-3PM?
Example query
Slide 23 of 44
![Page 24: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/24.jpg)
User is a faculty and is in front of Conf. room 1.
Is this room booked from
2PM-3PM?
Example query
Slide 24 of 44
![Page 25: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/25.jpg)
Conf. room 1 calendar has no events during that time.
Is this room booked from
2PM-3PM?
Example query
Slide 25 of 44
![Page 26: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/26.jpg)
Is this room booked from
2PM-3PM? No, would you like me to book it from
2PM – 3PM?
Example query
Slide 26 of 44
![Page 27: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/27.jpg)
Is this room booked from
2PM-3PM? No, would you like me to book it from
2PM – 3PM?Yes, please!
Example query
Slide 27 of 44
![Page 28: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/28.jpg)
Okay, the room has been booked
in your name from 2PM – 3PM
Is this room booked from
2PM-3PM? No, would you like me to book it from
2PM – 3PM?Yes, please!
Example query
Slide 28 of 44
![Page 29: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/29.jpg)
Example query
Slide 29 of 44
![Page 30: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/30.jpg)
Is Dr. Joshi here?
Example query
Slide 30 of 44
![Page 31: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/31.jpg)
Is Dr. Joshi here?
User could mean Dr. A. Joshi or Dr. K. Joshi.
Example query
Slide 31 of 44
![Page 32: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/32.jpg)
Is Dr. Joshi here?
But user is in front of Dr. A. Joshi’s office.
Example query
Slide 32 of 44
![Page 33: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/33.jpg)
Is Dr. Joshi here?
User is an advisee of Dr. A. JoshiExample query
Slide 33 of 44
![Page 34: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/34.jpg)
Is Dr. Joshi here?
Dr. Joshi is in a meeting till
3PM
Example query
Slide 34 of 44
![Page 35: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/35.jpg)
Example query
Slide 35 of 44
![Page 36: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/36.jpg)
Example query
Where is Dr. Finin’s office?
Slide 36 of 44
![Page 37: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/37.jpg)
Example queryUser is in CSEE building
Where is Dr. Finin’s office?
Slide 37 of 44
![Page 38: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/38.jpg)
Example query User is unknown to system
Where is Dr. Finin’s office?
Slide 38 of 44
![Page 39: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/39.jpg)
Example query
Where is Dr. Finin’s office?
Please see CSEE front desk for
required information
Slide 39 of 44
![Page 40: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/40.jpg)
Example1.@prefix crltn:<https://www.ebiquity.org/ontologies/carlton/0.1>.@prefix swrlb:<http://www.w3.org/2003/11/swrlb>.crltn: student(?requester)∧(
crltn: supervises(“Xavier”,?requester)∨(crltn: affiliatedWith(?requester,?labName) crltn: leads(“Xavier”,?labName))∧
)∧crltn: hasCurrentLocation(?requester,?aBldgLocation)∧crltn: room(?aBldgLocation) crltn: sitsIn(“Xavier”,?aBldgLocation)∧ ∧crltn: currentTime(?currTime) swrlb: Exists(?anEvent) crltn: speakingAt(“Xavier”,?anEvent)∧ ∧ ∧(
(crltn: startTime(?anEvent,?eventStartTime) swrlb: greaterThan(?eventStartTime,?∧currTime))∨
(crltn: endTime(?anEvent,?eventEndTime) swrlb: greaterThan(?currTime,?eventEndTime))∧) crltn: hasCurrentLocation(“Xavier”,?aLocation) crltn: Location(?aLocation)∧ ∧ ∧crltn: requestLocation(“Xavier”)⇒shareLocation(?aLocation)
Policy Example
Slide 40 of 44
![Page 41: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/41.jpg)
supervises(“Xavier”,?requester)OR(
affiliatedWith(?requester,?labName)ANDleads(“Xavier”,?labName)
)
Policy Example
Slide 41 of 44
![Page 42: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/42.jpg)
hasCurrentLocation(?requester,?aBldgLocation) ANDroom(?aBldgLocation) AND sitsIn(“Xavier”,?aBldgLocation)=>shareLocation(?aLocation)
Policy Example
Slide 42 of 44
![Page 43: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/43.jpg)
Future work Prototype system constantly adding
conversations Beacons on robots Reason over robots near you How robots respond to instructions?
“Can you take me to Prof. Matuszek now?”
“Show me the way to the ITE 346 conference room”
Slide 43 of 44
![Page 44: Semantic Knowledge and Privacy in the Physical Web](https://reader036.vdocuments.mx/reader036/viewer/2022070509/58a4b6de1a28ab2d688b5529/html5/thumbnails/44.jpg)
Summary We presented CARLTON A context-aware, NL question-answer BOT Context derived from the Physical Web (IoT) Semantic web technologies used to preserve
data privacyThanks to NSF for the travel grant!
and Thanks to Google for the gift of
beacons! Slide 44 of 44