visual web keynote at mmsp 2015
TRANSCRIPT
![Page 1: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/1.jpg)
Weaving the Visual Web
Ramesh Jain Dept. of Computer Science
University of California, Irvine [email protected]
.
![Page 2: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/2.jpg)
• Memories of experiences.
• Explicit contextual communicaFon. – Data is a new uFlity. – Text is too abstract in many situaFons.
Duality of Photos
Why do we take photos?
![Page 3: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/3.jpg)
Where do photos come from?
• Drawings • Pain6ngs • Cameras • Smart Cameras • Graphics
![Page 4: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/4.jpg)
Camera History
![Page 5: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/5.jpg)
The Role of Photos con6nues to Evolve
• Memory • Informa6on
– Notes – Documenta6on
• Ephemeral
Photos have ‘half life’!
![Page 6: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/6.jpg)
In 20th century, we tolerated photos in our textual documents.
In 21st century, you create visual documents that tolerate text.
![Page 7: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/7.jpg)
Major Disruption in Photos: From Memories to Information Sources.
Photos are the most compelling source of information.
![Page 8: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/8.jpg)
Experiences
Life =
Events
+
![Page 9: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/9.jpg)
Progress: Knowledge Creation and Propagation
Visual
Visual knowledge
Oral
Oral knowledge
Textual knowledge
![Page 10: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/10.jpg)
Berners-‐Lee: Suppose all the informa6on stored on computers everywhere were linked. Suppose I could program my computer to create a space in which anything could be linked to anything.
![Page 11: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/11.jpg)
Web: Human experiences, knowledge, and understanding captured using associative links in DOCUMENTS.
![Page 12: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/12.jpg)
Text based documents are not NATURAL. Language and Literacy come in the way.
![Page 13: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/13.jpg)
Imagine if every photo and video captured were connected to every other! And to other information!!
![Page 14: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/14.jpg)
What is a camera?
Captures intensity from a point in the world.
![Page 15: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/15.jpg)
Is a Smartphone camera sFll a camera?
Many sensors capture metadata related to the moment and capture. • Exposure Time • Aperture Diameter • Flash • Metering Mode • ISO Ratings • Focal Length • Time • Location • Face
Smartphone camera captures events.
![Page 16: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/16.jpg)
Computa6onal Representa6on of an event.
Experien6al Data: • Photos • Video • Audio • Accelerometer • Heart rate • …
![Page 17: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/17.jpg)
EMPT: Extractable Mobile Photo Tags
Exif
Content Analysis
EMPT
People Place Objects Events
…
![Page 18: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/18.jpg)
Krumbs: Capture and Connect holisFc experience of a moment.
What: Objects Who: People When: Events Where: Location Why: Intent/Emotions
![Page 19: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/19.jpg)
![Page 20: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/20.jpg)
![Page 21: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/21.jpg)
All photos and associated context
is automatically
Organized in a Web, and
Shared (if desired).
![Page 22: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/22.jpg)
![Page 23: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/23.jpg)
![Page 24: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/24.jpg)
![Page 25: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/25.jpg)
![Page 26: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/26.jpg)
Duality of Photos: Relevance to MM and CompuFng
• Memories: Search and Retrieval
• InformaFon communicaFon: Web, Big Data
![Page 27: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/27.jpg)
How do you search photos?
What Who When Where Why
AssociaFons related to these.
![Page 28: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/28.jpg)
Searching ‘For a Photo’ and
Searching ‘From a Photo’.
28
![Page 29: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/29.jpg)
29
![Page 30: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/30.jpg)
30
Who When Where What Why
AssociaFons related to these.
![Page 31: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/31.jpg)
Marking Moments: Micro Blogs
• Facebook Status and Tweets started Micro Blogs. – Now there are many – Instagram, Snapchat, …
• Problem with Tweets: More Noise less Data.
• Time to add Focused Micro Blogs – Sensors – Importance of marking a moment
![Page 32: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/32.jpg)
Waze
![Page 33: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/33.jpg)
Crowdsourced SituaFons
![Page 34: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/34.jpg)
Crowdsourced SituaFons
![Page 35: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/35.jpg)
A Photo is a Click in Real World.
• Remember Kodak Moment! • For each photo:
• Unique ID, • All metadata of the event, • Tags, • Links, • AnnotaFons.
![Page 36: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/36.jpg)
Crowdsourced SituaFons
At 435 Main 8:37 AM
![Page 37: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/37.jpg)
10/20/15 37
![Page 38: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/38.jpg)
10/20/15 38
Flood level - Shelter
Flood Level Shelter
Classify (Flood level - Shelter)
![Page 39: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/39.jpg)
Krumbs Used as Focused Micro Blog
• One Click upload to the FMB-‐App Server. • The client sends:
– Sender ID – Photo – GPS and Place – Time and Event – Emoji based Context and annota6on – Any addi6onal comments
• As JSON
![Page 40: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/40.jpg)
FMB-‐App Server
• FMB-‐App Server uses EventShop for appropriate aggrega6ons. – Loca6on – Geographic area (ward, town, city, …) – Also computes some rates of changes. – Determines Trends – Classifies areas based on evolving and current situa6ons.
• Allows drill-‐down to show even individual photos.
![Page 41: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/41.jpg)
Tweets VS Krumbs
Tweets 1. Tweet require thinking and
typing. 2. Loca6on of a tweet and the
corresponding event may be different.
3. Tweets are subjec6ve.
Krumbs 1. Krumbs have more
informa6on and are spontaneous.
2. Krumbs maintains event loca6on.
3. Krumbs are objec6ve.
![Page 42: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/42.jpg)
Seman6c Links for a Photo
• Photo Level – Automa6c crea6on – Manual Annota6on – Manual Crea6on
• Segment Level – Automa6c Crea6on – Manual Crea6on
![Page 43: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/43.jpg)
ObjecFve Self: From Personal Big Data
![Page 44: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/44.jpg)
Life Events relate disparate streams to life.
Personal photos on smart phones TELL a lot about you.
![Page 45: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/45.jpg)
Visual Web PlaZorm
Contextual Reasoning and Event ComputaFons
NavigaFon and Search
Krumbs
Next App
Knowledge Discovery: Event AnalyFcs and VisualizaFon
Agro-‐ Tech
Clean India Personal
Sharing and CommunicaFon
![Page 46: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/46.jpg)
Your Personal Visual Web on Smartphone
Photo Cloud
Moment Capture
Contextual Reasoning and
Event ComputaFons
Event AnalyFcs And VisualizaFon
NavigaFon and Search
• Available on Android and iOS.
• Your data remains on your phone unless shared.
Sharing
![Page 47: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/47.jpg)
Developing An App: Agro-‐Tech
Contextual Reasoning and Event ComputaFons
NavigaFon and Search
Krumbs for Agro-‐Tech
Agro Knowledge Discovery: Event AnalyFcs and VisualizaFon
Agro-‐ Tech
Sharing and CommunicaFon
Popular Selfie Food Agriculture Shopping
![Page 48: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/48.jpg)
Technology for Building Visual Web
• Visual Authoring Environment: HTML for Visual Data.
• Content Analysis • Deep Contextual Reasoning: From Smartphones, sensors, IoT, personal history, social, personal and all other events.
• Event Clustering and recogniFon. • Photo Ranking • Combine with Deep Learning based Content Analysis
• Visual NavigaFon • Cross sharing and integraFon with other PlaZorms. • Combine Algorithmic and InteracFve tools.
![Page 49: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/49.jpg)
For Visual Web, we need • Addressing (URI) • Transfer Protocol (HTTP) • Authoring and Presentation (HTML) • Ranking • Contextual Processing • Content Analysis • Privacy and Security • Information Vs Experience
Challenges
![Page 50: Visual Web keynote at MMSP 2015](https://reader033.vdocuments.mx/reader033/viewer/2022042907/586f74be1a28ab10258b5cdf/html5/thumbnails/50.jpg)
Sky is the Limit. ApplicaFons:
Lifestyle Health
Commerce Surveillance and Monitoring
Agriculture Research and development
ConstrucFon …