events in multimedia - theory, model, application
DESCRIPTION
Talk by Ansgar Scherp. Title: Events in Multimedia - Theory, Model, Application Event: Workshop on Event-based Media Integration and Processing, ACM Multimedia, 2013TRANSCRIPT
![Page 1: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/1.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 1
Events in Multimedia -
Theory, Model, Applicat
ionWorkshop on Event-based Media Integration and
Processing, ACM Multimedia, 2013
Juniorprof. Dr. habil. Ansgar Scherp
![Page 2: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/2.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 2
Motivation
• Events are a natural abstraction
of human experience
• Events are everywhere!
• Lifelogs
• Experience sharing
• Emergency response
• Cultural heritage
• News
• News
• Sports
• Surveillance
• …
![Page 4: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/4.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 4
Brothers at enmity with each other …
Event
Object
Object
Event
• Earlier: object-based and entity-based systems
• Now: applications that consider events at least
as important as objects
vs.
![Page 5: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/5.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 5
What is an event?
• Perduring entities that unfold
over time
• Occurrences in which humans
participate
• Subject to discussions and
interpretations by humans
• Enduring entities that unfold
over spaceObject
Event
![Page 6: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/6.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 6
Brothers at enmity with each other …
• Some philosophers consider objects as 4D
• Extend across time just as they do in space Casati R, Varzi A (2006) Events. Stanford encyclopedia of philosophy.
http://plato.stanford.edu/entries/events
Object
Event
![Page 7: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/7.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 7
Event Object
Brothers at enmity with each other …
• Events and objects as first class entities
• Events and objects require each other!
• For example, in DOLCE
‘is participant in’
‘has participant’
• … not necessarily!
![Page 9: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/9.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 9
Events need to be modeled …• … and are useful in a variety of domains
• Lifelogs
• Multimedia-based experience sharing
• Emergency response
• Cultural heritage
• News
• Sports
• Surveillance
• …
![Page 10: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/10.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 10
Emergency
Control Center
Forward
Liaison
Officer
Documentary
support
Calls to report about
a power outage
Creates incident
with audio recording
Request to
report about a
flooded cellar
Reports
by taking
photos
etc.
Emergency Response
Coordination
Emergency
Hotline
Fire Department
Police Department
Coordinate and
keep up to
dateReport
and update
about the incident
Coordinate
and keep up
to date
Report and update
about the incident
Citizen
• Several emergency response entities are involved
• Using different event-based systems
• Common understanding of multimedia information
needed to efficiently communicate between ERs
Snapped pole image from:
http://www.dailymail.co.uk/
Emergency Response Scenario
![Page 11: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/11.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 11
Requirements to a Common Event
Model*• Participative aspect
• Temporal aspect
• Spatial aspect
• Structural aspect
• Mereology (composition)
• Causality
• Correlation
• Interpretation
• Experiential aspect (documentation)
* Analysis of 21 models and systems [SM13, SSF+12, SSF+09]
![Page 12: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/12.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 12
Survey on Event Models & Systems
…
Part
icip
a-
tion
![Page 14: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/14.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 14
Ontology Patterns of Event-Model-F• Event-Model-F* defines six ontology patterns
• Ontology design pattern similar to SE
• Build on top of DOLCE+DnS Ultralight ontology
• Cf. theory on events and objects
• Provides Description and Situation pattern
• Specified in Web Ontology Language (OWL)
• Formalized in Description Logics
* Homage to event model E by Westermann and Jain
![Page 15: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/15.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 15
6 Patterns of the Event-Model-F
• (1) Participation pattern
• (2) Mereology pattern (composition)
• (3) Causality pattern
• (4) Correlation pattern
• (5) Documentation pattern
• (6) Interpretation pattern
• All based on Descriptions and Situations (DnS)
• Contextualization of events and objects w.r.t.
specific situations
![Page 16: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/16.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 16
Example: Descriptions ‘n’ Situations• DomesticPowerOutage-
Description defines roles
• AffectedObjectRole
• AffectedPersonRole
• AffectedPersonRole
• …
• DomesticPowerOutage-
Situation defines objects
house-1 : Building
paul-1 : NaturalPerson
sandy-1 : NaturalPerson
…
• In a different situation, NaturalPerson
paul-1 may play a FireFighterRole
• People may have differing opinions
about the cause of the power outage
• …
Image source: Wikipedia
Classify
![Page 17: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/17.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 17
(1) Participation Pattern• Participation of living and non-living objects in events
• Reuse of domain knowledge
Roles the
entities play
Real world
entities
EventParticipationDescription
defines exactly 1 DescribedEvent
defines min 1 Participant
defines some LocationParameter
defines some TimeParameter
defines only (DescribedEvent or Participant or
LocationParameter or TimeParameter)
isSatisfiedBy exactly 1 EventParticipationSituation
Example: Firemen and home owner are involved in an
incident of a house fire.
![Page 18: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/18.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 18
(3) Causality Pattern
• Event (cause) implies other event (effect)
• Causal relationship holds under some justification
• Causes and effects are events, and only events
Description
EventCausalityDescription
EventCausalitySituation
Situation
Cause Effect Justification
EventRole
Concept
Event Description
classifiesisRoleOf
defines
isEventIncludedIn
satisfies
isObjectIncludedIn
Role
EventCausalityDescription
defines exactly 1 Cause
defines exactly 1 Effect
defines exactly 1 Justification
defines only (Cause or Effect or Justification)
isSatisfiedBy exactly 1 EventCausalitySituation
Example: The event of a snapped power pole causes a
power outage.
![Page 19: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/19.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 19
Use Case: Emergency Response
Domain ontology by
City of Sheffield, UK
Events and
media metadata
Event details
[MTAP2012]
Web 2.0 content
and metadata
![Page 21: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/21.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 21
• Which bakery close by is open on Sunday?
• What could I do tonight?
• Which sights are in the area?
Are they still open?
• Existing applications
– Mostly focus on location,
i.e., points-of-interests
– No support for integrated
search for events Sun: 7am-11am
Problem: Events in Social Media
![Page 22: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/22.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 22
Different Sources of Social Events …• Opening hours of a shop like a bakery
• Concert of favorite rock star in my town
• Café serving English tea in my neighborhood
• Happy hour at a bar
• Special sales at the shopping mall
• …
…
![Page 23: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/23.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 23
Event Model for Social Media!
• Derive a specific model of social media events
• Use notions of events and objects defined in
Event-Model-F and specialize it for social media
• Focus on what is needed in the domain
– (1) Participation pattern
– (5) Documentation pattern
• Future work: (2) Mereology Pattern
![Page 24: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/24.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 24
• Exploration of POIs and
events in real-time
• Multithreaded integration
of social data sources:
KlickTel, DBpedia,
GeoNames, Eventful,
etc.
Pad
Phone
mobEx: Mobile Social Media Explorer
![Page 26: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/26.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 26
Update Event 003Add Event 003
Incremental Data Integration on Server
mobEx
Server
Client
Café ViennaPhone.: ???
Opening hours:
Mo – Fr. 10 – 2 Uhr
Provider 1 Provider 2
Café ViennaPhone.: 01234
Opening hours: ???
Café
Vienna
Mo – Fr.
10 – 2
o‘clock
Café Vienna
Tel.: 01234
Mo – Fr. 10 – 2
o‘clock
Entity
Resolution
Time
![Page 27: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/27.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 27
Provider …
Entity Resolution in Detail
Provider 1
Provider 2
….
Weighted comparison of attributes:
Location information
Time information
Similarity score computed from
single weights
Above threshold integration
Café ViennaTel.: 01234
Opening hours:
Mo – Fr. 10 – 2 Uhr
S1, 15
68161 Mannheim
Calculated GPS Data
www.cafevienna.de/
Description: XYZ
High Weight
Low Weight
Medium Weight
High Weight
Low Weight
High Weight
Heuristic filter (blocking):
GPS coordinates > 500m
distance
Except: same address
Different entity types are not
compared (Persons, Places,
Events)
High Weight
![Page 28: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/28.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 28
Entity Resolution vs. Data Delivery
• Percentage of resources the client receives
• Percentage of resolved resolution at that time
![Page 29: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/29.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 29
Faceted Exploration of Events & Objects
![Page 31: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/31.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 31
• 18 subjects, average age of 25.5 (SD=3.03)
and six female
• Tracked detailed activities over three weeks
Evaluation in the Field.
• More than 4000 single events in 234 sessions
• Session is defined as
• Active usage of application until it is closed
• Or: after 30 seconds of inactivity
![Page 32: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/32.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 32
Evaluation in the Field (2)• Session lasted on average 2:42 minutes.
• 57% of the participants used the application
daily, 72% every second day or more.
• Users spent more time on the map screen than
on the screen showing the facets (ca. x1.5)
• Time slider not used
more often in the course
of the three weeks
![Page 33: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/33.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 33
Conclusion
• Events are an important concept in multimedia
• Tremendous research conducted in the past
• But, we have a lack of
– Common theory of events in multimedia
– Integrated tool chain to deal with events
– From detection over representation to use
Acknowledgements:
• R. Jain, C. Saathoff, T. Franz, S. Staab, D. Schmeiß
• B. Opitz, T. Sztyler, B. Pfister, M. Jess, C. Bikar, F. Knip
![Page 34: Events in Multimedia - Theory, Model, Application](https://reader034.vdocuments.mx/reader034/viewer/2022042715/559cf48e1a28ab75438b4636/html5/thumbnails/34.jpg)
Events in Multimedia Ansgar Scherp – [email protected] Slide 34
But there is a long interest …
• … in this topic in the multimedia community
• 1st ACM Int. Workshop on Events in Multimedia,
ACM Multimedia 2009, Beijing, China
• 2nd ACM Int. Workshop on Events in Multimedia,
ACM Multimedia 2010, Firenze, Italy
• 3rd ACM Int. Workshop on Events in Multimedia,
ACM Multimedia 2011, Scottsdale, Arizona, USA
• "Multimedia Activity and Event Understanding" Area,
ACM Multimedia 2012, Nara, Japan
• Workshop on Event-based Media Integration and
Processing, ACM Multimedia 2013, Barcelona