joost de wit (media distillery) - technology update - watskeburt?!
TRANSCRIPT
![Page 1: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/1.jpg)
Watskeburt?! Onder de motorkap
Technology Update 6 oktober, 2016
![Page 2: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/2.jpg)
Founded by Geert Vos and Joost de Wit in 2014
![Page 3: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/3.jpg)
We still can’t search in video content!
![Page 4: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/4.jpg)
Adding metadata used to be manual labour
![Page 5: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/5.jpg)
We use advanced data mining to analyse video content
Photo courtesy of Google
![Page 6: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/6.jpg)
Both broadcast & on-line sources
Teletext Speech Subtitles Logos Faces Filmstrips
![Page 7: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/7.jpg)
Ingest
Analysis
Storage
Presentation / interaction
API
![Page 8: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/8.jpg)
cask still well
foundation
![Page 9: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/9.jpg)
Large Scale Video Storage
Online Video Service
StorageService
RSS feed Video scraper
RabbitMQ
SpeechRecognizer
SubtitleExtractor Filmstrip Creator …
individual frames chunked videoannotations
well still cask foundation
![Page 10: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/10.jpg)
SpeechRecognizer
SubtitleExtractor Filmstrip Creator …
RabbitMQ Indexer Elasticsearch
ContentService
Notification Service
SearchService
AuthenticationService
FeedService
well still cask foundation
annotations Large Scale Video Storage
![Page 11: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/11.jpg)
Large Vocabulary Automatic Speech Recognition (ASR)
Photo courtesy of IBM
![Page 12: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/12.jpg)
r eh k ao g n ay z s p iy ch
"recognize speech"
Speech / non-speech
Speaker diarisation
Phonemes (acoustic model)
Vocabulary (language model)
![Page 13: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/13.jpg)
Dialect de NS > Dennis
![Page 14: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/14.jpg)
SjoemelsoftwarePhone calls
![Page 15: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/15.jpg)
Subtitle extraction
![Page 16: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/16.jpg)
ROI selection
Text extraction
OCR
Utterance detection
Garbage detection
![Page 17: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/17.jpg)
![Page 18: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/18.jpg)
Logo recognition
![Page 19: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/19.jpg)
![Page 20: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/20.jpg)
Voor het kijken Tijdens het kijken
While browsingWhile watching
![Page 21: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/21.jpg)
Research questions
• How to “visually summarise” a video clip as a filmstrip? • #frames to show (fixed / variable)? • Which frames to show? • Size of the frames to show (fixed / variable)? • Part of the frame to show? • How to present the filmstrip? • How should users interact with it?
![Page 22: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/22.jpg)
Frame sampling
Shot detection
Frame selection
Merge
[Bar clipping]
Videoclip
Filmstrip & manifest
![Page 23: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/23.jpg)
Features for frame/bar selection
• Gezichten (en de beweging ervan over het scherm)
• Gezichtsuitdrukking • Open ogen • Tekst • Infographics • Scene-overgangen • Scherpte in het beeld • Rule-of-thirds • Aanwezigheid van muziek / spraak • Ondertiteling
• Visuele eigenschappen frame (sharpness, saturation, kleurhistogram)
• Programma specifiek (studio, naambordjes, …)
![Page 24: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/24.jpg)
Filmstrip as (eventually) tested
• Variable number of frames (one per shot) • Selected frame just after shot changed • Fixed width, no clipping • Focused on the presentation & interaction
![Page 25: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/25.jpg)
Examples
• Short clip • One long shot taken from a helicopter • No voice-over or text present
![Page 26: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/26.jpg)
Examples
• The subtitling tells the story • Selected images don’t contain much information
![Page 27: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/27.jpg)
Examples
• Great example • Reads like a comic book • Lucky shot
![Page 28: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/28.jpg)
Alternative filmstrip
• Variable number of frames • “bar” width based on length of shot • “bar” cut with respect to frame’s center
![Page 29: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/29.jpg)
Storing the filmstrip
• We store parts of the filmstrip up to a maximum length separately • Parts are stored in Cassandra • Efficient retrieval based on program ID and start time of the strip • Scalability and redundancy are build-in in Cassandra
Part 1
Part 2
![Page 30: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/30.jpg)
The API
• REST API (created using Jersey, running in Apache Tomcat) • JSON & JPG as output
![Page 31: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/31.jpg)
{ "height" : 480, // The height of the filmstrip "strips" : [ { "program_id" : "b214753f-5e17-46ec-afeb-c1b3d9ee6565", // The Id of the program from which // the filmstrip was created "start" : 0, // Start time of the strip (in milliseconds) "stop" : 12000, // Stop time of the strip (in milliseconds) "url" : "<base>/strip_001.jpg", // The URL of this strip's image "width" : 6700, // The width of this strip "segments" : [ // Segments are the tiles in the strip { "start_offset" : 0, // The start offset of this segment (in pixels) "stop_offset" : 100, // The stop offset of this segment (in pixels) "start_timestamp" : 0, // The start time of this segment (in milliseconds) "stop_timestamp" : 1234 // The stop time of this segment (in milliseconds) }, { "start_offset" : 101, "stop_offset" : 305, "start_timestamp" : 1235, "stop_timestamp" : 2345 } ] } ], "previous" : false, // A reference to the previous filmstrip file (when present) "next" : "<base>/next_strip.json" // A reference to the next filmstrip file (when present) }
/api/{version}/filmstrip/{program_id}
![Page 32: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/32.jpg)
Flexibility of the API (for other applications)
Parameter Required Description Defaultprogram_id yes The id of the program for which the filmstrip should
be returned-
start no The start time (in milliseconds) in the video for which the filmstrip is constructed
0
max_duration no The maximum duration (from {start}) of the video covered by the filmstrip
-1 (complete clip)
max_width no The maximum width (in pixels) of one single filmstrip
15000
size no The size of the filmstrip, either small, medium or large
small
![Page 33: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/33.jpg)
/api/{version}/filmstrip/{program_id}/strip
![Page 34: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/34.jpg)
UI challenges
• Cross platform & responsive (is a hassle) • We had to ‘fold’ the frames to shorten the strip • Interaction between the strip and the player
• Move strip while watching? • Interact with strip while watching?
• Equally sized frames represent different shot durations
• Implementing ‘the wiggle’ was challenging, but required to stimulate swiping
• IOS only plays video in fullscreen mode
![Page 35: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/35.jpg)
The filmstrip UI
• Runs in the (mobile) browser • Responsive design (using Bootstrap) • Heavily using jQuery + libraries
• jQuery UI • jQuery Mobile (Events) • Kinetic • SmoothDivScroll • Handlebars
• Thin backend that’s basically a proxy to the Media Distillery platform
![Page 36: Joost de Wit (Media Distillery) - Technology Update - Watskeburt?!](https://reader038.vdocuments.mx/reader038/viewer/2022110219/5874489b1a28ab0e6c8b8fbf/html5/thumbnails/36.jpg)
Any questions?