multimedia lab @ ghent university - iminds - organizational overview & outline research...

ELIS –Multimedia Lab

Multimedia Lab @ Ghent University - iMinds:Organizational Overview & Outline Research Activities

Research SeminarKAIST, 1 August 2014

Wesley De Neve@wmdeneve

Ghent University – iMinds & KAIST

2


• Organizational overview (15 minutes)

- Ghent University

- iMinds

- Multimedia Lab

• Outline research activities (45 minutes)

- social media analysis

- visual content understanding

- deep machine learning

Outline

3


• Organizational overview (15 minutes)

- Ghent University

- iMinds

- Multimedia Lab

• Outline research activities (45 minutes)




Outline

4


• A Dutch-speaking public university

- located in Ghent, Belgium

- established in 1817

Ghent University (1/3)

Ghent

Brussels

http://upload.wikimedia.org/wikipedia/commons/4/45/Map_of_Belgium_provinces.png

http://upload.wikimedia.org/wikipedia/commons/4/45/Map_of_Belgium_provinces.png

5


• Consists of 38,000 students and 8,000 staff members

- about 4,000 foreign students and 800 foreign staff members

• Consists of eleven faculties, composed of more than 130 departments

- campus buildings distributed all over the city


Congress Center‘Het Pand’

Faculty of Engineeringand Architecture

Aula Academia

6


• Ghent University Global Campus in Songdo

- offers academic programs in molecular biotechnology, environmental technology, and food technology

- operates together with the State University of New York (SUNY), George Mason University, and University of Utah


Songdo Global University Campus Visit to Samsung Biologics

7


• Organizational overview

- Ghent University

- iMinds

- Multimedia Lab

• Outline research activities




Outline

8


iMinds

Research institute founded in 2004 by the Flemish government, with the aim of creating lasting

economic and social value through ICT innovation

9


iMinds: A Virtual Research Institute

Leverages the strengths of 5 universities,20 research groups, and more than 850 researchers

10


iMinds’ Research Departments

ICT Media Health EnergySmart Cities

Manu-facturing

Internet Technologies

Digital Society

Multimedia Technologies

Security

Medical Information Technologies

11


From Idea to Business: The iMinds Innovation Toolbox

5+ years …1 yearTime-to-market

Strategic researchIncubation &

entrepreneurshipApplied research

Pre-competitivetesting

Knowledge-driven

Explorative

Basics for applied research

Training & coaching

Financing

Facilities

Networking

Internationali-zation

Business-driven

InterdisciplinaryCooperativeDemand-driven

Proof of Concept

ICON projects

Large-scale user trials & living labs

Evaluate technical feasibility

Simulations

12


• iRead+ – The intelligent reading companion

- January 2012 to December 2013

- finished project that built a text analysis pipeline for enriching digital news articlesin Dutch and French with links to Wikipedia,dictionary definitions, and images

• GiPA – Generic platform for augmented reality

- January 2014 to December 2015

- aims at building an interoperable platformfor augmented reality applications, rangingfrom games to simulations, addressing diverserequirements, from capturing to rendering

iMinds ICON: Example Projects

13



- Ghent University

- iMinds

- Multimedia Lab





Outline

14


People (Speech Lab excluded)

• Staff

- Rik Van de Walle – senior full professor, head of MMLab

- Peter Lambert – associate professor

- Piet Verhoeve – guest lecturer (ICON program manager at iMinds)

- Erik Mannens, Jan De Cock & Wesley De Neve – research management

- Ellen Lammens & Laura Smekens – administrative management

• 35 researchers

- 50% PhD students

• Miscellaneous

- about 15 master’s thesis students per year

- a few Summer internships each year

15


Research Activities (1/2)

• Cluster 1: Video Coding (Jan De Cock)

- compression and transport of video

- transcoding and scalable coding

- high-dynamic range video

• Cluster 2: Game Tech & Graphics (Peter Lambert)

- augmented and virtual reality

- texture and mesh compression

- path planning

16


Research Activities (2/2)

• Cluster 3: Semantic Web (SWTF; Erik Mannens)

- multimedia and interactivity on the Web

- knowledge representation and reasoning

- (big) data analytics and visualization

- digital publishing

• Cluster 4: Social & Visual Intelligence (SaVI; Wesley De Neve)


- visual content analysis

- machine learning

17


Teaching Activities

• Bachelor/Master Computer Science and Bachelor/Master Electronics (Faculty of Engineering and Architecture)

- Multimedia Techniques

- Design of Multimedia Applications

- Advanced Multimedia Applications

• Bachelor Informatics(Faculty of Sciences)

- Multimedia

- Internet Technology

• Bachelor Biotechnology(Songdo Global Campus)

- Structured Programming

+ New graduate course onBig Data Analytics(pending approval)

18


• W3C (World Wide Web Consortium)

- new Web techniques

- e.g., HTML5 and Media Annotations

• MPEG (Moving Picture Experts Group)

- new compression techniques

• e.g., H.264/AVC and 3-D Video Coding

- new storage and transport techniques

• e.g., MP4 file format and MPEG DASH

• VQEG (Video Quality Experts Group)

- measurement of video quality

- e.g., subjective quality evaluations

Standardization Activities

19



- Ghent University

- iMinds

- Multimedia Lab





Outline

20


• An online social network service that enables users to send and read short 140-character text messages, called "tweets" or "microposts"

Twitter

Tweet ormicropostRetweet

(sharing)

Favorite(like or

bookmark)

Mention(starts with @)

Hashtag(starts with #)

21


Note the presence of both textual and (embedded) visual information!

Famous Tweets

22


• Usage in general

- 271 million monthly active users

- 500 million Tweets are sent per day

- 78% of active users are on mobile

- expected revenue for 2014 is $1.33 billion

• mobile advertising + data licensing

• Usage during the World Cup 2014

- fans sent 672 million related tweets in total

- during the semi-final between Brazil and Germany, fans sent more than 35.6 million tweets

- during the final, the number of tweets sent by fans peaked at 618,725 Tweets Per Minute (TPM)

Twitter Statistics

23


• Research goal

- to make sense of the vast amounts of textual and visual information communicated on Twitter by means of machine learning

• Challenges

- microposts are noisy in nature

- microposts are short-form in nature

- microposts are multi-lingual in nature

- microposts come in highly varying quantities

- microposts are real-time in nature

- microposts are multi-modal in nature (textual & visual, a/o)

Twitter Research Goal and Challenges

24


• What?

- simply speaking: use of multi-layered neural networks that are able to learn complicated mappings between inputs and outputs

Deep Learning (1/4)

x y = hθ(x)

learned intermediate features

deep learning = (hierarchical) representation learning

25


• Example learned features

Deep Learning (2/4)

Supervised handwrittendigit recognition

Unsupervised visual object recognition(Google Brain)

26


• Why the resurgence of neural networks?

- availability of large data sets (cf. social media & Internet of Things)

- availability of cheap computing power (cf. GPU & cloud)

- availability of algorithmic improvements (cf. DropOut & max pooling)

• Current achievements

- top performance in handwritten digit recognition

- top performance in automatic speech recognition

- top performance in large-scale visual concept detection

• Attracts substantial private R&D investments

- Google (Geoffrey Hinton & Ray Kurzweil), Facebook (Yann LeCun), Baidu (Andrew Ng & Kai Yu), Microsoft, Twitter, Netflix, and so on

Deep Learning (3/4)

27


• Plenty of open research challenges

- how to tailor deep neural networks to novel applications?

- how to scale up deep neural networks?

- how to scale down neural networks at no cost in effectiveness?

- how to take advantage of massively parallel hardware?

- how to develop effective hybrid architectures?

- how to take into account long-term temporal dependencies?

- how to implement multi-modal approaches?

- how to establish solid theoretical foundations?

- how to bridge the gap between deep learning and strong A.I.?

Deep Learning (4/4)

28


• Hashtag recommendation

• Named entity recognition and disambiguation

• Sports analytics

• Social television

• Vine video classification

Ongoing Research Topics with a Twitter Focus

29


Social and Visual Intelligence (SaVI)

Abhineshwar Tomar [email protected]

Fréderic [email protected]

Baptist [email protected]

Wesley De [email protected]

Azarakhsh [email protected]

+ 3 master’s thesis students

mailto:[email protected]





30







Research Topics with a Twitter Focus

31


Hashtags on Twitter

Hashtag usage:

- topic-based indexing & search

• #socialnetwork

• #Reddit

- conversational/event clustering

• #www2014

Observation: only about 10% of tweets contain a hashtag

Research challenge: develop techniques for Twitter hashtag recommendation

32


• Training: learning the relation between tweets and hashtags

Twitter Hashtag RecommendationUsing Deep Learning (1/2)

300-D tweet vector

word2vec

300-D hashtag vector

word2vec

Deep feed-forward neural

network

300-D input layer1000-D hidden layer500-D hidden layer400-D hidden layer300-D output layer

Tweet HashtagElizabeth Warren Taking on Hillary as New Democratic Powerhouse

#politics

33


• Testing: recommending hashtags to tweets

Twitter Hashtag RecommendationUsing Deep Learning (2/2)

300-D tweet vector

word2vec

300-D hashtag vector

Deep feed-forward neural

network

300-D input layer1000-D hidden layer500-D hidden layer400-D hidden layer300-D output layer

TweetHouse Democrats suggestObama impeachment isimminent to raise cash

vec2word

HashtagHashtag

HashtagHashtags

#politics

#crisis

34


• Developed by Google Research

• Computes vector representations for words

- through the use of neural network technology

• trained on part of the Google News dataset (+/- 100 billion words)

• the model contains vectors for 3 million words and phrases

- capture the semantic meaning of a word

• Example word vector properties

- vector('Paris') - vector('France') + vector('Italy') ≈ vector('Rome')

- vector('king') - vector('man') + vector('woman') ≈ vector('queen')

word2vec

35


Tweet Recommended hashtags

1 Someone dm/text me bc I’m so bored madd, Oh noes, rainnwilson, sooooooo, fricken

2 The good life is one inspired by love and guided by knowledge.

Ahh yes, FIVE THINGS About, YANKEES TALK, Kinder gentler,Ya gotta love

3 Method of Losing Weight http://t.co/rs64CEuo5W Shape Shifting, Treat Acne, Detect Cancer, Warps, Calorie Burn

4 I hate today cause its room cleaning day for me!!! FAN ’S ATTIC, Puh leez, Mopping robot, % #F######## 3v.jsn, InterestEURO JAP

5 SPELLS AND SPELL-CASTING:ENCYCLOPEDIA OF 5000 SPELLS ( JUDIKA ILLES ):BLACKSMITH’S WATER HEALING SPELL: A... http://t.co/k0TfrqJFQW

DEBUTS NEW, NOW AVAILABLE FOR, TO PUBLISH, DESIGNED TO,IS READY TO

Experimental Results

36








37


• Named entity

- person

- location

- organization

- miscellaneous

• film/movie, entertainment award event, political event, programming language, sporting event and TV show

• Recognition

- identification of a named entity in a given text

• Disambiguation

- e.g., fruit ‘apple’ versus company ‘Apple’

Named Entity Recognition and Disambiguation

38


• Tools for named entity recognition and disambiguation have thus far been developed for long-form news articles using formal language

• Need for development of tools for named entity recognition and disambiguation for short-form microposts using informal language

Research Challenge

39


Natural Language Processing (NLP) for Twitter from Scratch

Tweet TokenizationPart-of-Speech Tagging (PoS)

Chunking

Named Entity Recognition and Disambiguation

Information Retrieval

Text-to-Speech

Artificial Intelligence(cf. Siri, Cortana, Google Now)

General Text Parsing

pronoun verb noun

Tom likes Sprite.

40


• Use of a feed-forward neural network for learning the mapping between a collection of word vector representations and a PoS tag

- feature learning and not feature engineering

• Use of word vector representations derived from Twitter

- not from Google News

Our Approach: Twitter PoS using Deep Learning

Word 1

Word 2

Word 3

Look

up

wordvector

wordvector

wordvector

Neural network

PoS tag of word 2

41


Twitter-based word2vec Examples (1/2)

Input: reddish

Word Cosine distance

-----------------------------------------------------------------redish 0.829081brownish 0.814688purple 0.812775burgundy 0.804166blueish 0.786641pastel 0.783559magenta 0.779790ombre 0.778065lilac 0.777773pink 0.775110

Captures spelling mistakes

42


Twitter-based word2vec Examples (2/2)

Input: :)

Word Cosinedistance

-----------------------------------------------:)) 0.918219

(: 0.870493:-) 0.855738=) 0.855088:))) 0.853806xo 0.852893xx 0.846706;)) 0.829732!:) 0.822094xox 0.819353

Input: :(

Word Cosinedistance

-----------------------------------------------:'( 0.865362;( 0.858428:(( 0.829048:-( 0.825194:(((( 0.812367!:( 0.807746)): 0.791888/: 0.769977:((( 0.758594:((((( 0.739779

43


Experimental Results

Dataset Vector size Accuracy

2 weeks (~5M tweets) 100 82%



6 months (~70M tweets) 300 81,5%

CMU ARK Tagger 91,6%

44



• Named entity recognition



• Video classification


45


• What?

- prediction of the outcome of football matchesin the English Premier League (EPL), using bothtraditional statistics and Twitter microposts

• Why?

- betting on football is a billion dollar industry

- Twitter is highly popular for real-time coverage of sports events

• How?

- fusion of the output of four simple methods, using different features and machine learning techniques

Rationale

46


• Method 1: Statistical features

- ranking in the league, the number of points gathered in the league, the number of points gathered during the last five games, the number of goals made, and the number of goals against

• Method 2: Twitter volume changes

• Method 3: Twitter sentiment analysis

• Method 4: Twitter user predictions

• Machine learning

- Naive Bayes, Logistic Regression, and SVM

Approach

social features derived from+50 million tweets

47


Experimental Results (1/2)

Method Accuracy

Baseline methods

Naive predictions 51%

Expert predictions 60%

Bookmaker predictions 67%

Individual methods

Statistical features 64%

Twitter volume changes 50%

Twitter sentiment analysis 52%

Twitter user predictions 63%

Combination of statistical features andTwitter user predictions

Majority voting 64%

Early fusion 68%

Late fusion 66%

48


Experimental Results (2/2)

Method Monetary profit (when betting 100 EUR)

Bookmaker predictions +18.55 EUR

Proposed method +29.70 EUR

49






• Video classification


50


• Social television (second screen)

- interaction between televised content and online social networks

• Breaking Bad finale: peak of 22,373 TPM

• Super Bowl 2014: peak of 382,000 TPM

• World Cup 2014 final: peak of 618,725 TPM

Rationale (1/2)

51


• Challenges

- how to measure engagement and reach on online social networks?

• cf. the Nielsen television ratings

- how to profile your audience?

• e.g., age, gender and location

• Addressing these challenges is important for the allocation of advertisement budgets and targeted advertisement strategies

Rationale (2/2)

versus

52


• Three major difficulties

- privacy concerns

- low usage of Twitter (at that time)

- identification of Flemish users of Twitter

Measurement of Engagement and Reach in Flanders

53


• What?

- classification of Flemish Twitter users into male and female classes

• Why?

- current user profiles do not contain gender information

- gender information is important for targeted advertising

• How?

- through (mostly n-gram) features extracted from the profile of the user, the tweets of the user, and the social network of the user

- through machine learning based on Naive Bayes and SVM

Twitter User Profiling: Gender Detection (1/3)

54



Male

Female

Ensemble

averaging ofprobabilities

Username Classifier

Name Classifier

Description Classifier

Tweet Content Classifier

Tweet Style Classifier

Friend Description Classifier

@wmdeneve

Wesley De Neve

Senior Researcher at Ghent University - iMinds & KAIST. Interested in social media analysis, visual content understanding and machine learning.

Attending "The Future of Metadata" at CONTEC. #TISP

Sports fan, basketball player, outdoor lover and a Ph.D. researcher #SocialTV and Natural

Language Processing (#NLP) @iMinds - @UGent

URL usage, emoticon usage, and punctuation

55



Classifier Accuracy

Username 78.86%

Name 87.54%

Description 65.74%

Tweet content 75.36%

Tweet style 66.34%

Friend description 75.34%

Test set TweetGenie Ensemble

Test set 2 82.15% 91.89%

Test set 3 86.44% 93.32%

56








57


• Platform for social & mobile video

- established in June 2012

• Allows creating & distributing videos of up to 6 seconds

- maximum video length resembles Twitter’s character limitation

• Acquired by Twitter in October 2012

- currently has more than 40 million users

• Has the potential to become a new social news platform

- cf. Ninja News in Belgium

What is Vine? (1/4)

58


What is Vine? (2/4)

59


What is Vine? (3/4)

60


What is Vine? (4/4)

61


Recognition of general concepts in video fragments

Categorize short and noisy video fragments

Localize and recognize named entities in video fragments

Localize and recognize products in video fragments

Automatic Understanding of Social Video Content (1/2)

+Neural

networkOutput

62


Representation learning for social video

Learn general noise-robust features

Exploitation of temporal information in video to improve classification

Investigate recurrent neural networks and reservoir computing networks

Automatic Understanding of Social Video Content (2/2)

63


Future Research Vision SaVI & SWTF

Machine-understandableinformation

Data(online social networks &

Internet of Things)

Human &machine action

Deeplearning

SemanticWeb

Visualization

Technology stacksApplication domains

Naturallanguage

understanding

Visualcontent

understanding

Cognitive computing? Strong A.I.? Technological singularity ;-)?

65


[1] F. Godin, B. Vandersmissen, A. Jalalvand, W. De Neve, and R. Van de Walle, “Alleviating manual feature engineering for Part-of-Speech tagging of Twitter microposts using distributed word representations,” Proceedings of the NIPS Workshop on Modern Machine Learning Methods and Natural Language Processing, Dec. 2014.

[2] A. Tomar, F. Godin, B. Vandersmissen, W. De Neve, and R. Van de Walle, “Towards Twitter hashtag recommendation using distributed word representations and a deep feed forward neural network,” Proceedings of the IEEE International Workshop on Cyber-Physical Systems and Social Computing (CSSC-2014) , Sep. 2014.

[3] F. Godin, J. Zuallaert, B. Vandersmissen, W. De Neve, and R. Van de Walle, "Beating the bookmakers: leveraging statistics and Twitter microposts for predicting soccer results,“ Proceedings of the 2014 KDD Workshop on Large-Scale Sports Analytics, Aug. 2014.

[4] B. Vandersmissen, F. Godin, A. Tomar, W. De Neve, and R. Van de Walle, "The rise of mobile and social short-form video: an in-depth measurement study of Vine," Proceedings of SoMuS2014 : Workshop on Social Multimedia and Storytelling (co-located with ICMR 2014), Apr. 2014.

References