forever young: a tribute to the grandmaster through a recount of personal journey

Forever YoungATributetotheGrandmaster&ARecountofPersonalJourney

Jiebo LuoUniversity of Rochester

2

Why Do We Need to Do Research?

3

Why Do We Need to Do Research?

4

Why Do We Need to Do Research?

5

Learned from the GrandmasterCuriosity

Open mindedNew problemsNew techniques

Passion Love, focus - sustained

ScholarshipUnaffected by noise, hypeUncompromised integrity

Never too old to learn

6

PhD ThesisImage Processing

Image and Video Codingwavelet transform

Scene-adaptive Coding primitive/budding scene understanding

7

Kodak Years • Intelligent Image Processing and Semantic Image Understanding

– Automatic Redeye Removal

– Automatic Image Orientation Detection

– Image Classification

– Image Annotation

– Geotagging

– Context + Content

From Classification to Description Recognizing Realistic Actions from Videos "in the Wild"UCF-11 to UCF-101(CVPR 2009)

Similarity btw Videos Cross-Domain Learning

Visual Event Recognition in Videos by Learning from Web Data (CVPR2010 Best Student Paper)

Heterogeneous Feature Machine For Visual Recognition(ICCV 2009)

Image Captioning with Semantic Attention

• Motivations– Real-world Usability

• Help visually impaired people, learning-impaired– Improving Image Understanding

• Classification, Objection detection– Image Retrieval

1. a young girl inhales with the intent of blowing out a candle2. girl blowing out the candle on an ice cream

1. A shot from behind home plate of children playing baseball

2. A group of children playing baseball in the rain

3. Group of baseball players playing on a wet field

Key Elements

• Additional textual information– Leverage noisy titles, tags or captions (Web)

Key Elements

• Additional textual information– Leverage noisy titles, tags or captions (Web)– Leverage visually similar nearest neighbor images

Key Elements

• Additional textual information– Leverage noisy titles, tags or captions (Web)– Leverage visually similar nearest neighbor images– Incorporate success of low-level tasks

• Visual attribute detection

Attention Model on Attributes

• Instead of using the same set of attributes at every step

• At each step, select the attributes (attention)

m mtmt kKwatt ),(

)softmax VK(wTtt

))],,(;([),( 11 tttttt hKwattxfhxfh

Overall Framework

• Training with a bilinear/bilateral attention model

ht

pt

xt

v

{Ai}

Yt~

RNN

Image

CNN

AttrDet 1

AttrDet 2

AttrDet 3

AttrDet N

t = 0

Word

Performance

• MS-COCO Image Captioning Challenge

TGIF: A New Video Dataset and Benchmark

Examples

a skate boarder is doing trick on his skate board.

a gloved hand opens to reveal a golden ring.

a sport car is swinging on the race playground

the vehicle is moving fast into the tunnel

Machine Generated Sentence Examples

19

Two Most Important Social Signals (IMO)• User

• Sentiment

• Cross‐modality Consistent Regression

Joint Visual-Textual Sentiment Analysis

21

Infer User Demographics & Interests

• Social interactions and social activities• Public health surveillance• Web sentiment analysis and trend prediction• Cyber terrorism, extremism, and activism• Fads and infectious ideas• Marketing intelligence analytics • Traffic and human mobility patterns• Human and environment• Social unrest, protest and riot

Understanding the Pulse of Society

23

•Correctly classified flattering expressions

•Correctly classified unflattering expressions

•Correctly classified neutral expressions

• Acts like a prism to reveal the spectrum of opinions

• Competitive Vector Autoregressive Model

2012: Calling the Swing States

2016: Fine-Grained Campaign Analysis

2016: Shifting Tide? Too Close to Call

25

August 26, 2016

September 15, 2016

Social Multimedia

Visual Data

Textual Data

Computer Vision

NLP

User Demographics

User Activities

Behavior Patterns

Using Social Multimedia to Study Social Problems

Time Pattens of Underage Alcohol Use

Temporal Patterns of Underage Drinking

Drug Use Pattern from Instagram

29

When We Were Young

29

30

Forever Young

30

Forever young,I want to be forever young.Do you really want to live forever?Forever, and ever

Forever young,I want to be forever young.Do you really want to research forever?Forever, and ever

Image Processing, Computer Vision, Multimedia, Social Media, Big Data, …

(A younger version of an old song)

……

Let’s Celebrate the Forever Young Huang Academic Tree!