Categorizing Multimedia Documents Using Associated Text

Thesis Proposal

By Carl Sable

Indoor vs. Outdoor

Denver Summit of Eight leaders begin their first official meeting in the Denver Public Library, June 21. They are clockwise from the top: Russian President Boris Yeltsin, U.S. President Bill Clinton, French President Jacques Chirac, Canadian Prime Minister Jean Chretien, Italian Prime Minister Romano Prodi, EU President Willem Kok, EC President Jacques Santer, British Prime Minister Tony Blair, Japanese Prime Minister Ryutaro Hashimoto and German Chancellor Helmut Kohl.

Villagers look at the broken tail-end of the Fokker 28 Biman Bangladesh Airlines jet December 23, a day after it crash-landed near the town of Sylhet, in northeastern Bangladesh. All 89 passengers and crew survived the accident, mostly with minor injuries. Most of the passengers were expatriate Bangladeshis returning home from London.

Event Categories

Politics Struggle

Disaster CrimeOther

Disaster Images

Workers Responding Affected People

Wreckage Other

Overview of Talk

I. Contributions.

II. Corpora.

III. Two previous systems.

IV. Harder categories.

V. Interactions between categories.

VI. Video.

VII. Schedule.

VIII. Conclusions.

Contributions

• Use of text categorization to categorize multimedia documents.

• Introduction of novel techniques.

• Use of NLP to handle tough categories.

• Exploration of interactions between various sets of categories.

Manual Categorization Tool

Reuters

• Common corpus for comparing methods.

• Over 10,000 articles, 90 topic categories.

• Binary categorization.

5

grain, wheat, corn, barley, oat, sorghum

9earn

448gold, acq, platinum

http://www.research.att.com/~lewis/reuters21578.html

Lots of Previous Literature

• “Bag of words” with weights.– Term frequency (TF).– Inverse document frequency (IDF).

• Variety of methods: Rocchio, k-nearest neighbors (KNN), naïve Bayes (NB), support vector machines (SVMs).

• Systems trained with labeled examples.

Density Estimation

• Start with advanced Rocchio system.

• For each test document, compute similarity to every category.

• Find all documents from training set with similar category similarities.

• Use categories of close training documents to predict categories of test documents.

Example

85, 35, 25, 95, 20

100, 75, 20, 30, 5

60, 95, 20, 30, 5

90, 25, 50, 110, 25

40, 30, 80, 25, 40

80, 45, 20, 75, 10

Category score vectorsfor training documents:Category score vector

for test document:

20.092.5

106.4

27.4

91.4

36.7

Predictions:Rocchio: StruggleDE: Crime (Probability .679)

100, 40, 30, 90, 10

Struggle

Politics

Disaster

Crim

e

Other

Distances:

679.0

7.361

4.271

0.201

7.361

0.201

(Crime)

(Struggle)

(Disaster)

(Struggle)

(Politics)

(Crime)

Actual Categories:

Bin System (AT&T)

• Group words with similar “features” together into a common “bin”.

• Based on training data, empirically estimate a term weight for words in each bin.– Smoothing, works well even if there is not

enough data for individual words.– Doesn’t assume simple relationships between

features.

Sample Words

Indoor Indicators“conference”“bed”

Outdoor Indicators“airplane”“earthquake”

Ambiguous“Gore”“ceremony”

Determine Bins for “airplane”

• Per category bins based on IDF and category counts.

• IDF(“airplane”) = 5.4.

• Examine first half of training data:– Appears in 0 indoor documents.– Appears in 2 outdoor documents.

Lambdas for “airplane”

• Determined at the bin level.

• Examine second half of training data:410*11.2)indoor|nobservatio(P 310*90.2)outdoor|nobservatio(P

)indoor|nobservatio(Plog2indoor )outdoor|nobservatio(Plog2outdoor

78.3outdoorindoor

Sample Words with Scores

Indoor Indicators“conference”

+5.91“bed”

+4.58

Outdoor Indicators“airplane”

-3.78“earthquake”

-4.86

Ambiguous“Gore”

+0.74“ceremony”

-0.32

Results

• Both systems did OK on Reuters.

• DE performed best for Indoor vs. Outdoor.

• Bin system performed best for Events.

Standard Evaluation Metrics (1)

• Per category measures:– Simple accuracy or error measures are

misleading for binary categorization.– Precision and recall.– F-measure, average precision, and

break-even point (BEP) combine precision and recall.

• Macro-averaging vs. micro-averaging.– Macro treats all categories equal,

micro treats all documents equal.– Macro usually lower since small

categories are hard.

Yes iscorrect

No iscorrect

AssignedYES

a b

AssignedNO

c d

p = a / (a + b)

r = a / (a + c)

contingency table:

rp

r*p*2F1

Results for Reuters

Micro-F1

0.7000

0.7500

0.8000

0.8500

0.9000

SVM KNN LSF NNet NB

Rocchio Columbia Combo Bin

Macro-F1

0.35000.40000.45000.50000.5500

SVM KNN LSF NNet NB

Rocchio Columbia Combo Bin

Standard Evaluation Metrics (2)

• Mutually exclusive categories:– Each test document has only one correct label.– Each test document assigned only one label.

• Performance measured by overall accuracy:

sprediction total#

spredictioncorrect #Accuracy

Results for Indoor vs. Outdoor

80.0%

81.0%

82.0%

83.0%

84.0%

85.0%

86.0%

87.0% Bin

Columbia

Rocchio

SVM

KNN

• Columbia system using density estimation shows best performance.

• Even beats SVMs.• System using bins

very respectable.

Results for Event Categories

82.0%

83.0%

84.0%

85.0%

86.0%

87.0%

88.0%

89.0%Bin

Columbia

Rocchio

KNN

• System using bins shows best performance.

• Columbia system respectable.

Improving Bin Method

• Experiment with more advanced binning rules.

• Fall back to single word term weights for frequent words.

Why are Disaster Images Hard?

• Small corpus (124 training images, 124 test images).

• Most words not important.

• Important words associated with test images have likely never occurred in training set.

Approach

• Extract important information (e.g. subjects and verbs).

• Compare test words to training words.• Previously seen words indicate strong evidence.• Consider large, unsupervised corpus of

subject/verb pairs.• Add evidence to categories for any verb or subject

ever paired with new subject or verb.

Subjects and VerbsSubject Verb Category

“two blocks” “have suffered” Wreckage

“aviation inspectors” “search” Workers Responding

“Moslems” “pray” Affected People

“rescue workers” “remove” Workers Responding

“this house” “is underwater” Wreckage

“passenger jet” “crashed” Other

“volunteers” “help clear” Workers Responding

“fuselage” “can be seen” Wreckage

“President Clinton” “hugs” Affected People

“officer” “tries on” Workers Responding

First Try

• Very simple subject/verb extraction.

• Rather small unsupervised corpus.

• Very simple similarity metric.

System Results

Rocchio 57.3%

Density Estimation 58.9%

First Attempt 59.7% (67.9% when prior evidence)

Results of Simple ExtractionSubject Verb Category

“blocks” “suffered” Wreckage

“aviation” “inspectors” Workers Responding

“moslems” “pray” Affected People

“rescue” “remove” Workers Responding

“house” “is” Wreckage

“passenger” “shown” Other

“volunteers” “including” Workers Responding

“fuselage” “be” Wreckage

“” “hugs” Affected People

“” “officer” Workers Responding

Doing It Right

• Better subject/verb extraction with parser.

• Much larger unsupervised corpus.

• Better similarity metric.

• Maybe stemming.

Previous Interactions

• Combining Pcut and DE for Reuters.

• Combining text system and image feature system for Indoor vs. Outdoor.

• Using number of people information to improve Indoor vs. Outdoor probabilities.

Those Results

Micro-average F1

Macro-average F1

Pcut 71.0 % 50.1%

DE 83.0% 40.5%

Combo 82.2% 51.2%Accuracy

Image Only 82.4%

Text Only 83.3%

Combo 86.2%

Standard Evaluation

Alternate Evaluation

Without # of people 79.9% 75.0%

With # of people 80.2% 77.2%

Pcut + DE for Reuters

Text + Image for In/Out

Adding # of People for In/Out

Future Interactions

• Improve accuracy.

• Determine new information.

• Likely use density estimation or bins.

For Example…

Indoor + Disaster Outdoor

Indoor + Politics Meeting / Press Conference

Closed Captioned Video

• Apply image system to video.

• New categories.

• New modality, new challenges.669 726 "HEADLINE NEWS" -- I'M KIMBERLEY KENNEDY, IN FOR DAVID GOODNOW.

750 878 THE FIRE HAS BEEN PUT OUT, BUT SO HAVE THE HOPES OF THOUSANDS OF PEOPLE COUNTING ON A VACATION TO MEXICO THIS WEEK.

930 1087 THE CARNIVAL CRUISE LINER "ECSTACY" WAS ONLY TWO MILES INTO ITS TRIP FROM MIAMI TO COZUMEL WHEN A FIRE BROKE OUT IN A LAUNDRY ROOM.

1134 1291 THE COAST GUARD CAME TO THE RESCUE AND DOUSED THE FLAMES AS HUNDREDS OF PASSENGERS DONNING LIFE JACKETS WATCHED ON.

1331 1388 THE FIRE CHARRED THREE LOWER DECKS BEFORE FIREFIGHTERS BROUGHT IT UNDER CONTROL.

...

Schedule

Conclusions

• Categorization of multimedia data.

• Applying novel techniques.

• Using NLP for hard categories.

• Exploring interactions between systems and categories.