building multilingual and crosslingual semantic resources with volunteer contributions over the web...
TRANSCRIPT
Building Multilingual and Crosslingual Semantic
Resources with Volunteer Contributions over the WebRada Mihalcea
University of North Texas
Facts
• Globalization– “Breaking down of political, cultural,
and trade barriers” (Thomas Friedman)
– Universal communication
• Dying languages – One language dying every other week
Some Figures (set 1)
• 7,000 languages spoken worldwide– + even more dialects– [http://ethnologue.com]
• Resources currently available for 15-20 languages (or less)
Some Figures (set 2)
Country Web users / TotalUnited States 150 mil. / 290 mil.China 100 mil. / 1.2 bil.Japan 50 mil. / 125 mil.India 40 mil. / 1 bil.Germany 35 mil. / 80 mil.United Kingdom 25 mil. / 60 mil.… …
• On average, an Internet user spends 11 h. 24’ / month
• United States users: 25 h. 25’ [home] + 74 h. 26’ [work] / month – [Lyman & Varian 2003]
10,773,000,000 hours spent online every month
[some 5 million man-years!]
• 945,000,000 total Web users
• [“The Main Thing”, June 2004]– http://www.rebron.org/
mozilla/archives/000085.html
• Internet population
Availability vs. Needs
• The Web as collective mind• A Different View of the Web:
WWW ≠ large set of pagesWWW = a way to ask millions of people
Users spending online 10,773,000,000 hours / mo. [~ 5,000,000 man-years]
Resources required for 7,000 languages
Outline
[building resources]• I: Building multilingual WordNets
• II: Building a (crosslingual) pictorial WordNet [using resources]• III: Applications
Outline
[building resources]• I: Building multilingual
WordNets• II: Building a (crosslingual)
pictorial WordNet [using resources]• III: Applications
Building WordNets
• Other WordNets:– Princeton WordNet– Euro WordNet– BalkaNet– …
• Methodology:– Manual– Lexicographers– Time-consuming and expensive
Romanian Semantic Dictionary
• Distributed / Web based• Non-expert users / expert
validators
WordNet RSDNET
Bilingual
Monolingual
Corpus
automatic
manual
non-expert expert
Resources Used
• WordNet semantic network
• English-Romanian dictionary
• Romanian dictionary
• Romanian corpus
Main Phases
words
meaningconfirm
choose
score!
• non-expert contributions– choose a WordNet
synset– pick the correct
translations (add other words to the synset)
– choose a sentence from the corpus that displays the appropriate meaning
– confirm the new synset– get points and rewards!
• expert validation– correct errors / remove
entries
Result
roSynset
synsetIDdefinitionexamplevalidated
roWords
IDwordsynsetIDengMatch
engSynset
synsetIDdefinitionexample
engWords
IDwordsynsetIDroMatch
Quantity
• Large number of contributions in short amount of time– 6 months: more than 2,000 synsets
from 150 contributors
QualityManual RoWN
RSDnet GenSynsets
• manual• depends on experts• takes longer to build
• semi-automatic• depends on Web users• errors corrected by experts
• automatic• depends on lexical resources• errors introduced
RSDnet GenSynsetsnoun adjectiv
enoun adjective
correct 96.6% 92.3%19% / 63%
21% / 71%
part.correct
3.3% 3.8% 2% / 2% 0% / 0%
erroneous
0% 0% 2% / 2% 4% / 4%
missing 0% 0%77% / 33%
75% / 25%
total 59 26 54 24
Pros / Cons• Pros
– faster than manual experts – more accurate than automatic– derived from WordNet => inherits
WordNet relations
• Limitations– bilingual users (English/Romanian)– capturing difficult concepts
Outline
[building resources]• I: Building multilingual WordNets
• II: Building a (crosslingual) pictorial WordNet [using resources]• III: Applications
A Picture is Worth 7,000 Words
An Image Dictionary
• Add image representations to concepts defined in WordNet– Encode word/image associations– Combine visual and linguistic
representations of world concepts
Typical entry in a dictionary
• pipe, tobacco pipe– a tube with a small
bowl at one end; used for smoking tobacco
• pipe, pipage, piping – a long tube made of
metal or plastic that is used to carry water or oil or gas etc.)
• pipe, tabor pipe – a tubular wind
instrument
+ pictorial representations
What for?• Language learning
– Children– Second (foreign) language– People with language disorders
• International language-independent knowledge base– Pictures are transparent to languages
• Applications– Pictorial translations (“Letters to my cousin”)
• Bridge the gap between research in image and text processing– Image retrieval/classification, natural language
Word/Image Associations
• Difficult• First iteration:
– Concrete nouns (flower, dog)– Concrete verbs (write, drink )
• Next:– Abstractions (friendship, love)– Object properties (red, large)
Building PicNet
• An illustrated semantic dictionary• Web-users perform the mapping• Resources
– WordNet• 150,000+ words, grouped in synsets• 250,000+ semantic relations
– Image Search Engines• PicSearch http://www.picsearch.com• AltaVista http://www.altavista.com/image• To date 72,000 images automatically
collected
Activities in PicNet
• Administrator functions • Word/image associations (Web-
users) – Free association– Competitive free association
(tournament)– Image validation / Scoring– Image donation– Word lookup (search)
Administrator functions
• Validate uploaded images– Determine whether to
allow the images into the system
– Does not verify the mapping
– Delete corporate, offensive, or unclear images
• Options– Ban User
• can delete all activity by a particular user from the database
• Word lookup• User
contribution– Contributing /
validating images
– Free association– Tournaments
(competitive free association)
Word Lookup (Search)
• Synsets with words matching the search phrase are displayed with their best image match.
• Finding the desired synset, a user may:– rate the validity of
the current synset – image mapping
– upload a new image to be attached to this synset.
Activity 1
Image Validation (Scoring)
• User is shown a synset-image pair – rank its appropriateness.
• Factors to consider:– fitness for the given synset.– quality of the image (size, clarity)
Activity 2
Scoring
• Score based on the user response– Not related ( -5 )– Loosely related ( 1 )– Some similarity ( 2 )– Well suited ( 3 )
• Result:– Determine a score for each synset-
image pair– Concept/image pairs that are not
related are quickly discovered• Typically after a response from one or two
users
Free Word Association
• Task: given an image, provide a word to match.
Activity 3
Free Word Association – problems
• Difficult to identify images with optimal specificity– E.g. violet vs. flower
• Sometimes tedious to find the intended word from the synset list
• However, the user can often determine a hypernym (more general concept) – useful information
• [Scoring] A free word association is considered to be “well suited” and scores 3
Image Upload
• Given a concept, upload a matching image– Search facilitated with shortcuts to three
search engines (PicSearch, AltaVista, Google)
• Scoring for uploaded images– An image uploaded for a particular synset
is considered “well suited” and scored at 5
– Account for the extra effort required from the user • Possible indicator of a stronger correlation.
Activity 4
User Motivation
• Points for each activity– Leaderboard
• Competitive activity – The PicNet Game
• Combine ideas into a competitive game
The PicNet Game
• Phase 1: Each player is shown an image and asked to provide a matching synset (as in free word association)
Activity 5
The PicNet Game
• Phase 2: Each player votes for the best match (cannot vote on her own entry).
Scoring and Winning• Each synset-image pair scores one point for
being entered, and one point for each vote received.
• If multiple players enter the same synset-image pair, the score is 2 * number of players entering that synset
• Players also receive a “game score”, which counts towards winning the game– A player receives 100 points for winning the round
• If multiple players entered the synset-image pair winning the best match, the score is split evenly
• A player reaching 300 points wins the tournament
Quality and Quantity
• [1 year] 6,200 concepts from 320 contributors
• Competitive free association– Number of users voting for the same
synset suggestion in each round– User concurrence: 43% (consistent
agreement)
• Random sampling 100 images– 85% correct associations
Sample Word/Image Associations
exodus, hegira, hejira – a journey by a large group to escape from a hostile environment
Sample Word/Image Associations
humerus – bone extending from the shoulder to the elbow
Sample Word/Image Associations
Castro, Fidel Castro – Cuban socialist leader who overthrew a dictator in 1959 and established a socialist state in Cuba (born in 1927)
Outline
[building resources]• I: Building multilingual WordNets
• II: Building a (crosslingual) pictorial WordNet [using resources]
• III: Applications
Translation with Pictures
• What do you understand by the following ?
The house has four bedrooms and one kitchen.
Understanding with Pictures: Pros
• Universal• Requires minimal learning• Intuitive• Cheap (free contribution by users
of PicNet)• Proven success (iconic languages
for augmentative communication)
Understanding with Pictures: Cons
• Complex information cannot be conveyed through pictures – e.g. “An inhaled form of insulin won
federal approval yesterday”• A large number of concepts with a
level of abstraction that prohibits a visual representation– e.g. politics, paradigm, regenerate
• Culture differences– e.g. some Latin American tribes do
not understand the concept of coffee
A First Cut
• Simple sentences– no complex states or evens (e.g. emotional
states, temporal markers, change) or their attributes (adjectives, adverbs)
– no linguistic structure (e.g. complex noun phrases, prepositional attachments, lexical order, certainty)
– basic concrete nouns and verbs translated “as is”
• Evaluate the amount of understanding achieved through pictures as opposed to words
Does It Work?
• Experiments carried out within a translation framework with simple sentences
• A communication process – a speaker of an “unknown” language– a listener of a “known” language– Chinese (unknown) to English (known)
• Three translation scenarios– fully pictorial representations (PicNet)– mixed pictorial/linguistic
representations– fully linguistic representations
Sample Pictorial and Linguistic Translations
this
this
Evaluation Study • Interpretations
– Users asked to provide an interpretation based on their first intuition
– Users’ background: Hispanics, Caucasians, Latin Americans
• Data set: 50 short sentences (10-15 words)– 30 sentences from language learning courses– 20 sentences from various domains (sports, politics,…)– Various levels of difficulty– 15 (average) interpretations for each sentence– One interpretation for each translation scenario– Total of 15*3*50=2,250 interpretations
Sample Interpretations
Evaluation Results
• Manual and automatic evaluations:– Adequacy– NIST [Bleu] – GTM
Evaluation Results
• Significant amount of information can be conveyed through pictures – 76%, compared to the baseline of 0%– Due to the intuitive visual
descriptions that can be assigned to some of the concepts in the text
– Due to humans’ ability to contextualize • Read a book is a more common
interpretation than read about a book• “He sees the riverbank illuminated by a
torch”
Evaluation Results• S1 (pictures) vs. S2 (pictures with words)
– 3.81 vs. 4.32– role played by context that cannot be
described with visual representations– adjectives, adverbs, prepositions, abstract
nouns, verbs cannot be translated into pictures but are important in the communication process
• S2 (pictures with words) vs S3 (words)– 4.32 vs. 4.40– advantage of words over pictures in
producing accurate interpretations
Outline
[building resources]• I: Building multilingual WordNets
• II: Building a (crosslingual) pictorial WordNet [using resources]• III: Applications
• IV: Conclusions
Conclusions• Multilingual and cross-lingual
semantic networks can be constructed with the help of volunteer contributions over the Web
• Advantages– faster than manual experts – more accurate than automatic approaches– taps on the “collective mind”
• Potentially infinite
– construct resources from scratch or validate automatically acquired knowledge
Conclusions
• Challenges– (multilingual networks) require
bilingual users– definition of difficult concepts– Quantity:
• User motivation• Disguise scientific tasks as games• Rewards
– Quality:• Multiple annotations• Human expert supervision
Conclusions
• Multilingual and cross-lingual semantic networks can be used as knowledge bases for building communication tools– machine translation– pictorial translation
• Rada Mihalcea, Ben Leong, Toward Communicating Simple Sentences Using Pictorial Representations, in Proceedings of the 7th Biennial Conference of the Association for Machine Translation in the Americas (AMTA), Boston, MA, August 2006.
• Andy Borman, Rada Mihalcea, Paul Tarau, PicNet: Pictorial Representations for Illustrated Semantic Networks, in Proceedings of the AAAI Spring Symposium on Knowledge Collection from Volunteer Contributors, Stanford, CA, March 2005.
• Nathaniel Ayewah, Rada Mihalcea, and Vivi Nastase, Building Multilingual Semantic Networks with Non-Expert Contributions over the Web, in Proceedings of the KCAP 2003 Workshop on Distributed and Collaborative Knowledge Capture, Sanibel Island, Florida, November 2003.
For More Information