from “meaning”s to words İlknur durgar el-kahlout
TRANSCRIPT
FROM “Meaning”s FROM “Meaning”s
TO WordsTO Words
İlknur DURGAR
EL-KAHLOUT
Problem
For a given definition, find the appropriate word (or words), that has a similar definition– traditional dictionary no use
Examples
Akımı ölçmek için kullanılan alet akımölçer(A device that is used to measure the current ammeter)
akımölçer: elektrik akımının şiddetini ölçmeye yarayan araç, ampermetre
(ammeter: a device that measures the intensity of electrical current, amperemeter)
Examples
Çalıştığı işten kendi isteği ile ayrılmak istifa(Leaving one’s job voluntarily resignation)
istifa: kendi isteği ile görevden ayrılma(resignation: leaving voluntarily, of a position)
Applications
Computer-assisted language learning Solving crossword puzzles Reverse dictionary
Outline
Problem Statement Challenges Our Approach Methods Results Result Summary Conclusion
Problem Statement
For example, one knows the meaning of the word akımölçer (ammeter):
Akımı ölçmek için kullanılan alet (A device that is used to measure the current)
However the actual definition of the word in the dictionary is:
elektrik akımının şiddetini ölçmeye yarayan araç, ampermetre
(a device that measures the intensity of electrical current, amperemeter)
Problem Statement
Find the similarity between two definitions Akımı ölçmek için kullanılan alet (A device that is used to measure the current)
elektrik akımının şiddetini ölçmeye yarayan araç, ampermetre
(a device that measures the intensity of electrical current, amperemeter)
Meaning-to-Word (MTW)
Meaning-to-Word System (MTW)– attacks the problem of finding the appropriate
word (or words), whose meaning “matches” the given definition
Challenges
Two challenging problems
– finding words whose definitions are "similar" to the query in some sense.
– ranking the candidate words using a variety of ways.
Information flow in MTW
User Definition
Search in Dictionary
Rank Candidates
query
candidates
List of words
Meanings To Words (MTW)
The problem of retrieving words from their "meaning"s at first sight seems to be an information retrieval problem
Information Retrieval (IR)
responds to the user's query by selecting documents from a database and ranking them in terms of relevance.
uses (mostly) statistical and symbolic techniques to retrieve documents for a given query, employing shallow natural language analysis.
Similarities between MTW and IR
Goals – Select relevant items from a collection based
on a query
Collections
– Collection Dictionary
Documents: – Documents Definitions
Similarities between MTW and IR
Approaches:– compare the user request with each of the
information in the collection Ranking:
– most important task– But ranking strategies are different
Differences between IR and MTW
Expected results:– Many relevant documents vs. only one correct word
Query Expression:– Keywords vs. sentence (or phrases)
Space size: – Long documents (avg. 300 - 400 words ) vs. one
sentence long definitions (avg. 10 - 20 words)– Huge collection(106-109doc) vs. medium dictionary
(105 word definitions)
Available Resources
Turkish Dictionary Turkish Wordnet
Normalization
User Definition
Search in Dictionary
Rank Candidates
query
candidates
List of words
Normalization
Normalization
Tokenization: – All inter-word (non-word, non-digit) symbols eliminated (ex.
Punctuation). – Each word is a term
Stemming: – same stem but different affixes– enables matching different morphological variants of the original
definition's words Stop Word Elimination:
– have little or no meaning– Frequency (very frequent words)– Linguistic (determiners, prepositions, pronouns,..)
Query Processing
User Definition
Search in Dictionary
Rank Candidates
query
candidates
List of words
Query Processing
Query Processing
Subset Generation:– Search with different set of words– Select informative words from user’s query
Query: hiç evlenmemiş kişi (a person who has never been married)
* {önce, evlenmemiş, kişi}(before, unmarried, person)
* {evlenmemiş, kişi} {önce, kişi} {önce, evlenmemiş} (unmarried, person) (before, person) (before, unmarried)
*{evlenmemiş} {önce} {kişi} (unmarried) (before) (person)
Query Processing
Subset Sorting:– Unordered list of subsets are insufficient
• Top-down sorting
– Rank the generated subsets 1) By the number of words
Ex: {önce,evlenmemiş, kişi} (before, unmarried, person) vs. {evlenmemiş, kişi} (unmarried, person)
2) By the sum of frequency logarithmEx:{evlenmemiş, kişi} (unmarried, person) vs. {önce, kişi} (before, person)
Searching for “Meaning”s
User Definition
Search in Dictionary
Rank Candidates
query
candidates
List of words
Searching for “Meaning”s
Two methods – Stem Match– Query Expansion
Stem Match
Morphological normalization of words– Find meanings that contain morphological
variants of the original definition
Stem Match (Ex.)
{A device that is used to measure the current}
{ akımı ölçmek için kullanılan alet }
ak (white) ölç (measure) için (to) kullan (use) alet (device)
akım (current) iç (drink) kul (slave)
akı (flux)
Stem Match
akımı ölçmek için kullanılan alet - A device that is used to measure the current
elektrik akımının şiddetini ölçmeye yarayan araç, ampermetre - a device that measures the intensity of electrical current, amperemeter
Stem Match Drawback:
– Conflate two words with very different meanings to the same stem
(ex: yüksek (high) yüksek (high), yük (load)
ilim (science, my city), ilde (in the city) il (city))
– Cant find relations between similar words
(ex: kimse (someone) kişi (person) ,
bölüm (part) kısım (portion))
Query Expansion
The users of retrieval systems often use different words to describe the concepts in their queries than the authors use to describe the same concept in their documents.
In experiments, two people use the same term to describe an object less than 20% of the time.(Furnas 1987).
Using Query Expansion
Two different approaches:• Expand query with relations (synonyms,
specializations, generalizations)• Expand query with unexpanded query’s
relevant answers
Synonym relation used in MTW Ex:{besin,gıda} (food, nourishment)
{iyileş,düzel} (to get better) /{iyileş,geliş} (to improve)
Query Expansion (Ex.)
{A device that is used to measure the current}
{ akımı ölçmek için kullanılan alet }
*ak (white) ölç (measure) için (to) ***kullan (use) alet (device)
akım (current) iç (drink)****kul (slave)
**akı (flux)
*beyaz ölçüm ***faydalan araç
**debi ***yararlan gereç
**akış ****köle
Query Expansion (Ex.)
akımı ölçmek için kullanılan alet - A device that is used to measure the current
elektrik akımının şiddetini ölçmeye yarayan araç, ampermetre - a device that measures the intensity of electrical current, amperemeter
Ranking
User Definition
Search in Dictionary
Rank Candidates
query
candidates
List of words
Ranking
The main goal of a retrieval system is to find the documents that are relevant to a query.
Documents that are likely to be more relevant
should be ranked at the top and documents that are likely to be less relevant should be ranked at the bottom of the ranked list. (Hiemstra 1999)
Ranking
Most important part of MTW– Having the right answer in the retrieved set is
not enough– Aim is to have the right answer at top of the
retrieved set (Ex: in first top 50 answers)
Ranking
Simple but effective methods– Subset informativeness (subset sorting)– Number of matched words (subset sorting)– Length of the candidate definition– Longest Common Subsequence
Some statistics Train sets:
– 50 queries from real users
– 50 queries from a dictionary Test sets:
– 50 queries from real users – 50 queries from a dictionary
Test set 1 Train set 2 Test set 1 Train set 2
# of queries 50 50 50 50
Avg. # of query words
5.66 4.64 9.24 13.98
Max. # of query words
17 12 23 45
Min. # of query words
2 1 1 6
Stem Match (all stems included)
Rank Test set 1 Train set 1 Test set 2 Train set 2
1-10 13 (26%) 18 (36%) 45 (90%) 41 (82%)
11-50 7 (14%) 12 (24%) 2 (4%) 5 (10%)
51-100 4 (8%) 1 (2%) 1 (2%) 2 (4%)
101-300 3 (6%) 3 (6%) 2 (4%) 1 (2%)
301-500 2 (4%) 2 (4%) 0 (0%) 1 (2%)
501-1000 6 (12%) 2 (4%) 0 (0%) 0 (0%)
Over 1000 4 (8%) 2 (4%) 0 (0%) 0 (0%)
Not found 11 (22%) 10 (20%) 0 (0%) 0 (0%)
Stem Match (longest stem included)
Rank Test set 1 Train set 1 Test set 2 Train set 2
1-10 14 (28%) 21 (42%) 46 (92%) 43 (86%)
11-50 5 (10%) 9 (18%) 1 (2%) 5 (10%)
51-100 4 (8%) 1 (2%) 1 (2%) 1 (2%)
101-300 3 (6%) 1 (2%) 2 (4%) 1 (2%)
301-500 2 (4%) 3 (6%) 0 (0%) 0 (0%)
501-1000 5 (10%) 2 (4%) 0 (0%) 0 (0%)
Over 1000 4 (8%) 2 (4%) 0 (0%) 0 (0%)
Not found 13 (26%) 11 (22%) 0 (0%) 0 (0%)
Query Expansion Match (all stems included)
Rank Test set 1 Train set 1 Test set 2 Train set 2
1-10 14 (28%) 24 (48%) 45 (90%) 41 (82%)
11-50 9 (18%) 9 (18%) 2 (4%) 5 (10%)
51-100 3 (6%) 3 (6%) 1 (2%) 2 (4%)
101-300 7 (14%) 2 (4%) 2 (4%) 1 (2%)
301-500 0 (0%) 1 (2%) 0 (0%) 1 (2%)
501-1000 4 (8%) 5 (10%) 0 (0%) 0 (0%)
Over 1000 4 (8%) 1 (2%) 0 (0%) 0 (0%)
Not found 9 (18%) 5 (10%) 0 (0%) 0 (0%)
Query Expansion Match (longest stem included)
Rank Test set 1 Train set 1 Test set 2 Train set 2
1-10 14 (28%) 24 (48%) 41 (82%) 39 (78%)
11-50 6 (12%) 8 (16%) 5 (10%) 6 (12%)
51-100 5 (10%) 5 (10%) 0 (0%) 2 (4%)
101-300 7 (14%) 2 (4%) 0 (0%) 2 (4%)
301-500 1 (2%) 1 (2%) 0 (0%) 0 (0%)
501-1000 5 (10%) 3 (6%) 0 (0%) 0 (0%)
Over 1000 3 (6%) 2 (4%) 1 (2%) 1 (2%)
Not found 9 (18%) 5 (10%) 0 (0%) 0 (0%)
Data fusion
No single method is better than all others in all cases
Merging results from different methods seems to be promising approach for achieving improved performance
Many data fusion methods including min, max, average, sum, weighted average and other linear combination functions
Data Fusion
Weighted Sum
21 *)(_*)(_)( wwscoreQEwwscoreSMwScore
Data Fusionc1= 0.7 (stem match const.)
c2= 0.3 (query expansion const.)
Rank Test set 1 Train set 1
1-10 15 (30%) 22 (44%)
11-50 10 (20%) 14 (28%)
51-100 4 (8%) 1 (2%)
101-300 3 (6%) 2 (4%)
301-500 3 (6%) 0 (0%)
501-1000 5 (10%) 3 (6%)
Over 1000 -- --
Not found 11 (22%) 8 (16%)
Result Summary
Stem Match (longest stem included)• 60% real user queries
• 96% dictionary queries
Query Expansion (all stems included)• 68% real user queries
• 92% dictionary queries
Data Fusion (longest stem included)• 72% real user queries
Conclusion
Meaning to Word system is implemented for Turkish language
Results on unseen data are rather satisfactory Query expansion is better
• Although, it can not find the words for all queries
• 68% of real user queries and 90% of dictionary queries are found in the first 50 results
Data fusion has a better performance • 72% of real user queries are found in first 50% results
THANK YOU !!THANK YOU !!