term processing & normalization major goal: find the best possible representation minor goals:...
TRANSCRIPT
Term Processing & NormalizationMajor goal:
Find the best possible representationMinor goals:
Improve storage and speed
First: Need to transform sequence of bytes into sequence of characters
Problems involved: Encoding, language, … Different formats (some might be proprietary) Content vs. structure information etc.
Lexical Analysis & TokenizationGoal: Transform stream of characters into
stream of words / terms / tokensTerms should be consistent and ‘reasonable’
Lots of potential problems here, e.g.Apostrophes
England’s: England s – Englands?haven’t: haven t – havent – have not?user’s vs. users’ vs. users’s?
Hyphenationstate-of-the-art vs. state of the artco-authors vs. coauthors vs. co authors
Lex. Analysis & Tokeniz. (Cont.)Special characters
Punctuation (Normally removed. But what about www.uni-freiburg.de?)Names: C++, A* Algorithm, M*A*S*H, …
Use just white spaces for segmentation?San Francisco? Los Angeles?Nuclear power plant?
Language specific issues, e.g.German: ‘Komposita’, e.g. Atomkraftwerk
Case sensitivity?
Conclusion: Implementation is easy, but what and how is critical (often heuristics, etc.)
Removal of Stop WordsGoal: Remove words with no relevance for any
search in order to reduce disk space, speed up search time, and improve result
Problems:What about ‘To be or not to be’?What about phrases, e.g. ‘The name of the rose’ (book/movie vs. flower names)?Usability?
Implementation:Often via manually generated stop words lists (depending on the task and data, e.g. small vs. big, general vs. topic specific, etc.)
ThesaurusThesaurus: A data structure composed of (1) a
pre-compiled list of important words in a given domain of knowledge and (2) for each word in the list, a list of related (synonym) words.(Glossary, page 453, Baeza-Yates & Ribeiro-Neto [1])
Implementation:Either manually generated (hard & expensive)or automatic (e.g. with statistical methods)
Usefulness for retrieval is controversialTopic-specific thesauri seem to work well, but limited usage with large, dynamic data base(?)
StemmingStemming: A technique for reducing words to
their grammatical roots.(Glossary, page 452, Baeza-Yates & Ribeiro-Neto [1])
Motivation: reduce different cases, singular and plural, etc. to the root (stem) of the word
Example: connected, connecting, connection, connections -> connect
But how about: operation research, operating system, operative dentistry -> opera???
Approaches for StemmingTable lookup
Generation is complexFinal tables are often incomplete
Affix removal Suffix vs. prefix (e.g. mega-volt)Doesn’t always work, esp. not in German
Successor variety stemming More complex than suffix removalUses (e.g.) linguistic approaches and techniques from morphology
N-gramsGeneral clustering approach which can also be used for stemming
The Porter AlgorithmSpecial algorithm for the English language
based on suffix removal Basic idea: Successive application of several rules
for suffix removal (5 phases of condition and action rules)
Example: Remove plural ‘s’ and ‘sses’Rules: sses -> ss, s -> NIL (obey order!)
Advantage: Easy algorithm with good resultsDisadvantage: Not always correct, e.g.
Same root for police – policy, execute –executive, …Different root for european – europe, search – searcher, …
Conventions for the pseudo language used to describe the - a consonant variable is represented by the symbol C which is used to refer to any letter
other than a,e,i,o,u and other than the letter y preceded by a consonant;- a vowel variable is represented by the symbol V which is used to refer to any letter which
is not a consonant; - a generic letter (consonant or vowel) is represented by the symbol L; - the symbol NIL is used to refer to an empty string (i.e., one with no letters); - combinations of C, V, and L are used to define patterns; - the symbol * is used to refer to zero or more repetitions of a given pattern; - the symbol + is used to refer to one or more repetitions of a given pattern; - matched parenthesis are used to subordinate a sequence of variables to the operators *
and +; - a generic pattern is a combination of symbols, matched parenthesis, and the operators *
and +; - the substitution rules are treated as commands which are separated by a semicolon
punctuation mark; - the substitution rules are applied to the suffixes in the current word; - a conditional if statement is expressed as ``if (pattern) rule'' and the rule is executed only
if the pattern in the condition matches the current word; - a line which starts with a % is treated as a comment; - curled brackets are used to form compound commands; - a “select rule with longest suffix” statement selects a single rule for execution among all
the rules in a compound command. The rule selected is the one with the largest matching suffix.
Porter Algorithm – Convention rules I
Porter Algorithm – Convention rules II
Thus, the expression (C)* refers to a sequence of zero or more consonants while the expression ((V)*(C)*)* refers to a sequence of zero or more vowels followed by zero or more consonants which can appear zero or more times. It is important to distinguish the above from the sequence (V*C) which states that a sequence must be present and that this sequence necessarily starts with a vowel, followed by a subsequence of zero or more letters, and finished by a consonant.
Finally, the command if (*V*L) then ed -> NILstates that the substitution of the suffix ed by nil (i.e., the removal of the suffix ed) only occurs if the current word contains a vowel and at least one additional letter.
Porter Algorithm – Phase 1
% Phase 1: Plurals and past participles. select rule with longest suffix { sses -> ss; ies -> i; ss -> ss; s -> NIL; } select rule with longest suffix { if ( (C)*((V)+(C)+)+(V)*eed) then eed -> ee; if (*V*ed or *V*ing) then { select rule with longest suffix { ed -> NIL; ing -> NIL; } select rule with longest suffix { at -> ate; bl -> ble; iz -> ize; if ((* C1C2) and ( C1 = C2) and (C1 not in {l,s,z})) then C1C2 -> C1; if (( (C)*((V)+(C)+)C1V1C2) and (C2 not in {w,x,y})) then C1V1C2 -> C1V1C2e; } } } if (*V*y) then y -> i;
Porter Algorithm – Phase 2
% Phase 2if ( (C)*((V)+(C)+)+(V)*) then select rule with longest suffix { ational -> ate; tional -> tion; enci -> ence; anci -> ance; izer -> ize; abli -> able; alli -> al; entli -> ent; eli -> e;
ousli -> ous; ization -> ize; ation -> ate; ator -> ate; alism -> al; iveness -> ive; fulness -> ful; ousness -> ous; aliti -> al; iviti -> ive; biliti -> ble; }
Porter Algorithm – Phase 3% Phase 3
if ( (C)*((V)+(C)+)+(V)*) then select rule with longest suffix { icate -> ic; ative -> NIL; alize -> al; iciti -> ic; ical -> ic; ful -> NIL; ness -> NIL; }
Porter Algorithm – Phase 4
% Phase 4if ( (C)*((V)+(C)+)((V)+(C)+)+(V)*) then select rule with longest suffix { al -> NIL; ance -> NIL; ence -> NIL; er -> NIL; ic -> NIL; able -> NIL; ible -> NIL; ant -> NIL; ement -> NIL;
ment -> NIL; ent -> NIL; ou -> NIL; ism -> NIL; ate -> NIL; iti -> NIL; ous -> NIL; ive -> NIL; ize -> NIL; if (*s or *t) then ion -> NIL; }
Porter Algorithm – Phase 5
% Phase 5select rule with longest suffix { if ( (C)*((V)+(C)+)((V)+(C)+)+(V)*) then e -> NIL; if (( (C)*((V)+(C)+)(V)*) and not (( *C1V1C2) and (C2 not in {w,x,y}))) then e -> nil; } if ( (C)*((V)+(C)+)((V)+(C)+)+V*ll) then ll -> l;
Source: Baeza-Yates [1], Appendix (available online at www.sims.berkeley.edu/~hearst/irbook/porter.html)
StemmingUsefulness for retrieval is controversialSeems useful for languages such as German
(lots of cases, compound nouns, etc.), i.e. languages where it is hard to do successfully!
Hard for large, multilingual data collections (such as the web)
Another problem: Usability (wrong results often hard to understand by the users)
Term Processing – SummaryTerm processing, e.g.
NormalizationLexical analysis & tokenizationRemoval of stop wordsThesaurusStemming (Porter’s Algorithm)
and much more
How is it usually done? Often depends on the application and data
Is it useful? How does it influence the result?We need a measure for that!
Evaluation – Precision & RecallEvaluation of IR systems is often done based on
some test data (benchmark tests), i.e. a set of (document-query-relevance)-labels
The most important measures to quantify IR performance are Precision and Recall
PRECISION =# FOUND & RELEVANT
# FOUND
RECALL =# FOUND & RELEVANT
# RELEVANT
Precision & Recall – Example
PRECISION =# FOUND & RELEVANT
# FOUND
RECALL =# FOUND & RELEVANT
# RELEVANT
RESULT: DOCUMENTS:
A CD B
F HGEJ
I
1. DOC. B
2. DOC. E
3. DOC. F
4. DOC. G
5. DOC. D
6. DOC. H
Relation Term Proc. – Prec. & Rec.How does term proc. influence IR performance?
Generally:More specific terms: Precision , RecallBroader terms: Precision , Recall
More specific:Thesaurus: Precision , RecallPhrases: Precision , RecallStemming: Precision , RecallCompound nouns: Precision , Recall
Note: Often increase in recall rather than precision. Maybe a reason why it’s hardly used for web search?
LOGICAL VIEW OF THEDOCUMENTS (INDEX)
Recap: IR System & Tasks Involved
INFORMATION NEED DOCUMENTS
User Interface
PERFORMANCE EVALUATION
QUERY
QUERY PROCESSING (PARSING & TERM
PROCESSING)
LOGICAL VIEW OF THE INFORMATION NEED
SELECT DATA FOR INDEXING
PARSING & TERM PROCESSING
SEARCHING
RANKING
RESULTS
DOCS.
RESULT REPRESENTATION
Documents: Selection & Segmentation
Which documents should be indexed?Big issue for web search (because we can’t do all of it: too big, changes too quickly, …)General problems: Confidential information, formats, etc.
What is a reasonable unit for indexing?In most cases: 1 document = 1 fileWhat about email archives w. more than one email in one file (file = folder)?What about attachments?What about one document split over several files (e.g. HTML version of PPT file)
LOGICAL VIEW OF THEDOCUMENTS (INDEX)
Recap: IR System & Tasks Involved
INFORMATION NEED DOCUMENTS
User Interface
PERFORMANCE EVALUATION
QUERY
QUERY PROCESSING (PARSING & TERM
PROCESSING)
LOGICAL VIEW OF THE INFORMATION NEED
SELECT DATA FOR INDEXING
PARSING & TERM PROCESSING
SEARCHING
RANKING
RESULTS
DOCS.
RESULT REPRESENTATION
References & Recommended Reading
[1] R. BAEZA-YATES, B. RIBEIRO-NETO: ‘MODERN INFORMATIN RETRIEVAL’, ADDISON WESLEY, 1999
CHAPTER 7 (TERMPROCESSING)CHAPTER 8.1, 8.2 (INDEX, INV. FILES)
[2] C. MANNING, P. RAGHAVAN, H. SCHÜTZ: ‘INTRODUCTION TO IR’ (TO APPEAR 2007)CHAPTER 1 (INDEX, INV. FILES)CHAPTER 2 (TERMPROCESSING)
AVAILABLE ONLINE AT http://www-csli.stanford.edu/~schuetze/ information-retrieval-book.html