cs349 on boolean retrieval query which plays of shakespeare contain the words brutus and caesar but...
TRANSCRIPT
![Page 1: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/1.jpg)
CS349
On Boolean Retrieval
![Page 2: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/2.jpg)
Query
Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia?
Could grep all of Shakespeare’s plays for Brutus and Caesar then strip out lines containing Calpurnia?
Slow (for large corpora)
NOT is hard to do
Grep does not help with other operations (e.g., find the Romans NEAR countrymen)
![Page 3: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/3.jpg)
Term-document incidence
Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 1 1 0 0 0 1
Brutus 1 1 0 1 0 0
Caesar 1 1 0 1 1 1
Calpurnia 0 1 0 0 0 0
Cleopatra 1 0 0 0 0 0
mercy 1 0 1 1 1 1
worser 1 0 1 1 1 0
1 if play contains word, 0 otherwise
![Page 4: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/4.jpg)
Incidence vectors
So we have a 0/1 vector for each term.
To answer query: bitwise AND. Brutus AND Caesar AND Calpurnia (complemented)
110100 AND 110111 AND 101111 = 100100.
Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 1 1 0 0 0 1
Brutus 1 1 0 1 0 0
Caesar 1 1 0 1 1 1
Calpurnia 0 1 0 0 0 0
Cleopatra 1 0 0 0 0 0
mercy 1 0 1 1 1 1
worser 1 0 1 1 1 0
![Page 5: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/5.jpg)
Answers to query
Antony and Cleopatra, Act III, Scene iiAgrippa [Aside to DOMITIUS ENOBARBUS]: Why, Enobarbus,
When Antony found Julius Caesar dead,
He cried almost to roaring; and he wept
When at Philippi he found Brutus slain.
Hamlet, Act III, Scene iiLord Polonius: I did enact Julius Caesar I was killed i' the
Capitol; Brutus killed me.
![Page 6: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/6.jpg)
Bigger corpora
Consider n = 1M documents,
each with about 1K terms. 1B terms
On average (English) 6 bytes/term (incl spaces/punctuation)
6GB of data.
Say there are m = 500K distinct terms among these.
![Page 7: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/7.jpg)
Can’t build the matrix
500K x 1M matrix has half-a-trillion 0’s and 1’s.
But it has no more than one billion 1’s. matrix is extremely sparse. i.e., the matrix has mostly 0’s
What’s a better representation for a sparse matrix?
Why?
![Page 8: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/8.jpg)
Inverted index: Step 1
I did enact JuliusCaesar I was killed
i' the Capitol; Brutus killed me.
Doc 1
So let it be withCaesar. The noble
Brutus hath told youCaesar was ambitious
Doc 2
Term Doc #I 1did 1enact 1julius 1caesar 1I 1was 1killed 1i' 1the 1capitol 1brutus 1killed 1me 1so 2let 2it 2be 2with 2caesar 2the 2noble 2brutus 2hath 2told 2you 2
caesar 2was 2ambitious 2
Documents are parsed to extract words;words are saved with the document ID.
![Page 9: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/9.jpg)
Inverted index: Step 2
Term Doc #ambitious 2be 2brutus 1brutus 2capitol 1caesar 1caesar 2caesar 2did 1enact 1hath 1I 1I 1i' 1it 2julius 1killed 1killed 1let 2me 1noble 2so 2the 1the 2told 2you 2was 1was 2with 2
Term Doc #I 1did 1enact 1julius 1caesar 1I 1was 1killed 1i' 1the 1capitol 1brutus 1killed 1me 1so 2let 2it 2be 2with 2caesar 2the 2noble 2brutus 2hath 2told 2you 2caesar 2was 2ambitious 2
Documents are parsed to extract words;words are saved with the document ID.
The inverted file is sorted by terms
![Page 10: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/10.jpg)
Inverted index: Step 3
Documents are parsed to extract words;words are saved with the document ID.
The inverted file is sorted by terms
Multiple term entries in a single document are merged and frequency information added
Term Doc # Freqambitious 2 1be 2 1brutus 1 1brutus 2 1capitol 1 1caesar 1 1caesar 2 2did 1 1enact 1 1hath 2 1I 1 2i' 1 1it 2 1julius 1 1killed 1 2let 2 1me 1 1noble 2 1so 2 1the 1 1the 2 1told 2 1you 2 1was 1 1was 2 1with 2 1
Term Doc #ambitious 2be 2brutus 1brutus 2capitol 1caesar 1caesar 2caesar 2did 1enact 1hath 1I 1I 1i' 1it 2julius 1killed 1killed 1let 2me 1noble 2so 2the 1the 2told 2you 2was 1was 2with 2
Term Doc #I 1did 1enact 1julius 1caesar 1I 1was 1killed 1i' 1the 1capitol 1brutus 1killed 1me 1so 2let 2it 2be 2with 2caesar 2the 2noble 2brutus 2hath 2told 2you 2caesar 2was 2ambitious 2
![Page 11: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/11.jpg)
The file is split into a Dictionary and a Postings file
Doc # Freq2 12 11 12 11 11 12 21 11 12 11 21 12 11 11 22 11 12 12 11 12 12 12 11 12 12 1
Term N docs Tot Freqambitious 1 1be 1 1brutus 2 2capitol 1 1caesar 2 3did 1 1enact 1 1hath 1 1I 1 2i' 1 1it 1 1julius 1 1killed 1 2let 1 1me 1 1noble 1 1so 1 1the 2 2told 1 1you 1 1was 2 2with 1 1
Term Doc # Freqambitious 2 1be 2 1brutus 1 1brutus 2 1capitol 1 1caesar 1 1caesar 2 2did 1 1enact 1 1hath 2 1I 1 2i' 1 1it 2 1julius 1 1killed 1 2let 2 1me 1 1noble 2 1so 2 1the 1 1the 2 1told 2 1you 2 1was 1 1was 2 1with 2 1
Dictionary Postings
![Page 12: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/12.jpg)
Where do we pay in storage?
Doc # Freq2 12 11 12 11 11 12 21 11 12 11 21 12 11 11 22 11 12 12 11 12 12 12 11 12 12 1
Term N docs Tot Freqambitious 1 1be 1 1brutus 2 2capitol 1 1caesar 2 3did 1 1enact 1 1hath 1 1I 1 2i' 1 1it 1 1julius 1 1killed 1 2let 1 1me 1 1noble 1 1so 1 1the 2 2told 1 1you 1 1was 2 2with 1 1
Pointers
Terms
![Page 13: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/13.jpg)
Two conflicting forces
A term like Calpurnia occurs in maybe one doc out of a million
would like to store this pointer using log2 1M ~ 20 bits.
A term like the occurs in virtually every doc,
so 20 bits/pointer is too expensive.
Prefer 0/1 vector in this case
(Or do not store it at all!)
![Page 14: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/14.jpg)
Postings file entry: Variable encoding
Store list of docs containing a term in increasing order of Doc #.
Brutus: 33,47,154,159,202 …
Therefore: suffices to store gaps. 33,14,107,5,43 …
Hope: most gaps are small numbers, encoded with far fewer than 20 bits.
For Calpurnia, use ~20 bits/gap entry.
For the, use ~1 bit/gap entry.
If the average gap for a term is G, want to use ~log2G bits/gap entry
Doc # Freq2 12 11 12 11 11 12 21 11 12 11 21 12 11 11 22 11 12 12 11 12 12 12 11 12 12 1
![Page 15: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/15.jpg)
codes for gap encoding
Length Offset
Represent a gap G as the pair <length,offset>
length is in unary and uses log2G +1 bits to specify the length of the binary encoding of
offset = G - 2log2G
e.g., 9 represented as 1110001
24 represented as 111101000
511 represented as 11111111011111111
Encoding G takes 2 log2G +1 bits.
What we have gained:Encoded each gap as tightly as possible, to within a factor of 2.
For better tuning (and a simple analysis) - need some handle on the distribution of gap values.
![Page 16: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/16.jpg)
Zipf’s law
An interesting (and surprising?) fact of life!
The k-th most frequent term has frequency proportional to 1/k.
Talks about the relative frequencies of terms (words in documents, etc)
Use this for a crude analysis of the space used by our postings file pointers.
Log-log plot
![Page 17: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/17.jpg)
Rough analysis based on Zipf’s law
Most frequent term occurs in n docs n gaps of 1 each.
Second most frequent term in n/2 docs n/2 gaps of 2 each …
kth most frequent term in n/k docs n/k gaps of k each - use 2log2k +1 bits for each gap;
net of ~(2n/k)*log2k bits for kth most frequent term.
![Page 18: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/18.jpg)
Sum over k from 1 to 500K
Do this by breaking values of k into groups: group i consists of 2i-1 k < 2i.
Group i has 2i-1 components in the sum, each contributing at most (2ni)/2i-1.
Summing over i from 1 to 19, we get a net estimate of 340Mbits ~ 45MB for our index.
Work outcalculation.
![Page 19: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/19.jpg)
Caveats
This is not the entire space for our index: it only refers to the postings file does not account for dictionary storage; as we get further, we’ll store even more stuff in the index.
Assumes Zipf’s law applies to occurrence of terms in docs.
All gaps for a term taken to be the same.
Does not talk about query processing.
![Page 20: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/20.jpg)
Issues with index we just built
How do we process a query? Much more complex than the boolean matrix What does it take to unpack such an encoding…
What terms in a doc do we index? All words or only “important” (“infrequent”) ones?
Stopword list: terms that are so common that they’re ignored for indexing.
e.g., the, a, an, of, to … language-specific Cannot find “to be or not to be” (so we need to do something
about it)
Exercise: Repeat postings size calculation if 100 mostfrequent terms are not indexed.
![Page 21: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/21.jpg)
Issues in what to index
Cooper’s vs. Cooper vs. Coopers.
Full-text vs. full text vs. {full, text} vs. fulltext.
Accents: résumé vs. resume.
Cooper’s concordance of Wordsworth was published in 1911. The applications of full-text retrieval are legion: they include résumé scanning, litigation support and searching published journals on-line.
![Page 22: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/22.jpg)
Punctuation
Ne’er: use language-specific, handcrafted “locale” to normalize.
State-of-the-art: break up hyphenated sequence.
U.S.A. vs. USA - use locale.
a.out
![Page 23: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/23.jpg)
Numbers
A pain: index size will blow up!
Maybe not index! People do not like it: computer models, years, …
3/12/91
Mar. 12, 1991
55 B.C.
B-52
100.2.86.144 Generally, don’t index as text Creation dates for docs
![Page 24: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/24.jpg)
Case folding
Reduce all letters to lower case exception: upper case in mid-sentence
e.g., General Motors Fed vs. fed SAIL vs. sail
![Page 25: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/25.jpg)
Thesauri and soundex
Handle synonyms and homonyms Hand-constructed equivalence classes
e.g., car = automobile your you’re
Index such equivalences, or expand query? More later ...
![Page 26: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/26.jpg)
Spell correction
Look for all words within (say) edit distance 3 (Insert/Delete/Replace) at query time
e.g., Alanis Morisette
Spell correction is expensive and slows the query (up to a factor of… 100)
Invoke only when index returns zero matches. What if docs contain mis-spellings? People exploit it:
they may use misspellings to find unique document!
![Page 27: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/27.jpg)
Lemmatization
Reduce inflectional/variant forms to base form
E.g., am, are, is be
car, cars, car's, cars' car
the boy's cars are different colors the boy car be different color
![Page 28: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/28.jpg)
Stemming
Reduce terms to their “roots” before indexing language dependent e.g., automate(s), automatic, automation all reduced to
automat.
for example compressed and compression are both accepted as equivalent to compress.
for exampl compres andcompres are both acceptas equival to compres.
![Page 29: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/29.jpg)
Porter’s stemmer algorithm
Commonest algorithm for stemming English There are for another ~40 languages E.g. Japanese is a pain
Conventions + 5 phases of reductions phases applied sequentially each phase consists of a set of commands sample convention: Of the rules in a compound command, select
the one that applies to the longest suffix.
Typical rules fr Porter’s stemmer: sses ss ies i ational ate tional tion
![Page 30: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/30.jpg)
Other stemmers
Other stemmers exist, e.g., Lovins stemmer http://www.comp.lancs.ac.uk/computing/research/stemming/general/lovins.htm
Single-pass, longest suffix removal (about 250 rules)
Motivated by Linguistics as well as IR
Full morphological analysis - modest benefits for retrieval
![Page 31: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/31.jpg)
Beyond term search
What about phrases?
Proximity: Find Gates NEAR Microsoft. Need index to capture position information in docs. Need heavier machinery
Zones in documents: Find documents with (author = Ullman) AND (text contains automata).
![Page 32: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/32.jpg)
Evidence accumulation
1 vs. 0 occurrence of a search term (e.g., “Lockheed” => it is about aerospace engineering)
2 vs. 1 occurrence 3 vs. 2 occurrences, etc.
Need term frequency information in docs
![Page 33: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/33.jpg)
Ranking search results
Boolean queries give inclusion or exclusion of docs.
Need to measure proximity from query to each doc.
Whether docs presented to user are singletons, or a group of docs covering various aspects of the query.
![Page 34: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/34.jpg)
Structured vs unstructured data
Structured data tends to refer to information in “tables”
Employee Manager Salary
Smith Jones 50000
Chang Smith 60000
50000Ivy Smith
Typically allows numerical range and exact match(for text) queries, e.g.,Salary < 60000 AND Manager = Smith.
![Page 35: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/35.jpg)
Unstructured data
Typically refers to free text
Allows Keyword queries including operators More sophisticated “concept” queries e.g.,
find all web pages dealing with drug abuse
Classic model for searching text documents
![Page 36: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/36.jpg)
Semi-structured data
But in fact almost no data is “unstructured”
E.g., this slide has distinctly identified zones such as the Title and Bullets
Facilitates “semi-structured” search such as Title contains data AND Bullets contain search
![Page 37: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/37.jpg)
More sophisticated semi-structured search
Title is about Object Oriented Programming AND Author something like stro*rup
where * is the wild-card operator
Issues: how do you process “about” how do you rank results
Will consider when studying XML search
![Page 38: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/38.jpg)
Clustering and classification
Given a set of docs, group them into clusters based on their contents.
Given a set of topics, plus a new doc D, classify it, e.g., decide which topic(s) D belongs to.
![Page 39: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/39.jpg)
The web and its challenges
Unusual and diverse documents
Unusual and diverse users, queries, information needs
Beyond terms, exploit ideas from social networks link analysis, clickstreams ...
![Page 40: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/40.jpg)
Evaluating an IR system – Part I
What are some measures for evaluating an IR system’s performance?
Speed of indexing Index/corpus size ratio Speed of query processing “Relevance” of results
Note: information need is translated into a query
Relevance is assessed relative to the information need not the query
![Page 41: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/41.jpg)
The classic search model (courtesy Andrei Broder)
Corpus
TASK
Info Need
Query
Verbal form
Results
SEARCHENGINE
QueryRefinement
Get rid of mice in a politically correct way
Info about removing micewithout killing them
How do I trap mice alive?
mouse trap
Mis-conception
Mis-translation
Mis-formulation
Polysemy
Synonymy
![Page 42: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/42.jpg)
Problems
Mis-conception = information thought necessary is not what is really needed for task at hand
Mis-translation = verbalization does not reflect information need
Mis-formulation = actual query does not reflect information need
Polysemy = one word has multiple meaningSynonymy = same concept might be expressed by
different words
![Page 43: CS349 On Boolean Retrieval Query Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia? Could grep all of Shakespeare’s plays](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c4c075503469d6a8b4989/html5/thumbnails/43.jpg)
Exercise
Observe your own information seeking behavior WWW University library Grocery store
Are you a searcher or a browser?How do you reformulate your query?
Read bad hits, then minus terms Read good hits, then plus terms Try a completely different query …