1 search strategy & tactics governed by effectiveness & feedback tefko saracevic...
Post on 21-Dec-2015
223 Views
Preview:
TRANSCRIPT
1
Search strategy& tacticsGoverned byeffectiveness
& feedback
Tefko Saracevic
tefkos@rutgers.edu; http://comminfo.rutgers.edu/~tefko/
Central ideas
• Searches are all about being effective – finding what is needed, desired
• To measure effectiveness common measures of precision and recall are used
• Various tactics are used to affect effectiveness• Application of tactics for a desired results is a second
nature of professional searchers
2Tefko Saracevic
Knowing searching = also knowing tactics to reach toward desired effectiveness
As a searcher you are striving toward doing effective searches
5
Some definitions
• Search statement (query):– set of search terms with logical connectors and attributes -
file and system dependent
• Search strategy (big picture):– overall approach to searching of a question
• selection of systems, files, search statements & tactics, sequence, output formats; cost, time aspects
• Search tactics (action choices):– choices & variations in search statements
• terms, connectors, attributes
Tefko Saracevic
6
Some definitions (cont.)
• Move :– modifications of search strategies or tactics that are aimed
at improving the results• e.g. from searching for digital AND libraries to digital(w)libraries
(Dialog) or “digital libraries” (Scopus)
• Cycle (particularly applicable to systems such as Dialog):– set of commands from start (begin) to viewing (type)
results, or from a viewing to a viewing command
Tefko Saracevic
Search strategy – big pictureall kinds of actions from start to end
• The entire approach to a search – selection of– files and sources to use– approaches in proceeding to search & combining
• search terms• operators to use• fields to search
– formats for viewing results– alternative actions if search yields
• too much• too little
– problem-solving heuristics
Tefko Saracevic 7
Search tactics – specific actionsconnected with a given search as it progresses
• A query - command line entered into a system in order to retrieve relevant information– terms, operators & attributes as allowed by a given system– vocabulary & syntax used in conjunction with connectors
&/or limiters to search a system• Again: depends on a system how it is done
– for example, a search statement might be: • in Dialog: b 47; ss (garbanzo? or chickpeas) and (hum?us or humus)• in Scopus you can enter: (garbanzo? or chickpeas) and (hum?us or
humus)• how would you do that in ?
Tefko Saracevic 8
9
Search performance is expressed in terms of:
• Effectiveness :– performance as to objectives
• to what degree did a search accomplish what desired?• how well done in terms of relevance?
• Efficiency :– performance as to costs
• at what cost and/or effort, time?
• But here we concentrate on effectiveness
Tefko Saracevic
Both are KEY concepts & criteria for selection of strategy, tactics & evaluation
10
Effectiveness criteria• Search tactics chosen & changed following some criteria
of accomplishment, such as:– none - no thought given – hard to imagine but happens sometimes
– relevance (most often) – is it relevant to a need, task, problem?
– magnitude (also very often)- is it to much retrieved to start with?
– output attributes- is it trustworthy? Authorities? Understandable? …
– topic – is it on the topic of the question?
• Tactics are chosen or altered interactively to match given criteria - feedback plays an important role
Tefko Saracevic
Knowing what choice of tactics may produce what results is key to professional searcher
11
Relevance:key concept in IR & key criterion for assessing effectiveness
“Relevant: having significant and demonstrable bearing on the matter at hand.”
“Relevance: the ability (as of an information retrieval system) to retrieve material that satisfies the needs of the user.” Merriam Webster (2005)
• Attribute/criterion reflecting effectiveness of exchange of inf. between people (users) & IR systems in communication contacts, based on valuation by people
Tefko Saracevic
Variation in names
• Relevance or relevant can be expressed in different terms– utility, pertinence, appropriate, significant, useful, germane,
applicable, valuable …
• But the concept still remains relevance That which we call relevance by any other word would
still be relevanceRelevance is relevance is relevance is relevance
thank you Shakespeare and Gertrude Stein
Tefko Saracevic 12
Some attributes of relevance
– in IR - user dependent – users are the final judges at the end
– multidimensional or faceted – users asses relevance on a number of factors, not only topic, but also authority, novelty, source …
– dynamic – users may change relevance assessments & criteria as search progresses or they learn or the problem is modified
– not only binary – users assess object not only as relevant/not relevant, but also more on a continuum (scale) - partially relevant included
– intuitively well understood – nobody has to explain to users what it is
Tefko Saracevic 13
ConsistencyIndividual differences
– consistency of assessment varies over time & between people to a (sometimes significant) degree – but (in)consistency is similar as in other processes e.g. indexing, classifying
• Subject expertise affects consistency of relevance judgments. Higher expertise results in higher consistency and stringency. Lower expertise results in lower consistency and more inclusion.
– relevance is not fixed; individual differences may be large
• It is most helpful to discuss with a user what kind of documents they may asses as relevant, what are their criteria and then try to incorporate that in searching & own assessments
Tefko Saracevic 14
15
Two major types of relevance – leading to a dichotomy of systems- & users-view of relevance:– Systems or algorithmic relevance
• the way a system or algorithm assesses relevance• relation between a query & objects in the file of a system as retrieved
or failed to be retrieved by a given algorithm
– User relevance• the way a user or user surrogate (searcher, specialist …) assesses
relevance• relation between information need or problem at hand of a user and
objects retrieved or out there in general
– But Topical relevance can be considered by both system & user • the degree to which topics or subjects of a query & topics or subjects
of objects (documents) in a file or retrieved match• relation between topic in the query & topic covered by the retrieved
objects, or objects in the file(s) of the system, or even in existence
Tefko Saracevic
16
User relevance can be further distinguished as to:
– Cognitive relevance or pertinence:• possible changes in user’s cognitive state due to objects retrieved• relation between state of knowledge & cognitive information need of a user
and the objects provided or in the file(s)
– Motivational or affective relevance• matching & satisfying user’s intentions, purposes, rationales, emotions• relation between intents, goals & motivations of a user & objects retrieved by a
system or in the file, or even in existence. Satisfaction
– Situational relevance or utility: • value of given objects or information for user’s situation or changes in situation• relation between the task or problem-at-hand & the objects retrieved (or in the
files). Relates to usefulness in decision-making, reduction of uncertainty ...
Tefko Saracevic
How?
The basic way by which effectiveness is established in IR searching is to compare
Tefko Saracevic 18
19
Effectiveness measuresTwo measures are used universally:Precision:
– of the stuff retrieved & given to user how much (what %) was relevant? – more formally: probability that given that an object is retrieved it is
relevant, or the ratio of relevant items retrieved to all items retrieved
Recall:– of the stuff that is relevant in the file how much (what %) was actually
retrieved?– more formally: probability that given that an object is relevant it is
retrieved, or the ratio of relevant items retrieved to all relevant items in a file
Tefko Saracevic
Tefko Saracevic 20
Items judgedRELEVANT
Items judgedNOT RELEVANT
ItemsRETRIEVED
aNo. of items retrieved & judged relevant
bNo. of items retrieved & judged not relevant (junk)
ItemsNOT RETRIEVED
cNo. of items not retrieved & relevant (missed relevant)
dNo. of items not retrieved & not relevant (missed junk)
Precision =a
a + b
Recall =a
a + c
High precision = maximize a, minimize b
High recall = maximize a, minimize c
Calculation
Examples of calculation
• If a system retrieved 16 documents and only 4 were assessed as relevant by a user then precision is 25%
• If a system had 40 documents in the file that were relevant but managed to retrieve only 12 of them then recall is 30%
• Precision is easy to establish, recall is not• union of retrievals is used as a “trick” to establish relative recall• you do a number of searches or use a number of tactics or
algorithms then you take together all that was retrieved (union) and have that assessed; then you can calculate a relative recall of each search, tactic or algorithm in respect to the union & see which one provides better or worse relative recall
Tefko Saracevic 21
22
Interpretation: PRECISION
• Precision= percent of relevant stuff you have in your set of answers retrieved– or conversely percent of junk (false drops) in the answer set– high precision = most stuff relevant– low precision = a lot of junk
• Some users demand high precision– do not want to wade through much stuff– but it comes at a price: relevant stuff may be missed
• Tradeoff almost always: high precision = low recall• we will get to tradeoff a bit later, but it is VERY important to consider
Tefko Saracevic
23
• A file may have a lot of relevant stuff• Recall = percent of that relevant stuff in the file that
you retrieved in your answer set– conversely percent of stuff you missed– high recall = you missed little– low recall = you missed a lot
• Some users demand high recall (e.g. PhD students doing a dissertation; patent lawyers; researchers writing a proposal or article)
– want to make sure that important stuff is not missed– but will have to pay a price of wading through a lot of junk
• Tradeoff almost always: high recall = low precision
Interpretation: RECALL
Tefko Saracevic
Tradeoff between precision and recallUsing it for different tactics
3. Search tactics
Tefko Saracevic 24
Aim of search tactics• Since there is no such thing as a perfect search:
– the aim is to search in a way that will insure a given, chosen or desired, level of effectiveness
• That means that we have to agree on– what do we mean by effectiveness in searching?
• general agreement is that retrieval of relevant answers is the major criterion for effectiveness of searching
– what measures do we use to express achievement of effectiveness in terms of relevance
• general agreement is that we use measures of precision and recall (or derivatives)
Tefko Saracevic 25
Thus many search tactics are geared toward achieving certain level of precision or recall
26
Precision-recall trade-off• USUALLY: precision & recall are inversely related
– higher recall usually means lower precision & vice versa
100 %
100 %0
Ideal
Usual
Impr
ovem
ents
Precision
Recall
Tefko Saracevic
27
Precision-recall trade-off
• It is like in life, usually:• you get some, lose some; you can't have your cake and eat it too
• Usually, but not always (keep in mind these are probabilities)
– when you have high precision most stuff you got is relevant or on the target but you also missed other stuff (could be a lot) that is relevant – it was left behind
– when you have high recall you did not miss much but you also got junk (could be a lot) - you have to wade through it
• There is price to pay either way– but then a lot of users are perfectly satisfied with and are aiming at high
precision • give me a few good things, that is all what I need
Tefko Saracevic
Effectiveness (Cleverdon’s) “laws”
High precision in retrieval is usually associated with low recall. High precision = low recall
High recall in retrieval is usually associated with low precision. High recall = low precision
You – with your user – have to decide if you are aiming toward high recall or high precision & you have to be aware of & explain tradeoffs
Tefko Saracevic 28
Originally formulated by Cyril Cleverdon,a UK librarian, in 1960s, in first IR tests done in Cranfield, UK, thus “Cranfield tests”You use different tactics for high
recall from those for high precision
Search tactics
Tefko Saracevic 29
Variable What variation possible?
1. LOGIC choice of connectors among terms AND, OR, NOT, W …
2. SCOPE no. of terms linked - ANDsA AND B vs A AND B AND C
3.EXHAUSTIVITY for each concept no. of related terms - OR connectionsA OR B vs. A OR B OR C
4. TERM SPECIFICITY for each concept level in hierarchybroader vs narrower terms
5. SEARCHABLE FIELDS choice for text terms & non-text attributese.g. titles only, limit as to years, various sources
6. FILE OR SYSTEM SPECIFIC CAPABILITIES
using capabilities, options availablee.g. given fields, ranking, sorting, linking
several ‘things’ or variables in a query can be selected & changed to affect effectiveness
Actions and consequences
Action ConsequenceSCOPE- adding more ANDs
Output size: downRecall: downPrecision: up
EXHAUSTIVITY- adding more ORs
Output size: upRecall: upPrecision: down
USE OF NOTs- adding more NOTs
Output size: downRecall: downPrecision: up
USING BROAD TERMS- low specificity
Output size: upRecall: upPrecision: down
USING PHRASES- high specificity
Output size: downRecall: downPrecision: up
Tefko Saracevic 30
BUT: each variation has consequence in outputif you do X then Y will happen
Tactics: what to do?
• These “laws” lead to precision & recall devices– tactics to increase/decrease precision or recall
Tefko Saracevic 31
To increase precision To increase recallSCOPE-add more ANDs-- you restrict
EXHAUSTIVITY-add more ORs- you enlarge
NOTs-add more NOTs-you eliminate
BROAD TERMS- you broaden terms
PHRASES- you become more specific
USE MORE TRUNCATION-you use grammatical variants of terms
Precision, recall devices
Tefko Saracevic 32
NARROWINGHigher precision
BOADENNINGHigher recall
More ANDsFewer ORsMore NOTsLess free textMore controlledLess synonymsNarrower termsMore specificLess truncationMore qualifiersMore limitsBuilding blocks
Fewer ANDsMore ORsFewer NOTsMore free textFewer controlledMore synonymsBroader termsLess specificMore truncationFewer qualifiers Fewer limitsCitation growing
With experience use of these devices will become second nature
33
Other tactics (but that gets us in the next unit on advanced searching)
• Citation growing:– find a relevant document– look for documents cited in– look for documents citing it– repeat on newly found relevant documents
• Building blocks– find documents with term A– review – add term B & so on
• Using different feedbacks– a most important tool
Tefko Saracevic
Feedback in searching
• Extremely important!• And constantly used, consciously or unconsciously• Formally:
– the process in which part of the output of a system is returned to its input in order to regulate its further output
• Simply put:– you search, find something that may be relevant, then look
at it, then on the basis of AHA! change your next query or tactic to get better or more stuff
Tefko Saracevic 35
36
Feedback in searching …• Any feedback implies loops
– a completion of a process provides information for modification, if any, for the next process
– information from output is used to change previous or create new input
• In searching:– some information taken from output of a search is used to
do something with next query (search statement)• examine what you got to decide what to do next in searching
– a basic tactic in searching
• Several feedback types used in searching– each used for different decisions
Tefko Saracevic
37
Feedback types• Content relevance feedback
– judge relevance of items retrieved– make decision what to do next
• switch files, change exhaustivity …
• Term relevance feedback– find relevant documents– examine what other terms used in those documents – search using additional terms
• also called query modification & in some systems done automatically
• Magnitude feedback– on the basis of size of output make tactical decisions
• often the size so big that documents are not examined but next search done to limit size
Tefko Saracevic
38
Feedback types (cont.)• Tactical review feedback
– after a number of queries (search statements) in the same search review tactics as to getting desired outputs
• review terms, logic, limits …– change tactics accordingly
• Strategic review feedback– after a while (or after consultation with user) review the
“big” picture on what searched and how• sources, terms, relevant documents, need satisfaction, changes in
question, query …– do next searches accordingly– used in reiterative searching
Tefko Saracevic
But then
• There are other ways to look at searching where many of these things are combined
• Here is one way that looks at the whole search as a process of complex wandering
• Shifting exploration & feedback are implied
Tefko Saracevic 39
40
Bates’ Berry-picking model of searching
“…moving through many actions towards a general goal of satisfactory completion of research related to information need.”– query is shifting (continually)
• as search progresses queries are changing• different tactics are used
– searcher (user) may move through a variety of sources
• new files, resources may be used
– think of your last serious search • isn’t this what you were doing?
Tefko Saracevic
Elaborated byMarcia BatesUCLA
41
Berry-picking …
– closer to the real behavior of information searchers– new information may provide new ideas, new directions
• feedback is used in various ways
– question is not satisfied by a single set of answers, but by a series of selections & bits of information found along the way
• results may vary & may have to be provided in appropriate ways & means
– you go as if berry-picking in a field
Tefko Saracevic
A berry-picking evolving search(from the article)
Tefko Saracevic 42
“1) The nature of the query is an evolving one, rather than single and unchanging, and 2) the nature of the search process is such that it follows a berrypicking pattern, instead of leading to a single best retrieved set.”
Conclusionssearch tactics have to be mastered
• Users are not concerned about searching but about finding
• Effective searching is a prerequisite for finding the right stuff
• Search tactics are critical for effective searching• They need to be understood and followed
if you do X you can expect Yin order to get to Y you have to know what X to do
Tefko Saracevic 43
top related