subject access to collections: introduction
DESCRIPTION
Subject Access to Collections: Introduction. University of California, Berkeley School of Information IS 245: Organization of Information In Collections. Review. Review of Description - PowerPoint PPT PresentationTRANSCRIPT
IS 257 – Fall 2007 2007.03.21 - SLIDE 1
Subject Access to Collections: Introduction
University of California, BerkeleySchool of Information
IS 245: Organization of Information In Collections
IS 257 – Fall 2007 2007.03.21 - SLIDE 2
Review
• Review of Description• Goal of IR is to retrieve all and only the
“relevant” documents in a collection for a particular user with a particular need for information
IS 257 – Fall 2007 2007.03.21 - SLIDE 3
Indexing Languages and Thesauri
• Origins and Uses of Controlled Vocabularies for Information Retrieval
• Types of Indexing Languages, Thesauri and Classification Systems
IS 257 – Fall 2007 2007.03.21 - SLIDE 4
Controlled Vocabularies
• Vocabulary control is the attempt to provide a standardized and consistent set of terms (such as subject headings, names, classifications, etc.) with the intent of aiding the searcher in finding information.
IS 257 – Fall 2007 2007.03.21 - SLIDE 5
What is a “Controlled Vocabulary”• “The greatest problem of today is how to teach
people to ignore the irrelevant, how to refuse to know things, before they are suffocated. For too many facts are as bad as none at all.” (W.H. Auden)
• Similarly, there are too many ways of expressing or explaining the topic of a document.
• Controlled vocabularies are sets of Rules for topic identification and indexing, and a THESAURUS, which consists of “lead-in vocabulary” and an limited and selective “Indexing Language” sometimes with special coding or structures.
IS 257 – Fall 2007 2007.03.21 - SLIDE 6
Uses of Controlled Vocabularies• Library Subject Headings, Classification and
Name Authority Files.• Commercial Journal Indexing Services and
databases• Yahoo, and other Web classification
schemes• Online and Manual Systems within
organizations– SunSolve– MacArthur
IS 257 – Fall 2007 2007.03.21 - SLIDE 7
Name Authority Files
ID:NAFL8057230 ST:p EL:n STH:a MS:c UIP:a TD:19910821174242 KRC:a NMU:a CRC:c UPN:a SBU:a SBC:a DID:n DF:05-14-80 RFE:a CSC: SRU:b SRT:n SRN:n TSS: TGA:? ROM:? MOD: VST:d 08-21-91 Other Versions: earlier 040 DLC$cDLC$dDLC$dOCoLC 053 PR6005.R517 100 10 Creasey, John 400 10 Cooke, M. E. 400 10 Cooke, Margaret,$d1908-1973 400 10 Cooper, Henry St. John,$d1908-1973 400 00 Credo,$d1908-1973 400 10 Fecamps, Elise 400 10 Gill, Patrick,$d1908-1973 400 10 Hope, Brian,$d1908-1973 400 10 Hughes, Colin,$d1908-1973 400 10 Marsden, James 400 10 Matheson, Rodney 400 10 Ranger, Ken 400 20 St. John, Henry,$d1908-1973 400 10 Wilde, Jimmy 500 10 $wnnnc$aAshe, Gordon,$d1908-1973
Different names for thesame person
IS 257 – Fall 2007 2007.03.21 - SLIDE 8
Name Authority Files
ID:NAFO9114111 ST:p EL:n STH:a MS:n UIP:a TD:19910817053048 KRC:a NMU:a CRC:c UPN:a SBU:a SBC:a DID:n DF:06-03-91 RFE:a CSC:c SRU:b SRT:n SRN:n TSS: TGA:? ROM:? MOD: VST:d 08-19-91 040 OCoLC$cOCoLC 100 10 Marric, J. J.,$d1908-1973 500 10 $wnnnc$aCreasey, John 663 Works by this author are entered under the name used in the item. For a listing of other names used by this author, search also under$bCrease y, John 670 OCLC 13441825: His Gideon's day, 1955$b(hdg.: Creasey, John; usage: J .J. Marric) 670 LC data base, 6/10/91$b(hdg.: Creasey, John; usage: J.J. Marric) 670 Pseuds. and nicknames dict., c1987$b(Creasey, John, 1908-1973; Britis h author; pseud.: Marric, J. J.)
IS 257 – Fall 2007 2007.03.21 - SLIDE 9
Name authority files
ID:NAFL8166762 ST:p EL:n STH:a MS:c UIP:a TD:19910604053124 KRC:a NMU:a CRC:c UPN:a SBU:a SBC:a DID:n DF:08-20-81 RFE:a CSC: SRU:b SRT:n SRN:n TSS: TGA:? ROM:? MOD: VST:d 06-06-91 Other Versions: earlier 040 DLC$cDLC$dDLC$dOCoLC 100 10 Butler, William Vivian,$d1927- 400 10 Butler, W. V.$q(William Vivian),$d1927- 400 10 Marric, J. J.,$d1927- 670 His The durable desperadoes, 1973. 670 His The young detective's handbook, c1981:$bt.p. (W.V. Butler) 670 His Gideon's way, 1986:$bCIP t.p. (William Vivian Butler writing as J .J. Marric)
Different people writing with the same name
IS 257 – Fall 2007 2007.03.21 - SLIDE 10
Indexing Languages
• An index is a systematic guide designed to indicate topics or features of documents in order to facilitate retrieval of documents or parts of documents.
• An Indexing language is the set of terms used in an index to represent topics or features of documents, and the rules for combining or using those terms.
IS 257 – Fall 2007 2007.03.21 - SLIDE 11
Types of Indexing Languages
• Uncontrolled Keyword Indexing
• Indexing Languages– Controlled, but not structured
• Thesauri– Controlled and Structured
• Classification Systems– Controlled, Structured, and Coded
• Faceted Classification Systems
IS 257 – Fall 2007 2007.03.21 - SLIDE 12
Indexing Languages
• Library of Congress Subject Headings
• Yellow Pages Topics
• Wilson Indexes (“Reader’s Guide”)
IS 257 – Fall 2007 2007.03.21 - SLIDE 13
Controlled Vocabulary
• Start with the text of the document• Attempt to “control” or regularize:
– The concepts expressed within• mutually exclusive• exhaustive
– The language used to express those concepts• limit the normal linguistic variations• regulate word order and structure of phrases• reduce the number of synonyms or near-synonyms
• Also, provide cross-references between concepts and their expression.
See Bates, 1988
IS 257 – Fall 2007 2007.03.21 - SLIDE 14
Subject Headings vs. Descriptors• Describe the contents
of an entire document• Designed to be
looked up in an alphabetical index– Look up document
under its heading
• Few (1-5) headings per document
• Describe one concept within a document
• Designed to be used in Boolean searching– Combine to describe
the desired document
• Many (5-25) descriptors per document
IS 257 – Fall 2007 2007.03.21 - SLIDE 15
Subject Heading vs. Descriptor Example
• WILSONLINE– Athletes– Athletes--Heath&Hygiene– Athletes--Nutrition– Athletes--Physical Exams– …– Athletics– Athletics -- Administration– Athletics -- Equipment --
Catalogs– …– Sports -- Accidents and
injuries– Sports -- Accidents and
injuries -- prevention
• ERIC– Athletes– Athletic Coaches– Athletic Equipment– Athletic Fields– Athletics– …– Sports psychology– Sportsmanship
IS 257 – Fall 2007 2007.03.21 - SLIDE 16
Assigning Headings vs. Descriptors
• Subject headings -- assign one (or a few) complex heading(s) to the document
• Descriptors -- mix and match– How would we describe recipes using each
technique?
IS 257 – Fall 2007 2007.03.21 - SLIDE 17
Thesauri
• A Thesaurus is a collection of selected vocabulary (preferred terms or descriptors) with links among Synonymous, Equivalent, Broader, Narrower and other Related Terms
IS 257 – Fall 2007 2007.03.21 - SLIDE 18
Thesauri (cont.)
• National and International Standards for Thesauri– ANSI/NISO z39.19--1994 -- American National
Standard Guidelines for the Construction, Format and Management of Monolingual Thesauri
– ANSI/NISO Draft Standard Z39.4-199x -- American National Standard Guidelines for Indexes in Information Retrieval
– ISO 2788 -- Documentation -- Guidelines for the establishment and development of monolingual thesauri
– ISO 5964-- Documentation -- Guidelines for the establishment and development of multilingual thesauri
IS 257 – Fall 2007 2007.03.21 - SLIDE 19
Thesauri (cont.)
• Examples:– The ERIC Thesaurus of Descriptors– The Art and Architecture Thesaurus– The Medical Subject Headings (MESH) of the
National Library of Medicine
IS 257 – Fall 2007 2007.03.21 - SLIDE 20
Development of a Thesaurus
• Term Selection.
• Merging and Development of Concept Classes.
• Definition of Broad Subject Fields and Subfields.
• Development of Classificatory structure
• Review, Testing, Application, Revision.
IS 257 – Fall 2007 2007.03.21 - SLIDE 21
Categorization Summary
• Processes of categorization underlie many of the issues having to do with information organization
• Categorization is messier than our computer systems would like
• Human categories have graded membership, consisting of family resemblances.
• Family resemblance is expressed in part by which subset of features are shared
• It is also determined by underlying understandings of the world that do not get represented in most systems
IS 257 – Fall 2007 2007.03.21 - SLIDE 22
Classification Systems
• A classification system is an indexing language often based on a broad ordering of topical areas. Thesauri and classification systems both use this broad ordering and maintain a structure of broader, narrower, and related topics. Classification schemes commonly use a coded notation for representing a topic and it’s place in relation to other terms.
IS 257 – Fall 2007 2007.03.21 - SLIDE 23
Classification Systems (cont.)
• Examples:– The Library of Congress Classification System– The Dewey Decimal Classification System– The ACM Computing Reviews Categories– The American Mathematical Society
Classification System
IS 257 – Fall 2007 2007.03.21 - SLIDE 24
Classification Schemes
• Classify possible concepts.
• Goals:– Completely distinct conceptual categories
(mutually exclusive)– Complete coverage of conceptual categories
(exhaustive)
IS 257 – Fall 2007 2007.03.21 - SLIDE 25
Hierarchical Classification
• Traditional “family-tree” – Each category is successively broken down
into smaller and smaller subdivisions– Each level divided out by a “character of
division”. Also known as a feature.• Example: distinguish Literature based on:
– Language– Genre– Time Period
IS 257 – Fall 2007 2007.03.21 - SLIDE 26
Hierarchical Classification
Literature
SpanishFrenchEnglish
DramaPoetryProse
18th17th16th
DramaPoetryProse
19th 18th17th16th 19th
...
... ... ...
...
IS 257 – Fall 2007 2007.03.21 - SLIDE 27
Labeled Categories for Hierarchical Classification• LITERATURE
– 100 English Literature• 110 English Prose
– English Prose 16th Century– English Prose 17th Century– English Prose 18th Century– ...
• 120 English Poetry– 121 English Poetry 16th Century– 122 English Poetry 17th Century– ...
• 130 English Drama– 130 English Drama 16th Century– …
– 200 French Literature
IS 257 – Fall 2007 2007.03.21 - SLIDE 28
Faceted Classification
• Create a separate, free-standing list for each characteristic of division (feature).
• Combine features to create a classification.
IS 257 – Fall 2007 2007.03.21 - SLIDE 29
Faceted Classification and Labeled Catgories
• A Language– a English– b French– c Spanish
• B Genre– a Prose– b Poetry– c Drama
• C Period– a 16th Century– b 17th Century– c 18th Century– d 19th Century
• Aa English Literature• AaBa English Prose• AaBaCa English
Prose 16th Century• AbBbCd French
Poetry 19th Century• BbCd Drama 19th
Century
IS 257 – Fall 2007 2007.03.21 - SLIDE 30
How to use such classification structures?
• How to look through them?
• How to use them in search?
IS 257 – Fall 2007 2007.03.21 - SLIDE 31
Automatic Indexing and Classification• Automatic indexing is typically the simple
deriving of keywords from a document and providing access to all of those words.
• More complex Automatic Indexing Systems attempt to select controlled vocabulary terms based on terms in the document.
• Automatic classification attempts to automatically group similar documents using either:– A fully automatic clustering method.– An established classification scheme and set of
documents already indexed by that scheme.
IS 257 – Fall 2007 2007.03.21 - SLIDE 32
Agglomerative Clustering
A B C D E F G HI
IS 257 – Fall 2007 2007.03.21 - SLIDE 33
Agglomerative Clustering
A B C D E F G HI
IS 257 – Fall 2007 2007.03.21 - SLIDE 34
Agglomerative Clustering
A B C D E F G HI
IS 257 – Fall 2007 2007.03.21 - SLIDE 35
Hierarchical Methods
2 .43 .4 .24 .3 .3 .35 .1 .4 .4 .1 1 2 3 4
Single Link Dissimilarity Matrix
Hierarchical methods: Polythetic, Usually Exclusive, OrderedClusters are order-independent
||||
||1
BA
BAitydissimilar
IS 257 – Fall 2007 2007.03.21 - SLIDE 36
Threshold = .1
Single Link Dissimilarity Matrix
2 .43 .4 .24 .3 .3 .35 .1 .4 .4 .1 1 2 3 4
2 03 0 04 0 0 05 1 0 0 1 1 2 3 4
2
1
35
4
IS 257 – Fall 2007 2007.03.21 - SLIDE 37
Threshold = .2
2 .43 .4 .24 .3 .3 .35 .1 .4 .4 .1 1 2 3 4
2 03 0 14 0 0 05 1 0 0 1 1 2 3 4
2
1
35
4
IS 257 – Fall 2007 2007.03.21 - SLIDE 38
Threshold = .3
2 .43 .4 .24 .3 .3 .35 .1 .4 .4 .1 1 2 3 4
2 03 0 14 1 1 15 1 0 0 1 1 2 3 4
2
1
35
4
IS 257 – Fall 2007 2007.03.21 - SLIDE 39
Clustering
Agglomerative methods: Polythetic, Exclusive or Overlapping, Unorderedclusters are order-dependent.
DocDoc
DocDoc
DocDoc
DocDoc
1. Select initial centers (I.e. seed the space)2. Assign docs to highest matching centers and compute centroids3. Reassign all documents to centroid(s)
Rocchio’s method
IS 257 – Fall 2007 2007.03.21 - SLIDE 40
Automatic Class Assignment
DocDoc
DocDoc
DocDoc
Doc
SearchEngine
1. Create pseudo-documents representing intellectually derived classes.2. Search using document contents3. Obtain ranked list4. Assign document to N categories ranked over threshold. OR assign to top-ranked category
Automatic Class Assignment: Polythetic, Exclusive or Overlapping, usually orderedclusters are order-independent, usually based on an intellectually derived scheme
IS 257 – Fall 2007 2007.03.21 - SLIDE 41
K-Means Clustering
• 1 Create a pair-wise similarity measure• 2 Find K centers using agglomerative clustering
– take a small sample – group bottom up until K groups found
• 3 Assign each document to nearest center, forming new clusters
• 4 Repeat 3 as necessary
IS 257 – Fall 2007 2007.03.21 - SLIDE 42
Scatter/Gather
• Cutting, Pedersen, Tukey & Karger 92, 93• Hearst & Pedersen 95
• Cluster sets of documents into general “themes”, like a table of contents
• Display the contents of the clusters by showing topical terms and typical titles
• User chooses subsets of the clusters and re-clusters the documents within
• Resulting new groups have different “themes”
IS 257 – Fall 2007 2007.03.21 - SLIDE 43
S/G Example: query on “star”
Encyclopedia text
14 sports
8 symbols 47 film, tv
68 film, tv (p) 7 music
97 astrophysics
67 astronomy(p) 12 steller phenomena
10 flora/fauna 49 galaxies, stars
29 constellations
7 miscelleneous
Clustering and re-clustering is entirely automated
IS 257 – Fall 2007 2007.03.21 - SLIDE 44
IS 257 – Fall 2007 2007.03.21 - SLIDE 45
IS 257 – Fall 2007 2007.03.21 - SLIDE 46