Identifying sets and Identifying sets and classes: taxonomies as classes: taxonomies as

finding aids finding aids

Alex HaigAlex Haig

NHS Education for ScotlandNHS Education for Scotland

2929thth September 2005 September 2005

A Case Study: A Case Study:

medical educationmedical education

What is a Taxonomy?What is a Taxonomy?

From the Greek From the Greek taxistaxis and and nomosnomos, (division , (division and law)and law)

““Division into ordered groups or Division into ordered groups or categories”categories”

Taxonomic schemas can be developed to Taxonomic schemas can be developed to order almost anythingorder almost anything

Carl Linnaeus (1701-1784)Carl Linnaeus (1701-1784)

Swedish botanistSwedish botanist12 volume 12 volume

Systema NaturaeSystema NaturaeDomain; Kingdom; Domain; Kingdom;

Phylum (animals) Phylum (animals) or Division or Division (plants); Class; (plants); Class; Order; Family; Order; Family; Genus; SpeciesGenus; Species

What is a Taxonomy?What is a Taxonomy?

Retain characteristics of classification, but Retain characteristics of classification, but not always based on standards used in not always based on standards used in library settingslibrary settings

““Taxonomies and vocabularies are Taxonomies and vocabularies are structured collections of terms that can structured collections of terms that can serve as values for the meta-data serve as values for the meta-data elements.”elements.” IMSIMS

Ironically Ambiguous …Ironically Ambiguous …

Controlled Vocabulary: (usually) enumerative list Controlled Vocabulary: (usually) enumerative list of all given terms/values in a subject areaof all given terms/values in a subject area

Taxonomy: top-down hierarchical arrangement Taxonomy: top-down hierarchical arrangement that does not necessarily define componentsthat does not necessarily define components

Thesaurus: defines components as well as Thesaurus: defines components as well as associative relationships; bound by international associative relationships; bound by international standardsstandards

Ontology: conceptual relationships (self-evident)Ontology: conceptual relationships (self-evident)

Controlled VocabularyControlled Vocabulary

(usually) enumerative list of all given terms/values in (usually) enumerative list of all given terms/values in a subject areaa subject area Arbovirus Infections Arbovirus Infections Bronchiolitis Bronchiolitis Encephalitis Encephalitis Eye Infections Eye Infections Fatigue Syndrome Fatigue Syndrome Hepatitis Hepatitis Meningitis Meningitis Pneumonia Pneumonia RNA Virus Infections RNA Virus Infections Sexually Transmitted Diseases Sexually Transmitted Diseases Skin Diseases Skin Diseases Tumor Virus Infections Tumor Virus Infections

TaxonomyTaxonomy

top-down hierarchical arrangement that does top-down hierarchical arrangement that does not necessarily define componentsnot necessarily define componentsViral DiseasesViral Diseases

HepatitisHepatitis Hepatitis AHepatitis A Hepatitis BHepatitis B Hepatitis CHepatitis C

Chronic Hepatitis CChronic Hepatitis C Hepatitis DHepatitis D Hepatitis EHepatitis E

ThesaurusThesaurus Chronic Fatigue SyndromeChronic Fatigue Syndrome: A syndrome characterized by persistent or : A syndrome characterized by persistent or

recurrent fatigue, diffuse musculoskeletal pain, sleep disturbances, and recurrent fatigue, diffuse musculoskeletal pain, sleep disturbances, and subjective cognitive impairment of 6 months duration or longer. Symptoms subjective cognitive impairment of 6 months duration or longer. Symptoms are not caused by ongoing exertion; are not relieved by rest; and result in a are not caused by ongoing exertion; are not relieved by rest; and result in a substantial reduction of previous levels of occupational, educational, social, or substantial reduction of previous levels of occupational, educational, social, or personal activitiespersonal activities

See Related:See Related: FIBROMYALGIA FIBROMYALGIA Used For:Used For: chronic fatigue syndrome chronic fatigue syndrome encephalomyelitis, myalgic encephalomyelitis, myalgic infectious mononucleosis-like syndrome, chronic infectious mononucleosis-like syndrome, chronic postviral fatigue syndrome postviral fatigue syndrome chronic fatigue disorder chronic fatigue disorder chronic fatigue and immune dysfunction syndrome chronic fatigue and immune dysfunction syndrome chronic fatigue-fibromyalgia syndrome chronic fatigue-fibromyalgia syndrome fatigue syndrome, postviral fatigue syndrome, postviral myalgic encephalomyelitis myalgic encephalomyelitis royal free disease royal free disease

OntologyOntology

Conceptual relationships (self-evident)Conceptual relationships (self-evident)Much more powerful way of describing Much more powerful way of describing

an entire domain in a variety of methodsan entire domain in a variety of methodsMetaphysical origins with the nature and Metaphysical origins with the nature and

relations of being relations of being Viral diseases by: aetiology (cause), Viral diseases by: aetiology (cause),

prognosis, diagnosis, protein prognosis, diagnosis, protein regulation, affectregulation, affect

Why Use or Create a Taxonomy?Why Use or Create a Taxonomy?

““I can call spirits from the vast deep.”I can call spirits from the vast deep.”

“Why, so can I, or so can any man; but “Why, so can I, or so can any man; but will they come when you do call for will they come when you do call for

them?”them?”

Shakespeare. Henry VI Part 1 3 iShakespeare. Henry VI Part 1 3 i

Where are taxonomies used?Where are taxonomies used?

A Case Study: A Case Study:

medical educationmedical education

(discipline wide effort)(discipline wide effort)

• Best Evidence Medical Education the dissemination of information which allows

medical teachers, institutions and all concerned with medical education to make decisions on the basis of the best evidence available

the production of appropriate systematic reviews of medical education which reflect the best evidence available and meet the needs of the user,

the creation of a culture of best evidence medical education amongst individual teachers, institutions and national bodies.

Searching for evidence in medical education …

… the need for a taxonomy …

Get a Measure of the Problem.

Association for Medical Education in Europe

Evidence retrieval in medical education: obstructions and

opportunities. Berlin, 2001.

Methods - topic

BEME pilot and consequent review groups (Barcelona/Tel Aviv)

Feedback in Assessment

Methods - software

• software used - Ovid [CGI version 7.8]

– permits design of rigorous strategies– consistency

Methods - databases selected

• Medline

• Embase

• ERIC

most relevant to medical education

Methods - journal selected

• limited time and resources• required a title that was most

comprehensively indexed• Academic Medicine 1996 - present

(2001)

Methods - strategies

• Three levels of strategy:I. standard (most users; limited search

syntax)II. enhanced (some use of search syntax)III. expert (full use of search syntax)

• syntax includes: free-text, controlled vocabulary, term explosions, phrase lists, subheadings, sub strings, filters, proximity operators, etc...

Methods - handsearching

• “...refers to the planned searching of a journal page by page (i.e. by hand), including editorials, letters, etc., to identify all relevant items.”

• time consuming and meticulous• produces the “gold standard” by

which search efficiency can be measured

Sensitivity

• Sensitivity (recall) - percentage of “gold standard”

• Sensitivity = total retrieved by search

total of the hand-search

Gold Gold Standard = Standard = 4646

Embase Sensitivity (n)Basic 4.3% 2Enhanced 10.7% 5Expert 15.2% 7

ERIC Sensitivity (n)Basic 0% 0Enhanced 4.3% 2Expert 6.5% 3

Medline Sensitivity (n)Basic 0% 0Enhanced 10.7% 5Expert 19.6% 9

GS=46GS=46

Specificity Specificity (precision) - positive

predictive value

Specificity = relevant records identified

total retrieved by search

Embase Specificity (n)Basic 40% 5Enhanced 33% 15Expert 30.4% 23

ERIC Specificity (n)Basic 0% 0Enhanced 40% 5Expert 37.5% 8

Medline Specificity (n)Basic 0% 0Enhanced 31.3% 16Expert 32.1% 28

A Note of Caution

• Academic MedicineAcademic Medicine is a journal that specialises in medical education :– more likely to be indexed for context– journal presents information for better retrieval

• Other journals will fare worse• Other specificity scores for BEME pilots

(not limited to one journal) ranged from 6 to 34%, with feedback in assessment at 17.8%

Reasons for shockingly poor

performance

1. Incomplete coverage of journals2. No indexed database for medical

education3. Existing controlled vocabularies are

inadequate for medical education

• Medical Education Taxonomy/Thesaurus Research Organisation

• Initial meeting in May 2002• Group originally coalesced around

special interest group discussing the subject area

Other Driving Factors

• GMC-driven reforms in 1990s highlight need for life-long learning and professionalisation of university teaching

• Consequently the literature expands (teachers, managers, researchers and students), with the expansion of medical education itself

• NHS Education for Scotland• University of Edinburgh• University of Newcastle• Royal College of Physicians (London)• University of Birmingham• Hull/York Medical School• University College London

Phases of Construction

1. Analysis / planning2. Design /development/ evaluation3. Implementation4. Maintenance

A Diverse Group enriches the entire effort

Funding

• Applied to LTSN01 for small grant funding

• £4000• Travel, communication and

dissemination

METRO 1 - ScopingMETRO 1 - Scoping

Prospective Applications• Primary - entities and processes directly

used in a medical educational setting: VLEs and frameworks such as Scottish Doctor & GMC’s Tomorrow’s Doctor

• Secondary - entities and processes involved with reporting and analysis: description, abstraction and synthesis of data; audit; evaluation

• Tertiary – applied to philosophical and ontological studies and activities surrounding medical education.

Stakeholder Contexts:education/e-learning

• Increased dependence on electronically supported activities and contexts

• RLOs• If there are to be efficiency gains

there needs to be robust semantic and symbolic representation of entities, activities, knowledge and competencies

Stakeholder Contexts: research

• BEME example: groups often spread across the world and divided by language and culture

• Taxonomy would aid: formulation of research question; evidence retrieval, data abstraction, data synthesis, publication and evaluation of results

• Both Primary and Secondary contexts

Review of Existing Schemas

• Structure: natural language (not), enumerative lists (e.g. glossary), semantically mono-dimension (taxonomy or ontology), semantic poly-dimension (thesaurus)

• Purpose: descriptive or indexing• Identity: non-controlled or controlled• Assche et al. IMS Global Learning

Consortia

Review of Existing Schemas

• BET: modelled on ERIC, health context at high level

• MeSH: global use; North American bias, freely maintained on behalf of users

• EMTREE: less educational depth than MeSH

• SNOMED: clinical specificity• EET: generalised educational thesaurus

Review of Existing Schemas

• IIME Glossary: Institute for International Medical Education in 2000 – Scoped not relational terms

(enumerative structure)– Many terms appear elsewhere

• Ninewells Thesaurus– For a print catalogue in early 1980s– Never piloted/evaluated– Many terms appear elsewhere

Why not use existing schemas?

• Nothing robust enough to support UK-specific contexts

• No comprehensive educational terms for undergraduate, postgraduate and CPD phases

• No comprehensive medical context• Yet BET and MESH were most

appropriate and stable

Pragmatic Approach

• Create a set of bridging terms and definitions between MeSH and BET, but only where terms are absent or require new definitions or extensions

• System must be dynamic– Contexts and cultures shift and evolve– Ongoing service not one-off product– Long-term viability means appropriate

rules and procedures

METRO Phase One: METRO Phase One: ProcessesProcesses

• Submission of seed terms• Discussion and agreement of terms• Initial scope notes from debate• Voting and resolution of scoped

terms and revision• Publishing of terms

Collaborative Work Environment

• CWE based on VLE at UoE Medical School– Commentary– Voting on terms; adding terms– Added and imbedded links and material– Forum– Simultaneous cross searching of MeSH,

BET and METRO

Phase One (Terms)

• 4 months, 180 terms, considered by 16 METRO members

• CWE enabled extremely valuable discussion

• Many terms too general/not specific, or synonymous with BET or MESH (dropped)

Phase One: Final Workshop• Typology of terms

– Generic education (BET)– Generic medicine (MeSH)– Role– Process– Teaching– Learning– Assessment– Periodicity– Design– Artefact

Knowledge Issues

• Procedures influence the product …• Was 16 reviewers enough?• Reasoning not always explicit – data

would have been illuminating• Pre/post coordination of terms• Limit to medical education?

Technical Issues

• How to render, apply and represent terms?• Structure hierarchically or flat• What data protocols and formats to employ• Interoperability is key: Zthes (protocol) and

XML (format), but will this accommodate all situations?

• Necessity to engage with wider interoperability debate for use in subscribing systems and applications

Organisational Issues

• By in from publishers/editors key• UK terms – transferability to other

English-speaking countries?• Need to encompass yet be sensitive• Terms can be divergent and even

contradictory within constituent communities

• Need single coherent entity

Procedural Issues

• All members were unpaid contributors

• One model would be to pay members or their organisations for work

• Relationship with MeSH• Relationship with BET

Investment of Time

METRO Phase 2

• A second batch of funding from LTSN01

• As with Phase 1, there was a core of 6 – 8 members, and slightly more in a supporting capacity

• Focus was on building the assessment branch of the thesaurus

Phase 2• Decision to submit terms for

inclusion in MeSH• Dictated the format:

– Heading– Tree Number (repeatable)– Annotation– Scope Note– See Also (repeatable)– Entry Term (repeatable)– Allowable Qualifiers– Previous Indexing (repeatable)– History Note

Phase 2

• Assessment seemed the logical place to start

• 104 terms: derived from Good Assessment Guide, Phase One, METRO members and interested parties, RCP seed terms, and existing schemas

• But where to start …• Design / Methods / Process

Working Environment

• There is software!• Blogger• Not the most user-friendly• Free and universally/publicly

accessible• http://metro2.blogspot.com/• (the nearly final version)

Creating Assessment Terms for MESH:

ConsiderationsDo we need a new MESH term?

• Assessment terms can be found in MESH branches other than Education e.g Behavioral Disciplines

• Existing terms and scope notes can be amended

• How likely is the concept you are describing likely to be encountered again

Creating Educational Terms for MESH:

Considerations (2)General rules:

• Identify a place in the MESH/METRO hierarchy where your proposed term will sit

• Terms should represent simple or unitary concepts as far as possible e.g. Portfolio

• Compound terms can also be created e.g. Test Reliability

• The number of simple concepts in a compound term should be kept to a minimum

Testing and Development 1

• Online Feedback• Approximately ten volunteers logged on and

recorded comments regarding the applicability of terms, scope notes and overall structure

• Several set practical exercises to see how the terms faired against their working areas

• Comments from non-logged-in persons appeared as well

• Differing backgrounds; debate; consolidation

Testing and Development 2

• Interactive Workshop• AMEE September 2004• Small group work to match

assessment articles against the vocabulary

• All comments from groups and the plenary discussion were reviewed and key revisions made

• Comprehensiveness and hierarchy

Testing and Development 3

• Inter-rater reliability• Following on from the workshop, an

exercise was developed to test IRR• Five objects (4 articles and 1

assessment instrument)• Five volunteers (each evaluate all 5)• Complete agreement on 4/5

Lessons from Phase 2

• Evolution and establishment of terminology demands flexibility and patience

• Consistency of description paramount• IRR improved by precise unambiguous

instruction, as with training• Success depends on continuing interest

and application by user community

Phases of Construction

1. Analysis / planning2. Design /development/ evaluation3. Implementation4. Maintenance

METRO - productsMETRO - products

Publications

• METRO--the creation of a taxonomy for medical education. Health Info Libr J. 2004 Dec;21(4):211-9. Haig A, Ellaway R, Dozier M, Liu D, McKendree J.

• METRO taxonomy -- progress report on assessment. Med Teach. 2005 Mar;27(2):155-7. Haig A, Dozier M, Liu D, McKendree J, Roper T, Selai C.

Applications

• Royal College of Physicians – catalogue at Education Resources Centre

• BEME Collaboration – data abstraction for research

• International Virtual Medical School (IVIMEDs) & other academic e-learning architectures

• NHS Education for Scotland – research database

Other Applications

• CoBaTrICE (Competency-Based Training in Intensive Care in Europe)

• AAMC curriculum database (CurrMIT)

• Ongoing international interest

What Next?

• Submission to Medline/National Library of Medicine

• Ensuring the effort continues …

“Folksonomy”

• user-generated classification, emerging through bottom-up consensus

• socially constructed classification schemes or communal categorisation

• Democratisation …

Folksonomy

• Del.icio.us., Flickr, and Furl are most famous examples with users categorising with tags (keywords)

• Reflect the language of users• Serendipity• Cheap – no training• Quick feedback

Fauxonomy?

• Grab-bag of keywords• Imprecision• No clearly defined relations between

terms• Ambiguity of “tags”• Lack of synonym control = same

concept, different tags• Trees are neat; piles of leaves are not.