searching intelligently: it’s no longer a nightmare

Bill G. Kelm - July 21, 2006

Searching Intelligently:Searching Intelligently: It’s No Longer a Nightmare It’s No Longer a Nightmare

Oregon Library Support Staff Division Gateways 2006 Conference


Today’s Goals – To LearnToday’s Goals – To Learn

How is the web indexed?– Google in particular.

Which tool to use?– Search engines, directories, hidden web,

listservs and online discussion groups.Drawbacks and advantages of the Web.Browser tips and research power tools.Horizontal searching.


How Search Engines WorkHow Search Engines Work

1. Discovery and Database

2. User Search

3. Presentation and Ranking

Source: http://www.webreference.com/content/search/


Google BackgroundGoogle Background“Google's mission is to organize the world's

information and make it universally accessible and useful.”

Google's founders Larry Page and Sergey Brin developed Google in a Stanford University dorm room and it is currently the world's largest search engine.

Source: http://www.google.com/corporate/


Google’s Discovery and Google’s Discovery and DatabaseDatabase

Google has programs called spiders (a.k.a. Google bots) constantly searching the web for new or updated web pages.

When a spider finds a new or updated page, it reads that entire page, reports back to Google, and then visits all of the other pages to which that new page links.


Google’s CacheGoogle’s CacheWhen the spider reports back to Google, it

doesn’t just tell Google the new or updated page’s URL.

The spider also sends Google a complete copy of the entire Web page – HTML, text, images, etc.

Google then adds that page and all of its content to Google’s cache.


How Google WorksHow Google WorksWhen you search for multiple keywords,

Google first searches for all of your keywords as a phrase.

So, if your keywords are baseball spring training, any pages on which those words appear as a phrase receive a score of X.


Google – Adjacency Google – Adjacency Google then measures the adjacency

between your keywords and gives those pages a score of Y.

A page with “baseball spring training” next to each other gets a higher score than one with “baseball” and then “spring training” farther down the page.


Google - WeightsGoogle - WeightsThen, Google measures the number of times

your keywords appear on the page (the keywords’ “weights”) and gives those pages a score of Z.


Presentation & RankingPresentation & Ranking Google takes

– The phrase hits (the Xs), – The adjacency hits (the Ys), – The weights hits (the Zs), and – About 100 other secret variables

Throws out everything but the top 2,000 Multiplies each remaining page’s individual score

by it’s “PageRank” And, finally, displays the top 1,000 in order.


Google – PageRank?Google – PageRank?There is a premise in higher education that

the importance of a research article can be judged by the number of citations to it from subsequent articles in the same field.

Google applies this premise to the Web: the importance of a Web page can be judged by the number of hyperlinks pointing to it from other pages.


Google Advanced & TricksGoogle Advanced & Tricks

CalculatorDefine~, +, -Advanced SearchingFinding Information on the Internet a

Tutorial


My Favorite Quote:My Favorite Quote:“Focus on users and their tasks, not the

technology.” – Jeff Johnson


When Searching the Web:When Searching the Web:

“Focus on your query, not the technology.”


Four tools:Four tools:1. Search Engines

2. Directories

3. Invisible Web (Deep Web)

4. Listservs and Online Discussion Groups


Which Tool to Use? Which Tool to Use? “It all Depends.”


When to use a Search Engine:When to use a Search Engine:

You are looking for the “Society of American Registered Architects.”

You have a specific phrase or unique keyword.


Which Search Engines are UsedWhich Search Engines are Used

Can you guess what percentage of people use the various search engines available?


Share of Searches 2005Share of Searches 2005

Source: http://searchenginewatch.com/reports/article.php/2156451


Rating Search EnginesRating Search EnginesSearch Engine Watch

Search Engine Showdown


Problems With Search Engines:Problems With Search Engines:

Speed response eliminates some documentsBias toward textUser expectation and skillsCosts of crawlingMetasearch engine: jux2


When to Use a Directory:When to Use a Directory:

“I’m looking for sites on American Architecture.”

Broad categoryEarly in your researchOpposing viewpoints


Sample DirectoriesSample DirectoriesGoogle DirectoryInternet Scout ProjectInternet Resources Columns

Targeted Directories: Classics Resources


Problems With Directories:Problems With Directories:

SmallEditorial policiesTimelinessCharging for listing


Hidden/Invisible WebHidden/Invisible WebSearchable databasesExcluded pages


When to use the Invisible Web:When to use the Invisible Web:

“I’m looking for a list of architects in Baltimore.”

“I need a specific statistic on the death rate of women with heart disease in 2002.”

“I’m looking for information on a plane crash in Salem, OR in 1979.”


How to Find the Hidden WebHow to Find the Hidden Web

Google:– Databases + your topic

Searching general web directories– Librarians Index– Infomine


When to use a Listserv?When to use a Listserv?

“If I’m looking for an opinion on a particular topic.”


How to Find a Listserv:How to Find a Listserv:Tile.netGoogle: “topic” and listservGoogle Groups


Browser Tips & ToolsBrowser Tips & ToolsBookmarks (personal toolbars)HistoryConQuery (search plugins)

– Journal Title List– Creative Commons– Open WorldCat via Google

BookmarkletsTabs, Tabs, Tabs


Horizontal SearchingHorizontal SearchingUse the web in conjunction with library

catalogs and databases.Search the Web for titles of articles.Locate more bibliographies that can be

incorporated into new searches for books, journal articles, etc.

Search for authors from books and articles.


Horizontal Searching:Horizontal Searching:Search a Library Database Search a Library Database


Horizontal Searching:Horizontal Searching:Search title of article on the WebSearch title of article on the Web


Horizontal Searching:Horizontal Searching:Follow citations from Web siteFollow citations from Web site


Horizontal Searching:Horizontal Searching:Search Book Title in the Library CatalogSearch Book Title in the Library Catalog


Horizontal Searching:Horizontal Searching:Follow subject headings from articleFollow subject headings from article


Horizontal Searching:Horizontal Searching:Follow cited references / and searchFollow cited references / and search


Horizontal Searching:Horizontal Searching:Organization Web sites and Official ReportsOrganization Web sites and Official Reports


Horizontal Searching:Horizontal Searching:Contact actual researchers on the topicContact actual researchers on the topic


Wrap Up:Wrap Up: Know how the web is indexed and collected. Choose the correct tool for your question. Realize more than one tool may be needed. Carefully evaluate whatever you find on the Web. Think horizontally in searching: library databases,

Web, bibliography, Web, library catalog, Web, reference book, Web…


BibliographyBibliography Cohen Laura (2001) 10 tips for teaching how to

surf the Web. American Libraries, 32, 44-46.

Sherman, C., Price, G. (2001). The Invisible Web: Uncovering Information Sources Search Engines Can't See. Medford, N.J.: Information Today, Inc.

Dale Vidmar’s: Horizontal Searching Linda Goff’s: Googling to the Max

searching intelligently: it’s no longer a nightmare

Documents

google adjacency google

google bots

google workswhen

google weightsthen

google pagerank

updated web pages

entire page

entire web page html