Exploring the InternetTopic: Searching the Web

91.113-021Instructor: Michael Krolak

91.113-031Instructor: Patrick Krolak

See Lecturer notes http://www.cs.uml.edu/~pkrolak/lab18/lab18.html

Authors: P. D. & M. S. Krolak Copyright 2005

Search for Information on the Web

Finding information on the web requires some concepts of how the various types search engines work.

Archives that capture the changes in the documents on the web are highly useful for those in the social sciences, technology, and business dynamics.

“Intelligence is not the ability to store information, but to know where to find it.“

- Albert Einstein

How do we find information?

• Memory

• Media – Books– Movies– Music– Art

• Observe

• Ask other people

The Problem with the Internet

• The “Surface Web” contains 2.5 Billion pages.• Each day 7.5 million web pages are added to the World Wide Web• Information is submitted to the web without any context or test of

validity

The Archives of the Web

1. Archival of the Web’s websites

2. Google’s archive of the Internet newsgroups.

3. Google’s archive of the world’s newspaper archives

The Way Back Machine

• Frustrated by dead links – there is an answer. The WayBack Machine at http://www.archive.org/

• Just fill in the URL of the dead link and the links history will give the history of the link (how the page changed over time) and allow you to view the dead link.

Google’s Newsgroup archivehttp://groups.google.com/

• Archives over 100,000 groups• Goes back for some groups over 30 years.• There are for fee sites that provide competitive services.• Depending on the group it can provide a treasure trove

of insight into the cyber information society and it early history.

• Not all messages in the database are true, have merit or redeeming value, or are appropriate for children.

Google Newsgroup

Google has a major effort to archive newsgroups

Google’s News Project

Google plans to digitize the 250 years of published news and other data online.

Google’s news archivehttp://news.google.com/archivesearch

• Google is creating an archive that will provide a free search of both free and fee publishers archives

• The goal is to provide search for articles going back to 1700.

• The project involves a partnership of many major information publishers

Google’s News archive

• The archive is located at:

http://news.google.com/archivesearch

• The archive works like any other Google search and the results can be requested in time line order.

Searching for Information on the web

1. Search engines

2. Directory searches

3. Meta-Search engines

What is a Search Engine?

search enginen. 1. A software program that searches a database and gathers and reports information that contains or is related to specified terms.

2. A website whose primary function is providing a search engine for gathering and reporting information available on the Internet or a portion of the Internet.

Source: The American Heritage® DictionaryCopyright © 2002, 2001, 1995 by Houghton Mifflin Company. Published by Houghton Mifflin Company.

Search Engines

Search engines have two parts:

1. The search sends out onto the Internet a software called a spider or bot (robot).

• Traces all the links and returns all the pages found.• The pages are characterized by algorithms and stored in

databases

2. The retrieval system that takes a query and maps against the databases.

• The retrieval rank orders the responses by relevance• Each search engine uses a unique technique for retrieval and

ranking.

What is a spider?

n.

1. An automated program which crawls over the World Wide Web, gathering web pages for search engines. Spiders will ignore sites that explicitly state not be indexed by the search engines.

Also referred to as a webcrawler, crawler, or bot

What are Meta Tags

meta tags

n.

1. Attributes that describe information about the content of the document. Some spiders use these tags to determine the relevance of a site to future queries.

Example

<META NAME="keywords" CONTENT=“red sox world champions schilling manny damon">

How do search engines work?

Meta Search Engines

• Meta search engines are search engines that use their own resources for answering the question

• but they mostly form the query from the user input and package it and send it off  to many other search engines simultaneously (the process is called spawning) and then wait until the replies come back.

• After a fixed time the meta takes the responses received and pulls them together into a report.

• There are many ways to create a meta search based on the idea. Some allow you to search only the web, others newsgroups, newspapers, and scientific journal.

Why is an understanding of how a search engine works important?

• From the view of a user:– The user wants to find the information with as few downloads

as possible. – The easier to use and the more accurate the ranking the better.

• From the view of a web site developer:– The developer wants the site to found by in the first 5-10

ranked responses to a query.– The merit of a web design is often based on the search

rankings. This requires a knowledge of a given search engine ranks a page.

When in doubt ask a librarian:

• The librarian is a trained professional and are well versed in using the various WWW resources for finding answers to a vast array of subjects.

• The librarian should be used for difficult searches; but the student will wisely observe, learn, and contemplate the librarian's techniques, resources, and methods.

What is a Subject Directory?

subject directoryn. 1. An Internet research tool on the World Wide Web that organizes Internet resources by subject headings and subheadings. Subject directories are usually compiled by human beings who apply some selection criteria to resources included in the database.

Examples of Subject Directories

• www.yahoo.com Yahoo!

• http://bubl.ac.uk/ BUBL

• http://www.ipl.org/ Internet Public Library

• www.about.com About.com

• www.jumpcity.com Jump City

• http://www.joeant.com/ Joe Ant

What is a Meta Search Engine?

meta search enginen. 1. Meta search engines are search engines that use their own database as well as sending the query to many other search engines simultaneously (called spawning) and report the unique responses from other search engines.2. Meta search engines that are limited to only the web, newsgroups, newspapers, and scientific journals.

Examples of Meta Search Engines

• Ask Jeeves -- frequently get the answer in the first pass. Jeeves allows queries in natural language.

• Dogpile  -- for its variety of sources (web, newsgroups, newspapers)

• Ixquick

• Metacrawler

• ProFusion

Current Information on search engines

• Search Engine Watch Is a source for comparing search engines and keeping up with innovations as they occur in the field.

• Recently Google was asked to turn over records about their customers search topics and the number of times pornographic information was accessed. The Federal government was looking to prove its case for protecting children. MSN and Yahoo had complied with the request.

• Google also recently complied with the Chinese government request to censure political inquiries on the Chinese version of Google.

The Deep Web

What is the invisible or deep web

• Invisible Web (n.) Also referred to as the deep Web, the term refers to either Web pages that cannot be indexed by a typical search engine or Web pages that a search engine purposely does not index, rendering the data “invisible” to the general user. One of the most common reasons that a Web site’s content is not indexed is because of the site’s use of dynamic databases, which opens the door for a potential spider trap. Web pages can also fall into the invisible Web if there are no links leading to them, since search engine spiders typically crawl through links that lead them from one destination to another. Data on the invisible Web is not inaccessible; the information is out there—it is stored on a Web server somewhere and can be accessed using a browser—but the data must be found using means other than the general-purpose search engines, such as Google and Yahoo!.

Source: http://www.webopedia.com/TERM/I/invisible_Web.html

The deep web

• The deep web is not mysterious, it simply means that normal search engines that use spiders that go from one link to another will not work with pages that are generated on the fly from data requested from a database, or not linked to other data, etc.

• Example of a deep website are the yellow or white pages, catalogues, and patents.

• Google can index search pdf, text, and word documents

What is the Deep Web?

• Estimated to be 500 times (1.25 trillion web sites) the size of the surface web.

Using the Search tools to find information of the web

Successful searching

Plan your search:1. What are the words that will only be on the right

web page. Should they all be there or are there alternatives. The most specific concept is the best.

2. If you do not know your ideal topic well, use a meta search engine to get the smart. Then refine your search with a search engine like google or altavista.

3. Use a virtual library site to find information reviewed by experts if it is technical.

What is Boolean Logic?

We use Boolean Logic to evaluate the truth of one or more propositions. There are three important operators: AND, OR, NOT

•AND – only true if A and B are both true.•OR - only true if either A or B is true.•NOT - only true when A is false.

When searching for information, we use Boolean logic to find results that are relevant to our search terms. If a web page is relevant to a search term, the search engine evaluates the page as true.

Examples of Searching with Boolean Logic

• Yankees and Choke– All web pages that contain the

terms Yankees and Choke.• Yankees or Choke

– All web pages that contain the word Yankees.

– All web pages that contain the word Choke

– All web pages that contain the terms Yankees and Choke

• Choke and not Yankees– All web pages that contain the

word Choke, but don’t contain the word Yankees

More Advanced Usesof Boolean Logic

• If you are looking for a proper name, a phrase, or an other collection of words that normally are found together, then enclose them in double quotes, i.e. "President Gerald Ford".

• If the web page should have one or more words that must be on the page, then use the logical And, i.e. President And Ford And "United States".

• If the web page may have different forms of the name, or titles, etc. then use the logical Or, i.e. President Or "Vice President" Or Representative And "Gerald Ford".

• If document should exclude a word or phrase, then use the logical Not, i.e. "Gerald Ford" Not "Ford automotive" and Not "Ford car" and Not "Ford truck".

Other Helpful Hints

• While not Boolean logic, some search engines allow concepts like -- NEAR and FOLLOWED BY are also allowed, to indicate the relationship of the words or phrases other words and phrases. Normally these relations can be which comes first or whether the word is within a certain number of words to the first word. This concept is called proximity logic.

• Not all search engines use the  AND, OR, NOT notation some like Alta Vista use " +"  for AND and "-" for NOT.

Tips for Using Search Engines

• When searching for a large scale database, it is important to be extremely precise.

• Avoid using vague or common words that will only produce millions of pages.

• Read the instructions for each new search engine you use. There are many different methods of searching between the search engines and subject directories.

Finding Audio and Video

• http://www.alltheweb.com/ video, audio, news• http://images.google.com – Good source of

images• www.dogpile.com – One of the few search

engines that provides searches for video.• www.fazzle.com – Provides limited video and

image searching capabilities• http://video.google.com/ -- A new beta product

may have bugs.• http://www.tssphoto.com/ Stock photo images

Finding Movies and Films

• http://www.archive.org/details/movies

Sources of Audio Information

• An Audio Archive (software & Music, etc.) http://archive.museophile.org/audio/

• Speeches – http://www.historychannel.com/speeches/

Dogpile for finding non-text based files

The number of sites that allow so called “anonyms or guest” ftp directories is now greatly diminished. Due to security considerations most sites do not have non-text directories that are open to search and file download. Hence Dogpile no longer maintains a search engine that can find files in ftp sites, but it still allows searches for images, audio, and videos.

Similarly tools like Archie and Gopher are now obsolete