History of Internet Searching

The Internet

Built in 1969.Built in 1969. Funded by the DoD for scientific research, Funded by the DoD for scientific research,

built by BBN Technologies.built by BBN Technologies. Originally called Originally called ARPANETARPANET

AAdvanced dvanced RResearch esearch PProjects rojects AAgencygency First nodes (connections) were at First nodes (connections) were at

universities (UCLA, UCSB, Stanford, Univ. universities (UCLA, UCSB, Stanford, Univ. of Utah)of Utah)

Bolt, Beranek, and Newman (BBN)

Ray TomlinsonInvented Email in 1971.Invented Email in 1971.

Wasn’t supposed to be working on it, he thought it would be a Wasn’t supposed to be working on it, he thought it would be a ““neat ideaneat idea””..

The Internet

Computers that made up the ARPANET were Computers that made up the ARPANET were called called IMPIMP’’ss (Internet Message Processor) (Internet Message Processor)

By 1971 there were 23 sites on the ‘net.

Protocols

Methods of using the Internet:Methods of using the Internet: Telnet – Access and Control ComputersTelnet – Access and Control Computers FTP – File Transfer ProtocolFTP – File Transfer Protocol HTTP – HyperText Transfer ProtocolHTTP – HyperText Transfer Protocol Gopher – File Access & DownloadingGopher – File Access & Downloading EmailEmail

History of Internet Searching

FTP – File Transfer ProtocolFTP – File Transfer Protocol Protocol established in 1985.Protocol established in 1985. FTP Servers provide files to FTP ClientsFTP Servers provide files to FTP Clients

Problems with FTP No organization of FTP Servers User had to know an FTP Server existed User had to visit FTP Server to see files

History of Internet Searching

ARCHIEARCHIE 1990 (No WWW)1990 (No WWW) Alan Emtage @ McGill Univ. in MontrealAlan Emtage @ McGill Univ. in Montreal Searchable directory of FTP filesSearchable directory of FTP files Searched FTP Servers and indexed their filesSearched FTP Servers and indexed their files User searched the IndexUser searched the Index Required Telnet and FTPRequired Telnet and FTP

History of Internet Searching

GopherGopher 1991 (WWW Began)1991 (WWW Began) Paul Lindner & Mark P. McCahill of Paul Lindner & Mark P. McCahill of

Univ. of MinnesotaUniv. of Minnesota Named after the Univ. of Minn. MascotNamed after the Univ. of Minn. Mascot Connected Gopher servers through the Connected Gopher servers through the

Gopher hierarchy (gopherspace)Gopher hierarchy (gopherspace)

History of Internet Searching

WandererWanderer (Matthew Gray(Matthew Gray’’s World Wide s World Wide Web Wanderer)Web Wanderer) First WWW EngineFirst WWW Engine Designed to track the size of the WWWDesigned to track the size of the WWW Captured URLCaptured URL’’s and entered into s and entered into

database (Wandex)database (Wandex) First Robots First Robots ““botsbots””

Search Engine Technology

Three parts to a Search EngineThree parts to a Search Engine

Bots (Robots)Bots (Robots)DatabaseDatabaseUser InterfaceUser Interface

Search Engine Technology

Bots (Robots)Bots (Robots) Also called SpidersAlso called Spiders Computer programs sent out by Query Computer programs sent out by Query

ServersServers Search the Internet for serversSearch the Internet for servers Identify servers & collect informationIdentify servers & collect information Uses links from websites to find other sitesUses links from websites to find other sites

Search Engine Technology

DatabaseDatabase Collects the information from Query Collects the information from Query

Server and organizes it.Server and organizes it.

Search Engine Technology

User InterfaceUser Interface Allows users to search the database and Allows users to search the database and

returns the information from it.returns the information from it.

Search Engine Technology

Relevance RankingRelevance Ranking Search engine measures the relevance of Search engine measures the relevance of

the information found to your requestthe information found to your request First search engine to use Relevance First search engine to use Relevance

Ranking was the Repository-Based Ranking was the Repository-Based Software Engine (RBSE) in 1993Software Engine (RBSE) in 1993

Search Engine Technology

Relevance Ranking (Techniques)Relevance Ranking (Techniques) How often do the search terms appearHow often do the search terms appear How close are the search terms to each How close are the search terms to each

otherother Where do the search terms appearWhere do the search terms appear How often do the search terms appear How often do the search terms appear

compared to the length of the web pagecompared to the length of the web page