Exploration versus Exploitation in Topic Driven Crawlers

Gautam Pant@ Padmini Srinivasan@!

Filippo Menczer@

@Department of Management Sciences!School of Library and Information Science

The University of IowaIowa City, IA 52242

{gautam-pant,padmini-srinivasan,filippo-menczer}@uiowa.edu

Abstract

The dynamic nature of the Web highlights thescalability limitations of universal search en-gines. Topic driven crawlers can address theproblem by distributing the crawling processacross users, queries, or even client computers.The context available to a topic driven crawlerallows for informed decisions about how to prior-itize the links to be visited. Here we focus on thebalance between a crawler’s need to exploit thisinformation to focus on the most promising links,and the need to explore links that appear sub-optimal but might lead to more relevant pages.We investigate the issue for two different tasks:(i) seeking new relevant pages starting from aknown relevant subset, and (ii) seeking relevantpages starting a few links away from the relevantsubset. Using a framework and a number of qual-ity metrics developed to evaluate topic drivencrawling algorithms in a fair way, we find that amix of exploitation and exploration is essentialfor both tasks, in spite of a penalty in the earlystage of the crawl.

1 Introduction

A recent projection estimates the size of the vis-ible Web today (March 2002) to be around 7billion “static” pages [10]. The largest searchengine, Google, claims to be “searching” about2 billion pages. The fraction of the Web cov-

ered by search engines has not improved muchover the past few years [16]. Even with increas-ing hardware and bandwidth resources at theirdisposal, search engines cannot keep up with thegrowth of the Web and with its rate of change[5].

These scalability limitations stem from searchengines’ attempt to crawl the whole Web, and toanswer any query from any user. Decentralizingthe crawling process is a more scalable approach,and bears the additional benefit that crawlerscan be driven by a rich context (topics, queries,user profiles) within which to interpret pages andselect the links to be visited. It comes as nosurprise, therefore, that the development of topicdriven crawler algorithms has received significantattention in recent years [9, 14, 8, 18, 1, 19].

Topic driven crawlers (also known as focusedcrawlers) respond to the particular informationneeds expressed by topical queries or interestprofiles. These could be the needs of an indi-vidual user (query time or online crawlers) orthose of a community with shared interests (top-ical search engines and portals).

Evaluation of topic driven crawlers is diffi-cult due to the lack of known relevant sets forWeb searches, to the presence of many conflict-ing page quality measures, and to the need forfair gauging of crawlers’ time and space algorith-mic complexity. In recent research we presentedan evaluation framework designed to support thecomparison of topic driven crawler algorithms

188

under specified resource constraints [19]. In thispaper we further this line of research by inves-tigating the relative merits of exploration versusexploitation as a defining characteristic of thecrawling mechanism.

The issue of exploitation versus exploration isa universal one in machine learning and artificialintelligence, since it presents itself in any taskwhere search is guided by quality estimations.Under some regularity assumption, one can as-sume that a measure of quality at one point inthe search space provides some information onthe quality of nearby points. A greedy algo-rithm can then exploit this information by con-centrating the search in the vicinity of the mostpromising points. However, this strategy canlead to missing other equally good or even bet-ter points, for two reasons: first, the estimatesmay be noisy; and second, the search space mayhave local optima that trap the algorithm andkeep it from locating global optima. In otherwords, it may be necessary to visit some “bad”points in order to arrive at the best ones. Atthe other extreme, algorithms that completelydisregard quality estimates and continue to ex-plore in a uniform or random fashion do not riskgetting stuck at local optima, but they do notuse the available information to bias the searchand thus may spend most of their time exploringsuboptimal areas. A balance between exploita-tion and exploration of clues is obviously calledfor in heuristic search algorithms, but the op-timal compromise point is unknown unless thetopology of the search space is well understood— which is typically not the case.

Topic driven crawlers fit into this picture verywell if one views the Web as the search space,with pages as points and neighborhoods as de-fined by hyperlinks. A crawler must decide whichpages to visit based on the cues provided by linksfrom nearby pages. If one assumes that a rele-vant page has a higher probability to be nearother relevant pages than to any random page,then quality estimate of pages provide cues thatcan be exploited to bias the search process. How-ever, given the short range of relevance clues onthe Web [17], a very relevant page might be onlya few links behind an apparently irrelevant one.

Balancing the exploitation of quality esti-mate information with exploration of subopti-mal pages is thus crucial for the performance oftopic driven crawlers. It is a question that westudy empirically with respect to two differenttasks. In the first, we seek relevant pages start-ing from a set of relevant links. Applications ofsuch a task are query-time search agents thatuse results of a search engine as starting pointsto provide a user with recent and personalizedresults. Since we start from relevant links, wemay expect an exploratory crawler to performreasonably well. The second task involves seek-ing relevant pages while starting the crawl fromlinks that are a few links away from a relevantsubset. Such a task may be a part of Web min-ing or competitive intelligence applications (e.g.,a search starting from competitors’ home pages).If we do not start from a known relevant subset,the appropriate balance of exploration vs. ex-ploitation becomes an empirical question.

2 Evaluation Framework

2.1 Topics, Examples and Neighbors

In order to evaluate crawler algorithms, weneed topics, some corresponding relevant exam-ples, and neighbors. The neighbors are URLsextracted from neighborhood of the examples.We obtain our topics from the Open Direc-tory (DMOZ). We ran randomized Breadth-Firstcrawls starting from each of the main cate-gories on the DMOZ site.1 The crawlers identifyDMOZ “leaves,” i.e., pages that have no childrencategory nodes. Leaves with five or more exter-nal links are then used to derive topics. We thuscollected 100 topics.

A topic is represented by three types of infor-mation derived from the corresponding leaf page.First, the words in the DMOZ hierarchy form thetopic’s keywords. Second, up to 10 external linksform the topic’s examples. Third, we concate-nate the text descriptions and anchor text of thetarget URLs (written by DMOZ human editors)to form a topic description. The difference be-

1http://dmoz.org

289

Table 1: A sample topic. The description is truncated for space limitations.Topic Keywords Topic Description ExamplesRecreation

Hot Air Ballooning

Organizations

Aerostat Society of Australia Varied collec-tion of photos and facts about ballooningin Australia, Airships, Parachutes, BalloonBuilding and more. Includes an article onthe Theory of Flight. Albuquerque AerostatAscension Association A comprehensive sitecovering a range of ballooning topics includ-ing the Albuqeurque Balloon Fiesta, local ed-ucation and safety programs, flying events,club activities and committees, and club his-tory. Arizona Hot Air Balloon Club [...]

http://www.ozemail.com.au/~p0gwil

http://www.hotairballooning.org/

http://www.ballooningaz.com

http://www.aristotle.net/~mikev/

http://www89.pair.com/techinfo/ABAC/abac.htm

http://www.prairienet.org/bagi

http://www.atu.com.au/~balloon/club1.html

http://communities.msn.com/BalloonCrews

http://www.bfa.net

http://www.ask.ne.jp/~kanako/ebst.html

tween topic keywords and topic descriptions isthat we give the former to the crawlers, as mod-els of (short) query-like topics, while we use thelatter, which are much more detailed representa-tions of the topics, to gauge the relevance of thecrawled pages in our post-hoc analysis. Table 1shows a sample topic.

The neighbors are obtained for each topicthrough the following process. For each of theexamples, we obtain the top 20 inlinks as re-turned by Google.2 Next, we get the top 20inlinks for each of the inlinks obtained earlier.Hence, if we had 10 examples to start with, wemay now have a maximum of 4000 unique URLs.A subset of 10 URLs is then picked at randomfrom this set. The links in such a subset arecalled the neighbors.

2.2 Architecture

We use the a previously proposed evaluationframework to compare different crawlers [19].The framework allows one to easily plug in mod-ules implementing arbitrary crawling algorithms,which share data structures and utilities to op-timize efficiency without affecting the fairness ofthe evaluation.

As mentioned before, we use the crawlersfor two different tasks. For the first task, thecrawlers start from the examples while for thesecond the starting points are the neighbors. Ineither case, as the pages are fetched their com-ponent URLs are added to a list that we call thefrontier. A crawler may use topic’s keywords to

2http://google.yahoo.com

guide the selection of frontier URLs that are tobe fetched at each iteration. For a given topic,a crawler is allowed to crawl up to MAX PAGES =2000 pages. However, a crawl may end sooner ifthe crawler’s frontier should become empty. Weuse a timeout of 10 seconds for Web downloads.Large pages are chopped so that we retrieve onlythe first 100 KB. The only protocol allowed isHTTP (with redirection allowed), and we alsofilter out all but “static pages” with text/htmlcontent. Stale links yielding HTTP error codesare removed as they are found (only good linksare used in the analysis).

We constrain the space resources a crawler al-gorithm can use by restricting the frontier size toMAX BUFFER = 256 URLs. If the buffer becomesfull then the crawler must decide which links areto be replaced as new links are added.

3 Crawling Algorithms

In this paper we study the notion of explo-ration versus exploitation. We begin with asingle family of crawler algorithms with a sin-gle greediness parameter to control the explo-ration/exploitation behavior. In our previous ex-periments [19] we found that a naive Best-Firstcrawler displayed the best performance amongthree crawlers considered. Hence, in this studywe explore variants of the Best-First crawler.More generally, we examine the Best-N-Firstfamily of crawlers where the parameter N con-trols the characteristic of interest.

Best-First crawlers have been studied before[9, 14]. The basic idea is that given a frontier of

390

Best_N_First(topic, starting_urls, N) {

foreach link (starting_urls) {

enqueue(frontier, link);

}

while (#frontier > 0 and visited < MAX_PAGES) {

links_to_crawl := dequeue_top_links(frontier, N);

foreach link (randomize(links_to_crawl)) {

doc := fetch_new_document(link);

score := sim(topic, doc);

foreach outlink (extract_links(doc)) {

if (#frontier >= MAX_BUFFER) {

dequeue_link_with_min_score(frontier);

}

enqueue(frontier, outlink, score);

}

}

}

}

Figure 1: Pseudocode of Best-N-First crawlers.

links, the best link according to some estimationcriterion is selected for crawling.

Best-N-First is a generalization in that at eachiteration a batch of top N links to crawl are se-lected. After completing the crawl of N pagesthe crawler decides on the next batch of N andso on. As mentioned above, the topic’s keywordsare used to guide the crawl. More specificallythis is done in the link selection process by com-puting the lexical similarity between a topic’skeywords and the source page for the link. Thusthe similarity between a page p and the topicis used to estimate the relevance of the pageslinked from p. The N URLs with the best es-timates are then selected for crawling. Cosinesimilarity is used by the crawlers and the linkswith minimum similarity score are removed fromthe frontier if necessary in order to not exceedthe MAX BUFFER size. Figure 1 offers a simplifiedpseudocode of a Best-N-First crawler.

Best-N-First offers an ideal context for ourstudy. The parameter N controls the greedy be-havior of the crawler. Increasing N results incrawlers with greater emphasis on explorationand consequently a reduced emphasis on ex-ploitation. Decreasing N reverses this; selectinga smaller set of links is more exploitative of theevidence available regarding the potential mer-its of the links. In our experiments we test fivemutants of the crawler by setting N to 1, 16, 64,128 and 256. We refer to them as BFSN whereN is one of the above values.

4 Evaluation Methods

Table 2 depicts our overall methodology forcrawler evaluation. The two rows of Table 2 in-dicate two different methods for gauging pagequality. The first is a purely lexical approachwherein similarity to the topic description is usedto assess relevance. The second method is pri-marily linkage based and is an approximation ofthe retrieval/ranking method used by Google [6];it uses PageRank to discriminate between pagescontaining the same number of topic keywords.

The columns of the table show that our mea-sures are used both from a static and a dynamicperspective. The static approach examines crawlquality assessed from the full set of (up to 2000)pages crawled for each query. In contrast thedynamic measures provide a temporal charac-terization of the crawl strategy, by consideringthe pages fetched while the crawl is in progress.More specifically, the static approach measurescoverage, i.e., the ability to retrieve “good” pageswhere the quality of a page is assessed in twodifferent ways (corresponding to the rows of thetable). Our static plots show the ability of eachcrawler to retrieve more or fewer highly relevantpages. This is analogous to plotting recall as afunction of generality.

The dynamic approach examines the quality ofretrieval as the crawl progresses. Dynamic plotsoffer a trajectory over time that displays the dy-namic behavior of the crawl. The measures arebuilt on average (quality-based) ranks and aregenerally inversely related to precision. As theaverage rank decreases, an increasing proportionof the crawled set can be expected to be relevant.

It should be noted that scores and ranks usedin each dynamic measure are computed omni-sciently, i.e., all calculations for each point intime for a crawler are done using data generatedfrom the full crawl. For instance, all PageR-ank scores are calculated using the full set ofretrieved pages. This strategy is quite reason-able given that we want to use the best possibleevidence when judging page quality.

491

Table 2: Evaluation Schemes and Measures. The static scheme is based on coverage of top pages(ranked by quality metric among all crawled pages, S). Scrawler is the set of pages visited by acrawler. The dynamic scheme is based on the ranks (by quality metric among all crawled pages,S) averaged over the crawl sets at time t, Scrawler(t).

Static Scheme Dynamic SchemeLexical |Scrawler ∩ toprankSMART

(S)|∑p∈Scrawler(t) rankSMART (p)/|Scrawler(t)|

Linkage |Scrawler ∩ toprankKW,PR(S)|∑p∈Scrawler(t) rankKW,PR(p)/|Scrawler(t)|

4.1 Lexical Based Page Quality

We use the SMART system [23] to rank the re-trieved pages by their lexical similarity to thetopic. The SMART system allows us to poolall the pages crawled by all the crawlers for atopic and then rank these against the topic de-scription. The system utilizes term weightingstrategies involving term frequency and inversedocument frequency computed from the pooledpages for a topic. SMART computes the simi-larity between the query and the topic as a dotproduct of the topic and page vectors. It out-puts a ranked set of pages based on their topicsimilarity scores. That is, for each page we geta rank which we refer to as rankSMART (cf. Ta-ble 2). Thus given a topic, the percentage of topn pages ranked by SMART (where n varies) thatare retrieved by each crawler may be calculated,yielding the static evaluation metric.

For the dynamic view we use the rankSMART

values for pages to calculate mean rankSMART

at different points of the crawl. If we letScrawler(t) denote the set of pages retrieved up totime t, then we calculate mean rankSMART overScrawler(t). The set Scrawler(t) of pages increasesin size as we proceed in time. We approximate tby the number of pages crawled. The trajectoryof mean rankSMART values over time displaysthe dynamic behavior of a crawler.

4.2 Linkage Based Page Quality

It has been observed that content alone does notgive a fair measure of the quality of the page[15]. Algorithms such as HITS [15] and PageR-ank [6] use the linkage structure of the Web to

rank pages. PageRank in particular estimatesthe global popularity of a page. The computa-tion of PageRanks can be done through an it-erative process. PageRanks are calculated onceafter all the crawls are completed. That is, wepool the pages crawled for all the topics by allthe crawlers and then calculate the PageRanksaccording to the algorithm described in [13]. Wesort the pages crawled for a given topic, by allcrawlers, first based on the number of topic key-words they contain and then sort the pages withsame number of keywords by their PageRank.The process gives us a rankKW,PR for each pagecrawled for a topic.

Once again, our static evaluation metric mea-sures the percentage of top n pages (ranked byrankKW,PR) crawled by a crawler on a topic. Inthe dynamic metric, mean rankKW,PR is plottedover each Scrawler(t) where t is the number ofpages crawled.

5 Results

For each of the evaluation schemes and metricsoutlined in Table 2, we analyzed the performanceof each crawler on the two tasks.

5.1 Task 1 : Starting from Examples

For the first task the crawlers start from a rele-vant subset of links, the examples, and use thehyperlinks to navigate and discover more rele-vant pages. The results for the task are sum-marized by the plots in Figure 2. For read-ability, we are only plotting the performance ofa selected subset of the Best-N-First crawlers(N = 1, 256). The behavior of the remaining

592

35

40

45

50

55

60

65

70

75

80

0 100 200 300 400 500

% c

raw

led

�

number of top pages

a) Static Lexical Performance

BFS1BFS256

1400

1600

1800

2000

2200

2400

2600

2800

3000

3200

3400

0 500 1000 1500 2000

aver

age

rank

�

pages crawled

b) Dynamic Lexical Performance

BFS1BFS256

35

40

45

50

55

60

65

0 100 200 300 400 500

% c

raw

led

�

number of top pages

c) Static Linkage Performance

BFS1BFS256

2100

2200

2300

2400

2500

2600

2700

2800

2900

3000

3100

0 500 1000 1500 2000

aver

age

rank

�

pages crawled

d) Dynamic Linkage Performance

BFS1BFS256

Figure 2: Static evaluation (left) and dynamic evaluation (right) of representative crawlers on Task1. The plots correspond to lexical (top) and linkage (bottom) quality metric. Error bars correspondto ±1 standard error across the 100 topics in this and the following plots.

crawlers (BFS16, BFS64 and BFS128) can beextrapolated between the curves correspondingto BFS1 and BFS256.

The most general observation we can drawfrom the plots is that BFS256 achieves a sig-nificantly better performance under the staticevaluation schemes, i.e., a superior coverage ofthe most highly relevant pages based on bothquality metrics and across different numbers oftop pages (cf. Figure 2a,c). The difference be-tween the coverage by crawlers for different Nincreases as one considers fewer highly relevantpages. These results indicate that explorationis important to locate the highly relevant pageswhen starting from relevant links, whereas toomuch exploitation is harmful.

The dynamic plots give us a richer picture.(Recall that here lowest average rank is best.)

BFS256 still does significantly better than othercrawlers on the lexical metric (cf. Figure 2b).However, the linkage metric shows that BFS256pays a large penalty in the early stage of thecrawl (cf. Figure 2d). However, the crawler re-gains quality over the longer run. The bettercoverage of highly relevant pages by this crawler(cf. Figure 2c) may help us interpret the im-provement observed in the second phase of thecrawl. We conjecture that by exploring subop-timal links early on, BFS256 is capable of even-tually discovering paths to highly relevant pagesthat escape more greedy strategies.

5.2 Task 2: Starting from Neighbors

The success of a more exploratory algorithm onthe first task may not come as a surprise sincewe start from known relevant pages. However, in

693

18

20

22

24

26

28

30

32

34

36

38

40

0 100 200 300 400 500

% c

raw

led

�

number of top pages

a) Static Lexical Performance

BFS1BFS256

BreadthFirst

3500

4000

4500

5000

5500

6000

6500

7000

7500

0 500 1000 1500 2000

aver

age

rank

�

pages crawled

b) Dynamic Lexical Performance

BFS1BFS256

BreadthFirst

10

15

20

25

30

35

40

0 100 200 300 400 500

% c

raw

led

�

number of top pages

c) Static Linkage Performance

BFS1BFS256

BreadthFirst

4000

4500

5000

5500

6000

6500

7000

0 500 1000 1500 2000

aver

age

rank

�

pages crawled

d) Dynamic Linkage Performance

BFS1BFS256

BreadthFirst

Figure 3: Static evaluation (left) and dynamic evaluation (right) of representative crawlers on Task2. The plots correspond to lexical (top) and linkage (bottom) quality metric.

the second task we use the links obtained fromthe neighborhood of relevant subset as the start-ing points with the goal of finding more relevantpages. We take the worst (BFS1) and the best(BFS256) crawlers on Task 1, and use them forTask 2. In addition, we add a simple Breadth-First crawler that uses the limited size frontieras a FIFO queue. The Breadth-First crawler isadded to observe the performance of a blind ex-ploratory algorithm. A summary of the resultsis shown through plots in Figure 3. As for Task1, we find that the more exploratory algorithm,BFS256, performs significantly better than BFS1under static evaluations for both lexical and link-age quality metrics (cf. Figure 3a,c). In the dy-namic plots (cf. Figure 3b,d) BFS256 seems tobear an initial penalty for exploration but recov-ers in the long run. The Breadth-First crawlerperforms poorly on all evaluations. Hence, as a

general result we find that exploration helps anexploitative algorithm, but exploration withoutguidance goes astray.

Due to the availability of relevant subsets (ex-amples) for each of the topics in the current task,we plot the average recall of the relevant exam-ples against number of pages crawled (Figure 4).The plot illustrates the target-seeking behaviorof the three crawlers if the examples are viewedas the targets. We again find BFS256 outper-forming BFS1 while Breadth-First trails behind.

6 Related Research

Research on the design of effective focusedcrawlers is very vibrant. Many different typesof crawling algorithms have been developed. Forexample, Chakrabarti et al. [8] use classifiersbuilt from training sets of positive and negative

794

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0 500 1000 1500 2000

aver

age

reca

ll

�

pages crawled

BFS1BFS256

BreadthFirst

Figure 4: Average recall of examples when thecrawls start from the neighbors.

example pages to guide their focused crawlers.Fetuccino [3] and InfoSpiders [18] begin theirfocused crawling with starting points generatedfrom CLEVER [7] or other search engines. Mostcrawlers follow fixed strategies, while some canadapt in the course of the crawl by learning toestimate the quality of links [18, 1, 22].

The question of exploration versus exploita-tion in crawler strategies has been addressedin a number of papers, more or less directly.Fish-Search [11] limited exploration by bound-ing the depth along any path that appeared sub-optimal. Cho et al. [9] found that exploratorycrawling behaviors such as implemented in theBreadth-First algorithm lead to efficient discov-ery of pages with good PageRank. They also dis-cuss the issue of limiting the memory resources(buffer size) of a crawler, which has an impact onthe exploitative behavior of the crawling strategybecause it forces the crawler to make frequentfiltering decisions. Breadth-First crawlers alsoseem to find popular pages early in the crawl [20].The exploration versus exploitation issue contin-ues to be studied via variations on the two majorclasses of Breadth-First and Best-First crawlers.For example, in recent research on Breadth-Firstfocused crawling, Diligenti et al. [12] addressthe “shortsightedness” of some crawlers when as-sessing the potential value of links to crawl. Inparticular, they look at how to avoid short-termgains at the expense of less-obvious but larger

long-term gains. Their solution is to build clas-sifiers that can assign pages to different classesbased on the expected link distance between thecurrent page and relevant documents.

The area of crawler quality evaluation hasalso received much attention in recent research[2, 9, 8, 19, 4]. For instance, many alterna-tives for assessing page importance have beenexplored, showing a range of sophistication. Choet al. [9] use the simple presence of a word suchas “computer” to indicate relevance. Amento etal. [2] compute the similarity between a pageand the centroid of the seeds. In fact content-based similarity assessments form the basis ofrelevance decisions in several examples of re-search [8, 19]. Others exploit link information toestimate page relevance with methods based onin-degree, out-degree, PageRank, hubs and au-thorities [2, 3, 4, 8, 9, 20]. For example, Cho etal. [9] consider pages with PageRank score abovea threshold as relevant. Najork and Wiener [20]use a crawler that can fetch millions of pages perday; they then calculate the average PageRankof the pages crawled daily, under the assumptionthat PageRank estimates relevance. Combina-tions of link and content-based relevance estima-tors are evident in several approaches [4, 7, 18].

7 Conclusions

In this paper we used an evaluation frameworkfor topic driven crawlers to study the role of ex-ploitation of link estimates versus exploration ofsuboptimal pages. We experimented with a fam-ily of simple crawler algorithms of varying greed-iness, under limited memory resources for twodifferent tasks. A number of schemes and qual-ity metrics derived from lexical features and linkanalysis were introduced and applied to gaugecrawler performance.

We found consistently that exploration leadsto better coverage of highly relevant pages, inspite of a possible penalty during the early stageof the crawl. An obvious explanation is that ex-ploration allows to trade off short term gains forlonger term and potentially larger gains. How-ever, we also found that a blind exploration

895

when starting from neighbors of relevant pages,leads to poor results. Therefore, a mix of ex-ploration and exploitation is necessary for goodoverall performance. When starting from rele-vant examples (Task 1), the better performanceof crawlers with higher exploration could be at-tributed to their better coverage of documentsclose to the relevant subset. The good perfor-mance of BFS256 starting away from relevantpages, shows that its exploratory nature com-plements its greedy side in finding highly rele-vant pages. Extreme exploitation (BFS1) andblind exploration (Breadth-First), impede per-formance. Nevertheless, any exploitation seemsto be better than none. Our results are basedon short crawls of 2000 pages. The same maynot hold for longer crawls; this is an issue to beaddressed in future research. The dynamic eval-uations do suggest that for very short crawls itis best to be greedy; this is a lesson that shouldbe incorporated into algorithms for query time(online) crawlers such as MySpiders3 [21].

The observation that higher exploration yieldsbetter results can motivate parallel and/ordistributed implementations of topic drivencrawlers, since complete orderings of the linksin the frontier, as required by greedy crawler al-gorithms, do not seem to be necessary for goodperformance. Therefore crawlers based on localdecisions seem to hold promise both for the per-formance of exploratory strategies and for the ef-ficiency and scalability of distributed implemen-tations. In particular, we intend to experimentwith variations of crawling algorithms such asInfoSpiders [18], that allow for adaptive and dis-tributed exploratory strategies.

Other crawler algorithms that we intendto study in future research include Best-Firststrategies driven by estimates other than lexi-cal ones. For example we plan to implement aBest-N-First family using link estimates basedon local versions of the rankKW,PR metric usedin this paper for evaluations purposes. We alsoplan to test more sophisticated lexical crawlerssuch as InfoSpiders and Shark Search [14], whichcan prioritize over links from a single page.

3http://myspiders.biz.uiowa.edu

A goal of present research is to identify op-timal trade-offs between exploration and ex-ploitation, where either more exploration ormore greediness would degrade performance. Alarge enough buffer size will have to be usedso as not to constrain the range of explo-ration/exploitation strategies as much as hap-pened in the experiments described here due tothe small MAX BUFFER. Identifying an optimalexploration/exploitation trade-off would be thefirst step toward the development of an adaptivecrawler that would attempt to adjust the level ofgreediness during the crawl.

Finally, two things that we have not done inthis paper are to analyze the time complexity ofthe crawlers and the topic-specific performanceof each strategy. Regarding the former, clearlymore greedy strategies require more frequent de-cisions and this may have an impact on the ef-ficiency of the crawlers. Regarding the latter,we have only considered quality measures in theaggregate (across topics). It would be useful tostudy how appropriate trade-offs between explo-ration and exploitation depend on different char-acteristics such as topic heterogeneity. Both ofthese issues are the object of ongoing research.

References

[1] C. Aggarwal, F. Al-Garawi, and P. Yu. In-telligent crawling on the World Wide Webwith arbitrary predicates. In Proc. 10th Intl.World Wide Web Conference, pages 96–105,2001.

[2] B. Amento, L. Terveen, and W. Hill. Does”authority” mean quality? Predicting ex-pert quality ratings of web documents. InProc. 23rd ACM SIGIR Conf. on Researchand Development in Information Retrieval,pages 296–303, 2000.

[3] I. Ben-Shaul, M. Herscovici, M. Jacovi,Y. Maarek, D. Pelleg, M. Shtalhaim,V. Soroka, and S. Ur. Adding support fordynamic and focused search with Fetuccino.Computer Networks, 31(11–16):1653–1665,1999.

996

[4] K. Bharat and M. Henzinger. Improved al-gorithms for topic distillation in hyperlinkedenvironments. In Proc. 21st ACM SIGIRConf. on Research and Development in In-formation Retrieval, pages 104–111, 1998.

[5] B. E. Brewington and G. Cybenko. Howdynamic is the Web? In Proc. 9th Interna-tional World-Wide Web Conference, 2000.

[6] S. Brin and L. Page. The anatomy ofa large-scale hypertextual Web search en-gine. Computer Networks, 30(1–7):107–117,1998.

[7] S. Chakrabarti, B. Dom, P. Raghavan,S. Rajagopalan, D. Gibson, and J. Klein-berg. Automatic resource compilation byanalyzing hyperlink structure and associ-ated text. Computer Networks, 30(1–7):65–74, 1998.

[8] S. Chakrabarti, M. van den Berg, andB. Dom. Focused crawling: A new approachto topic-specific Web resource discovery.Computer Networks, 31(11–16):1623–1640,1999.

[9] J. Cho, H. Garcia-Molina, and L. Page. Ef-ficient crawling through url ordering. InProc. 7th Intl. World Wide Web Confer-ence, Brisbane, Australia, 1998.

[10] Cyveillance. Sizing the inter-net. White paper, July 2000.http://www.cyveillance.com/web/us/corporate/white papers.htm.

[11] P. De Bra and R. Post. Information retrievalin the World Wide Web: Making client-based searching feasible. In Proc. 1st Intl.World Wide Web Conference, 1994.

[12] M. Diligenti, F. Coetzee, S. Lawrence,C. L. Giles, and M. Gori. Focused crawl-ing using context graphs. In Proc. 26thInternational Conference on Very LargeDatabases (VLDB 2000), pages 527–534,Cairo, Egypt, 2000.

[13] T. Haveliwala. Efficient computation ofpagerank. Technical report, StanfordDatabase Group, 1999.

[14] M. Hersovici, M. Jacovi, Y. S. Maarek,D. Pelleg, M. Shtalhaim, and S. Ur. Theshark-search algorithm — An application:Tailored Web site mapping. In Proc. 7thIntl. World-Wide Web Conference, 1998.

[15] J. Kleinberg. Authoritative sources in ahyperlinked environment. Journal of theACM, 46(5):604–632, 1999.

[16] S. Lawrence and C. Giles. Accessibility ofinformation on the Web. Nature, 400:107–109, 1999.

[17] F. Menczer. Links tell us about lexical andsemantic web content. arXiv:cs.IR/0108004.

[18] F. Menczer and R. Belew. Adaptive re-trieval agents: Internalizing local contextand scaling up to the Web. Machine Learn-ing, 39(2–3):203–242, 2000.

[19] F. Menczer, G. Pant, M. Ruiz, andP. Srinivasan. Evaluating topic-driven Webcrawlers. In Proc. 24th Annual Intl. ACMSIGIR Conf. on Research and Developmentin Information Retrieval, 2001.

[20] M. Najork and J. L. Wiener. Breadth-firstsearch crawling yields high-quality pages. InProc. 10th International World Wide WebConference, 2001.

[21] G. Pant and F. Menczer. Myspiders: Evolveyour own intelligent web crawlers. Au-tonomous Agents and Multi-Agent Systems,5(2):221–229, 2002.

[22] J. Rennie and A. K. McCallum. Using re-inforcement learning to spider the Web effi-ciently. In Proc. 16th International Conf. onMachine Learning, pages 335–343. MorganKaufmann, San Francisco, CA, 1999.

[23] G. Salton. The SMART Retrieval System –Experiments in Automatic Document Pro-cessing. Prentice-Hall, Englewood Cliffs,NJ, 1971.

1097