1 algorithms for large data sets ziv bar-yossef lecture 6 april 13, 2005

1

Algorithms for Large Data Sets

Ziv Bar-YossefLecture 6

April 13, 2005

http://www.ee.technion.ac.il/courses/049011

2

Web Structure II:Bipartite Cores

and Bow Tie Structure

3

Outline

Bipartite cores The copying model Bow-tie structure of the web

4

Web as a Social Network

Small-world network: Low (average) diameter High clustering coefficient

vMany of v’s neighbors are neighbors of each other.

Reason: Web is built of communities.

5

Cyber Communities

Cyber community: A group of people sharing a common interest. Web pages authored/cited by these people.

Examples Israeli student organizations in the United States Large automobile manufacturers Oil spills off the coast of Japan Britney Spears fans

6

Structure of Cyber Communities [Kumar et al, 1999]

Hubs: resource pages about the community’s shared interest Examples:

Directory of Israeli Student Organizations in the US Yahoo! Autos Oil spills near Japan: bookmarks Donna’s Britney Spears links

Authorities: central pages on the community’s shared interest Examples:

ISO: Stanford’s Israeli Student Organization Mazda.com Britney Spears: The official site

7

Dense Bipartite Subgraphs

Hubs Cite many authorities Have overlapping citations

Authorities Cited by many hubs Frequently co-cited

Therefore: a cyber community is characterized by a dense directed bipartite subgraphs

hubs authorities

8

Bipartite Cores

(i,j)-bipartite core: (H’,A’) H’: a subset of H of size i A’: a subset of A of size j Subgraph induced on (H’,A’) is a

complete bipartite graph

Hypothesis: “Most” dense bipartite subgraphs of the web have cores.

Therefore: bipartite cores are footprints of cyber communities.

H A

9

Finding Cyber Communities Bipartite cores can be found efficiently from a crawl

A few one-pass scans of the dataA few sorts

Web is rife with cyber communitiesAbout 200k disjoint (3,*)-cores in a 1996 crawlCrawl had ~200M pages A random graph of this size is not likely to have even a single (3,3) core !

10

The Copying Model[Kleinberg et al 1999] [Kumar et al 2000]

Initialization: A single node Evolution: At every step, a new node v is added

to the graph. v connects to d out-neighbors. Prototype selection: v chooses a random node u

from the graph. Bernoulli copying: For each i = 1,…,d,

v tosses a coin with heads probability of If coin is heads, v connects to a random node If coin is tails, v connects to the i-th out-neighbor of u

11

The Copying Model: Motivation When a new page is created, author has

some “topic” in mind Author chooses links from a “prototype” u

about the topic Author introduces his own spin on the

topic, by linking to new “random” pages.

12

The Copying Model: Degree Distribution If = 0, then i-th neighbor of v is u with

probability indeg(u)/w indeg(w) Identical to the preferential attachment model In the limit, fraction of pages with in-degree k is

1/k2. For arbitrary

Fraction of pages with in-degree k is 1/k(2-)/(1 - )

Similar analysis

13

Erdős-Rényi Random Graph: Bipartite Cores Gn,p with p = d/n

Fix any A,B Gn,p, |A| = i, |B| =j

Probability A,B form a complete bipartite graph:

# of such pairs A,B:

Expected # of (i,j)-bipartite cores is at most

14

The Copying Model: Bipartite Cores Consider the graph after n steps

Theorem: For any i < log n, expected # of (i,d) bipartite cores is (n/ci)

Definition: v is a duplicator of u, if it copies all its neighbors from the prototype u.

Observation: If v1,…,vi are duplicators of u, then v1,…,vi and their neighbors form an (i,d)-bipartite core.

15

The Copying Model: Bipartite Cores (cont.) Lemma: w.h.p. (almost all) the first O(n/ci) nodes added

to the graph have at least i duplicators.

Probability a new node v is a duplicator: (1-)d

Define: c = 21/(1-)d

Let u be any node born at some step t < O(n/ci) Probability a node v born at step t’ > t chooses u as a

prototype: 1/(t’ – 1).

16

The Copying Model: Bipartite Cores (cont.) Split steps t+1,…,n into O(log(n/t)) “epochs”: (t,2t], (2t,4t],(4t,8t],…,

(n/2,n] Probability at least one node at the first epoch chooses u as a

prototype:

Same for the other epochs Expected # of duplicators of u is at least

# of duplicators is sharply concentrated about the mean

17

Bow Tie Structure of the Web[Broder et al 2000]

18

Random Samplingof

Web Pages

19

Outline

Problem definition Random sampling of web pages according

to their PageRank Uniform sampling of web pages

(Henzinger et al) Uniform sampling of web pages (Bar-

Yossef et al)

20

Random Sampling of Web Pages W = a snapshot of the “indexable

web” Consider only “static” HTML web

pages

= a probability distribution over W

Goal: Design an efficient algorithm for generating samples from W distributed according to .

Our focus: = PageRank = Uniform

Indexable web

21

Random Sampling of Web Pages: Motivation Compute statistics about the web

Ex: What fraction of web pages belong to .il? Ex: What fraction of web pages are written in Chinese? Ex: What fraction of hyperlinks are advertisements?

Compare coverage of search engines Ex: Is Google larger than MSN? Ex: What is the overlap between Google and Yahoo?

Data mining of the web Ex: How frequently computer science pages cite

biology pages? Ex: How are pages distributed by topic?

22

Random Sampling of Web Pages: Challenges Naïve solution: crawl, index, sample

Crawls cannot get complete coverage Web is constantly changing Crawling is slow and expensive

Our goals: Accuracy: generate samples from a snapshot of the

entire indexable web Speed: samples should be generated quickly Low cost: sampling procedure should run on a desktop

PC

23

A Random Walk Approach

Design a random walk on W whose stationary distribution is P = Random walk’s probability transition matrix P =

Run random walk for sufficiently many steps Recall: For any initial distribution q, Mixing time: # of steps required to get close to the limit

Use reached node as a sample

Repeat for as many samples as needed

24

A Random Walk Approach:Advantages & Issues Advantages:

Accuracy: random walk can potentially visit every page on the web

Speed: no need to scan the whole web Low cost: no need for large storage or multiple

processors

Issues: How to design the random walk so it converges to ? How to analyze the mixing time of the random walk?

25

PageRank Sampling [Henzinger et al 1999]

Use the “random surfer” random walk:Start at some initial node v0

When visiting a page v Toss a coin with heads probability If coin is heads, go to a uniformly chosen page If coin is tails, go to a random out-neighbor of v

Limit distribution: PageRank Mixing time: fast (will see later)

26

PageRank Sampling:Reality Problem: how to pick a page uniformly at

random? Solutions:

Jump to a random page from the history of the walk

Creates bias towards dense web-sitesPick a random host from the hosts in the walk’s

history and jump to a random page from the pages visited on that host

Not converging to PageRank anymore Experiments indicate it is still fine

27

Uniform Sampling via PageRank Sampling [Henzinger et al 2000]

Sampling algorithm:1. Use previous random walk to generate a sample w

according to the PageRank distribution

2. Toss a coin with heads probability

3. If coin is heads, output w as a sample

4. If coin is tails, goto step 1

Analysis: Need C/|W| iterations until getting a single sample

28

Uniform Sampling via PageRank Sampling: Reality How to estimate PR(w)?

Use the random walk itself: VR(w) = visit ratio of w (# of times w was visited by

the walk divided by length of the walk) Approximation is very crude

Use the subgraph spanned by nodes visited to compute PageRank

Bias towards to neighborhood of the initial pageUse Google

29

Uniform Sampling by RW on Regular Graphs [Bar-Yossef et al 2000]

Fact: A random walk on an undirected, connected, non-bipartite, and regular graph converges to a uniform distribution.

Proof: P: random walk’s probability transition matrix

P is stochastic 1 is a right eigenvector with e.v. 1: P1 = 1

Graph is connected RW is irreducible Graph is non-bipartite RW is aperiodic Hence, RW is ergodic, and thus has a stationary

distribution : is a left eigenvector of P with e.v. 1: P =

30

Random Walks on Regular Graphs

Proof (cont.):d: graph’s degree,A: graph’s adjacency matrix

Symmetric, because graph is undirected

P = (1/d) A Hence, also P is symmetric Its left eigenvectors and right eigenvectors are the

same = (1/n) 1

31

Web as a Regular Graph

Problems Web is not connected Web is directed Web is non-regular

Solutions Focus on indexable web, which is connected Ignore directions of links Add a weighted self loop to each node

weight(w) = degmax – deg(w)

All pages then have degree degmax

Overestimate on degmax doesn’t heart

32

Mixing Time Analysis

Theorem

Mixing time of a random walk is log(|W|) / (1 - 2) 1 - 2: spectral gap of P

Experiment (over a large web crawl): 1 – 2 ~ 1/100,000 log(|W|) ~ 34

Hence: mixing time ~ 3.4 million steps Self loop steps are free About 1 in 30,000 steps is not a self loop step (degmax

~ 300,000, degavg ~ 10) Actual mixing time: ~ 115 steps!

33

Random Walks on Regular Graphs: Reality

How to get incoming links? Search engines

Potential bias towards search engine index Do not provide full list of in-links? Costly communication

Random walk’s history Important for avoiding dead ends Requires storage

How to estimate deg(w)? Solution: run random walk on the sub-graph of W

spanned by the available links Sub-graph may no longer have the good mixing time

properties

34

Top 20 Internet Domains (Summer 2003)

10.36%

5.57%4.15%3.01%

0.61%

9.19%

51.15%

0%

10%

20%

30%

40%

50%

60%

.com

.org

.net

.edu .d

e .uk

.au .u

s.e

s .jp .ca .nl .it .ch .p

l .il .nz

.gov

.info .m

x

35

68%

54%50% 50%

48%

38%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Google AltaVista Fast Lycos HotBot Go

Search Engine Coverage(Summer 2000)

36

End of Lecture 6