Search Engines

Indexing

Page Ranking

The W W W

Page 1

Page 3Page 2

Page 1

Page 2

Page 1

Page 5

Page 6

Page 4

Page 1

Page 2

Page 1

Page 3

WebSite4

WebSite5

WebSite3

WebSite1

WebSite2

The Web Search Problem

Search Engine

Query: set of key words or phrase

Response: list of documents (pages) containing the key words or phrase

Important requirements:•Response must be quick

•Documents must be relevant

Tasks of a Search Engine

• Discover documents around the WWW

• Search keywords in documents

• Filter/rank documents according to their relevance

WebCrawlers (spiders, bots, wanderers, etc)

Based on graph searching algorithms (BFS or DFS ?)

For obvious performance reasons, this cannot be done by

string searching after every query !

Solution: Indexing; Web Search Engine Architectures

Web Search Engine Architecture

Page Repository

Text Index PageRank

Text Analysis Link Analysis

Ranker

WebCrawler

Query

Outline

• Data structures and algorithms for indexing the web

• The PageRank algorithm

Outline

• Data structures and algorithms for indexing the web

• The PageRank algorithm

Indexing the web

• Once a crawl has collected pages, their text is compressed and stored in a repository

• Each URL mapped to a unique ID• A lexicon (sorted list of all words) is created• A hit list (“Inverted index”) is created for every

word in the lexicon• Terminology:

– Forward index: Document -> list of contained words

– Inverted index: Word -> list of containing documents

Simple Inverted IndexingWords -> PageIDs

Using Simple Inverted Indexes for Queries

• Simple indexes help searching for keywords or sets of keywords– Example:

• Search “cat” => found in pages 1 and 3• Search “cat” AND “dog” => found in page 3

• Simple indexes cannot help performing phrase queries:– Example:

• Search “cat sat” => found in pages 1 and 3, but actually only page 1 contains the phrase “cat sat”

– Solution: indexing contains also the in-page location

Fully Inverted Indexing Words -> PageID’s + in-page locations

Using Fully Inverted Indexing for Queries

• Performing queries for phrases:– Search “cat sat”

• “cat” found at 1-2, 3-2• “sat” found at 1-3, 3-7• “cat” AND “sat”:

– in page 1, at 1-2 AND 1-3 => distance 1 between words

– in page 3, at 3-2 AND 3-7 => distance 5 between words

– Using the distance between words, only page 1 matches the search phrase

Using Metainformation

If the searched word is part of a title, the document is probably more relevant for the query

Indexing the web

• Once a crawl has collected pages, their text is compressed and stored in a repository

• Each URL (document) mapped to a unique ID• A lexicon (sorted list of all words) is created• A hit list (“Inverted index”) is created for every

word in the lexicon– Occurrences of a word in a particular document,

including position, font, capitalization, metainformation (part of titles)

Google’s Indexing – Step 1

• Each document is parsed an transformed into a collection of “hit lists” that are put into “barrels”, sorted by docID.

• Hit: <wordID, position in doc, font info, hit type>– Hit type: Plain or fancy.– Fancy hit: Occurs in URL, title, anchor text, metatag.

Forward Barrels

Google’s Forward Barrels

Wordid #hits Hit, hit, hit, hit, hitWordid #hits Hit

Wordid #hits HitWordid #hits Hit, hit, hit

Wordid #hits Hit, hit

Barrel i

Barrel i+1

Docid

Wordid #hits Hit, hitDocid

Wordid #hits Hit, hit

Wordid #hits Hit, hitWordid #hits Hit, hit, hitDocid

Google’s Indexing – Step 2

• Each barrel is then sorted by wordID to create the inverted index. This sorting also creates the lexicon file.– Lexicon: <wordID, offset into inverted index>– Lexicon is mostly cached in-memory

wordid #docswordid #docswordid #docs

Lexicon (in-memory) Postings (“Inverted Barrels”, on disk)

Google’s Inverted Index

Sorted by wordid

Docid #hits Hit, hit, hit, hit, hitDocid #hits Hit

Docid #hits HitDocid #hits Hit, hit, hit

Docid #hits Hit, hit

Barrel i

Barrel i+1

Outline

• Data structures and algorithms for indexing the web

• The PageRank algorithm

Motivation

• Efficient matching: Indexing helps finding pages that contain the search phrase, giving priority to the pages that contain it in titles or other privileged positions. Still there can be a huge number of such matches !

• Also needed for an effective search: a measure of importance of the pages that matched the search criteria

• Problem: Assessing the importance of web pages without human evaluation of the content – First solution: the PageRank algorithm

PageRank History

• History:– Proposed by 2 PhD students, Sergey Brin and

Lawrence Page in 1998 at Stanford.– “The Anatomy of a Large-Scale Hypertextual Web

Search Engine”.– “The PageRank citation ranking: Bringing order to the

web ”, http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf

– Algorithm of the first generation of Google Search Engine.

PageRank Principles

• Measure the importance of Web page based on the link structure alone.

• The importance of a page is given by the number of pages linking to it (number of “votes” received) as well as their importance (the importance of the voters)

• If a page contains links to a number of l pages, its contribution to the importance of each page is a fraction 1/l of its own importance (it “splits” its votes)

PageRank Principles - Example

Importance(P1)=100Outdegree(P1)=2

P1

Importance(P2)=9Outdegree(P2)=3

P2

Importance(P3)=53

P3

Importance(P4)=3

P4

50

50

3

3

3

Issues with Computing PageRank

• The simplified PageRank computation principles presented before cannot be directly applied:– Pages without inlinks: which should be their PR value?

(it cannot be zero, otherwise nothing gets propagated)– Cycles in page graphs: we cannot go forever round the

cycle, always increasing the scores

• The solution to this problem can be formulated from one of the possible viewpoints on PageRank:– Algebraic point of view– Probabilistic point of view

PageRank – The Probabilistic Point of View

• The Random Surfer Model• Since the importance of a web page P is

measured by its popularity (how many incoming links it has) we can view the importance of the page as the probability that a random surfer that starts browsing the net at any page arrives at the page P following hyperlinks.

• If the random surfer is at a page having k outlinks, he has 1/k probability to go next to any of the k pages

The Random Surfer Model

• Initial data:• The page graph contains N pages Pi, i=1..N

• We denote by Bi the set of all pages Pj that have links to Pi

• We denote by lj the outdegree of page Pj (the number of its outgoing links)

• Initially, each page Pi has 1/N probability to be choosen as a start page. This is the initial probability (at moment 0) of the page to be reached, PR(i, 0)

The Random Surfer Model

• Updating probabilities:• At a moment t, each page Pi has a probability

PR(i, t)• At next moment t’, the probability of page Pi is

PR(i, t’) and it is the weighted sum of the probabilities of its incoming pages, weighted by their outdegrees:

ij BP jl

tjPRtiPR

),()',(

The Random Surfer Model

• Updating probabilities:

PR(j, t)Outdegree(Pj)=lj

Pj

PiPR(j, t)/lj

PR(i, t’)

The Random Surfer Model• Convergence:• The values PR(i, t), when t→∞, converge to PR(i)

• The fact that PR converges to a unique probabilistic vector (the stationary distribution) can be mathematically proved (see: stochastic matrices, eigenvectors, the power method for finding eigenvector)

Example

• N=4

• l1=3, l2=2, l3=1, l4=2

• Initially (t=0):– PR(1,0)=1/4– PR(2,0)=1/4– PR(3,0)=1/4– PR(4,0)=1/4

P1

P2 P4

P31

1/2

1/3 1/2

1/3

1/21/3

1/2

PR(1,0)=1/4 PR(3,0)=1/4

PR(2,0)=1/4 PR(4,0)=1/4

Example (cont)

• t=1;• PR(1,1)=1*PR(3,0)+1/2*PR(4,0) =

1 * 0.25 + 1/2 * 0.25 = 0.37• PR(2,1)=1/3*PR(1,0)= 1/3 * 0.25 =

0.08• PR(3,1)=1/3*PR(1,0)+1/2*PR(2,0)

+1/2*PR(4,0) = 1/3 * 0.25 + 1/2 * 0.25 + 1/2 * 0.25 = 0.33

• PR(4,1)=1/3*PR(1,0)+1/2*PR(2,0)= 1/3 * 0.25 + 1/2 * 0.25 = 0.20

P1

P2 P4

P31

1/2

1/3 1/2

1/3

1/21/3

1/2

PR(1,0)=0.25 PR(3,0)=0.25

PR(2,0)=0.25 PR(4,0)=0.25

Example (cont)

• t=2;• PR(1,2)=1*PR(3,1)+1/2*PR(4,1) =

1 * 0.33 + 1/2 * 0.20 = 0.43• PR(2,2)=1/3*PR(1,1)= 1/3 * 0.37 =

0.12• PR(3,2)=1/3*PR(1,1)+1/2*PR(2,1)

+1/2*PR(4,1) = 1/3 * 0.37 + 1/2 * 0.08 + 1/2 * 0.20 = 0.27

• PR(4,2)=1/3*PR(1,1)+1/2*PR(2,1)= 1/3 * 0.37 + 1/2 * 0.08 = 0.16

P1

P2 P4

P31

1/2

1/3 1/2

1/3

1/21/3

1/2

PR(1,1)=0.37 PR(3,1)=0.33

PR(2,1)=0.08 PR(4,1)=0.20

Example (cont)

• t=3;• PR(1,3)=1*PR(3,2)+1/2*PR(4,2) =

1 * 0.27 + 1/2 * 0.16 = 0.35• PR(2,3)=1/3*PR(1,2)= 1/3 * 0.43 =

0.14• PR(3,3)=1/3*PR(1,2)+1/2*PR(2,2)

+1/2*PR(4,2) = 1/3 * 0.43 + 1/2 * 0.12 + 1/2 * 0.16 = 0.29

• PR(4,3)=1/3*PR(1,2)+1/2*PR(2,2)= 1/3 * 0.43 + 1/2 * 0.12 = 0.20

P1

P2 P4

P31

1/2

1/3 1/2

1/3

1/21/3

1/2

PR(1,2)=0.43 PR(3,2)=0.27

PR(2,2)=0.12 PR(4,2)=0.16

Example (cont)

• The values of PR calculated until now:• t=0: [0.25, 0.25, 0.25, 0.25]• t=1: [0.37, 0.08, 0.33, 0.20]• t=2: [0.43, 0.12, 0.27, 0.16]• t=3: [0.35, 0.14, 0.29, 0.20]• We can continue the iterations, and get:• t=4: [0.39, 0.11, 0.29, 0.19] • t=5: [0.39, 0.13, 0.28, 0.19]• t=6: [0.38, 0.13, 0.29, 0.19]• t=7: [0.38, 0.12, 0.29, 0.19]• t=8: [0.38, 0.12, 0.29, 0.19]

PR(1)=0.38PR(2)=0.12PR(3)=0.29PR(4)=0.19

Dangling Nodes and Disconnected Components

• Problems with the initial Random Surfer Model:– If the random web surfer arrives at a page Pj that has

no outlinks (a dangling node), he has nowhere to go. The accumulated importance of Pj “gets lost”, since it is not transferred further to any other pages

– If the web is formed by several connected components, the random web surfer will never reach pages that are in a different connected component than the initial random node

Example – The Dangling Node Problem

• N=3• l1=2, l2=2, l3=0• Initially (t=0):

– PR(1,0)=1/3– PR(2,0)=1/3– PR(3,0)=1/3

• Update rules:– PR(1,t’)=1/2 *PR(2,t)– PR(2,t’)=1/2*PR(1,t)– PR(3,t’)=1/2*PR(1,t)+1/2*PR(2,t)

P1

P2

P31/2

1/2

1/2

1/2

Example – The Dangling Node Problem (cont)

Applying the update rules we get:• t=0: [1/3, 1/3. 1/3]• t=1: [1/6, 1/6, 1/3]• t=2: [1/12, 1/12, 1/6]• t=3: [1/24, 1/24, 1/12]• ….

• Result: PR(1)=PR(2)=PR(3)=0 !

P1

P2

P31/2

1/2

1/2

1/2

This result has no meaning as a ranking -> a solution must be found for dangling nodes

Solution for dangling nodes and disconnected components

• The PageRank Random Surfer model is updated as follows:– Most of the time (a percentage d) a surfer will follow links from a

page, as in the model before. If a page has no outlinks, he will continue after it with a random page (a page with no outlinks will be considered to have N outlinks to any other page).

– A smaller, but positive percentage of time (the rest of the percentage 1-d) the surfer will dump the current page and choose arbitrarily a different page from the web and “teleport” there

Computing PageRank

N

d

N

tjPR

l

tjPRdtiPR

jij PBP j

1)

),(),(()',(

0)(outdegre

The probability of reaching a page Pi

The probability of arriving from a page Pj

that has a link to Pi

The probability of arriving from a page Pj

that has no outlinks

The probability of arriving through

teleporting at a random time

d=dumping factor, heuristic

The dumping factor

• Dumping factor (d) can have values in [0,1]• If d=0: all the web surfer moves are random jumps

(teleports), no links are followed• If d=1: the web surfer makes no teleports, he only follows

links, except for the case of dangling nodes • The value of d also influences how fast the vector

converges to the stationary distribution (the number of needed iterations)

• Usual value (proposed by Brin and Page): d=0.85• Convergence is reached in less than 100 iterations

public Map<Vertex, Double> computePageRank(Digraph<Vertex> g) { double d=0.85; int iterations=100; int N=g.getNumberOfNodes(); List<Vertex> nodes= g.getAllNodes(); List<Vertex> nodesWithoutOutlinks = g.getNodesWithoutOutlinks();

Map<Vertex, Double> opr = new HashMap<Vertex, Double>(); // old pageranks Map<Vertex, Double> npr = new HashMap<Vertex, Double>(); // new pageranks for (Vertex n:nodes) npr.put(n, 1.0/N); // init pageranks with 1/N for (Vertex n:nodes) opr.put(n, 1.0/N); while (iterations>0) { double dp=0; for (Vertex p:nodesWithoutOutlinks) dp=dp+opr.get(p)/N; for (Vertex p:nodes) { double nprp; nprp=dp+(1-d)/N; for (Vertex ip: g.inboundNeighbors(p)) nprp=nprp+d*opr.get(ip)/g.outDegree(ip); npr.put(p,nprp); } Map<Vertex, Double> temp; temp=opr; opr=npr; npr=temp; iterations=iterations-1; } return npr;}

PageRank – the Algebraic Point of View

• Initial data:• The page graph contains N pages Pi, i=1..N

• We denote by Bi the set of all pages Pj that have links to Pi

• We denote by lj the outdegree of page Pj (the number of its outgoing links)

• The Hyperlink matrix A: a square matrix with the rows and column corresponding to web pages, where A[i,j] = 1/lj if there is a link from j to i and A[i,j] = 0 if not.

Example –The Hyperlink Matrix

P1

P2 P4

P31

1/2

1/3 1/2

1/3

1/21/3

1/2

0 0 1 1/2

1/3 0 0 0

1/3 1/2 0 1/2

1/3 1/2 0 04

3

2

1

1 2 3 4

Properties of the Hyperlink Matrix

• Properties of the Hyperlink Matrix– All entries are nonnegative– The sum of the entries in a column j is 1, if j has

outgoing links . – All elements of a column j are 0 if j has no outgoing

links (j is a dangling node)

• If the web has no dangling nodes, the Hyperlink matrix is stochastic

Stochastic Matrices

• A column stochastic matrix (probability matrix, Markov matrix)  is a square matrix of nonnegative real numbers, with each column summing to 1.

Stochastic Matrices

• The Perron-Frobenius Theorem: Every positive column stochastic matrix A has a unique stationary column vector X (an eigenvector with eigenvalue 1): A*X=X

• The Power Method Convergence Theorem: Let A be a positive column stochastic matrix of size n*n and X its stationary column vector. Then X can be calculated by following procedure: Initialize the column vector Z with all entries equal to 1/n. Then the sequence Z, A*Z, A2*Z ….,Ak*Z converges to the vector X.

The Google Matrix

• A= Transition matrix• S= a matrix obtained from A, by setting the

elements of the columns where all elements of the column are 0, to 1/N

• G= the Google matrix:• G[i,j]=d*S[i,j]+(1-d)/N • Property: the Google matrix is a stochastic

matrix

• The stationary vector of G contains the PageRank values

PageRank and the History of Search Engines

• PageRank (1998) was the first algorithm to introduce the concept of “importance of a webpage” and calculate it without relying on external information– crucial factor in Google ascension

• Drawbacks:– PageRank can be manipulated– SEO (“Search Engine Optimisation”)

PageRank and the Future of Search Engines

• 2011: Google Panda: – introduce filters that prevent low quality sites and/or

pages from ranking well in the search results, identifying

– use human feedback and machine learning algorithms

• 2012: Google Penguin:– decrease ranking of sites identified as using “black-hat

SEO techniques” 

• 2013: Google Hummingbird – Judge the context of a query - thereby judging the

intent of a person carrying out a search, to determine what they are trying to find out

Other Uses of PageRank

• Ranking scientific articles according to their citations

• Ranking streets for predicting human movement and street congestion

• Automatic summarization – extracting the most relevant sentences from a text

Tool Project #3

• Optional: Automatic Summarization Tool, based on PageRank– Text is represented as a graph of sentences– Edges are given by the “similarity” of two sentences

(what can be used as a form of “recommendation” or “vote” between sentences ?)

– Apply PageRank (or a modified version, able to cope with undirected, maybe weighted graphs) and take the top x% sentences to form the abstract

http://bigfoot.cs.upt.ro/~ioana/algo/lab_pagerank.html

Bibliography

• John Mac Cormick: Nine Algorithms that changed the Future, Chapters 2 & 3

• Page, L., Brin, S., Motwani, R., Winograd, T. (1999). The PageRank citation ranking: Bringing order to the web. http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf

• David Austing, How Google Finds Your Needle in the Web's Haystack, AMS Feature Column http://www.ams.org/samplings/feature-column/fcarc-pagerank