presented by: - chandrika b n. agenda technology overview introduction link structure of the web...
TRANSCRIPT
Presented By:- Chandrika B N
Agenda Technology Overview Introduction Link Structure of the Web Simplified PageRank Eigenvalue and Eigenvector PageRank Definition Random Surfer Model Dangling Links PageRank Implementation Convergence Searching with PageRAnk Personalized PageRank Application Conclusion
Technology OverviewRecognized the need for a new kind of server setup
Linked PC’s to quickly find each query’s answersThis resulted in: Faster Response Time
Greater Scalability Lower costs
Google uses more than 200 signals (including PageRank algorithm) to determine which pages are important
Google then performs hypertext-matching- Google Corporate Information
- Google Corporate Information
Life of a Google Query
The mechanism
•Web Crawler: Finds and retrieves pages on the web•Repository: web pages are compressed and stored here•Indexer: each index entry has a list of documents in which the term appears
and the location within the text where it occurs
IntroductionWWW is very large and heterogeneous
The web pages are extremely diverse
Problem: How can the most relevant pages be ranked at the top?
Answer: Take advantage of the link structure of the Web to produce ranking of every web page known as PageRank
Link Structure of the Web
A and B are Backlinks of C
•Every page has some number of forward links (outedges) and backlinks (inedges)
•We can never know all the backlinks of a page, but we know all of its forward links
•Generally, highly linked pages are more “important”
PageRank• PageRank - a method for computing a ranking for every web page
based on the graph of the web• A page has high rank if the sum of the ranks of its backlinks is high
• Page has many backlinks• Page has a few highly ranked backlinks
• Page rank is a link analysis algorithm that assigns a numerical weight that represents how important a page is on the web
• The web is democratic i.e., pages vote for pages
Google interprets a link from page A to page B as a vote, by page A, for page B.It also analyses the page that cast the vote.
A page is important if important pages refer to it
Simple Ranking Function: u: web pageBu: backlinksNu = |Fu| number of links from uc: factor used for normalization
The PageRanks form a probability distribution over web pages, so the sum of all web pages’ PageRanks will be one
Simplified PageRank Calculation
Eigenvalue and EigenvectorEigenvalues and Eigenvectors are properties of a matrix In general, a matrix acts on a vector by changing both its
magnitude and directionHowever, a matrix may act on certain vectors by changing
only their magnitude, and leaving their direction unchanged – Eigenvector
A matrix acts on an eigenvector by multiplying its magnitude by a factor called the Eigenvalue
Given a linear transformation A, a non-zero vector x is defined to be an eigenvector of the transformation if it satisfies the eigenvalue equation
In this situation, the scalar λ is called an eigenvalue of A corresponding to the eigenvector x
Given a square matrix A, the eigenvalue eq can be expressed as
The eigenvector equation for A can be written as
λ is the eigenvalueSolving this eq we get λ = 1 and λ = 3
Example
A =
Considering first the eigenvalue λ = 3, we have
After matrix-multiplication
This can be represented as 2 linear equations: 2x + y = 3x and x + 2y = 3y
The equations can be reduced to x = yWe can choose any value for x. Taking x=1, we get y=1
Eigenvector with eigenvalue 1
Eigenvector with eigenvalue 3
Computing PageRank given a Directed Graph
The Transition matrix A =
We get the eigenvalue λ = 1
Calculating the eigenvector
On substituting we get,
so the vector u is of the form
Choose v to be the unique eigenvector with the sum of all entries equal to 1
PageRank vector
Calculating the PageRankFinding the Eigenvalue and Eigenvector
Let Au,v = 1/Nu , if there is an edge from u to v
0, otherwiseIf R is a vector over the web pages,
then R = cAR where , R: eigenvector of A
c: eigenvalue
•Consider two web pages that point to each other but to no other page
•Suppose there is some web page which points to one of them, then
•During iteration, this loop will accumulate rank but will never distribute any rank
•This forms a trap called the RANK SINK. This can be overcome by introducing a Rank Source
Problem: Rank Sink
PageRank Definition:Let E(u) be some vector over the Web pages that corresponds to a source of rank. Then, the PageRank of a set of Web pages is an assignment, R’, to the Web pages which satisfies
such that c is maximized and ||R’||1 = 1 (||R’||1 denotes the L1 norm of R’).
PageRank of document u
Number of outlinks from document v
PageRank of document vthat links to u
Normalizationfactor
Vector of web pages that the Surfer randomly jumps to
Computing PageRank
S: any vector over the web pages
Loop:
Calculate the Ri+1 vector using Ri
Calculate the normalizing factor
Find the vector Ri+1 using d
Find the norm of the difference of 2 vectors
while Loop until convergence
SR 0
ii ARR 1
111 ii RRd
dERR ii 11
ii RR 1
Random Surfer Model
The “Random surfer” simply keeps clicking on successive links at random
A Real Web Surfer will unlikely continue in a loop forever
The surfer periodically “gets bored” and jumps to another random page
Dangling Links
Links that point to any page with no outgoing links
They do not affect the ranking of any other page directly
Problem: It is not clear where their weight should be distributed
Solution: They can be removed from the system until all the PageRanks are calculated
PageRank Implementation
Convert each URL into a unique integer IDSort the link structure by IDRemove the dangling linksMake an initial assignment of ranksIteratively compute PageRank until ConvergenceAdd the dangling links back Recompute the rankings
NOTE: After adding the dangling links back, we need to iterate as many times as was required to remove the dangling links
ConvergencePR (322 Million Links): 52 iterationsPR (161 Million Links): 45 iterationsScaling factor is roughly linear in logn
Convergence
The web is an expander-like graph
A graph is said to be an expander if:Every subset of nodes S has a neighborhood that is
larger than some factor α times |S| α is called the expansion factor
A graph has a good expansion factor if and only if the largest eigenvalue is sufficiently larger than the second-largest eigenvalue
Searching with PageRank• Two search engines:
– Title-based search engine– Full text search engine
• Title-based search engine– Searches only the “Titles”– Finds all the web pages whose titles contain all the
query words– Sorts the results by PageRank– Very simple and cheap to implement– Title match ensures high precision, and PageRank
ensures high quality
• Full text search engine– Called Google– Examines all the words in every stored document and
also performs PageRank (Rank Merging)
Title-based search for University
Personalized PageRank
Important component of PageRank calculation is EA vector over the web pages (used as source of rank)Powerful parameter to adjust the page ranks
E vector corresponds to the distribution of web pages that a random surfer periodically jumps to
Having an E vector that is uniform over all the web pages results in some web pages with many related links receiving an overly high rank eg: copyright page or forums General Search over the internet
Instead in Personalized PageRank E consists of a single web page
ApplicationsEstimating Web Traffic
On analyzing the statistics, it was found that there are some sites that have a very high usage, but low PageRank.eg: Links to pirated software
PageRank as Backlink PredictorThe goal is to try to crawl the pages in as close to the optimal order as possible i.e., in the order of their rank.PageRank is a better predictor than citation counting
User Navigation: The PageRank ProxyThe user receives some information about the link before they click on itThis proxy can help users decide which links are more likely to be interesting
ConclusionPageRank is a global ranking of all web pages base of
their location in the Web’s graph structure
PageRank uses information which is external to the Web pages – backlinks
Backlinks from important pages are more significant than backlinks from average pages
The structure of the Web graph is very useful for information retrieval tasks.
References L. Page, S. Brin, R. Motwani, T. Winograd. The PageRank Citation
Ranking: Bringing Order to the Web, 1998 L. Page and S. Brin. The anatomy of a large-scale hypertextual
web search engine, 1998 THE $25,000,000,000 EIGENVECTOR THE LINEAR ALGEBRA
BEHIND GOOGLE by KURT BRYAN AND TANYA LEISE Google Corporate Information:
http://www.google.com/corporate/tech.html http://en.wikipedia.org/wiki/PageRank http://en.wikipedia.org/wiki/
Eigenvalue,_eigenvector_and_eigenspace http://www.googleguide.com/google_works.html http://www.math.cornell.edu/~mec/Winter2009/RalucaRemus/
Lecture3/lecture3.html http://pr.efactory.de/