locality sensitive hashing basics and applications
TRANSCRIPT
Locality Sensitive Hashing
Basics and applications
A well-known problem
Given a large collection of documents Identify the near-duplicate documents
Web search engines Proliferation of near-duplicate documents
Legitimate – mirrors, local copies, updates, … Malicious – spam, spider-traps, dynamic URLs, …
30% of web-pages are near-duplicates
[1997]
Natural Approaches
Fingerprinting: only works for exact matches Karp Rabin (rolling hash) – collision probability
guarantees MD5 – cryptographically-secure string hashes
Edit-distance metric for approximate string-matching expensive – even for one pair of documents impossible – for billion web documents
Random Sampling sample substrings (phrases, sentences, etc) hope: similar documents similar samples But – even samples of same document will differ
Basic Idea: Shingling [Broder 1997]
dissect document into q-grams (shingles)
T = I leave and study in Pisa, ….
If we set q=3 the 3-grams are:<I leave and><leave and study><and study in><study in
Pisa>…
represent documents by sets of hash[shingle]
The problem reduces to set intersection among int
Basic Idea: Shingling [Broder 1997]
Set intersection Jaccard similarity
DocB SB
SADocA
• Claim: A & B are near-duplicates if sim(SA,SB) is high
BA
BABA SS
SS )S,sim(S
Sketching of a document
From each shingle-set we build a “sketch vector” (~200
size)
Postulate: Documents that share ≥ t components of their sketch-vectors are claimed to be near duplicates
Sec. 19.6
Sketching by Min-Hashing
Consider SA, SB P = {0,…,p-1}
Pick a random permutation π of the whole set P (such as ax+b mod p)
Pick the minimal element of SA : = min{π(SA)}
Pick the minimal element of SB : = min{π(SB)}
Lemma:
BA
BA
SS
SS β]P[α
Strengthening it…
Similarity sketch sk(A) d minimal elements under π(SA) Or take d permutations and the min of each
Note: we can reduce the variance by using a larger d
Typically d is few hundred mins (~200)
Computing Sketch[i] for Doc1
Document 1
264
264
264
264
Start with 64-bit f(shingles)
Permute with i
Pick the min value
Sec. 19.6
Test if Doc1.Sketch[i] = Doc2.Sketch[i]
Document 1 Document 2
264
264
264
264
264
264
264
264
Are these equal?
Use 200 random permutations (minimum), and thus create one 200-dim vector per document and evaluate the fraction of shared components
A B
Sec. 19.6
Claim: This happens with probability Size_of_intersection / Size_of_union
It’s even more difficult…
So we have squeezed few Kbs of data
(web page) into few hundred bytes
But you still need a brute-force
comparison (quadratic time) to compute all
nearly-duplicate documents This is too much even if it is executed in RAM
Locality Sensitive Hashing
The case of the Hamming distance
How to compute fast the fraction of different compo. in d-dim vectors
How to compute fast the hamming distance between d-dim vectors
Fraction different components = HammingDist/d
A warm-up
Consider the case of binary (sketch) vectors, thus living in the hypercube {0,1}d
Hamming distance
D(p,q)= # coord on which p and q differ
Define hash function h by choosing a set I of k random coordinates
h(p) = p|I = projection of p on I
Example: if p=01011 (d=5), a pick I={1,4} (with k=2), then h(p)=01 Note similarity with the Bloom Filter
A key property
Pr[to pick an equal component]= (d - D(p,q))/d
We can vary the probability by changing k
k=1 k=2
distance distance
Pr Pr
1 2 …. d
k
d
qpDqhph )
),(1()]()(Pr[
equal
What about FalseNegatives ?
Reiterate
Repeat L times the k-projections hi
Declare a «match» if at least one hi matches
Example: d=5, k=2, p = 01011 and q = 00101
•I1 = {2,4}, we have h1(p) = 11 and h1(q)=00
•I2 = {1,4}, we have h2(p) = 01 and h2(q)=00
•I3 = {1,5}, we have h3(p) = 01 and h3(q)=01
We set g( ) = < h1( ), h2( ), h3( )>
p and qmatch !!
Measuring the error prob.
The g() consists of L independent hashes hi
Pr[g(p) matches g(q)] =1 - Pr[hi(p) ≠ hi(q), i=1, …, L]
L
k
d
qpDqgpg
)
),(1(11)()(Pr
kii d
qpDqhph )
),(1()]()(Pr[
Lksqgpg 11)()(Prs
Pr
(1/L)^(1/k)
Find groups of similar items SOL 1: Buckets provide the candidate similar items
«Merge» similar sets if they share items
h1(p) h2(p) hL(p)
TLT2T1
pPoints in a bucket are possibly similar objects
Find groups of similar items
SOL 1: Buckets provide the candidate similar items
SOL 2: Sort items by the hi(), and pick as similar candidate
the equal ones Repeat L times, for all hi()
«Merge» candidate sets if they share items.
What about clustering ?
Check candidates !!!
LSH versus K-means
What about optimality ? K-means is locally optimal [recently, some researchers showed how to introduce some guarantee]
What about the Sim-cost ? K-means compares items in (d) time and space [notice that d may be bi/millions]
What about the cost per iteration and their number? Typically K-means requires few iterations, each costs K n time: I K n d
What about K ? In principle have to iterate K=1,…, n
LSH needs sort(n) time hence, on disk, few passes over the data and with
guaranteed error bounds
Also on-line query
Given a query q, check the buckets of hj(q) for j=1,…,
L
h1(q) h2(q) hL(q)
TLT2T1
q
Locality Sensitive Hashingand its applications
More problems, indeed
Another classic problem
The problem: Given U users, the goal is to find groups of similar users (or, smilar to user Q)
Features = Personal data, preferences, purchases, navigational behavior, followers/ing or +1,…
A feature is typically a numerical value: binary or real
1 2 3 4 5
U1 0 1 0 1 1
U2 0 1 1 0 0
U3 0 1 1 1 1
Hamming distance: #different components
More than Hamming distance
Example:
q P*
q is the query point. P* is its Nearest Neighbor
Approximation helps
q
p*
r
A slightly different problem
Approximate Nearest Neighbor
Given an error parameter >0 For query q and nearest-neighbor p’, return p such that
Justification Mapping objects to metric space is heuristic anyway Get tremendous performance improvement
p )ε )D (q ,(1)p 'D (q ,
A workable approach
Given an error parameter >0, distance threshold t>0 (t,)-Approximate NN Query
If no point p with D(q,p)<t, return FAILURE Else, return any p’ with D(q,p’)< (1+)t
Application: Approximate Nearest Neighbor Assume maximum distance is T Run in parallel for
Time/space – O(log1+ T) overhead
T,,ε )(1,ε )(1 ε ),(1 1 , t 32
Locality Sensitive Hashingand its applications
The analysis
LSH Analysis
For a fixed threshold r, we distinguish between Near D(p,q) < r Far D(p,q) > (1+ε)r
A locality-sensitive hash h should guarantee
Near points are hashed together with Pr[h(a)=h(b)] ≥ P1
Far points may be mapped together but Pr[h(a)=h(c)] ≤ P2
where, of course, we have that P1 > P2
c ab
ε)r(1
r
Family hi(.) = p|c1,…,ck ,where the ci are chosen randomly
If D(a,b) ≤ r, then Pr[hi(a)= hi(b)] = (1 – D(a,b)/d)k
≥ ( 1 – r/d )k = ( p1 )k = P1
If D(a,c) > (1+)r, then Pr[hi(a)= hi(c)] = (1 – D(a,c)/d)k
< ( 1 – r(1+)/d )k = ( p2 )k = P2
where, of course, we have that p1 > p2 (as P1 > P2)
What about hamming distance?
LSH Analysis
The LSH-algorithm with the L mappings hi() correctly solves the (r,)-NN problem on a point query q if the following hold:
I. The total number of points FAR from q and belonging to the bucket hi(q) is a constant.
II. If p* NEAR to q, then hi(p*)= hi(q) for some I (p* is in a visited bucket)
Theorem. Take k=log_{1/p2} n and L=n(ln p1/ ln p2), then the two
properties above hold with probability at least 0.298.
Repeating the process (1/) times, we ensure a probability of success of at least 1-.
Space ≈ nL = n1+, where = (ln p1 / ln p2 ) < 1
Query time ≈ L buckets accessed, they are n
Proof
p* is a point near to q: D(q,p*) < r
FAR(q) = set of points p s.t. D(q,p) > (1+) r
BUCKETi (q) = set of points p s.t. hi(p)= hi(q)
Let us define the following events:
E1 = Num of far points in the visited buckets ≤ 3 L
E2 = p* occurs in some visited bucket, i.e. j s.t. hj(q) = hj(p*)
Lj
j LqBUCKETqFAR,1
*3)(
Bad collisions more than 3L
Let p is a point in FAR(q):
Pr[fixed j, hj(p) = hj(q)] < P2 = ( p2 )k
Given that k = log1/p2 n
Pr[fixed j, a far point p satisfies hj(p) = hj(q)] = 1/n
By Markov inequality Pr(X > 3E[x]) ≤ 1/3, follows:
Lj
L
j
L
jjj LnnqBUCKETqFAREqBUCKETqFARE
,1 11
)/1(*])([])([
Good collision: p* occursFor any hj the probability that
Pr[hj(p*) = hj(q)] ≥ P1 = ( p1 )k = ( p1 )^{ log1/p2 n } =
Given that L=n(ln p1/ ln p2), this is actually 1/L.
So we have that Pr[not E2] = Pr[not finding p* in q’s buckets] =
= (1 - Pr[hj(p*) = hj(q)])L = (1 – 1/L)L ≤ 1/e
Finally Pr[E1 and E2] ≥ 1 – Pr[not E1 OR not E2] ≥ 1 – (Pr[not E1]
+ Pr [not E2]) ≥ 1 - (1/3) - (1/e) = 0.298