improving minhashing: de bruijn sequences and primitive roots for counting trailing zeroes mark...
TRANSCRIPT
Improving minhashing:Improving minhashing:De Bruijn sequences and De Bruijn sequences and
primitive roots for primitive roots for counting trailing zeroescounting trailing zeroes
Mark ManasseFrank McSherry
Kunal Talwar
Microsoft Research
Why things you didn’t think you cared about are actually practical means of bit-twiddling
MinhashingMinhashing
This all comes out of a perennial quest to make minhashing faster
Minhashing is a technique for sampling an element from a stream which is◦Uniformly random (every element equally
likely)◦Consistent (similar stream ↔ similar sample)
P(S(A) = S(B))=
BA
BA
Locality sensitive hashingLocality sensitive hashing
View documents, photos, music, etc. as a set of features
View feature set as high-dimensional vectorFind closely-matching vectors
◦Most of the time◦Proportionally close
In L2, this leads us to cosine similarity (Indyk, Motwani)◦A hash function whose bits match proprtionally to
the cosine of the angle between two vectors◦Allows off-line computation of hash, and faster
comparison of hashes
Working in LWorking in L11::Jaccard similarityJaccard similarityGiven two sets, define
sim(A,B) to equal the cardinality of the intersection of the sets divided by the cardinality of the union
Proven useful, when applied to the set of phrases in a web page, when testing for near duplicates
A BA∩B
Basic idea, and old speedupBasic idea, and old speedup
Pick ~100 such samples, by picking ~100 random one-to-one functions mapping to a well-ordered range
For each function, select the pre-image of the smallest image under that function
Naively, takes 100 such evaluations per input item (one for each function)
Improve by factor of almost 8, by ◦ Choosing a 64 bit function◦ Lead 8 bits of 8 images generated by carving 64 into 8◦ Compute more bits, but only when needed
Why this approximates JaccardWhy this approximates Jaccard
Assume a uniformly chosen random function, mapping injectively to infinite binary sequences.
Order sequences lexicographicallyGiven sets A and B, and a random function f,
argmin(f”(AB)) is certain to be an element of A or B, and is in the intersection with exactly the Jaccard coefficient.
Sampling requires uniformity and consistency◦Uniformity so that probability mass is spread◦Consistency small perturbations don’t matter
New idea to speed up and reduce New idea to speed up and reduce collisionscollisions
Carve 64 into expected 32, by dividing at 1’s into 10…0
32, because expect half of bits to be 1Better yet, the number of maximal length
sequences is bounded by 2, independent of length of input
But, how do we efficiently divide at 1’s?
Dividing at 1’sDividing at 1’s
Could go bit at a time, shifting and testingLots of missed branch predictions, lots of
tests
Could look at low-order byte; if zero, shift by 8, if not do table look-up
Almost certainly good enough, in practice
But we can do mathematically better….
Dividing at 1’s smoothlyDividing at 1’s smoothly
Reduce to a simpler problem: taking the logarithm base 2 of a power of 2
Given x in twos-complement, x&-x is the smallest power of 2 in binary expansion of x◦ Works because -x = ~x + 1
~x is 01..1 below smallest power of 2 in x◦ x & (x-1) removes the least power of 2 (not useful here)◦ x ^ -x is all ones above the least power of 2◦ x ^ (x-1) is all ones at and below the least power of 2
So all 64-bit numbers can be reduced to only 65 possibilities, depending on least power of 2
How to we figure out which one?How to we figure out which one?
Naïve: binary search in sorted tableBetter: perfect hashingUsing x&-x, all 65 possible values are
powers of 2, or 0◦2 is a primitive root of unity modulo 67 (kudos
to Peter Montgomery for noting this)◦So, the powers of 2 generate the multiplicative
group (1, 2, …, 66) modulo 67◦That is, first 66 powers of 2 are distinct mod 67
So, take (x&-x) % 67, and look in a table
Perfect, maybe. But optimal?Perfect, maybe. But optimal?
Leiserson, Prokop, and Randall noticed that De Bruijn sequences are even better
Like Gray codes, only folded◦ De Bruijn sequences are vaguely like Gray codes
Hamiltonian circuit of hypercube is Gray code Hamiltonian circuit of De Bruijn graph is ….
De Bruijn sequences allow candy necklaces where any sequence of k candies occurs at exactly one starting point in clockwise order
Always exist (even generalized to higher dimension, but we don’t need that)
(00011101)* is such a sequence for 3-bit binary
More de BruijnMore de Bruijn
Any rotation of De Bruijn is De BruijnReversal of De Bruijn is De BruijnCanonicalize sequences by rotating to leastThree canonical sequences for binary sequences
of length 6; one is own reversal (6 is even)Starting with 6 zeroes, the first five bits needed
in rotation are zero, so shift is good enoughJust look at high-order 6 bits after multiplying by
constant DB=0x0218a392cd3d5dbfULDoesn’t handle 0, just powers of 2
Few branch missesFew branch misses
#define DB 0x0218a392cd3d5dbfUL#define NRES 100unsigned short dblookup[64]; // initialized to dblookup[(DB << i) >> 58] = iunsigned result[NRES + 64]; // answers plus spill-over spaceunsigned n=0, rb=0, elog=0; // quantity produced, remaining bits of
randomness, left-over zeroesunsigned long long cur;while (n < NRES) {
cur = newrandom(key);elog += rb;rb = 64;while (cur != 0) {
unsigned short log = dblookup[((cur & (1+~cur)) * DB) >> 58];cur >>= log + 1;rb -= log + 1;result[n++] = log + elog;elog= 0;
}}
Selecting randomly and Selecting randomly and repeatably from weighted repeatably from weighted
distributionsdistributionsMark Manasse
Frank McSherryKunal Talwar
Microsoft Research
Jaccard, extended to multisetsJaccard, extended to multisets
Jaccard, as defined, doesn’t work when the number of copies of an element is a distinguishing factor
If we generalize a little, we get sum of lesser number of occurrences divided by sum of greater
◦ Still works even for non-integral counts
◦ Allows for weighting of elements by importance
Same as before, for integer counts, if we replace items with iteminstance◦ <cat, cat, dog>
{cat1, cat2, dog1}Sample is (item,
instance), not just item
Sampling, instead of pairwise Sampling, instead of pairwise computation, for setscomputation, for sets
To allow for faster computation, we estimate similarity by sampling
Pick some number of samples, where for any sets A and B, each sample agrees with probability equal to sim(A,B)
Count the average number of matching samples
To get a good sample, pick a random one-to-one mapping of set elements to a well-ordering, and pick preimage of the (unique!) smallest.
Multiset Jaccard, one Multiset Jaccard, one implementationimplementation
Given a good way to approximate Jaccard for sets, we can convert a multiset (but not a distribution) into a set by replacing 100 occurrences of “cat” by “cat1”, “cat2”, …, “cat100”.
Requires (if streaming) remembering how many prior occurrences of elements have been seen.
Multiset reduction considered Multiset reduction considered inefficientinefficient
Previous technique is linear in input size, if input is {cat, cat, cat, dog}
Exponential in input size if input is (cat, 100)
Probability to the rescue!◦If our random map is to a real number between
0 and 1, we don’t need to generate 100 random values to find the smallest
◦CDF(X > x) = 1-x◦CDF(min_k > x) = (1-x)k
Not so fast!Not so fast!
That’s the probability, but not one that lets us pick samples to test for agreement
Not good enough to pick an element, have to pick an instance of the element◦(cat, 100) and (cat, 200) are only .5 similar
Has to be repeatable◦If cat7 is chosen from (cat, 100), mustn’t choose
cat73 from (cat, 200) (but cat104 would be OK)
Properties for repeatable Properties for repeatable samplingsampling
A sampling process must pick an element of the input◦For discrete things, an integer occurrence at
most equal to the number of occurrences◦For non-negative real valued things, a real
value at most equal to the inputMust pick uniformly by weightMust pick same sample from any subset
containing the sample
UniformityUniformity
To be uniform in a column, we have to pick a number smaller than a given number
Variant of reservoir sampling suffices◦ Given n, pick a random number
below n uniformly◦ Given that number, pick a random
number below that, and repeat◦ To make repeatable & expected con-
stant time, break into powers of 2◦ Given n, round up to next higher
power of 2, repeat downward process until below half
1.0
2.0
4.0
8.0
0.5
S(8,1) = 7.632…
S(8,2) = 4.918…
S(4,1) = 3.054…
S(1,1) = 0.783…
n = 3.724…
For this choice of n, S(4,1) is the downward selection; for slightly smaller n, S(1,1) would be selected
Scaling upScaling up
Same process, but we have to first round up, by finding smallest chosen number above n
First check the power of 2 range containing n
Next level up contains something if first selected number < 2k+1 is > 2k
If this happens, take smallest number in range
Otherwise, repeat at next up power of 2
1.0
2.0
4.0
8.0
0.5
S(8,1) = 7.632…
S(8,2) = 4.918…
S(4,1) = 3.054…
S(1,1) = 0.783…
n = 3.724…
For this choice of n, S(8,2) is the upward selection
Picking a columnPicking a column
Given a scaled-up column to n, need to construct the right distribution for second smallest number (assuming the n’th is the smallest)
In the discrete case (if we consider only integers as valid choices) IDF (2nd smallest > x) proportional to (1-x)n-1×x, so CDF = (n+1)xn-nxn+1= xn(1-n(x-1))
In the continuous case (which we can get by scaling the discrete case), CDF = xn(1-nlnx)
Pick a random luckiness factor for a column, p, solve for x in CDF = p by iteration
Pick column with smallest x value
Reducing randomness and Reducing randomness and improving numerical accuracyimproving numerical accuracy
We can just use one bit to decide if a power of 2 range has any selected values
So use a single random value to decide which of 64 powers of 2 are useful◦Either by computing 64 samples in parallel or◦Computing 64 intervals at once
Use logarithms of CDF rather than CDF to keep things reasonable; look at 1-x instead of x to keep log away from 0
Partially evaluate convergence to save time, and compare preimages to CDF when possible