Data Streams and Applications in Computer Science

David Woodruff

IBM Almaden

Presburger lecture, ICALP, 2014

Thanks to my advisors

Prof. Ron Rivest Prof. Piotr Indyk Prof. Andy Yao

Thanks for your mentorship and research advice, andearly guidance on a path in theoretical computer science

and my amazing summer interns

Arnab Bhattacharyya

JelaniNelson

HuyNguyen

MarcoMolinaro

YiLi

EricPrice

GrigoryYaroslavtsev

and my awesome collaborators!

in the theory group at IBM

and throughout the world…

My current research interests

• Communication Complexity

• Data Stream Algorithms and Lower Bounds

• Graph Algorithms

• Machine learning

• Numerical Linear Algebra

• Sketching

• Sparse Recovery

Talk Outline

• Data Stream Model and Sample Results– Distinct Elements– Frequency Moments– Characterization of Algorithms

• Connections to Other Areas– Compressed Sensing– Linear Algebra– Machine Learning

Data Streams

• A data stream is a sequence of data that is too large to be stored in available memory

• Examples

– Internet search logs

– Network Traffic

– Financial transactions

– Sensor networks

– Scientific data streams (astronomical, genomics, physical simulations)…

Streaming Model

• Stream of elements a1, …, am each in {1, …, n}

• Single or small number of passes over the data

• Algorithms should work for any ordering of elements

• Almost all algorithms are randomized and approximate– Usually necessary to achieve efficiency– Randomness is in the algorithm, not the input

• Goals: minimize space complexity (in bits), processing time

…0113734

Vector InterpretationStream: 8 2 1 9 1 9 2 4 4 9 4 2 5 4 2 5 8 5 2 5

Vector x:

1 2 3 4 5 6 7 8 9

• Think of x as an n-dimensional vector– Initially, x = 0n

• Insertion of i is interpreted as xi = xi + 1

• Output an approximation to f(x) with high probability

(1) Distinct Elements• Streaming model originated in work of Flajolet and Martin, ‘85

– Studied the distinct elements question

– # of distinct elements, denoted F0, is |{i | xi > 0}|

– Output a number Z with F0 · Z · (1+ε) F0 with 99% probability

– Can we do better than just storing all the coordinates of x?

– Yes, and tight bounds are known [Indyk,W],[W],[Kane,Nelson,W]: £(1/ ε2 + log n) bits of space, O(1) processing time

– Connections: to prove the tight lower bound, the gap-hamming communication problem was introduced

Gap-Hamming Problem

x 2 {0,1}n y 2 {0,1}n

• Promise: Hamming distance satisfies Δ(x,y) > n/2 + εn or Δ(x,y) < n/2 - εn

• Lower bound of Ω(ε-2) for randomized 1-way communication [Indyk, W], [W], [Jayram, Kumar, Sivakumar]

• Same for 2-way communication [Chakrabarti, Regev]• Applications: in information complexity, functional monitoring, embeddings, linear algebra, differential privacy, sparsifiers, …

(Andoni, Brody, Clarkson, de Wolf, Jayram, Krauthgamer, McGregor, Mironov, Pitassi, Reingold, Sherstov, Talwar, Vadhan, Vidick, W, Zhang…)

(2) Frequency Moments• Streaming model revived in work of Alon, Matias, and Szegedy, ’96

[AMS]

• Consider more general turnstile streaming model [coined by Muthukrishnan]– positive and negative updates, so xi = xi + 1 or xi = xi – 1– summarize statistics of difference x-y of two streams of insertions

Frequency Moments• [AMS] study frequency moments Fp = sumi=1

n |xi|p , or equivalently lp-norms– Summarize skewness of an empirical distribution– F2 used in computing self-join sizes, geometry and

linear algebra– F1 used for measuring distance between distributions,

and in “robust” algorithms (regression, subspace approximation)

Flat Skewed

Frequency Moments

• Output a number Z with Fp · Z · (1+ε) Fp with 99% probability

• Near-tight bounds known (Andoni, Bar-Yossef, Braverman, Chakrabarti, Coppersmith, Cormode, Ganguly, Gronemeier, Indyk, Jayram, Kane, Krauthgamer, Kumar, Li, Nelson, Porat, Sivakumar, Sun, W, …)

Any guesses on how the space bounds depend on p?

Frequency Moments• F2 is the “breaking point”

– Fp for p · 2 doable in O~(1) bits of space– Fp for p > 2 requires £~(n1-2/p) bits of space

• Algorithms achieve O~(1) processing times

• Connections: “sub-sampling + heavy hitters” technique for the upper bound – Used in many data stream, embedding, and linear

algebra problems: earthmover distance, mixed norms, sampling in the turnstile model, compressed sensing, graph sparsifiers, regression

– Optimally solves sumi=1n G(xi) problems

[Braverman, Ostrovsky]

Subsampling + Heavy Hitters• CountSketch [Charikar,Chen,Farach-Colton]:

– Give each coordinate i a random ¾(i) 2 {-1,1}– Randomly partition coordinates into B buckets,

maintain cj = Σi s.t. h(i) = j ¾(i)¢xi in j-th bucket

. Σi s.t. h(i) = 2 ¾(i)¢xi

. .

– Estimate xi as ch(i)

x1 x2 x3 x4 x5 x6 x7 x8 x9 x10

–Estimation error ¼ |x|2/B

–Can be used to find the “heavy hitters”–It is a linear map x -> S ¢ x

•Easy to maintain under updates

–Estimation error ¼ |x|2/B

–Can be used to find the “heavy hitters”–It is a linear map x -> S ¢ x

•Easy to maintain under updates

Subsampling + Heavy Hitters

• Subsampling [Indyk,W]: – Create nested sequence of subsets of [n]

– [n] = Llog n ¶ Llog n - 1 ¶ … ¶ L0

– Li contains about 2i random coordinates

– Run CountSketch to find heavy hitters of each xLi

– Estimate number of coordinates “at every scale”– Obtain a rough approximation x’ to x

n1/4 n1/31

1

n1/2

(n)

Value

Number ofcoordinatesx 2 Rn:

(3) Characterization of Turnstile Algorithms

• All known algorithms in the turnstile model have the form:

1. Choose a random matrix A independent of x

2. Maintain the “linear sketch” Ax in the stream

3. Output a function of Ax

• Question (?!): does the optimal algorithm for any function in the turnstile model have this form?

• [Li, Nguyen, W] Yes, up to a factor of log n in the space

– Some caveats, e.g., can’t necessarily store A in low space

– For lower bounds doesn’t matter, gives simpler proof strategy since just need to rule out linear sketches

=Ax

Ax

Talk Outline

• Data Stream Model and Sample Results– Distinct Elements– Frequency Moments– Characterization of Algorithms

• Connections to Other Areas– Compressed Sensing– Linear Algebra– Machine Learning

Compressed Sensing• Compute a sketch A¢x with a small number

of rows (a.k.a. measurements)

• Output x’ which approximates x in the sense that

|x’-x|p · (1+ε) |x-xk|qwhere xk is the best k-sparse approximation

to x

• Similar to heavy hitters problem solved by CountSketch

• Variations of CountSketch + subsampling:• Can design algorithms with near-optimal

number of measurements as a function of various ε, k, p, q [Price, W]

• For p = q = 2, can reduce number of measurements by adaptively invoking CountSketch [Indyk, Price, W]

x

x2

Linear Algebra• Least squares regression

– Fitting points to a line, or more generally a subspace

– minx |Ax-b|2 for n x d matrix A, n x 1 vector b

– Typically n >> d, i.e., the problem is over-constrained

Linear Algebra• If S is a random projection matrix:

– compute S*A and S*b, – solve minx |SAx-Sb|2

– Intuition: randomly rotate the column span of [A°b], then drop all but first O(d) coordinates

• (0, 0, 0, …, 0, 1) 2 Rn

• After rotation approximately:(± 1/n1/2, …, ± 1/n1/2)

• Drop all but first d coordinatesand rescale by (n/d)1/2

(± 1/d1/2, …, ± 1/d1/2) 2 Rd

Linear Algebra• 1+ε approximation in O(nd log n) + poly(d/ε) time using Fast

Johnson Lindenstrauss Transforms (restricted family of projections)

• If replace S with CountSketch, this still works! [Clarkson, W]– Leads to running time O(nnz(A)) + poly(d/ε) time, where nnz(A)

is the number of non-zero entries of A

• Low Rank Approximation– Using CountSkech instead of Fast Johnson Lindenstrauss

improves running time from O(nd log n) to O(nnz(A)) [Clarkson, W]

• Beautiful followup works by Li, Mahoney, Meng, Miller, Nelson, Nguyen, Peng

Machine Learning• CountSketch can be used to estimate inner products

– Estimate <x,y> as <Sx, Sy>– E[<Sx, Sy>] = <x,y>– Var[<Sx, Sy>] · |x|2 |y|2/B

• Replace expensive inner product computations in classification algorithms with approximations via CountSketch – perceptron and minimum enclosing ball [Clarkson, Hazan, W]

• Often interested in non-linear kernel transformations of input points: x1, …, xn -> f(x1), …, f(xn)– “Tensor product” CountSketch of Pagh gives subspace embeddings of

the polynomial kernel [Avron, Nguyen, W]

Conclusions

• Many data stream and sketching techniques give efficient ways of “compressing” big data – a broadly applicable goal in computer science– Compressed sensing, graph algorithms, linear

algebra, machine learning… – Recently been looking at shape-fitting and clustering

problems, etc.– Also useful for proving lower bounds in other areas,

e.g., number of measurements in sparse recovery [DoBa,Indyk,Price,W]

– I’m sure there are many other unexplored areas

• Thank you!