lower bounds for read/write streams paul beame joint work with trinh huynh (dang-trinh huynh-ngoc)...
TRANSCRIPT
Lower Bounds for Read/Write Streams
Paul Beame
Joint work with Trinh Huynh (Dang-Trinh Huynh-
Ngoc)
University of Washington
Data stream Algorithms
• Many huge successes– No need to remind people at this workshop!
• Some problems provably hard
– E.g. Frequency moments Fk, k > 2 require space Ω(n1-2/k) [Bar-Yossef-Jayram-Kumar-Sivakumar 02], [Chakrabarti-Khot-Sun 03]
Beyond Data Streams
• Disk storage can be huge– Can stream data to/from disks in real time
• Sequential access hides latency– Motivates multipass streams
• Analyzed by similar methods to single pass
• Why stop at a single copy?– Working with more than one copy at once may
make computations easier
• Why stream the data onto disks exactly as read?– Can make modifications to data while writing
0 1 0 0 1 1 1 0 1 0 0 0 1 0 1 0 1 0 0 0 1 0
0 0 0 0 1 0 1 1 1 1 0 0 1 1 1 1 0 1 0 0 0 0
Read/write streams model
• Disks read/write streams– Key Parameters: space, #passes=reversals– Assume #streams is constant
• Introduced by [Grohe-Schweikardt 05]
0 1 0 0 1 1 1 0 1 0 0 0 1 0 1 0 1 0 0 0 1 0
memory
0 0 1 1 1 1 0 1 0
Read/write streams model
• Much more powerful than data-stream model– Sort with O(log n) passes, O(log n) space, 3
streams• MergeSort
– Exactly compute any frequency moment• Data-stream requires passes space = Ω(n)
– Θ(log n) passes, O(1) space gives all of LOGSPACE [Hernich-Schweikardt 08]
What can be computed in o(log n) passes + small space?
Previous lower bounds for R/W streams
• In o(log n) passes need Ω(n1-ε) space to– Sort n numbers
[Grohe-Schweikardt 05]– Test set-equality A=B, multiset equality,
XQuery, XPath
[Grohe-Hernich-Schweikardt 06]
• Same lower bounds apply for randomized algorithms with one-sided error [Grohe-Hernich-Schweikardt 06]
Previous lower bounds for R/W streams
• Lower bounds for general randomness and two-sided error:– In o(log nlog log n) passes, need Ω(n1-ε)
space to:• Approximate F
* within factor 2 • Find Empty-Join, XQuery/XPath-Filtering etc.
[B-Jayram-Rudra 07]
What about approximating frequency moments Fk for k 2 ?
Our Main Result
Theorem: Any randomized R/W-stream algorithm using o(log n) passes needs Ω(n1-4/k-ε) space to 2-approximate Fk
• Implies polynomial space for k>4
• Compare with: Θ(n1-2/k) on data streamsR/W streams with o(log n) passes don’t help
much for approximating frequency moments.R/W streams with o(log n) passes don’t help much for approximating frequency moments.
Methods
1. Reduce testing t-party set-disjointness to Fk
Easy!
2. Simulate any data-stream algorithm by amulti-party number-in-hand communication game
Trivial!
3. Apply Ω(n/t) communication lower bound on t-party set-disjointness
[AMS 96,Saks-Sun 02,Bar-Yossef-Jayram-Kumar-Sivakumar
02, Chakrabarti-Khot-Sun 03,Grönemeier 09] (tight!)
[Alon-Matias-Szegedy 96] approach to lower bounding Fk in data streams
Fails for R/W streams!
Fails for R/W streams!
Solved easily by R/W streams!Solved easily by R/W streams!
Cannot be applied to R/W streams!
Cannot be applied to R/W streams!
Promise Set-Disjointness (DISJ)
0, x1,…,xt are pair-wise disjoint
DISJn,t(x1,…,xt) = 1, a s.t. a xi for every i
Undefined otherwise
0 1 0 1 0 0 1 0 1 0 0 0 1 0 01 0 0 0 1 0 0 0 1 0 0 0 0 0 10 0 1 0 0 0 0 1 1 0 1 0 0 0 00 0 0 0 0 1 0 0 1 0 0 1 0 0 00 0 0 0 0 0 0 0 1 0 0 0 0 1 0
x1
x2
x3
x4
x5
• t-party NIH communication: Ω(nt)• Approximating Fk testing DISJn,t for t n1/k
xtxt-1x2x1
• Testing DISJn,t with 2 streams,3 passes,O(log n)
space
• Input: x1,x2,…,xt{0,1}n
R/W streams easily solve DISJn,t
x1 x2 xt-1 xt
• Lower bounds [GS05], [GHS05], [BJR07] for R/W streams don’t use [AMS96] outline
– Introduce permuted 2-party versions of problems
– Employ ad-hoc combinatorial arguments
How to prove lower bounds in R/W streams?
We take a more general approach related to [AMS96] directly using NIH comm. complexity
Our approach to lower bound Fk
R/W streams algorithm for
t-party-permuted-DISJ
on input size n
Number-in-hand communication protocol for t-party-DISJ
on input size nt2
1.Reduce testing t-party set-disjointness to Fk
Easy!
2.Simulate data-stream algorithms bymulti-party number-in-hand communication game
Apply our simulation
3.Apply communication lower bound on t-party set-disjointness
[AMS96,SS02,B-YJKS02,CKS03,G09] (tight!)
2. Simulate R/W streams for permuted DISJ by NIH comm. for DISJ on slightly smaller input size
1. Reduce testing permuted t-party DISJ to Fk
[Alon,Matias,Szegedy 96]’s approach to lower bound Fk in data streamOur approach to lower bound Fk
in R/W streams
Ideas from the proof
Segmenting DISJn,t
Input: x1,x2,…,xt{0,1}n
• View DISJn,t as an OR of m subproblems DISJn/m,t
x1 x2 xt-1 xt
1 2 m
nm 1 2 m
nm
Fix 1,2,…,t permutations on [m]
Permuted-DISJn,m,t
• View Permuted-DISJn,m,t as an OR of m subproblems
DISJn/m,t
Permuted DISJ
1(1) 1(2) 1(m)
1(x1) 2(x2) t(xt)
1 2 m
DISJn/m,tDISJn/m,t
nm
DISJn/m,tDISJn/m,t
1 2 m
nm
t(1) t(2) t(m)
• Intuitively, to solve a subproblem (e.g. blue), we need to compare at least two blue
segments
• Need to compare at least two segments of every color
• If segments are shuffled, many passes are needed
Why is permuted-DISJ hard?
i(xi) j(xj) l(xl)
DISJn/m,tDISJn/m,t
Permuted DISJ• Good subproblem: computation always depends
only on at most one of its t segments (and the memory/state)
• If segments are randomly shuffled:With o(log m) passes, t=o(m1/2) parties,
99% of the m subproblems are good• Reduction idea: Try to embed an ordinary
DISJn/m,t in one of the good subproblems
Catch: Which subproblems are good depends on input
t players on input y1,y2,…,yt:1. Generate m-1 DISJn/m,t’s
that look like* y1,y2,…,yt
2. Shuffle with 1,2,…,t
• (y1,y2,…,yt) is good w.h.p
3. Run A on 1(x1),…,t(xt)
Simulation
s-space R/W streams algo A for permuted-
DISJn,m,t
NIH comm. protocol
for DISJn/m,t
y1
y2
1(x1)
2(x2)
x1
x2
*same sizes but don’t intersect
Generating the extended input
Given y1,y2,…,yt, players– Exchange the sizes of each of the sets
• O(t log n) bits– Choose random consistent reordering of the indices
of each y1,y2,…,yt
– Generate m-1 random inputs to DISJn/m,t with same set sizes as y1,y2,…,yt but that are disjoint
– Place y1,y2,…,yt in random position and then shuffle
Key observation: If y1,y2,…,yt are disjoint then this resolves the catch– After shuffling, all the subproblems look the same
so the probability that the subproblem where y1,y2,…,yt lands is good does not depend on the input
Simulating R/W stream algorithm A using NIH
communication• As A executes on input v=1(x1),…,t(xt) players
know all inputs except y1,…,yt – each player builds up copy of a dependency graph
σ(v) for the elements of each stream so far• Using σ(v), at each step all players either
– know the next move, or – know which one player knows next block of moves
• that player communicates – know that need two players’ info: simulation
“fails” • If subproblem y1,…,yt is good for v then simulation
does not fail• If players detect failure they output “not disjoint”
– If input was disjoint then only 1% chance of this
Dependency Graph
pass j
pass j+1
Stream R to L Stream L to R
Stream L to R
Vertices: Elements of each stream in each passEdges: From element to elements in previous pass that contained heads at same time it did
pass j -1
pass 0
pass 1
Why most subproblems are good
• Simple case: algorithm just makes copies of the input stream and compares them– # of subproblems with > 1 segment read at same
time on single pass through the streams (L-to-R or R-to-L on each stream)
• ≤ # segments appearing in the same (or reversed) order
– Almost surely, for random permutations 1,2,…,t
no pair has a common subsequence or inverted subsequence longer than 2em1/2
– When t is o(m1/2) the total is o(m).
Why most subproblems are good
• General case: May combine information about all streams onto a single stream in single pass– What is combined may depend on the
input values
– Each element depends on the segments that it can reach in the input stream via the dependency graph
• For each fixed v, after p=o(log m) passes: – Each element can depend on only 2O(p) different
input segments
– For any one stream, the sequence of its
elements’ dependencies on input segments is
the interleaving of 2O(p) monotone
subsequences from 1,2,…,t
Only 2O(p) t m1/2=mo(1) bad subproblems on
input v
Why most subproblems are good
Communication Cost of Simulation
• For each fixed v, after p=o(log m) passes: – Only 2O(p) t elements depend on a segment and
have a neighbor that does not depend on it
• Players only need to communicate when segment dependencies change – only happens 2O(p)t times at cost of O(ps) bits
per time
Limitations and Future Work
• Gap from data stream due to loss in input size
• Most of this loss is necessary– Need nm (t2) to use Ω(n/t) CC lower bound for
DISJn/m,t
– Efficient R/W algo for permuted-DISJn,m,t unless m ≥ t32
– Implies that n is Ω(mt2) which is Ω(t3.5)
Since we need t≈n1/k, the lower bound Ω(n/t) is trivial for k 3.5
Limitation of using permuted-DISJ
R/W streams algo for
permuted-DISJn,m,t
NIH CC protocol for DISJn/m,t
• Algorithm for permuted-DISJn,m,t follows from the following theorem:
Proof: For each i [m] define a triple ti of integers:
For each of the 3 pairs of permutations put length of the longest common subsequence for that pair that ends with value i. Can show that all m triples are different.
So some triple must contain a coordinate ≥ m1/3
• Tight even for 4 permutations
In any 3 permutations on [m] there is a pair
with
longest common subsequence length ≥
m1/3.
In any 3 permutations on [m] there is a pair
with
longest common subsequence length ≥
m1/3.
A longest-common-subsequence problem on
permutations
t m2/3, any : Testing permuted-DISJn,m,t
with 2 streams, 3 passes, O(log nmt) space
R/W stream algorithm for permuted-DISJn,m,t for large t
In any three permutations on [m] there is a pair
with
longest common subsequence length ≥ m1/3.
In any three permutations on [m] there is a pair
with
longest common subsequence length ≥ m1/3.
1(x1) 2(x2) 3(x3) 4(x4) 5(x5) 6(x6)
1(x1) 2(x2) 3(x3) 4(x4) 5(x5) 6(x6)
• Compare m1/3 blocks each time
Open problems
• Is Ω(n1-4/k-ε) lower bound for R/W streams tight?– Gap from O(n1-2/k) upper bound in data stream
• Can’t use permuted-DISJn,m,t to close it
– Polynomial space to compute Fk for 2 < k ≤ 4 ?
• Other problems on R/W streams?• L(m,k) maximum LCS length that can be guaranteed
between some pair in any set of k permutations on [m].
– We show L(m,3) L(m,4) m1/3
– What is L(m,k) for other values of k?
– [B-Blais-Huynh 08] L(m,k) = m1/3+o(1) for k mO(1)