beyond set disjointness: the communication complexity of finding the intersection grigory...
TRANSCRIPT
![Page 1: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/1.jpg)
Beyond Set Disjointness: The Communication Complexity of Finding the
Intersection
Grigory Yaroslavtsevhttp://grigory.us
Joint with Brody, Chakrabarti, Kondapally and Woodruff
![Page 2: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/2.jpg)
Communication Complexity [Yaoβ79]
Alice: Bob:
π (π ,π )=?
Shared randomness
β¦
π (π ,π )β’ = min. communication (error ) β’ min. -round communication (error )
![Page 3: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/3.jpg)
Set Intersection
π=πΊ ,π=π» , π (π , π )=πΊβ©π»πΊβ [π ] ,|π|β€π π» β [π ] ,|π|β€π = ?
(-Intersection) = ?
is big, n is huge, where huge big
![Page 4: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/4.jpg)
Our results
Let
β’ (-Intersection) = [Brody, Chakrabarti, Kondapally, Woodruff, Y.; PODCβ14]β’ (-Intersection) = [Saglam-Tardos FOCSβ13; Brody, Chakrabarti, Kondapally, Woodruff, Y.β; RANDOMβ14]
{
times
(-Intersection) = for
![Page 5: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/5.jpg)
Applications
β’ Exact Jaccard index (for -approximate use MinHash [Broderβ98; Li-Konigβ11; Path-Strokel-Woodruffβ14])β’ Rarity, distinct elements, joins,β¦β’ Multi-party set intersection (later)β’ Contrast:
![Page 6: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/6.jpg)
1-round -protocol
π : [π ]β[π3]
πΊ π»
π(πΊ) π(π» )
[π ] [π ]
[π3] [π3]
![Page 7: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/7.jpg)
Hashing
log π
=# of buckets
π : [π ]β[π / logπ]
Expected # of elements
![Page 8: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/8.jpg)
Secondary Hashing
= # of hash functions
log 3π where
![Page 9: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/9.jpg)
2-Round -protocol
log 3π
log 3π
|hπ (πΊ )|,|hπ (π» )|=π ( logπ log logπ )
Total communication = = O()
![Page 10: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/10.jpg)
Collisions
πlogπ
log 3πPr [ππππππ πππ ]=π( 1logπ )
![Page 11: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/11.jpg)
Collisions
log 3π
log 3π
Key fact: If then also =
![Page 12: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/12.jpg)
Collisions
β’ Second round: β For each bucket send -bit equality check (total -
communication)β Correct intersection computed in buckets where
β Expected # items in incorrect buckets β Use 1-round protocol for incorrect bucketsβ Total communication
![Page 13: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/13.jpg)
Main protocol
π (1)
=# of buckets
π : [π ]β[π]
Expected # of elements
![Page 14: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/14.jpg)
Verification tree -degree
β¦i logπ β1π
buckets = leaves of the verification tree
![Page 15: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/15.jpg)
Verification bottom-up
πΊπβ ,ππ
β πΊπβ ,ππ
β
πΊπββͺπΊπ ,ππ
ββͺπ» π
πΊπββ©ππ
βπΊπββ©ππ
β
(πΊπββͺπΊπ )β©(π ΒΏΒΏπββͺπ» π)ΒΏ
![Page 16: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/16.jpg)
EQUALITY CHECK
Verification bottom-up
πΊπββ©ππ
βπΊπββ©ππ
β
(πΊπββͺπΊπ )β©(π ΒΏΒΏπββͺπ» π)ΒΏ
Correct Incorrect
Incorrect
πΊπββ©ππ
βπΊπββ©ππ
β
(πΊπββͺπΊπ )β©(π ΒΏΒΏπββͺπ» π)ΒΏ
Correct Incorrect
![Page 17: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/17.jpg)
Correct
Verification bottom-up
πΊπββ©ππ
βπΊπββ©ππ
β
(πΊπββͺπΊπ )β©(π ΒΏΒΏπββͺπ» π)ΒΏ
Correct Incorrect
EQUALITY CHECK FAILS =>RESTART THE SUBTREE
πΊπββ©ππ
βπΊπββ©ππ
β
(πΊπββͺπΊπ )β©(π ΒΏΒΏπββͺπ» π)ΒΏ
Correct Incorrect
Correct
![Page 18: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/18.jpg)
Verification bottom-up
ππ βπ
β¦ππ
πΊππ ,ππ
π β¦ πΊππ ,π π’
ππΊππ ,ππ
π πΊππ ,ππ
πβ¦
ππ βπ
![Page 19: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/19.jpg)
Analysis of Stage
β’ = [node at stage computed correctly]β’ Set = β Run equality checks and basic intersection
protocols with success probability β Key lemma: [# of restarts per leaf => Cost of
Intersection in leafs = β Cost of Equality =
β’ [protocol succeeds] =
![Page 20: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/20.jpg)
Multi-party extensions
players: , where
β’ Boost error probability of 2-player protocol to β’ Average per player (using coordinator):
in roundsβ’ Worst-case per player (using a tournament)
in rounds
![Page 21: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/21.jpg)
Open Problems
β’ (-Intersection) = ?β’ Better protocols for the multi-party setting?
![Page 22: Beyond Set Disjointness: The Communication Complexity of Finding the Intersection Grigory Yaroslavtsev Joint with Brody, Chakrabarti,](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d765503460f94a57287/html5/thumbnails/22.jpg)
-Disjointnessβ’ , iff β’ [Razborovβ92; Hastad-Wigdersonβ96] β’ [Folklore + Dasgupta, Kumar, Sivakumar; Buhrmanβ12, Garcia-Soriano, Matsliah, De Wolfβ12]
β’ [Saglam, Tardosβ13]β’ [Braverman, Garg, Pankratov, Weinsteinβ13]