analysis - university of pennsylvaniaryrogers/leverage_talk.pdf · at the end of may, the imagenet...
TRANSCRIPT
![Page 1: Analysis - University of Pennsylvaniaryrogers/Leverage_Talk.pdf · At the end of May, the ImageNet Large Scale Visual Recognition Competition, or LSVRC (currently the word's top annual](https://reader036.vdocuments.mx/reader036/viewer/2022071405/60f9d4f2df7c006471449bad/html5/thumbnails/1.jpg)
![Page 2: Analysis - University of Pennsylvaniaryrogers/Leverage_Talk.pdf · At the end of May, the ImageNet Large Scale Visual Recognition Competition, or LSVRC (currently the word's top annual](https://reader036.vdocuments.mx/reader036/viewer/2022071405/60f9d4f2df7c006471449bad/html5/thumbnails/2.jpg)
Analysis 𝑡
𝑡
𝑡 𝑋
![Page 3: Analysis - University of Pennsylvaniaryrogers/Leverage_Talk.pdf · At the end of May, the ImageNet Large Scale Visual Recognition Competition, or LSVRC (currently the word's top annual](https://reader036.vdocuments.mx/reader036/viewer/2022071405/60f9d4f2df7c006471449bad/html5/thumbnails/3.jpg)
𝑡
𝑡 𝑋
Analysis 𝑡# ← 𝑡 𝑋
![Page 4: Analysis - University of Pennsylvaniaryrogers/Leverage_Talk.pdf · At the end of May, the ImageNet Large Scale Visual Recognition Competition, or LSVRC (currently the word's top annual](https://reader036.vdocuments.mx/reader036/viewer/2022071405/60f9d4f2df7c006471449bad/html5/thumbnails/4.jpg)
𝑡 𝑋
𝑡
Analysis 𝑡# ← 𝑡 𝑋𝑡# 𝑋#
A lot of existing theory assumes tests are selected independently of the data.
𝑡#
![Page 5: Analysis - University of Pennsylvaniaryrogers/Leverage_Talk.pdf · At the end of May, the ImageNet Large Scale Visual Recognition Competition, or LSVRC (currently the word's top annual](https://reader036.vdocuments.mx/reader036/viewer/2022071405/60f9d4f2df7c006471449bad/html5/thumbnails/5.jpg)
Ideal World
How can we provide statistically valid answers to adaptively chosen analyses?
Real World
![Page 6: Analysis - University of Pennsylvaniaryrogers/Leverage_Talk.pdf · At the end of May, the ImageNet Large Scale Visual Recognition Competition, or LSVRC (currently the word's top annual](https://reader036.vdocuments.mx/reader036/viewer/2022071405/60f9d4f2df7c006471449bad/html5/thumbnails/6.jpg)
Ideal World
How can we provide statistically valid answers to adaptively chosen analyses?
Real World
I
n
t
r
o
d
u
c
t
i
o
n
M
o
d
e
l
R
e
s
u
l
t
s
K
e
y
I
d
e
a
s
P
r
o
o
f
S
k
e
t
c
h
Adaptivity causes r
eal problems
I
n
t
r
o
d
u
c
t
i
o
n
M
o
d
e
l
R
e
s
u
l
t
s
K
e
y
I
d
e
a
s
P
r
o
o
f
S
k
e
t
c
h
Adaptivity causes real problems
Introduction Model Results Key Ideas Proof Sketch
Adaptivity causes real problems
![Page 7: Analysis - University of Pennsylvaniaryrogers/Leverage_Talk.pdf · At the end of May, the ImageNet Large Scale Visual Recognition Competition, or LSVRC (currently the word's top annual](https://reader036.vdocuments.mx/reader036/viewer/2022071405/60f9d4f2df7c006471449bad/html5/thumbnails/7.jpg)
𝑋~𝑃(
How can we provide statistically valid answers to adaptively chosen analyses?
𝑡𝑡(𝑋)
𝑡′ ← 𝑡 𝑋
𝑡′𝑡′(𝑋)
![Page 8: Analysis - University of Pennsylvaniaryrogers/Leverage_Talk.pdf · At the end of May, the ImageNet Large Scale Visual Recognition Competition, or LSVRC (currently the word's top annual](https://reader036.vdocuments.mx/reader036/viewer/2022071405/60f9d4f2df7c006471449bad/html5/thumbnails/8.jpg)
𝑋~𝑃(
𝑡𝑡(𝑋)
𝑡′ ← 𝑎
𝑡′𝑡′(𝑋)
Answer: Limit the info learned about the dataset with each analysis [Dwork,Feldman,Hardt,Pitassi,Reingold,Roth’15].
𝑎
![Page 9: Analysis - University of Pennsylvaniaryrogers/Leverage_Talk.pdf · At the end of May, the ImageNet Large Scale Visual Recognition Competition, or LSVRC (currently the word's top annual](https://reader036.vdocuments.mx/reader036/viewer/2022071405/60f9d4f2df7c006471449bad/html5/thumbnails/9.jpg)
[Dwork,McSherry,Nissim,Smith’06]
𝐴:𝐷( → 𝑌 𝜀, 𝛿𝑥, 𝑥′ ∈ 𝐷( 𝑆 ⊆ 𝑌
𝑃 𝐴 𝑥 ∈ 𝑆 ≤ 𝑒;𝑃 𝐴 𝑥# ∈ 𝑆 + 𝛿
![Page 10: Analysis - University of Pennsylvaniaryrogers/Leverage_Talk.pdf · At the end of May, the ImageNet Large Scale Visual Recognition Competition, or LSVRC (currently the word's top annual](https://reader036.vdocuments.mx/reader036/viewer/2022071405/60f9d4f2df7c006471449bad/html5/thumbnails/10.jpg)
[R, Roth, Smith, Thakkar’16].
[Gaboardi, Lim, R, Vadhan’16], [Kifer,R’16].
[R,Roth,Ullman,Vadhan’16].