deep learning with differential privacylxiong/cs573/share/slides/06_dp_dl.pdf · 2018-10-16 ·...
TRANSCRIPT
![Page 1: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/1.jpg)
CS573 Data Privacy and Security
Differential Privacy – Machine Learning
Li Xiong
![Page 2: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/2.jpg)
Big Data + Machine Learning
+
![Page 3: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/3.jpg)
Machine Learning Under Adversarial Settings
• Data privacy/confidentiality attacks
• membership attacks, model inversion attacks
• Model integrity attacks
• Training time: data poisoning attacks
• Inference time: adversarial examples
![Page 4: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/4.jpg)
Differential Privacy for Machine Learning
• Data privacy attacks
• Model inversion attacks
• Membership inference attacks
• Differential privacy for deep learning
• Noisy SGD
• PATE
![Page 5: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/5.jpg)
Neural Networks
![Page 6: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/6.jpg)
![Page 7: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/7.jpg)
Learning the parameters: Gradient Descent
![Page 8: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/8.jpg)
Stochastic Gradient Descent
Gradient Descent (batch GD)
The cost gradient is based on the complete training set, can be costly and longer to converge to minimum
Stochastic Gradient Descent (SGD, iterative or online-GD)
Update the weight after each training sample
The gradient based on a single training sample is a stochastic approximation of the true cost gradient
Converges faster but the path towards minimum may zig-zag
Mini-Batch Gradient Descent (MB-GD)
Update the weights based on small group of training samples
![Page 9: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/9.jpg)
FacialRecognitio
n Model
Private training datasetPhilip
Jack
Monica
unknown
Input (facial image)
Output (label)
…
Training-data extraction attacks
Fredrikson et al. (2015) :
![Page 10: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/10.jpg)
Membership Inference Attacks
against Machine Learning Models
Reza Shokri, Marco Stronati, Congzheng Song, Vitaly Shmatikov
![Page 11: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/11.jpg)
Membership Inference Attack
5
Model
Training
DATA
Prediction
Input
data
Classification
Was this specific
data record part of
the training set?
airplane
automobile
…
ship
truck
![Page 12: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/12.jpg)
Membership Inference Attack
8
on Summary Statistics
• Summary statistics (e.g., average) on each attribute
• Underlying distribution of data is known
[Homer et al. (2008)], [Dwork et al. (2015)], [Backes et al. (2016)]
on Machine Learning Models
Black-box setting:
• No knowledge about the models’ parameters
• No access to internal computations of the model
• No knowledge about the underlying distribution of data
![Page 13: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/13.jpg)
9
Model
Training API
DATA
Prediction API
Exploit Model’s Predictions
Main insight:
ML models overfit to
their training data
![Page 14: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/14.jpg)
9
Model
Training API
DATA
Prediction API
Exploit Model’s Predictions
Input from
the training set Classification
Main insight:
ML models overfit to
their training data
![Page 15: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/15.jpg)
9
Model
Training API
DATA
Prediction API
Exploit Model’s Predictions
Input from
the training set
Input NOT from
the training setClassification
Classification
Main insight:
ML models overfit to
their training data
![Page 16: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/16.jpg)
9
Model
Training API
DATA
Prediction API
Exploit Model’s Predictions
Input from
the training set
Input NOT from
the training setClassification
Classification
Recognize the difference
![Page 17: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/17.jpg)
10
Model
Training API
DATA
Prediction API
Input from
the training set
Input not from the training set
Classification
Classification
recognize the differenceTrain a ML model to
ML against ML
![Page 18: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/18.jpg)
11
…
IN OUT IN OUT IN OUT
cla
ssific
atio
n
cla
ssific
atio
n
cla
ssific
atio
n
Shadow
Model 2
Shadow
Model k
Shadow
Model 1
Train Attack Model using
Shadow Models
Train the attack model
Train 1 Test 1 Train 2 Test 2 Train k Test k
to predict if an input was a member of the
training set (in) or a non-member (out)
![Page 19: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/19.jpg)
Obtaining Data for Training
Shadow Models
• Real: similar to training data of the target model
(i.e., drawn from same distribution)
• Synthetic: use a sampling algorithm to obtain data
classified with high confidence by the target model
12
![Page 20: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/20.jpg)
Constructing the Attack Model
14
Model
Prediction API
DATA
SYNTHETIC
ShadowShadowShadowShadowShadowShadowShadow
Models
DATA
AT TA C K Tr a i n i n g
Attack
Model
![Page 21: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/21.jpg)
Constructing the Attack Model
14
Model
Prediction API
Attack
Model membership
probabilityclassification
one single
data record
Using the Attack Model
Model
Prediction API
DATA
SYNTHETIC
ShadowShadowShadowShadowShadowShadowShadow
Models
DATA
AT TA C K Tr a i n i n g
Attack
Model
![Page 22: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/22.jpg)
15
Purchase Dataset — Classify Customers (100 classes)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
Cu
mu
lative
Fra
ctio
n o
f C
lasse
s
Real DataMarginal-Based Synthetic
Model-Based Synthetic
shadows trained
on real data
overall accuracy:
0.93
shadows trained
on synthetic data
overall accuracy:
0.89
Membership inference precision
![Page 23: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/23.jpg)
16
Privacy Learning
data universe
training set
Model
![Page 24: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/24.jpg)
16
Privacy Learning
data universe
training set
Model
Does the model leak
information about data
in the training set?
![Page 25: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/25.jpg)
16
Privacy Learning
data universe
training set
Model
Does the model leak
information about data
in the training set?
Does the model
generalize to data
outside the training set?
![Page 26: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/26.jpg)
16
Privacy Learning
data universe
training set
Model
Overfitting is
the common enemy!
Does the model leak
information about data
in the training set?
Does the model
generalize to data
outside the training set?
![Page 27: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/27.jpg)
Not in a Direct Conflict!17
Privacy-preserving
machine learning
Privacy
Utility
(prediction accuracy)
![Page 28: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/28.jpg)
Differential Privacy for Machine Learning
• Data privacy attacks
• Model inversion attacks
• Membership inference attacks
• Differential privacy for deep learning
• Noisy SGD
• PATE
![Page 29: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/29.jpg)
DEEP LEARNING WITH
DIFFERENTIAL PRIVACYMartin Abadi, Andy Chu, Ian Goodfellow*,
Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang
Google* OpenAI
![Page 30: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/30.jpg)
Differential Privacy
(ε, δ)-Differential Privacy: The distribution of the output
M(D) on database D is (nearly) the same as M(D′):
∀S : Pr[M(D)∊S] ≤ exp(ε) ∙ Pr[M(D′)∊S]+δ.
quantifies information leakage
allows for a small probability of failure
![Page 31: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/31.jpg)
Interpreting Differential Privacy
DD′
Training Data ModelSGD
![Page 32: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/32.jpg)
Differential Privacy: Gaussian Mechanism
If ℓ2-sensitivity of f:D→ℝn:
maxD,D′
||f(D) − f(D′)||2 < 1,
then the Gaussian mechanism
f(D) + Nn(0, σ2)
offers (ε, δ)-differential privacy, where δ ≈ exp(-(εσ)2/2).
Dwork, Kenthapadi, McSherry, Mironov, Naor, “Our Data, Ourselves”, Eurocrypt 2006
![Page 33: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/33.jpg)
Basic Composition Theorem
If f is (ε1, δ
1)-DP and g is (ε
2, δ
2)-DP, then
f(D), g(D) is (ε1+ε
2, δ
1+δ
2)-DP
![Page 34: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/34.jpg)
Simple Recipe for CompositeFunctions
Tocompute composite f with differential privacy
1. Bound sensitivity of f’scomponents
2. Apply the Gaussian mechanism to each component
3. Compute total privacy via the composition theorem
![Page 35: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/35.jpg)
Deep Learning with DifferentialPrivacy
![Page 36: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/36.jpg)
Differentially Private Deep Learning
softmax loss
MNIST andCIFAR-10
PCA+ neural network
1. Loss function
2. Training / Test data
3. Topology
4. Training algorithm
5. Hyperparameters
Differentially private SGD
tune experimentally
![Page 37: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/37.jpg)
![Page 38: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/38.jpg)
![Page 39: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/39.jpg)
![Page 40: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/40.jpg)
Naïve Privacy Analysis
1. Choose
2. Each step is (ε, δ)-DP
3. Number of steps T
4. Composition: (Tε, Tδ)-DP
= 4
(1.2, 10-5)-DP
10,000
(12,000, .1)-DP
![Page 41: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/41.jpg)
Advanced Composition Theorems
![Page 42: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/42.jpg)
Composition theorem
+ε for Blue
+.2ε for Blue
+ ε for Red
![Page 43: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/43.jpg)
Strong Composition Theorem
Dwork, Rothblum, Vadhan, “Boosting and Differential Privacy”, FOCS 2010
Dwork, Rothblum, “Concentrated Differential Privacy”, https://arxiv.org/abs/1603.0188
1. Choose = 4
2. Each step is (ε, δ)-DP
3. Number of steps T
(1.2, 10-5)-DP
10,000
4. Strong comp: ( , Tδ)-DP (360, .1)-DP
![Page 44: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/44.jpg)
Amplification by Sampling
1. Choose
2. Each batch is q fraction of data
3. Each step is (2qε, qδ)-DP
4. Number of steps T
5. Strong comp: ( , qTδ)-DP
= 4
1%
(.024, 10-7)-DP
10,000
(10, .001)-DP
S. Kasiviswanathan, H. Lee, K. Nissim, S. Raskhodnikova, A. Smith, “What Can We Learn Privately?”, SIAM J. Comp, 2011
![Page 45: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/45.jpg)
Moments Accountant
1. Choose
2. Each batch is q fraction of data
3. Keeping track of privacy loss’s moments
4. Number of steps T
5. Moments: ( , δ)-DP
= 4
1%
10,000
(1.25, 10-5)-DP
![Page 46: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/46.jpg)
Results
![Page 47: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/47.jpg)
Our Datasets: “Fruit Flies of Machine Learning”
MNIST dataset:
70,000 images
28⨉28 pixels each
CIFAR-10 dataset:
60,000 color images
32⨉32 pixels each
![Page 48: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/48.jpg)
Summary of Results
Baseline
no privacy
MNIST 98.3%
CIFAR-10 80%
![Page 49: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/49.jpg)
Summary of Results
Baseline [SS15] [WKC+16]
no privacyreports ε per
parameterε =2
MNIST 98.3% 98% 80%
CIFAR-10 80%
![Page 50: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/50.jpg)
Baseline [SS15] [WKC+16] this work
no privacyreports ε per
parameter ε =2ε =8
δ = 10-5
ε =2
δ = 10-5
ε =0.5
δ = 10-5
MNIST 98.3% 98% 80% 97% 95% 90%
CIFAR-10 80% 73% 67%
Summary of Results
![Page 51: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/51.jpg)
Contributions
● Differentially private deep learning applied to publicly
available datasets and implemented in TensorFlow
○ https://github.com/tensorflow/models
● Innovations
○ Bounding sensitivity ofupdates
○ Moments accountant to keep tracking of privacy loss
● Lessons
○ Recommendations for selection ofhyperparameters
● Full version: https://arxiv.org/abs/1607.00133
![Page 52: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/52.jpg)
Differential Privacy for Machine Learning
• Data privacy attacks
• Model inversion attacks
• Membership inference attacks
• Differential privacy for deep learning
• Noisy SGD
• PATE
![Page 53: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/53.jpg)
In their work, the threat model assumes:• Adversary can make a potentially unbounded number of queries• Adversary has access to model internals
![Page 54: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/54.jpg)
Private Aggregation of Teacher Ensembles (PATE)
Intuitive privacy analysis:• If most teachers agree on the label, it does not depend on specific partitions, so
the privacy cost is small.• If two classes have close vote counts, the disagreement may reveal private
information
1. Count votes2. Take maximum
![Page 55: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/55.jpg)
Noisy aggregation
![Page 56: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/56.jpg)
The aggregated teacher violates the threat model:• Each prediction increases total privacy loss.
privacy budgets create a tension between the accuracy and number of predictions
• Inspection of internals may reveal private data.Privacy guarantees should hold in the face of white-box adversaries
Private Aggregation of Teacher Ensembles (PATE)
1. Count votes2. Take maximum
![Page 57: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/57.jpg)
Private Aggregation of Teacher Ensembles (PATE)
Privacy Analysis:• Privacy loss is fixed after the student model is done training.• Even if white-box adversary can inspect the model parameters, the
information can be revealed from student model is unlabeled public dataand labels from aggregate teacher which is protected with privacy
![Page 58: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/58.jpg)
Generator:Input: noise sampled from randomdistribution
Output: synthetic input close to theexpected training distribution
Discriminator:Input: output from generator ORexample from real trainingdistribution
Output: in distribution OR fake
Gaussian
sample
Fakesample Sample
P(real) = …P(fake)= …
GANsIJ Goodfellow et al. (2014) Generative Adversarial Networks
2 computing models
![Page 59: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/59.jpg)
Generator:Input: noise sampled from randomdistribution
Output: synthetic input close to theexpected training distribution
Discriminator:Input: output from generator ORexample from real trainingdistribution
Output: in distribution (which class)OR fake
Gaussian
sample
Fakesample Sample
P(real0)= …P(real1)= ……P(realN)= …P(fake) = …
Improved Training of GANsT Salimans et al. (2016) Improved Techniques for Training GANs
![Page 60: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/60.jpg)
Private Aggregation of Teacher Ensembles using GANs (PATE-G)
Generator
Discriminator
PublicData
Queries
Not available to the adversary Available to the adversary
![Page 61: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/61.jpg)
Aggregated Teacher Accuracy Before the Student Model is Trained
![Page 62: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/62.jpg)
(2, 10−5)
(8, 10−5) 97%
95%
(0.5, 10−5) 90%
M Abadi et al. (2016) Deep Learning with Differential Privacy
Evaluation
increase # teachers will increase privacy guarantee, but decrease model accuracy# teachers is constrained by task’s complexity and the available data
![Page 63: DEEP LEARNING WITH DIFFERENTIAL PRIVACYlxiong/cs573/share/slides/06_DP_DL.pdf · 2018-10-16 · Machine Learning Under Adversarial Settings • Data privacy/confidentiality attacks](https://reader034.vdocuments.mx/reader034/viewer/2022042318/5f07ed2f7e708231d41f7431/html5/thumbnails/63.jpg)
Differential Privacy for Machine Learning
• Data privacy attacks
• Model inversion attacks
• Membership inference attacks
• Differential privacy for deep learning
• Noisy SGD
• PATE