private systems

Private Systems, a Columbia University Research Seminar, Instructor: Roxana Geambasu

Private Systems

CUCS Research Seminar https://columbia.github.io/private-systems-class/

Instructor: Roxana Geambasu

1


2

Differential Privacy Basics

Part 3: Algorithms and Implementations


DP Algorithms

●  There are some good DP libraries that implement many basic algorithms for statistical data analysis: ○  Google’s library (basic statistics): [Wilson+20],

https://github.com/google/differential-privacy/blob/master/differential_privacy. ○  IBM’s library (k-means, linear/logistic regression, PCA): [Holohan+19],

https://github.com/IBM/differential-privacy-library. ○  Tensorflow privacy (stochastic gradient descent, or SGD): [McMahan+18],

https://github.com/tensorflow/privacy.

●  Let’s look at the implementations of some of these algorithms. We’ll describe them in words, but best is to read them in code. ○  We focus on Google’s statistics library in these slides.

3


Bounded Sum https://github.com/google/differential-privacy/blob/master/differential_privacy/algorithms/bounded-sum.h

4

●  Goal: Compute sum over a set of values bounded in range [L,U]. The range is assumed either to be public knowledge, or obtained through a DP “approximate bounds” algorithm that we’ll see later. The range is enforced, i.e., any value that doesn’t fit in it is “clamped.”

●  Method: ○  Laplace mechanism with sensitivity max(|L|,|U|). ○  If L and U need to be approximated, use part of the privacy budget for

the approximate bounds algorithm, and the remainder for the Laplace mechanism.


Bounded Mean https://github.com/google/differential-privacy/blob/master/differential_privacy/algorithms/bounded-mean.h

5

●  Goal: Compute mean over a set of values bounded in range [L,U] (as before).

●  One option: ○  Compute DP sum: S (with half the budget) ○  Compute DP count: N (with half the budget) ○  Return S/N. But Sum has sensitivity max(|U|,|L|).

●  Observe: ○  For fixed N, the mean has L1-sensitivity 1/N*max(|U|,|L|). ○  So it’s tempting to compute the mean and apply Laplace for that sensitivity. ○  But that assumes the count is public information, but usually it isn’t.


6

Bounded Mean https://github.com/google/differential-privacy/blob/master/differential_privacy/algorithms/bounded-mean.h

●  Google’s implementation rewrites the average in terms relative to the middle of the interval [L,U].

●  That enables calculating the sum of all input values with lower sensitivity than it would take if doing noisy sum / noisy count: |U-L|/2.

average =xi∑N

=middle+(xi −middle)∑

N

noisy _ average =middle+xi − N ⋅middle∑ + Lap(|U − L | /ε)

N + Lap(2 / ε)

function 1, sensitivity: |U-L|/2

function 2, sensitivity: 1

middle = L+U2

⎛

⎝⎜

⎞

⎠⎟


7

Bounded Variance https://github.com/google/differential-privacy/blob/master/differential_privacy/algorithms/bounded-variance.h

●  Goal: Compute variance over a set of values bounded in [L,U] (as before).

●  Method:

○  Variance can be written as [mean of squares – mean squared]

○  DP variance can therefore be computed by applying the preceding DP Bounded Mean algorithm twice, each time with half of the budget.

variance =(xi −µ)

2∑n

=xi2∑

N−

xi∑N

⎛

⎝⎜⎜

⎞

⎠⎟⎟

2


Approximate Bounds https://github.com/google/differential-privacy/blob/master/differential_privacy/algorithms/approx-bounds.h

●  Goal: Establish approximate [L, U] range over a set of values. ●  Method (described for non-negatives, but can be adapted for negatives too):

○  Organize data into a given number of logarithmic histogram bins: e.g., [0,1], (1,2], (2,4], (4,8],…

○  Each bin keeps a DP count of the number of values in its range. Bin i holds the DP count of values in range

○  Given a confidence parameter c, we choose a threshold t after which to declare a bin as non-empty with probability >=c. The formula for t is:

(justified in [Wilson+20]).

○  L := leftmost bin with DP count > t. U := rightmost bin with DP count > t. 8

scale ⋅basei−1,scale ⋅basei( ⎤⎦.

t = 1εlog(1− c

1base−1 )


Approximate Bounds https://github.com/google/differential-privacy/blob/master/differential_privacy/algorithms/approx-bounds.h

●  Example: scale = 1, base = 2, num_bins = 4, inputs = {0, 0, 0, 0, 1, 3, 7, 8, 8, 8} ○  The bins and (non-DP) counts are: [0, 1]: 5 ; (1, 2]: 0 ; (2, 4]: 1 ; (4, 8]: 4 . ○  If success_probability=.9 and epsilon=1 we get threshold t=3.5. ○  Since the count of bin (4, 8] > t we would return max := 2^3 = 8. ○  Since the count of bin [0,1] > t we would return min := 0. ○  With DP counts, the procedure gives approximate values of course.

●  The parameters of this scheme – num_bins, scale, base – are not trivial to set without some external knowledge of the rough range you want to capture. But if you have a rough range, you can get a data-based approximate range privately with this method. 9


10

Binary Search https://github.com/google/differential-privacy/blob/master/differential_privacy/algorithms/binary-search.h

●  No slides, please refer to code.


Some ML Algorithms

●  Let’s review DP versions of some ML algorithms. ●  These are not exactly the algos implemented in IBM’s/Tensorflow’s DP

ML libraries, but they help us understand the basics.

11


Stochastic Gradient Descent (SGD)

●  Without privacy, SGD is: ○  Model function fθ ○  Training dataset D= {(x1,y1), …, (x|D|,y|D|)}

○  Repeat: ■  Sample a batch from D. ■  Learn from the batch by calculating update Δθ. ■  θ := θ + Δθ.

●  With privacy, one method is to noise the gradients Δθ [Abadi+16]. This is what we will describe next.

15


16

Deep Learning With DP


17

Deep Learning With DP


Privacy Analysis ●  The preceding algorithm receives as arguments a noise scale factor ,

gradient norm C, batch size L, and number of iterations T. ●  Because each step uses the Gaussian mechanism, the gradient at each

step is (ε,δ)-DP with respect to the batch. ε,δ are determined from σ. ●  Question: What’s the DP guarantee after many steps for the gradient w.r.t. the

dataset? Answer: apply Amplification Theorem and Advanced Composition.

18

σ

One step, Within the group

(ε,δ)-DP

Amplification Theorem One step,

Within the dataset

Advanced Composition

Many steps, Within the dataset

(2qε, qδ)-DP O 2qε T log(1/ δ)( ),Tqδ( )−DP


Determining ε,δ from σ ●  To make a real-valued function DP with the Gaussian

mechanism, we add noise from a normal distribution with standard deviation shown to the right.

●  In the preceding algorithm, we added noise with standard deviation σC, where C=Δ2.

●  Fixing δ to something reasonable, we can now compute ε based on σ, as shown on the right.

Δ2ε

2ln 1.25δ

⎛

⎝⎜

⎞

⎠⎟

ε =1σ

2ln 1.25δ

⎛

⎝⎜

⎞

⎠⎟


(ε,δ)-DP


Within the dataset



(2qε, qδ)-DP O 2qε T log(1/ δ)( ),Tqδ( )−DP19


Amplification Theorem

●  N: data size; L: size of each batch ●  Let q := L/N

●  Amplification Theorem: if gradient is (ε,δ)-DP within the batch, then it is (2qε, qδ)-DP within the dataset.


(ε,δ)-DP


Within the dataset

(2qε, qδ)-DP



O 2qε T log(1/ δ)( ),Tqδ( )−DP20



●  Applying the same (ε,δ)-DP algorithm T times will give an

-DP algorithm.

21


(ε,δ)-DP


Within the dataset



O ε T log(1/ δ)( ),Tδ( )

(2qε, qδ)-DP O 2qε T log(1/ δ)( ),Tqδ( )−DP


More Wrinkles ●  There’s finer analysis that can be done given that the noise is Gaussian. See

moments accountant in [Abadi+16], which ultimately gives:

●  Training pipelines usually involve more processes than (e.g.) SGD. Those also need to be made DP. For example, data preprocessing (e.g., normalization, centering, etc.) often transforms the data based on quantities computed from the same dataset. Those quantities need to be estimated DP [McMahan+18]. Hyperparameter optimizations . Even within a DNN, some layers (e.g., batch norm) rely on batch-level aggregates and need to be replaced with instance-level versions (e.g., instance norm). Thus, in practice making an ML pipeline DP will involve more changes to an ML pipeline than just switching to DP SGD. 22

2qε T ,δ( )−DP


Alternative Approaches

●  There are alternatives for DP-training models in a more black box way.

23

•  One is Google’s PATE [Papernot+18], which trains an ensemble of vanilla (non-DP) models on disjoint portions of the data. Then it uses a DP voting function to either do DP predictions from a non-DP model or to help train another “student“ model on public data.


24

Differential Privacy Basics

Part 4: Testing DP Implementations


Testing DP Implementations

●  Testing is a critical component of any software engineering process.

●  Testing that algorithm implementations satisfy DP is still a very new form of “art” for which no well established patterns and tools exist.

●  A good set of initial guidelines are provided in [Kifer+20], which I strongly recommend reading.

○  This includes: discussion of code review, white box testing and black-box testing of various components of a DP system.

●  In terms of tools, Google’s DP library includes a black-box, stochastic tester of the DP property, which we’ll review here.

25


Black-Box Testing: Key Ideas

●  The goal of black-box testing is to identify a neighboring pair of input databases, x and x’, plus a set S of possible outputs with property that:

●  If such x,x’,S are foud, they constitute a counter example that demonstrates violations of DP.

●  Searching for appropriate x,x’,S requires running f many times. Heuristic techniques for identifying promising choices for x,x’ exist [Ding+18].

●  Typically it suffices to use small input datasets with few rows/columns. You probably want to test datasets whose records are as different from one another as possible, so they can “leak” easier if they do leak. 26

Pr( f (x)∈ S) > eε Pr( f (x ')∈ S)+δ.


Black-Box Testing

●  Example, for sum/avg: ○  Dataset 1: empty ○  Dataset 2: {0} ○  Dataset 3: {100, 0} ○  Dataset 4: {100,50,0} ○  …

●  S might be [0,100], [0, inf],…

27


Black-Box Testing

●  Once a set of {<x,x’,S>} to test are chosen, verifying the property can be done statistically by running the mechanism on both inputs many times and using a hypothesis test to determine if there is enough evidence to conclude that:

●  In general, it’s best to test small computations this way (e.g., individual sum, avg, …), and then to use more standard unit testing to test for privacy loss accounting at coarser granularity (with composition and post-processing).

28

Pr( f (x)∈ S) > eε Pr( f (x ')∈ S)+δ.


Google’s Stochastic Tester https://github.com/google/differential-privacy/tree/master/differential_privacy/testing

Described in [Wilson+20]: 1.  Database generation 2.  Search procedure to find database pairs 3.  Output generation 4.  Predicate verification

29


Database Generation

●  All their mechanisms are scale independent, so they can check only in a unit range [−r, r].

●  Of course, they can’t enumerate all possible databases [−r, r]N, where N is the size of the database.

●  Instead, they try to generate a diverse set of databases.

●  They use the Halton sequence to do so. ○  Unlike uniform random sampling, Halton

sequences ensure that databases are evenly distributed and not clustered together.

30

Di�erentially Private SQL 14

the engine). Thus, in a completely untrusted environ-ment, the engine should catch all errors and silently ig-nore them, to avoid leaking information in the sameway; and it should be hardened against crashes. We donot think that we can completely mitigate this prob-lem, and silently catching all errors severely impedesusability. Thus, is it a good idea to add additional riskmitigation techniques, like query logging and auditing.

Interestingly, fixing the floating-point issue in [36]leads to a di�erent issue when using the Laplace mech-anism in · -thresholding. The secure version of Laplacemechanism requires rounding the result to the near-est r, where r is the smallest power of 2 larger than1/Á. If the · -thresholding check is implemented asif (noisy_count >= tau), then a noisy count of e.g.38.1 can be rounded up to e.g. 40 (with r = 4). If thethreshold · is 39, and the ” calculation is based on atheoretical Laplace distribution, then a noisy count of38.1 shouldn’t pass the threshold, but will: this leadsto underestimating the true ”. This can be fixed by us-ing a non-rounded version of the Laplace mechanism forthresholding only; as the numerical output is never dis-played to the user, attacks described in [36] don’t apply.

5.3 Stochastic testing

While the operations used in the engine are theoreti-cally proven to be di�erentially private, it is crucial toverify that these operations are implemented correctly.Since the number of possible inputs is unbounded, it isimpossible to exhaustively test this. Thus we fall backto stochastic testing and try to explore the space ofdatabases as e�ciently as possible. This does not giveus a guarantee that an algorithm passing the test is dif-ferentially private, but it is a good mechanism to detectviolations.

Note that we focus on testing DP primitives (ag-gregation functions) in isolation, which allows us to re-strict the scope of the tests to row-level DP. We thenuse classical unit testing to independently test contri-bution bounding. We leave it as future work to extendour system to handle end-to-end user-level DP testing.

Our testing system contains four components:database generation, search procedure to find databasepairs, output generation, and predicate verification.

5.3.1 Database generation and testing

What databases should we be generating? All DP aggre-gation functions are scale-invariant, so without loss ofgenerality, we can consider only databases with values

in a unit range [≠r, r]. Of course, we can’t enumerateall possible databases [≠r, r]S , where S is the size of thedatabase. Instead, we try to generate a diverse set ofdatabases. We use the Halton sequence [18] to do so.As an example, Figure 5 plots databases of size 2 gen-erated by a Halton sequence. Unlike uniform randomsampling, Halton sequences ensure that databases areevently distributed and not clustered together.

Fig. 5. 256 points over a [≠0.5, 0.5]2 unit square: Halton sequence

A database is a set of records: we consider its powerset, and find database pairs by recursively removingrecords. This procedure is shown in Figure 6.

Fig. 6. Database Search Graph for a database {e1, e2, e3}

5.3.2 DP predicate test

Once we have pairs of adjacent databases, we describehow we test each pair (D1, D2). The goal is to checkthat for all possible outputs S of mechanism f :

Pr[f(D1) œ S] Æ eÁ Pr[f(D2) œ S] + ”.

By repeatedly evaluating f on each database, weestimate the density of these probability distributions.We then use a simple method to compare these distri-butions: histograms.

We illustrate this procedure in Figure 7a and Fig-ure 7b. The upper curves (in orange) are the upper DP


Search for Neighboring DBs

●  A database is a set of records: they consider its power set, and find database pairs by recursively removing records.

●  This procedure is shown to the side.

31


the engine). Thus, in a completely untrusted environ-ment, the engine should catch all errors and silently ig-nore them, to avoid leaking information in the sameway; and it should be hardened against crashes. We donot think that we can completely mitigate this prob-lem, and silently catching all errors severely impedesusability. Thus, is it a good idea to add additional riskmitigation techniques, like query logging and auditing.

Interestingly, fixing the floating-point issue in [36]leads to a di�erent issue when using the Laplace mech-anism in · -thresholding. The secure version of Laplacemechanism requires rounding the result to the near-est r, where r is the smallest power of 2 larger than1/Á. If the · -thresholding check is implemented asif (noisy_count >= tau), then a noisy count of e.g.38.1 can be rounded up to e.g. 40 (with r = 4). If thethreshold · is 39, and the ” calculation is based on atheoretical Laplace distribution, then a noisy count of38.1 shouldn’t pass the threshold, but will: this leadsto underestimating the true ”. This can be fixed by us-ing a non-rounded version of the Laplace mechanism forthresholding only; as the numerical output is never dis-played to the user, attacks described in [36] don’t apply.

5.3 Stochastic testing

While the operations used in the engine are theoreti-cally proven to be di�erentially private, it is crucial toverify that these operations are implemented correctly.Since the number of possible inputs is unbounded, it isimpossible to exhaustively test this. Thus we fall backto stochastic testing and try to explore the space ofdatabases as e�ciently as possible. This does not giveus a guarantee that an algorithm passing the test is dif-ferentially private, but it is a good mechanism to detectviolations.

Note that we focus on testing DP primitives (ag-gregation functions) in isolation, which allows us to re-strict the scope of the tests to row-level DP. We thenuse classical unit testing to independently test contri-bution bounding. We leave it as future work to extendour system to handle end-to-end user-level DP testing.

Our testing system contains four components:database generation, search procedure to find databasepairs, output generation, and predicate verification.

5.3.1 Database generation and testing

What databases should we be generating? All DP aggre-gation functions are scale-invariant, so without loss ofgenerality, we can consider only databases with values

in a unit range [≠r, r]. Of course, we can’t enumerateall possible databases [≠r, r]S , where S is the size of thedatabase. Instead, we try to generate a diverse set ofdatabases. We use the Halton sequence [18] to do so.As an example, Figure 5 plots databases of size 2 gen-erated by a Halton sequence. Unlike uniform randomsampling, Halton sequences ensure that databases areevently distributed and not clustered together.

Fig. 5. 256 points over a [≠0.5, 0.5]2 unit square: Halton sequence

A database is a set of records: we consider its powerset, and find database pairs by recursively removingrecords. This procedure is shown in Figure 6.

{e1, e2, e3}

{e1, e2} {e1, e3} {e2, e3}

{e1} {e2} {e3}

{}

Fig. 6. Database Search Graph for a database {e1, e2, e3}

5.3.2 DP predicate test

Once we have pairs of adjacent databases, we describehow we test each pair (D1, D2). The goal is to checkthat for all possible outputs S of mechanism f :

Pr[f(D1) œ S] Æ eÁ Pr[f(D2) œ S] + ”.

By repeatedly evaluating f on each database, weestimate the density of these probability distributions.We then use a simple method to compare these distri-butions: histograms.

We illustrate this procedure in Figure 7a and Fig-ure 7b. The upper curves (in orange) are the upper DP


Output Generation & Predicate Verification

●  Once they have pairs of adjacent databases, they need to test each pair (x, x’). The goal is to check that for all possible outputs S of mechanism f, the DP formula holds.

●  By repeatedly evaluating f on each database, they estimate the density of these probability distributions. They then use histograms to compare these distributions.

32


Example

●  Orange: DP upper bounds, created by multiplying the probability estimate of each bin for database x by eε and adding δ.

●  Blue (underneath): the unmodified probability estimate of x’ for each bin.

●  In (a), orange >= blue in all bins. In (b), orange < blue in three bins.

●  (b) provides a counterexample to DP property: x,x’,S (S being those bins where it is violated).

●  Since all of this is stochastic, one needs to assess that the differences are not by mere chance, so confidence intervals are used. 33


(a) Passing test (b) Failing test

Fig. 7. Histogram examples for DP testing, given one pair ofdatabases

bound, created by multiplying the probability estimateof each bin for database D1 by e

Á and adding ”. Thelower curve (in blue) is the unmodified probability esti-mate of D2. In Figure 7a, all blue buckets are less thanthe upper DP bound: that the DP predicate is not vio-lated. In Figure 7b, 3 buckets exceed this upper bound:the DP predicate is been violated. For symmetry, wealso repeat this check with D1 swapped with D2.

It is su�cient to terminate once we find a single pairof databases which violate the predicate. However, sincethe histogram is subject to sampling error, a correctlyimplemented algorithm can fail this test with non-zeroprobability. To address this, we relax our test by usingconfidence intervals as bounds [42]. We can also param-eterize the tester with a parameter – that tolerates apercentage of failing buckets per histogram comparison.

5.3.3 DP stochastic tester algorithm

The overall approach is an algorithm that iterates overdatabases and performs a DFS on each of the databasesearch graphs, where each edge is a DP predicate test.See Appendix A for more details.

5.3.4 Case study: noisy average

We were able to detect that an algorithm was imple-mented incorrectly, violating DP. When we first imple-mented ANON_AVG, we used the Noisy Average with Ac-curate Count algorithm from [31]: we used our ANON_SUMimplementation to compute the noisy sum and then di-vided it by the un-noised count. Our first version ofANON_SUM used a Laplace distribution with scale |U≠L|

Á,

where U and L are the upper and lower clampingbounds, which is the correct bound when used as acomponent of ANON_AVG. However, this was not correctfor noisy sum in the case when adjacent databases dif-fer by the presence of a row. We updated the scaleto max(|U |,|L|)

Á, as maximum change in this case is the

largest magnitude. This change created a regression inDP guarantee for ANON_AVG, which was detected by thestochastic tester.

Fig. 8. Example histogram comparison for the Noisy Averagealgorithm with incorrect noise.

Orange: eÁP r[f({≠0.375, ≠0.055, 0.3})]

Blue: P r[f({≠0.375, ≠0.055})]

Figure 8 shows a pair of datasets where thestochastic tester detected a violation of the DP pred-icate: {≠0.375, ≠0.055, 0.3} and {≠0.375, ≠0.055}. Wecan clearly see that several buckets violate the predi-cate. Once the stochastic tester alerted us to the errorwe quickly modified ANON_AVG to no longer depend onANON_SUM so that it could use the correct sensitivity.

6 Conclusion and future work

We presented a generic system to answer SQL querieswith user-level di�erential privacy. This system is ableto capture most data analysis tasks based on aggrega-tions, performs well for typical use-cases, and provides amechanism to deduce privacy parameters from accuracyrequirements, allowing a principled decision between re-identification risk and the required utility of a particu-lar application. All implemented mechanisms are testedwith a stochastic checker that prevents regressions andincreases our level of confidence in the robustness ofthe privacy guarantee. By releasing components of oursystem as open-source software after we validated its vi-ability on internal use-cases, we hope to encourage fur-ther adoption and research of di�erentially private dataanalysis.

The algorithms presented in this work are relativelysimple, but empirical evidence show that this approachis useful, robust and scalable. Future work could in-clude usability studies to test the success of the methodswe used to explain the system and its inherent accura-cy/privacy trade-o�s. In addition, we see room for sig-nificant accuracy improvements: using Gaussian noiseand better composition theorems is a natural next step.


(a) Passing test (b) Failing test

Fig. 7. Histogram examples for DP testing, given one pair ofdatabases

bound, created by multiplying the probability estimateof each bin for database D1 by e

Á and adding ”. Thelower curve (in blue) is the unmodified probability esti-mate of D2. In Figure 7a, all blue buckets are less thanthe upper DP bound: that the DP predicate is not vio-lated. In Figure 7b, 3 buckets exceed this upper bound:the DP predicate is been violated. For symmetry, wealso repeat this check with D1 swapped with D2.

It is su�cient to terminate once we find a single pairof databases which violate the predicate. However, sincethe histogram is subject to sampling error, a correctlyimplemented algorithm can fail this test with non-zeroprobability. To address this, we relax our test by usingconfidence intervals as bounds [42]. We can also param-eterize the tester with a parameter – that tolerates apercentage of failing buckets per histogram comparison.

5.3.3 DP stochastic tester algorithm

The overall approach is an algorithm that iterates overdatabases and performs a DFS on each of the databasesearch graphs, where each edge is a DP predicate test.See Appendix A for more details.

5.3.4 Case study: noisy average

We were able to detect that an algorithm was imple-mented incorrectly, violating DP. When we first imple-mented ANON_AVG, we used the Noisy Average with Ac-curate Count algorithm from [31]: we used our ANON_SUMimplementation to compute the noisy sum and then di-vided it by the un-noised count. Our first version ofANON_SUM used a Laplace distribution with scale |U≠L|

Á,

where U and L are the upper and lower clampingbounds, which is the correct bound when used as acomponent of ANON_AVG. However, this was not correctfor noisy sum in the case when adjacent databases dif-fer by the presence of a row. We updated the scaleto max(|U |,|L|)

Á, as maximum change in this case is the

largest magnitude. This change created a regression inDP guarantee for ANON_AVG, which was detected by thestochastic tester.

Fig. 8. Example histogram comparison for the Noisy Averagealgorithm with incorrect noise.

Orange: eÁP r[f({≠0.375, ≠0.055, 0.3})]

Blue: P r[f({≠0.375, ≠0.055})]

Figure 8 shows a pair of datasets where thestochastic tester detected a violation of the DP pred-icate: {≠0.375, ≠0.055, 0.3} and {≠0.375, ≠0.055}. Wecan clearly see that several buckets violate the predi-cate. Once the stochastic tester alerted us to the errorwe quickly modified ANON_AVG to no longer depend onANON_SUM so that it could use the correct sensitivity.

6 Conclusion and future work

We presented a generic system to answer SQL querieswith user-level di�erential privacy. This system is ableto capture most data analysis tasks based on aggrega-tions, performs well for typical use-cases, and provides amechanism to deduce privacy parameters from accuracyrequirements, allowing a principled decision between re-identification risk and the required utility of a particu-lar application. All implemented mechanisms are testedwith a stochastic checker that prevents regressions andincreases our level of confidence in the robustness ofthe privacy guarantee. By releasing components of oursystem as open-source software after we validated its vi-ability on internal use-cases, we hope to encourage fur-ther adoption and research of di�erentially private dataanalysis.

The algorithms presented in this work are relativelysimple, but empirical evidence show that this approachis useful, robust and scalable. Future work could in-clude usability studies to test the success of the methodswe used to explain the system and its inherent accura-cy/privacy trade-o�s. In addition, we see room for sig-nificant accuracy improvements: using Gaussian noiseand better composition theorems is a natural next step.


Cited References (no particular order)

●  [Wilson+20] Royce J Wilson, Celia Yuxin Zhang, William Lam, Damien Desfontaines, Daniel Simmons-Marengo, and Bryant Gipson. Differentially Private SQL with Bounded User Contribution. PETs, 2020.

●  [Holohan+19] Naoise Holohan, Stefano Braghin, Pól Mac Aonghusa, Killian Levacher. Diffprivlib: The IBM Differential Privacy Library. arXiv:1907.02444, 2019.

●  [McMahan+18] H. Brendan McMahan, Galen Andrew, Ulfar Erlingsson, Steve Chien, Ilya Mironov, Nicolas Papernot, Peter Kairouz. A General Approach to Adding Differential Privacy to Iterative Training Procedures. arXiv:1812.06210, 2018.

●  [Karp+07] Noisy binary search and its applications. SODA, 2007.

●  [Kifer+20] Kifer, Messing, Roth, Thakurta, Zhang. Guidelines for Implementing and Auditing Differentially Private Systems. arXiv:2002.04049, 2020.

●  [Ding+18] Zeyu Ding, Yuxin Wang, Guanhong Wang, Danfeng Zhang, and Daniel Kifer. Detecting violations of differential privacy. CCS, 2018. 34


Cited References (no particular order)

●  [Abadi+16] Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang. Deep Learning with Differential Privacy. CCS 2016.

●  [Papernot+18] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable Private Learning with PATE. ICLR 2018.

35


Acknowledgements ●  Many slides in this lecture are constructed from the source code and

comments in Google’s DP library: https://github.com/google/differential-privacy/tree/master/differential_privacy

●  The DP SGD slides were inspired by a talk from UIUC’s Advanced Security Course website: https://courses.engr.illinois.edu/cs563/fa2018/slides/cs563-21-xiaojun.pdf

●  The testing slides and graphs were constructed from the paper associated with Google’s DP library [Wilson+20].

●  Figure on the PATE slide is from: http://www.cleverhans.io/privacy/2018/04/29/privacy-and-machine-learning.html.

36

private systems

Documents