Privacy Odometers and Filters: Pay-as-you-Go Composition

Introduction

Differential privacy is a stability condition on a randomized algorithm, designed to guarantee individual-level privacy during data analysis. Informally, an algorithm is differentially private if any pair of close inputs map to similar probability distributions over outputs, where similarity is measured by two parameters $\varepsilon$ and $\delta$. Informally, $\varepsilon$ measures the amount of privacy and $\delta$ measures the failure probability that the privacy loss is much worse than $\varepsilon$. A signature property of differential privacy is that it is preserved under composition—combining many differentially private subroutines into a single algorithm preserves differential privacy and the privacy parameters degrade gracefully. Composability is essential for both privacy and for algorithm design. Since differential privacy is composable, we can design a sophisticated algorithm and prove it is private without having to reason directly about its output distribution. Instead, we can rely on the differential privacy of the basic building blocks and derive a privacy bound on the whole algorithm using the composition rules.

The composition theorem for differential privacy is very strong, and holds even if the choice of which differentially private subroutine to run is adaptive—that is, the choice of the next algorithm may depend on the output of previous algorithms. This property is essential in algorithm design, but also more generally in modeling unstructured sequences of data analyses that might be run by a human data analyst, or even by many data analysts on the same data set, while only loosely coordinating with one another. Even setting aside privacy, it can be very challenging to analyze the statistical properties of general adaptive procedures for analyzing a dataset, and the fact that adaptively chosen differentially private algorithms compose has recently been used to give strong guarantees of statistical validity for adaptive data analysis .

However, all the known composition theorems for differential privacy have an important and generally overlooked caveat. Although the choice of the next subroutine in the composition may be adaptive, the number of subroutines called and choice of the privacy parameters $\varepsilon$ and $\delta$ for each subroutine must be fixed in advance. Indeed, it is not even clear how to define differential privacy if the privacy parameters are not fixed in advance. This is generally acceptable when designing a single algorithm (that has a worst-case analysis), since worst-case eventualities need to be anticipated and budgeted for in order to prove a theorem. However, it is not acceptable when modeling the unstructured adaptivity of a data analyst, who may not know ahead of time (before seeing the results of intermediate analyses) what he wants to do with the data. When controlling privacy loss across multiple data analysts, the problem is even worse.

The “basic” composition theorem asserts that Example1 is $(\varepsilon k,0)$-differentially private. The “advanced” composition theorem gives a more sophisticated bound and asserts that (provided that $\varepsilon$ is sufficiently small), the algorithm satisfies $(\varepsilon\sqrt{8k\ln(1/\delta)},\delta)$-differential privacy for any $\delta>0$. There is even an “optimal” composition theorem , although its precise statement is too complicated to describe here. These analyses crucially assume that both the number of iterations $k$ and the parameter $\varepsilon$ are fixed up front, even though it allows for the queries $q_{i}$ to be adaptively chosen.The same analysis holds for hetereogeneous parameters $(\varepsilon_{1},\dots,\varepsilon_{k})$ are used in each round as long as they are all fixed in advance. For basic composition $\varepsilon k$ is replaced with $\sum_{i=1}^{k}\varepsilon_{i}$ and for advanced composition $\varepsilon\sqrt{k}$ is replaced with $\sqrt{\sum_{i=1}^{k}\varepsilon_{i}^{2}}$.

Now consider a similar example where the number of iterations is not fixed up front, but actually depends on the answers to previous queries. This is a special case of a more general setting where the privacy parameter $\varepsilon_{i}$ in every round may be chosen adaptively—halting in our example is equivalent to setting $\varepsilon_{i}=0$ in all future rounds.

Example2 cannot be said to be differentially private ex ante for any non-trivial fixed values of $\varepsilon$ and $\delta$, because the computation might run for an arbitrarily long time that depends on the data itself and privacy may degrade indefinitely. What can we say about privacy after we run the algorithm? If the algorithm/data-analyst happens to stop after $k$ rounds, can we apply the composition theorem ex post to conclude that it is $(\varepsilon k,0)$- and $(\varepsilon\sqrt{8k\log(1/\delta)},\delta)$-differentially private, as we could if the algorithm were constrained to always run for at most $k$ rounds?

In this paper, we study the composition properties of differential privacy when everything—the choice of algorithms, the number of rounds, and the privacy parameters in each round—may be adaptively chosen. We show that this setting is much more delicate than the settings covered by previously known composition theorems, but that these sorts of ex post privacy bounds do hold with only a small (but in some cases unavoidable) loss over the standard setting.

We give a formal framework for reasoning about the adaptive composition of differentially private algorithms when the privacy parameters themselves can be chosen adaptively. When the parameters are chosen non-adaptively, a composition theorem gives a high probability bound on the worst case privacy loss that results from the output of an algorithm. In the adaptive parameter setting, it no longer makes sense to have fixed bounds on the privacy loss. Instead, we propose two kinds of primitives capturing two natural use cases for composition theorems:

A privacy odometer takes as input a global failure parameter $\delta_{g}$. After every round $i$ in the composition of differentially private algorithms, the odometer outputs a number $\tau_{i}$ that may depend on the realized privacy parameters $\varepsilon_{i},\delta_{i}$ in the previous rounds. The privacy odometer guarantees that with probability $1-\delta_{g}$, for every round $i$, $\tau_{i}$ is an upper bound on the privacy loss in round $i$.

A privacy filter is a way to cut off access to the dataset when the privacy parameters are too large. It takes as input a global privacy “budget” $(\varepsilon_{g},\delta_{g})$. After every round, it either outputs $\mathtt{CONT}$ (“continue”) or $\mathtt{HALT}$ depending on the privacy parameters from the previous rounds. The privacy filter guarantees that with probability $1-\delta_{g}$, it will output $\mathtt{HALT}$ before the privacy loss exceeds $\varepsilon_{g}$. When used, it guarantees that the resulting interaction is $(\varepsilon_{g},\delta_{g}$)-DP.

A tempting heuristic is to take the realized privacy parameters $\varepsilon_{1},\delta_{1},\dots,\varepsilon_{i},\delta_{i}$ and apply one of the existing composition theorems to those parameters, using that value as a privacy odometer or implementing a privacy filter by halting when getting a value that exceeds the global budget. However this heuristic does not necessarily give valid bounds.

We first prove that the heuristic does work for the basic composition theorem in which the parameters $\varepsilon_{i}$ and $\delta_{i}$ add up. We prove that summing the realized privacy parameters yields both a valid privacy odometer and filter. The idea of a privacy filter was also considered in , who show that basic composition works in the privacy filter application.

We then show that the heuristic breaks for the advanced composition theorem . However, we give a valid privacy filter that gives the same asymptotic bound as the advanced composition theorem, albeit with worse constants. On the other hand, we show that, in some parameter regimes, the asymptotic bounds given by our privacy filter cannot be achieved by a privacy odometer. This result gives a formal separation between the two models when the parameters may be chosen adaptively, which does not exist when the privacy parameters are fixed. Finally, we give a valid privacy odometer with a bound that is only slightly worse asymptotically than the bound that the advanced composition theorem would give if it were used (improperly) as a heuristic. Our bound is worse by a factor that is never larger than $\sqrt{\log\log(n)}$ (here, $n$ is the size of the dataset) and for some parameter regimes is only a constant.

Privacy Preliminaries

Differential privacy is defined based on the following notion of similarity between two distributions.

Two random variables $X$ and $Y$ taking values from domain $\mathcal{D}$ are $(\varepsilon,\delta)$-indistinguishable, denoted as $X\approx_{\varepsilon,\delta}Y$, if $\forall S\subseteq\mathcal{D}$,

There is a slight variant of indistinguishability, called point-wise indistinguishability, which is nearly equivalent, but will be the more convenient notion for the generalizations we give in this paper.

Two random variables $X$ and $Y$ taking values from $\mathcal{D}$ are $(\varepsilon,\delta)$-point-wise indistinguishable if with probability at least $1-\delta$ over either $a\sim X$ or $a\sim Y$, we have

Let $X$ and $Y$ be two random variables taking values from $\mathcal{D}$. If $X$ and $Y$ are $(\varepsilon,\delta)$-point-wise indistinguishable, then $X\approx_{\varepsilon,\delta}Y$. Also, if $X\approx_{\varepsilon,\delta}Y$ then $X$ and $Y$ are $\left(2\varepsilon,\frac{2\delta}{e^{\varepsilon}\varepsilon}\right)$-point-wise indistinguishable.

We say two databases $\mathbf{x},\mathbf{x}^{\prime}\in\mathcal{X}^{n}$ are neighboring if they differ in at most one entry, i.e. if there exists an index $i\in[n]$ such that $\mathbf{x}_{-i}=\mathbf{x}_{-i}^{\prime}$. We can now state differential privacy in terms of indistinguishability.

A randomized algorithm $\mathcal{M}:\mathcal{X}^{n}\to\mathcal{Y}$ with arbitrary output range $\mathcal{Y}$ is $(\varepsilon,\delta)$-differentially private (DP) if for every pair of neighboring databases $\mathbf{x},\mathbf{x}^{\prime}$:

We then define the privacy loss $\mathtt{Loss}_{\mathcal{M}}(a;\mathbf{x},\mathbf{x}^{\prime})$ for outcome $a\in\mathcal{Y}$ and neighboring datasets $\mathbf{x},\mathbf{x}^{\prime}\in\mathcal{X}^{n}$ as

We note that if we can bound $\mathtt{Loss}_{\mathcal{M}}(a;\mathbf{x},\mathbf{x}^{\prime})$ for any neighboring datasets $\mathbf{x},\mathbf{x}^{\prime}$ with high probability over $a\sim\mathcal{M}(\mathbf{x})$, then Lemma 2.3 tells us that $\mathcal{M}$ is differentially private. Moreover, Lemma 2.3 also implies that this approach is without loss of generality (up to a small difference in the parameters). Thus, our composition theorems will focus on bounding the privacy loss with high probability.

A useful property of differential privacy is that it is preserved under post-processing without degrading the parameters:

Let $\mathcal{M}:\mathcal{X}^{n}\to\mathcal{Y}$ be $(\varepsilon,\delta)$-DP and $f:\mathcal{Y}\to\mathcal{Y}^{\prime}$ be any randomized algorithm. Then $f\circ\mathcal{M}:\mathcal{X}^{n}\to\mathcal{Y}^{\prime}$ is $(\varepsilon,\delta)$-DP.

We next recall a useful characterization from : any DP algorithm can be written as the post-processing of a simple, canonical algorithm which is a generalization of randomized response.

For any $\varepsilon,\delta\geq 0$, we define the randomized response algorithm $\mathtt{RR}_{\varepsilon,\delta}:\{0,1\}\to\{0,\top,\bot,1\}$ as the following (Note that if $\delta=0$, we will simply write the algorithm $\mathtt{RR}_{\varepsilon,\delta}$ as $\mathtt{RR}_{\varepsilon}$.)

Kairouz, Oh, and Viswanath show that any $(\varepsilon,\delta)$–DP algorithm can be viewed as a post-processing of the output of $\mathtt{RR}_{\varepsilon,\delta}$ for an appropriately chosen input.

For every $(\varepsilon,\delta)$-DP algorithm $\mathcal{M}$ and for all neighboring databases $\mathbf{x}^{0}$ and $\mathbf{x}^{1}$, there exists a randomized algorithm $T$ where $T(\mathtt{RR}_{\varepsilon,\delta}(b))$ is identically distributed to $\mathcal{M}(\mathbf{x}^{b})$ for $b\in\{0,1\}$.

This theorem will be useful in our analyses, because it allows us to without loss of generality analyze compositions of these simple algorithms $\mathtt{RR}_{\varepsilon,\delta}$ with varying privacy parameters.

We now define the adaptive composition of differentially private algorithms in the setting introduced by and then extended to heterogenous privacy parameters in , in which all of the privacy parameters are fixed prior to the start of the computation. The following “composition game” is an abstract model of composition in which an adversary can adaptively select between neighboring datasets at each round, as well as a differentially private algorithm to run at each round – both choices can be a function of the realized outcomes of all previous rounds. However, crucially, the adversary must select at each round an algorithm that satisfies the privacy parameters which have been fixed ahead of time – the choice of parameters cannot itself be a function of the realized outcomes of previous rounds. We define this model of interaction formally in Algorithm 1 where the output is the view of the adversary $\mathcal{A}$ which includes any random coins she uses $R_{\mathcal{A}}$ and the outcomes $A_{1},\cdots,A_{k}$ of every round.

We say that the sequence of parameters $\varepsilon_{1},\cdots,\varepsilon_{k}\geq 0$, $\delta_{1},\cdots,\delta_{k}\in[0,1)$ satisfies $(\varepsilon_{g},\delta_{g})$-differential privacy under adaptive composition if for every adversary $\mathcal{A}$, and $\mathcal{E}=(\mathcal{E}_{1},\cdots,\mathcal{E}_{k})$ where $\mathcal{E}_{i}$ is the class of $(\varepsilon_{i},\delta_{i})$-DP algorithms, we have $\mathtt{FixedParamComp}(\mathcal{A},\mathcal{E},\cdot)$ is $(\varepsilon_{g},\delta_{g})$-DP in its last argument, i.e. $V^{0}\approx_{\varepsilon_{g},\delta_{g}}V^{1}$.

We first state a basic composition theorem which shows that the adaptive composition satisfies differential privacy where “the parameters just add up.”

The sequence $\varepsilon_{1},\cdots,\varepsilon_{k}$ and $\delta_{1},\cdots\delta_{k}$ satisfies $(\varepsilon_{g},\delta_{g})$-differential privacy under adaptive composition where

We now state the advanced composition bound from which gives a quadratic improvement to the basic composition bound.

For any $\hat{\delta}>0$, the sequence $\varepsilon_{1},\cdots,\varepsilon_{k}$ and $\delta_{1},\cdots\delta_{k}$ where $\varepsilon=\varepsilon_{i}$ and $\delta=\delta_{i}$ for all $i\in[k]$ satisfies $(\varepsilon_{g},\delta_{g})$-differential privacy under adaptive composition where

This theorem can be easily generalized to hold for values of $\varepsilon_{i}$ that are not all equal (as done in ). However, this is not as all-encompassing as it would appear at first blush, because this straightforward generalization would not allow for the values of $\varepsilon_{i}$ and $\delta_{i}$ to be chosen adaptively by the data analyst. Indeed,the definition of differential privacy itself (Definition 2.4) does not straightforwardly extend to this case. The remainder of this paper is devoted to laying out a framework for sensibly talking about the privacy parameters $\varepsilon_{i}$ and $\delta_{i}$ being chosen adaptively by the data analyst, and to prove composition theorems (including an analogue of Theorem 2.10) in this model.

Composition with Adaptively Chosen Parameters

We now introduce the model of composition with adaptive parameter selection, and define privacy in this setting.

We want to model composition as in the previous section, but allow the adversary the ability to also choose the privacy parameters $(\varepsilon_{i},\delta_{i})$ as a function of previous rounds of interaction. We will define the view of the interaction, similar to the view in $\mathtt{FixedParamComp}$, to be the tuple that includes $\mathcal{A}$’s random coin tosses $R_{\mathcal{A}}$ and the outcomes $A=(A_{1},\cdots,A_{k})$ of the algorithms she chose. Formally, we define an adaptively chosen privacy parameter composition game in Algorithm 2 which takes as input an adversary $\mathcal{A}$, a number of rounds of interaction $k$,Note that in the adaptive parameter composition game, the adversary has the option of effectively stopping the composition early at some round $k^{\prime}<k$ by simply setting $\varepsilon_{i}=\delta_{i}=0$ for all rounds $i>k^{\prime}$. Hence, the parameter $k$ will not appear in our composition theorems the way it does when privacy parameters are fixed. This means that we can effectively take $k$ to be infinite. For technical reasons, it is simpler to have a finite parameter $k$, but the reader should imagine it as being an enormous number (say the number of atoms in the universe) so as not to put any constraint at all on the number of rounds of interaction with the adversary. and an experiment parameter $b\in\{0,1\}$.

We then define the privacy loss with respect to $\mathtt{AdaptParamComp}(\mathcal{A},k,b)$ in the following way for a fixed view $v=(r,a)$ where $r$ represents the random coin tosses of $\mathcal{A}$ and we write $v_{<i}=(v_{0},\cdots,v_{i-1}):=(r,a_{1},\cdots,a_{i-1})$:

Note that the privacy parameters $(\varepsilon_{i},\delta_{i})$ depend on the previous outcomes that $\mathcal{A}$ receives. We will frequently shorten our notation $\varepsilon_{t}=\varepsilon_{t}(v_{<t})$ and $\delta_{t}=\delta_{t}(v_{<t})$ when the outcome is understood.

It no longer makes sense to claim that the privacy loss of the adaptive parameter composition experiment is bounded by any fixed constant, because the privacy parameters (with which we would presumably want to use to bound the privacy loss) are themselves random variables. Instead, we define two objects which can be used by a data analyst to control the privacy loss of an adaptive composition of algorithms.

The first object, which we call a privacy filter, is a stopping-time rule. It takes two global parameters $(\varepsilon_{g},\delta_{g})$ and will at each round either output $\mathtt{CONT}$ or $\mathtt{HALT}$. A privacy filter can be used to guarantee that with high probability, the stated privacy budget $\varepsilon_{g}$ is never exceeded – the data analyst at each round $k^{\prime}$ simply queries $\mathtt{COMP}_{\varepsilon_{g},\delta_{g}}(\varepsilon_{1},\delta_{1},\ldots,\varepsilon_{k^{\prime}},\delta_{k^{\prime}},0,0,\ldots,0,0)$ before she runs algorithm $k^{\prime}$, and runs it only if the filter returns $\mathtt{CONT}$. Again, this is guaranteed because the continuation is a feasible choice of the adversary, and the guarantees of both a filter and an odometer are quantified over all adversaries. We give the formal description of this interaction where $\mathcal{A}$ uses the privacy filter in Algorithm 3.

are $(\varepsilon_{g},\delta_{g})$-point-wise indistinguishable.

The second object, which we call a privacy odometer will be parameterized by one global parameter $\delta_{g}$ and will provide a running real valued output that will, with probability $1-\delta_{g}$, upper bound the privacy loss at each round of any adaptive composition in terms of the realized values of $\varepsilon_{i}$ and $\delta_{i}$ selected at each round.

We note that a valid privacy odometer can be used to provide a running upper bound on the privacy loss at each intermediate round: the privacy loss at round $k^{\prime}<k$ must with high probability be upper bounded by $\mathtt{COMP}_{\delta_{g}}\left(\varepsilon_{1},\delta_{1},\ldots,\varepsilon_{k^{\prime}},\delta_{k^{\prime}},0,0,\ldots,0,0\right)$. That is, the bound that results by setting all future privacy parameters to 0. This is because setting all future privacy parameters to zero is equivalent to stopping the computation at round $k^{\prime}$, and is a feasible choice for the adaptive adversary $\mathcal{A}$.

2 Focusing on Pure DP

Throughout our analysis we will be focusing on pure DP mechanisms, i.e. the adversary selects a $\varepsilon_{i}$-DP mechanism at each round $i$ with $\delta_{i}=0$. We show that we can, with worse constants, convert the case where an adversary can select $\delta_{i}>0$ each round, while still using our formulas for $\delta_{i}=0$. Specifically, we show that we can simulate $\mathtt{AdaptParamComp}(\mathcal{A},k,b)$ by defining a new adversary that chooses the differentially private algorithm $\mathcal{M}_{i}$ of adversary $\mathcal{A}$, but uses a pure DP mechanism.We thank Valentin Hartmann for pointing out an issue with an earlier version of the following result.

If $\widetilde{\mathtt{COMP}}_{\delta_{g}}$ is a valid privacy odometer when $\{\delta_{i}\}\equiv 0$, then for every $\delta_{g}^{\prime}\geq 0$, $\mathtt{COMP}_{\delta_{g}+\delta_{g}^{\prime}}$ is a valid privacy odometer where

If $\widetilde{\mathtt{COMP}}_{\varepsilon_{g},\delta_{g}}$ is a valid privacy filter when $\{\delta_{i}\}\equiv 0$, then for every $\delta_{g}^{\prime}\geq 0$, $\mathtt{COMP}_{\varepsilon_{g},\delta_{g}+\delta_{g}^{\prime}}$ is a valid privacy filter where

Let $V^{(b)}=(V^{(b)}_{1},\cdots,V^{(b)}_{k})$ be the view of $\mathcal{A}$ in $\mathtt{AdaptParamComp}(\mathcal{A},k,b)$. Recall that we have $\mathtt{Loss}(V^{(b)})=\sum_{i=1}^{k}\mathtt{Loss}_{i}(V^{(b)}_{\leq i})$. From Lemma 2.3 and given any $V_{0}^{(b)}=v_{0},V_{1}^{(b)}=v_{1},\cdots,V_{i-1}^{(b)}=v_{i-1}$, we have that $\mathtt{Loss}_{i}(V^{(b)}_{\leq i})\leq 2\varepsilon_{i}(v_{<i})$ with probability at least $1-\tfrac{2\delta_{i}(v_{<i})}{\varepsilon_{i}(v_{<i})e^{\varepsilon_{i}(v_{<i})}}$ over $v_{i}\sim V_{i}^{(b)}$.

Let $V$ be another copy of $V^{(b)}$. We have for any realizations of the view $V_{<k}=v_{<k}$ that the following holds,

Note that the event in the probability is a pure DP bound, i.e. $\delta_{i}=0$, for each round $i$. Hence, we use the pure DP privacy filter or odometer to bound the sum of privacy losses. Let $\mathcal{G}(v_{<k})$ be the event in the above probability statement. We write $\mathcal{V}_{<k}$ to be the space of all possible views of an adversary $\mathcal{A}$ in $\mathtt{AdaptParamComp}(\mathcal{A},k,0)$. We then first consider the statement about privacy odometers,

Next, we turn to the statement on privacy filters, which follows a similar argument as above by conditioning on the events where each loss can be bounded by $2\varepsilon_{i}$ at round $i$,

We will state our results for the pure DP case, where each $\delta_{i}=0$. In the case where $\delta_{i}>0$, we can translate pure DP filters and odometers to ones that apply in the more general setting by using Lemma 3.3.

3 Basic Composition

We first give an adaptive parameter version of the basic composition in Theorem 2.9.

For each nonnegative $\delta_{g}$, $\mathtt{COMP}_{\delta_{g}}$ is a valid privacy odometer where

Additionally, for any $\varepsilon_{g},\delta_{g}\geq 0$, $\mathtt{COMP}_{\varepsilon_{g},\delta_{g}}$ is a valid privacy filter where

The proof follows simply from the definition of (pure) differential privacy, so for all possible views $v$ of the adversary in $\mathtt{AdaptParamComp}(\mathcal{A},k,b)$, we have:

where we explicitly write the dependence of the choice of $\varepsilon_{i}$ by $\mathcal{A}$ at round $i$ on the view from the previous rounds as $\varepsilon_{i}(v_{<i})$ ∎

Concentration Preliminaries

We give a useful concentration bound that will be pivotal in proving an improved valid privacy odometer and filter from that given in Theorem 3.4. We first present a concentration bound for self normalized processes.

If $A$ and $B>0$ are two random variables such that

To put this bound into context, suppose that $B$ is a constant and we apply the bound with $\beta=B^{2}$. Then the bound simplifies to

which is just a standard concentration inequality for any subgaussian random variable $A$ with standard deviation $B$. Another bound which will be useful for our results is the following.

We will apply both Theorem 4.1 and Theorem 4.2 to random variables coming from martingales defined from the privacy loss functions.

We then use the following result which gives us a pair of random variables to which we can apply Theorem 4.1.

For $M_{k}$ defined in (3), if there exists two random variables $C_{i}<D_{i}$ that are $\mathcal{F}_{i-1}$-measurable for $i\geq 1$

We then obtain the following result from combining Theorem 4.1 with Theorem 4.3.

Let $M_{k}$ be defined as in (3) and satisfy the hypotheses of Theorem 4.3. Then for every fixed $k\geq 1$, $x>0$ and $\delta\leq 1/e$, we have

Similarly, we can obtain the following result by combining Theorem 4.2 with Theorem 4.3.

Let $M_{k}$ be defined as in (3) and satisfy the hypotheses of Theorem 4.3. Then for every fixed $k\geq 1$, $0<\beta<1$ $c>0$ and $t\geq 1$, we have

We use the fact that $x\log(1/x)$ is maximized at $x=1/e$ and has value no more than $0.37$. ∎

Given Lemma 3.3, we will focus on finding a valid privacy odometer and filter when $\{\delta_{i}\}\equiv 0$. Our analysis will then depend on the privacy loss $\mathtt{Loss}(V)$ from (1) where $V$ is the view of the adversary in $\mathtt{AdaptParamComp}(\mathcal{A},k,0).$ We then focus on the following martingale in our analysis:

We can then bound the conditional expectation $\mu_{i}$ with the following result from that improves on an earlier result from by a factor of 2.

For $\mu_{i}$ defined in (5), we have $\mu_{i}\leq\varepsilon_{i}\left(e^{\varepsilon_{i}}-1\right)/2$.

Advanced Composition for Privacy Filters

We next show that we can essentially get the same asymptotic bound as Theorem 2.10 for the privacy filter setting using the bound in Theorem 4.4 for the martingale given in (5).

We define $\mathcal{K}$ as the following where we set $x=\frac{\varepsilon_{g}^{2}}{28.04\cdot\log(1/\delta_{g})}$ and,We thank Daniel Winograd-Cort for catching an incorrectly set constant in an earlier version of this theorem.

$\mathtt{COMP}_{\varepsilon_{g},\delta_{g}}$ is a valid privacy filter for $\delta_{g}\in\left(0,1/e\right)$ and $\varepsilon_{g}>0$ where

Note that if we have $\sum_{i=1}^{k}\varepsilon_{i}^{2}=O\left(1/\log(1/\delta_{g})\right)$ and set $\varepsilon_{g}=\Theta\left(\sqrt{\sum_{i=1}^{k}\varepsilon_{i}^{2}\log(1/\delta_{g})}\right)$ in (6), we are then getting the same asymptotic bound on the privacy loss as in and in Theorem 2.10 for the case when $\varepsilon_{i}=\varepsilon$ for $i\in[k]$. If $k\varepsilon^{2}\leq\frac{1}{8\log(1/\delta_{g})}$, then Theorem 2.10 gives a bound on the privacy loss of $\varepsilon\sqrt{8k\log(1/\delta_{g})}$. Note that there may be better choices for the constant $28.04$ that we divide $\varepsilon_{g}^{2}$ by in (6), but for the case when $\varepsilon_{g}=\varepsilon\sqrt{8k\log(1/\delta_{g})}$ and $\varepsilon_{i}=\varepsilon$ for every $i\in[n]$, it is nearly optimal.

We focus on the martingale $M_{k}$ given in (5). In order to apply Theorem 4.4 we set the lower bound for $M_{i}$ to be $C_{i}=\left(-\varepsilon_{i}-\mu_{i}\right)$ and upper bound to be $D_{i}=\left(\varepsilon_{i}-\mu_{i}\right)$ in order to compute $U_{k}^{2}$ from (4). We then have for the martingale in (5) that

We can then directly apply Theorem 4.4 to get the following for $x=\left(\frac{\varepsilon_{g}}{\sqrt{28.04\cdot\log(1/\delta_{g})}}\right)^{2}>0$ with probability at least $1-\delta_{g}$

We can then obtain a bound on the privacy loss with probability at least $1-\delta_{g}$ over $v\sim V^{0}$

Advanced Composition for Privacy Odometers

One might hope to achieve the same sort of bound on the privacy loss from Theorem 2.10 when the privacy parameters may be chosen adversarially. However we show that this cannot be the case for any valid privacy odometer. In particular, even if an adversary selects the same privacy parameter $\varepsilon=o(\sqrt{\log(\log(n)/\delta_{g})/k})$ each round but can adaptively select a time to stop interacting with $\mathtt{AdaptParamComp}$ (which is a restricted special case of the power of the general adversary – stopping is equivalent to setting all future $\varepsilon_{i},\delta_{i}=0$), then we show that there can be no valid privacy odometer achieving a bound of $o(\varepsilon\sqrt{k\log\left(\log(n)/\delta_{g}\right)})$. This gives a separation between the achievable bounds for a valid privacy odometers and filters. But for privacy applications, it is worth noting that $\delta_{g}$ is typically set to be (much) smaller than $1/n$, in which case this gap disappears (since $\log(\log(n)/\delta_{g})=(1+o(1))\log(1/\delta_{g})$ ).

For any $\delta_{g}\in(0,O(1))$ there is no valid $\mathtt{COMP}_{\delta_{g}}$ privacy odometer where

In order to prove Equation 7, we use the following anti-concentration bound for a sum of random variables.

For $\gamma\in[1/2,1)$, we define the random variables $\xi_{i}\in\{-1,1\}$ where

We then apply Lemma 6.2 to prove an anti-concentration bound for the martingale given above.

Consider the partial sums $M_{t}$ defined in (9) for $t\in[n]$. There exists a constant $C$ such that for all $\delta\in(0,O(1))$ and $n>\Omega\left(\log(1/\delta)\cdot\left(\frac{1+\mu}{\sigma}\right)^{2}\right)$ we have

By Lemma 6.2, we know that there exists constants $C_{1},C_{2},C_{3}$ and large $N$ such that for all $m>N\cdot\left(\frac{1+\mu}{\sigma}\right)^{2}$ and $x\in\left[1,C_{3}\sqrt{m}\frac{\sigma}{1+\mu}\right]$, we have

Thus, we set $C=\frac{C_{1}}{2\sqrt{C_{2}}}$ and then for any $\delta$ such that $\sqrt{C_{2}}<\sqrt{\log(1/\delta)}<C_{3}\sqrt{C_{2}}\sqrt{m_{\delta}}\frac{\sigma}{1+\mu}$, we have

We next use Lemma 6.3 to prove that we cannot have a bound like Theorem 2.10 in the adaptive privacy parameter setting, which uses the stopping time adversary given in Algorithm 4.

Consider the stopping time adversary $\mathcal{A}_{\varepsilon,\delta_{g}}$ from Algorithm 4 for a constant $C$ that we will determine in the proof. Let the number of rounds $k=n$ and $\varepsilon=1/n$. In order to use Lemma 6.3 we define $\gamma=\frac{e^{\varepsilon}}{1+e^{\varepsilon}}$ from (8). Because we let $\varepsilon$ depend on $n$, we have $\mu\equiv\mu_{n}=\frac{e^{1/n}-1}{e^{1/n}+1}=O(1/n)$ and $\sigma\equiv\sigma_{n}=1-\mu_{n}^{2}=1-O(1/n^{2})$ which gives $\frac{1+\mu_{n}}{\sigma_{n}}=\Theta(1)$. We then relate the martingale in (9) with the privacy loss for this particular adversary in $\mathtt{AdaptParamComp}(\mathcal{A}_{\varepsilon,\delta_{g}},n,0)$ with view $V$ who sets $X_{t}=\pm\varepsilon$ each round,

Hence, at any round $t$ if $\mathcal{A}_{\varepsilon,\delta_{g}}$ finds that

then she will set all future $\varepsilon_{i}=0$ for $i>t$. To find the probability that (10) holds in any round $t\in[n]$ we use Lemma 6.3 with the constant $C$ from the lemma statement to say that (10) occurs with probability at least $\delta_{g}$.

Assume that $\mathtt{COMP}_{\varepsilon_{g}}$ is a valid privacy odometer and (7) holds. We then know that with probability at least $1-\delta_{g}$ over $v\sim V^{b}$ where $V^{b}$ is the view for $\mathtt{AdaptParamComp}(\mathcal{A}_{1/n,\delta_{g}},n,b)$

But this is a contradiction given that the bound in (10) at any round $t\in[n]$ occurs with probability at least $\delta_{g}$. ∎

We now utilize the bound from Theorem 4.4 to obtain a concentration bound on the privacy loss.

$\mathtt{COMP}_{\delta_{g}}$ is a valid privacy odometer for $\delta_{g}\in\left(0,1/e\right)$ where for any $x>0$,

We will follow a similar argument as in Theorem 5.1 where we use the same martingale $M_{k}$ from (5). We can then directly apply Theorem 4.4 to get the following for any $\beta>0$ with probability at least $1-\delta_{g}$

This above result is only useful if we can plug in a constant $x>0$. If we were to set $x=\sum_{i=1}^{k}\varepsilon_{i}^{2}$, then we would get asymptotically close to the same bound as in Theorem 2.10, however, the $\varepsilon_{i}$ are random variables, and their realizations cannot be used in setting $x$; further, we know from Equation 7 that such a bound cannot hold in this setting.

We now give our main positive result for privacy odometers, which is similar to our privacy filter in Theorem 5.1 except that $\delta_{g}$ is replaced by $\delta_{g}/\log(n)$, as is necessary from Equation 7. Note that the bound incurs an additive $1/n^{2}$ loss to the $\sum_{i}\varepsilon_{i}^{2}$ term that is present without privacy. In any reasonable setting of parameters, this translates to at most a constant-factor multiplicative loss, because there is no utility running any differentially private algorithm with $\varepsilon_{i}<\frac{1}{10n}$ (we know that if $A$ is $(\varepsilon_{i},0)$-DP then $A(\mathbf{x})$ and $A(\mathbf{x}^{\prime})$ for any pair of inputs have statistical distance at most $e^{\varepsilon_{i}n}-1<0.1$, and hence the output is essentially independent of the input - note that a similar statement holds for $(\varepsilon_{i},\delta_{i})$-DP.)

and if $\sum_{i=1}^{k}\varepsilon_{i}^{2}\notin[1/n^{2},1]$ then $\mathtt{COMP}_{\delta_{g}}(\varepsilon_{1},0,\varepsilon_{2},0,\cdots,\varepsilon_{k},0)$ is equal to

We first focus on proving the bound in (11). For the martingale $M_{k}$ in (5), we can use Theorem 4.5 with $c=1/n$ and $t=n$ to get the following for any $\beta>0$

We then solve for $\beta$ so that $\delta_{g}/2=2\sqrt{e}\left(\beta+2\log(n)\sqrt{0.74\ \beta}\right)$, which yields,

where the inequality is due to $\sqrt{1+x}\geq 1+x/3$ for $0<x<1$. This gives the stated bound in (11).

For the bound given in (12), we set $x=1/n^{2}$ in Lemma 6.4. Hence, we would have with probability at least $1-\delta_{g}$ when $\sum_{i=1}^{k}\varepsilon_{i}^{2}\notin[1/n^{2},1]$,

In the above theorem, we only allow privacy parameters such that $\sum_{i=1}^{k}\varepsilon_{i}^{2}\in[1/n^{2},1]$. This assumption is not too restrictive, since the output of a single $(\ll 1/n)$-differentially private algorithm is nearly independent of its input. More generally, we can replace $1/n^{2}$ with an arbitrary “granularity parameter” $\gamma$ and require that $\sum_{i=1}^{k}\varepsilon_{i}^{2}\in[\gamma,1]$. When doing so, $\log_{2}(n^{2})/\delta_{g}$ in (11) will be replaced with $\log_{2}(1/\gamma)/\delta_{g}$. For example, we could require that $\varepsilon_{1}\geq\delta_{g}$, in which case we can choose $\gamma=\delta_{g}^{2}$, which would not affect our bound substantially.

Acknowledgements

The authors are grateful Jack Murtagh for his collaboration in the early stages of this work, and for sharing his preliminary results with us. We thank Andreas Haeberlen, Benjamin Pierce, and Daniel Winograd-Cort for helpful discussions about composition. We further thank Daniel Winograd-Cort for catching an incorrectly set constant in an earlier version of Theorem 5.1. Special thanks to Valentin Hartmann for pointing out an error in a earlier version of Lemma 3.3. The revised version translates pure DP filters/odometers to general DP filters/odometers, but with worse constants than the earlier version.

References