From dc71903533013a8a62c80fcee11fd3bf1bd7e3ed Mon Sep 17 00:00:00 2001 From: Kousha Kalantari <25445970+KoushaKalantari@users.noreply.github.com> Date: Fri, 13 Nov 2026 10:17:12 -0800 Subject: [PATCH 1/5] =?UTF-8?q?feat(Privacy):=20differential=20privacy=20?= =?UTF-8?q?=E2=80=94=20definition,=20post-processing,=20group=20privacy,?= =?UTF-8?q?=20composition=20(Dwork=E2=80=93Roth)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- Statlib.lean | 1 + Statlib/Privacy/DP/Basic.lean | 625 ++++++++++++++++++++++++++++++++++ 2 files changed, 626 insertions(+) create mode 100644 Statlib/Privacy/DP/Basic.lean diff --git a/Statlib.lean b/Statlib.lean index 3ee4ad5..2f13f19 100644 --- a/Statlib.lean +++ b/Statlib.lean @@ -138,6 +138,7 @@ import Statlib.ForMathlib.Probability.StrongLaw import Statlib.ForMathlib.Topology.Order.MonotoneConvergence import Statlib.Inference.Inference import Statlib.Inference.QMD +import Statlib.Privacy.DP.Basic import Statlib.Testing.Binary import Statlib.Testing.BinaryRenyi import Statlib.Testing.ChangeMeasure diff --git a/Statlib/Privacy/DP/Basic.lean b/Statlib/Privacy/DP/Basic.lean new file mode 100644 index 0000000..a73c038 --- /dev/null +++ b/Statlib/Privacy/DP/Basic.lean @@ -0,0 +1,625 @@ +/- +Copyright (c) 2026 Kousha Kalantari. All rights reserved. +Released under Apache 2.0 license as described in the file LICENSE. +Authors: Kousha Kalantari +-/ +module + +public import Mathlib.Analysis.SpecialFunctions.Exp +public import Mathlib.MeasureTheory.Integral.Layercake +public import Mathlib.MeasureTheory.Measure.Real +public import Mathlib.Probability.Kernel.Composition.CompMap +public import Mathlib.Probability.Kernel.Composition.CompProd +public import Mathlib.Probability.Kernel.Composition.Prod + +/-! +# Differential privacy: basic theory + +This file develops the basic theory of `(ε, δ)`-differential privacy following C. Dwork and +A. Roth, *The Algorithmic Foundations of Differential Privacy*, Found. Trends Theor. Comput. Sci. +9(3–4), 2014; result numbers follow that monograph. The definition is that of C. Dwork, +F. McSherry, K. Nissim and A. Smith, *Calibrating noise to sensitivity in private data analysis*, +TCC 2006 (with the `δ` of Dwork–Kenthapadi–McSherry–Mironov–Naor, Eurocrypt 2006). + +A mechanism is a kernel `M : Kernel 𝒟 Y` from datasets to outputs, and `Adj` is an arbitrary +neighbouring relation on datasets (Dwork–Roth use `‖x - y‖₁ ≤ 1` on histograms). The predicate +`IsEpsDeltaDP M Adj ε δ` says `P[M(S) ∈ E] ≤ e^ε P[M(S') ∈ E] + δ` for +all `Adj S S'` and all sets `E` (Definition 2.4); pure `ε`-DP is the case `δ = 0`. Unless stated +otherwise, the output spaces are **arbitrary measurable spaces**. + +`IsEpsDeltaDP` is stated with `Measure.real`, which sends `∞` to `0`, so the predicate is only +meaningful for **finite kernels** (in practice Markov kernels, as mechanisms are). The results +that use measure values assume finite kernels (basic API, post-processing) or Markov kernels +(composition); the purely order-theoretic ones (`IsEpsDeltaDP.mono`, `IsEpsDeltaDP.group`, +`IsEpsDeltaDP.comap`, …) hold for any kernel but carry content only for finite ones. + +## Main definitions + +* `IsEpsDeltaDP M Adj ε δ`: `(ε, δ)`-differential privacy (Definition 2.4). +* `IsEpsDP M Adj ε`: pure `ε`-DP, an abbreviation for `IsEpsDeltaDP M Adj ε 0`. +* `DP.composeFin M`: the `k`-fold composition `S ↦ (M₀(S), …, M_{k-1}(S))` of mechanisms + `M_i : Kernel 𝒟 (R i)`, with values in `(i : Fin k) → R i`. +* `DP.ParallelAdj Adj₁ Adj₂`: neighbouring datasets made of two disjoint parts. +* `DP.AddRemoveAdj`: datasets `X, X' : Set α` of records that differ in one record. +* `IsKNeighbour Adj k`: datasets joined by a chain of `k` neighbouring (or identical) steps. + +## Main statements (monograph result → Lean name) + +* Basic API: `IsEpsDeltaDP.mono` (monotonicity in `ε` and `δ`), `IsEpsDeltaDP.of_le`, + `IsEpsDeltaDP.nonneg`, `isEpsDeltaDP_iff_forall_measurableSet` (measurable events suffice), + `isEpsDeltaDP_iff_measure_le` (`ℝ≥0∞` form), `isEpsDP_iff_le_smul` (`M(S) ≤ e^ε M(S')`), + `isEpsDP_iff_forall_singleton` (pointwise form on countable `Y`), `isEpsDeltaDP_const`, + `isEpsDeltaDP_of_one_le`, `IsEpsDeltaDP.comap` (change of dataset space). +* Proposition 2.1 (post-processing by an arbitrary randomized mapping): `IsEpsDeltaDP.comp`, + deterministic case `IsEpsDeltaDP.map`. +* Theorem 2.2 (group privacy, `(ε, 0)`): `IsEpsDP.group`; the `(ε, δ)` version of footnote 1 to + Theorem 2.2 (`(kε, k e^{(k-1)ε} δ)`): `IsEpsDeltaDP.group'`. Both derive from + `IsEpsDeltaDP.group` (`(kε, δ ∑_{i 2` parts, and the bridges from the two-part set + version to PINQ's setting: multiset datasets, and the guarantee for datasets at any + symmetric-difference distance. +* Everything from Section 3.5.1 on (privacy loss, advanced composition, Theorem 3.20). +-/ + +@[expose] public section + +open MeasureTheory ProbabilityTheory Set Real +open scoped ENNReal + +namespace InformationTheory + +variable {Y Z 𝒟 : Type*} {mY : MeasurableSpace Y} {mZ : MeasurableSpace Z} + {m𝒟 : MeasurableSpace 𝒟} + +/-! ### Pure differential privacy and the measure-level form of the definition -/ + +/-- **`(ε, δ)`-differential privacy** (Definition 2.4 of Dwork–Roth): for all neighbouring +`S, S'` and every event `E`, `P[M(S) ∈ E] ≤ e^ε P[M(S') ∈ E] + δ`. The inequality is required for +every set `E`; measurable events suffice (`isEpsDeltaDP_iff_forall_measurableSet`). + +**Markov requirement (documented, not enforced).** `M` is an arbitrary `Kernel`, not necessarily +Markov, so this is an unrestricted algebraic extension whose privacy interpretation requires +`[IsMarkovKernel M]`: e.g. the zero kernel satisfies `IsEpsDeltaDP 0 Adj 0 0`, and for infinite +measures `Measure.real` reads `∞` as `0`. -/ +def IsEpsDeltaDP (M : Kernel 𝒟 Y) (Adj : 𝒟 → 𝒟 → Prop) (ε δ : ℝ) : Prop := + ∀ ⦃S S'⦄, Adj S S' → ∀ E : Set Y, (M S).real E ≤ exp ε * (M S').real E + δ + +/-- **Pure `ε`-differential privacy** (Definition 2.4 of Dwork–Roth with `δ = 0`): for all +neighbouring `S, S'` and every event `E`, `P[M(S) ∈ E] ≤ e^ε P[M(S') ∈ E]`. -/ +abbrev IsEpsDP (M : Kernel 𝒟 Y) (Adj : 𝒟 → 𝒟 → Prop) (ε : ℝ) : Prop := + IsEpsDeltaDP M Adj ε 0 + +variable {M : Kernel 𝒟 Y} {Adj : 𝒟 → 𝒟 → Prop} {ε ε' δ δ' : ℝ} + +lemma isEpsDP_iff_isEpsDeltaDP : IsEpsDP M Adj ε ↔ IsEpsDeltaDP M Adj ε 0 := Iff.rfl + +lemma isEpsDP_iff : IsEpsDP M Adj ε ↔ + ∀ ⦃S S'⦄, Adj S S' → ∀ E : Set Y, (M S).real E ≤ exp ε * (M S').real E := by + simp only [IsEpsDP, IsEpsDeltaDP, add_zero] + +/-- The `δ` of an `(ε, δ)`-DP mechanism is nonnegative as soon as some pair of datasets is +neighbouring (take `E = ∅`). -/ +lemma IsEpsDeltaDP.nonneg {S S' : 𝒟} (h : IsEpsDeltaDP M Adj ε δ) (hS : Adj S S') : 0 ≤ δ := by + simpa using h hS ∅ + +/-- `(ε, δ)`-DP is monotone in `ε` and `δ`. -/ +lemma IsEpsDeltaDP.mono (h : IsEpsDeltaDP M Adj ε δ) (hε : ε ≤ ε') (hδ : δ ≤ δ') : + IsEpsDeltaDP M Adj ε' δ' := fun _ _ hS E ↦ (h hS E).trans <| by gcongr + +/-- `(ε, δ)`-DP is antitone in the neighbouring relation. -/ +lemma IsEpsDeltaDP.of_le {Adj' : 𝒟 → 𝒟 → Prop} (h : IsEpsDeltaDP M Adj ε δ) + (hAdj : ∀ ⦃S S'⦄, Adj' S S' → Adj S S') : IsEpsDeltaDP M Adj' ε δ := + fun _ _ hS ↦ h (hAdj hS) + +/-- It suffices to check `(ε, δ)`-DP on measurable events: the inequality for an arbitrary set +`E` (where `(M S).real E` is an outer measure) follows from that for a common measurable hull +of `E` for `M S` and `M S'`. -/ +theorem isEpsDeltaDP_iff_forall_measurableSet : IsEpsDeltaDP M Adj ε δ ↔ + ∀ ⦃S S'⦄, Adj S S' → ∀ E : Set Y, MeasurableSet E → + (M S).real E ≤ exp ε * (M S').real E + δ := by + refine ⟨fun h S S' hS E _ ↦ h hS E, fun h S S' hS E ↦ ?_⟩ + obtain ⟨F, -, hF, h₁, h₂⟩ := exists_measurable_superset₂ (M S) (M S') E + simpa only [measureReal_def, h₁, h₂] using h hS F hF + +namespace DP + +variable {μ ν : Measure Y} {E : Set Y} + +/-- The real `(ε, δ)` inequality for one event implies its `ℝ≥0∞` form. -/ +lemma measure_le_of_measureReal_le [IsFiniteMeasure μ] + (h : μ.real E ≤ exp ε * ν.real E + δ) : + μ E ≤ ENNReal.ofReal (exp ε) * ν E + ENNReal.ofReal δ := by + calc μ E = ENNReal.ofReal (μ.real E) := (ofReal_measureReal).symm + _ ≤ ENNReal.ofReal (exp ε * ν.real E + δ) := ENNReal.ofReal_le_ofReal h + _ ≤ ENNReal.ofReal (exp ε * ν.real E) + ENNReal.ofReal δ := ENNReal.ofReal_add_le + _ ≤ ENNReal.ofReal (exp ε) * ν E + ENNReal.ofReal δ := by + rw [ENNReal.ofReal_mul (exp_pos ε).le, measureReal_def] + gcongr + exact ENNReal.ofReal_toReal_le + +/-- The `ℝ≥0∞` form of the `(ε, δ)` inequality for one event implies its real form. -/ +lemma measureReal_le_of_measure_le [IsFiniteMeasure ν] (hδ : 0 ≤ δ) + (h : μ E ≤ ENNReal.ofReal (exp ε) * ν E + ENNReal.ofReal δ) : + μ.real E ≤ exp ε * ν.real E + δ := by + have hne : ENNReal.ofReal (exp ε) * ν E + ENNReal.ofReal δ ≠ ∞ := by finiteness + calc μ.real E ≤ (ENNReal.ofReal (exp ε) * ν E + ENNReal.ofReal δ).toReal := + ENNReal.toReal_mono hne h + _ = exp ε * ν.real E + δ := by + rw [ENNReal.toReal_add (by finiteness) ENNReal.ofReal_ne_top, ENNReal.toReal_mul, + ENNReal.toReal_ofReal (exp_pos ε).le, ENNReal.toReal_ofReal hδ, measureReal_def] + +end DP + +/-- An `(ε, δ)`-DP finite kernel satisfies `M(S)(E) ≤ e^ε M(S')(E) + δ` in `ℝ≥0∞`. -/ +lemma IsEpsDeltaDP.measure_le [IsFiniteKernel M] (h : IsEpsDeltaDP M Adj ε δ) {S S' : 𝒟} + (hS : Adj S S') (E : Set Y) : + M S E ≤ ENNReal.ofReal (exp ε) * M S' E + ENNReal.ofReal δ := + DP.measure_le_of_measureReal_le (h hS E) + +/-- For a finite kernel and `δ ≥ 0`, `(ε, δ)`-DP is equivalent to its `ℝ≥0∞` form on measurable +events. -/ +theorem isEpsDeltaDP_iff_measure_le [IsFiniteKernel M] (hδ : 0 ≤ δ) : + IsEpsDeltaDP M Adj ε δ ↔ ∀ ⦃S S'⦄, Adj S S' → ∀ E : Set Y, MeasurableSet E → + M S E ≤ ENNReal.ofReal (exp ε) * M S' E + ENNReal.ofReal δ := by + rw [isEpsDeltaDP_iff_forall_measurableSet] + exact ⟨fun h S S' hS E hE ↦ DP.measure_le_of_measureReal_le (h hS E hE), + fun h S S' hS E hE ↦ DP.measureReal_le_of_measure_le hδ (h hS E hE)⟩ + +/-- For a finite kernel, pure `ε`-DP says that `M(S) ≤ e^ε M(S')` as measures. -/ +theorem isEpsDP_iff_le_smul [IsFiniteKernel M] : + IsEpsDP M Adj ε ↔ ∀ ⦃S S'⦄, Adj S S' → M S ≤ ENNReal.ofReal (exp ε) • M S' := by + simp only [isEpsDeltaDP_iff_measure_le le_rfl, ENNReal.ofReal_zero, add_zero, Measure.le_iff, + smul_apply, smul_eq_mul] + +/-- On a countable output space with measurable singletons, pure `ε`-DP of a finite kernel is +equivalent to the pointwise inequality `P[M(S) = y] ≤ e^ε P[M(S') = y]`. -/ +theorem isEpsDP_iff_forall_singleton [Countable Y] [MeasurableSingletonClass Y] + [IsFiniteKernel M] : IsEpsDP M Adj ε ↔ + ∀ ⦃S S'⦄, Adj S S' → ∀ y : Y, (M S).real {y} ≤ exp ε * (M S').real {y} := by + refine ⟨fun h S S' hS y ↦ isEpsDP_iff.1 h hS {y}, fun h ↦ ?_⟩ + refine isEpsDP_iff_le_smul.2 fun S S' hS ↦ Measure.le_iff.2 fun E hE ↦ ?_ + have hy (y : Y) : M S {y} ≤ ENNReal.ofReal (exp ε) * M S' {y} := by + simpa using DP.measure_le_of_measureReal_le (δ := 0) (by simpa using h hS y) + rw [smul_apply, smul_eq_mul, ← (M S).tsum_indicator_apply_singleton E hE, + ← (M S').tsum_indicator_apply_singleton E hE, ← ENNReal.tsum_mul_left] + refine tsum_le_tsum fun y ↦ ?_ + by_cases hyE : y ∈ E <;> simp [hyE, hy] + +/-- A constant mechanism (ignoring its input) with a finite output distribution is +`(ε, δ)`-DP for all `ε, δ ≥ 0`. The finiteness hypothesis is not needed by the proof; it excludes +the degenerate case of an infinite `μ`, where `Measure.real` is `0` on every set of infinite +measure. -/ +@[nolint unusedArguments] +lemma isEpsDeltaDP_const (μ : Measure Y) [IsFiniteMeasure μ] (hε : 0 ≤ ε) (hδ : 0 ≤ δ) : + IsEpsDeltaDP (Kernel.const 𝒟 μ) Adj ε δ := fun _ _ _ E ↦ by + simp only [Kernel.const_apply] + nlinarith [one_le_exp hε, measureReal_nonneg (μ := μ) (s := E)] + +/-- Every Markov kernel is `(ε, δ)`-DP when `δ ≥ 1`. -/ +lemma isEpsDeltaDP_of_one_le [IsMarkovKernel M] (hδ : 1 ≤ δ) : IsEpsDeltaDP M Adj ε δ := + fun S S' _ E ↦ by + nlinarith [measureReal_le_one (μ := M S) (s := E), measureReal_nonneg (μ := M S') (s := E), + exp_pos ε] + +/-! ### Post-processing -/ + +namespace DP + +variable {μ ν : Measure Y} {c c₁ c₂ d d₁ d₂ : ℝ≥0∞} + +/-- An `(ε, δ)`-type inequality `μ(E) ≤ c ν(E) + d` on all measurable events extends to all +measurable functions `g` with values in `[0, 1]`: `∫ g dμ ≤ c ∫ g dν + d`. The proof integrates +the event inequality over the superlevel sets of `g` (layer-cake formula). -/ +lemma lintegral_le_of_forall_measure_le (h : ∀ E, MeasurableSet E → μ E ≤ c * ν E + d) + {g : Y → ℝ≥0∞} (hg : Measurable g) (hg₁ : ∀ y, g y ≤ 1) : + ∫⁻ y, g y ∂μ ≤ c * ∫⁻ y, g y ∂ν + d := by + rw [lintegral_eq_lintegral_measure_ofReal_le μ hg.aemeasurable, + lintegral_eq_lintegral_measure_ofReal_le ν hg.aemeasurable] + have hmeas : Measurable fun t : ℝ ↦ ν {x | ENNReal.ofReal t ≤ g x} := + Antitone.measurable fun t t' htt' ↦ measure_mono fun x hx ↦ + (ENNReal.ofReal_le_ofReal htt').trans hx + calc ∫⁻ t in Ioi 0, μ {x | ENNReal.ofReal t ≤ g x} + ≤ ∫⁻ t in Ioi 0, (c * ν {x | ENNReal.ofReal t ≤ g x} + (Iic 1).indicator (fun _ ↦ d) t) := by + refine lintegral_mono fun t ↦ ?_ + by_cases ht : t ≤ 1 + · simpa [ht] using h _ (measurableSet_le measurable_const hg) + · have : {x | ENNReal.ofReal t ≤ g x} = ∅ := eq_empty_iff_forall_notMem.2 fun x hx ↦ + (not_le.2 (ENNReal.one_lt_ofReal.2 (not_le.1 ht))) ((hx : _ ≤ _).trans (hg₁ x)) + simp [this] + _ = c * (∫⁻ t in Ioi 0, ν {x | ENNReal.ofReal t ≤ g x}) + d := by + rw [lintegral_add_right _ (measurable_const.indicator measurableSet_Iic), + lintegral_const_mul _ hmeas, lintegral_indicator_const measurableSet_Iic, + Measure.restrict_apply measurableSet_Iic, inter_comm, Ioi_inter_Iic, Real.volume_Ioc] + simp + +/-- The integral step of the composition theorems: if `μ(E) ≤ c₁ μ'(E) + d₁` on measurable +events and `k ≤ min (1, c₂ g + d₂)` pointwise, then `∫ k dμ ≤ c₁ c₂ ∫ g dμ' + d₁ + d₂`. -/ +lemma lintegral_le_mul_add [IsProbabilityMeasure μ] (h : ∀ E, MeasurableSet E → μ E ≤ c₁ * ν E + d₁) + {k g : Y → ℝ≥0∞} (hg : Measurable g) (hk₁ : ∀ y, k y ≤ 1) + (hk : ∀ y, k y ≤ c₂ * g y + d₂) : + ∫⁻ y, k y ∂μ ≤ c₁ * c₂ * ∫⁻ y, g y ∂ν + (d₁ + d₂) := by + have hpt (y : Y) : k y ≤ min 1 (c₂ * g y) + d₂ := by + rcases le_total 1 (c₂ * g y) with h' | h' + · rw [min_eq_left h'] + exact (hk₁ y).trans le_self_add + · rw [min_eq_right h'] + exact hk y + calc ∫⁻ y, k y ∂μ ≤ ∫⁻ y, (min 1 (c₂ * g y) + d₂) ∂μ := lintegral_mono hpt + _ = ∫⁻ y, min 1 (c₂ * g y) ∂μ + d₂ := by + rw [lintegral_add_right _ measurable_const, lintegral_const, measure_univ, mul_one] + _ ≤ c₁ * ∫⁻ y, min 1 (c₂ * g y) ∂ν + d₁ + d₂ := by + gcongr + exact lintegral_le_of_forall_measure_le h (measurable_const.min (hg.const_mul c₂)) + fun y ↦ min_le_left _ _ + _ ≤ c₁ * ∫⁻ y, c₂ * g y ∂ν + d₁ + d₂ := by + gcongr with y + exact min_le_right _ _ + _ = c₁ * c₂ * ∫⁻ y, g y ∂ν + (d₁ + d₂) := by + rw [lintegral_const_mul _ hg, mul_assoc, add_assoc] + +end DP + +/-- **Proposition 2.1 of Dwork–Roth** (post-processing): if `M` is `(ε, δ)`-DP, then so is +`f ∘ₖ M` for every Markov kernel `f` (an arbitrary randomized mapping of the output), on +arbitrary measurable spaces. -/ +theorem IsEpsDeltaDP.comp [IsFiniteKernel M] (h : IsEpsDeltaDP M Adj ε δ) (f : Kernel Y Z) + [IsMarkovKernel f] : IsEpsDeltaDP (f ∘ₖ M) Adj ε δ := by + rw [isEpsDeltaDP_iff_forall_measurableSet] + intro S S' hS E hE + refine DP.measureReal_le_of_measure_le (h.nonneg hS) ?_ + rw [Kernel.comp_apply' _ _ _ hE, Kernel.comp_apply' _ _ _ hE] + exact DP.lintegral_le_of_forall_measure_le (fun F _ ↦ h.measure_le hS F) + (Kernel.measurable_coe f hE) fun y ↦ prob_le_one + +/-- Post-processing by a measurable function (the deterministic case of Proposition 2.1 of +Dwork–Roth). -/ +theorem IsEpsDeltaDP.map [IsFiniteKernel M] (h : IsEpsDeltaDP M Adj ε δ) {φ : Y → Z} + (hφ : Measurable φ) : IsEpsDeltaDP (M.map φ) Adj ε δ := by + rw [← Kernel.deterministic_comp_eq_map hφ] + exact h.comp _ + +/-! ### Group privacy -/ + +/-- `S` and `S'` are `k`-neighbours (J. Dong, A. Roth and W. J. Su, *Gaussian differential +privacy*, J. R. Stat. Soc. Ser. B 84(1), 2022, Section 2.5): they are joined by a chain +`S = S₀, S₁, …, S_k = S'` in which consecutive datasets are neighbouring or identical. -/ +inductive IsKNeighbour (Adj : 𝒟 → 𝒟 → Prop) : ℕ → 𝒟 → 𝒟 → Prop + | zero (S : 𝒟) : IsKNeighbour Adj 0 S S + | succ {k : ℕ} {S T U : 𝒟} : + IsKNeighbour Adj k S T → Adj T U ∨ T = U → IsKNeighbour Adj (k + 1) S U + +/-- **Group privacy for `(ε, δ)`-DP, chained form**: if `M` is `(ε, δ)`-DP with `ε, δ ≥ 0`, then +for datasets joined by a chain of `k` neighbouring (or identical) steps (`IsKNeighbour Adj k`), +`P[M(S) ∈ E] ≤ e^{kε} P[M(S') ∈ E] + δ ∑_{i < k} e^{iε}`. This is the constant that chaining the +definition `k` times gives; it is sharper than the constant `k e^{(k-1)ε} δ` of footnote 1 to +Theorem 2.2 of Dwork–Roth (see `IsEpsDeltaDP.group'`), is not stated in the book, and is not +claimed to be tight. -/ +theorem IsEpsDeltaDP.group (h : IsEpsDeltaDP M Adj ε δ) (hε : 0 ≤ ε) (hδ : 0 ≤ δ) (k : ℕ) : + IsEpsDeltaDP M (IsKNeighbour Adj k) (k * ε) (δ * ∑ i ∈ Finset.range k, exp (i * ε)) := by + intro S S' hSS' E + induction hSS' with + | zero S => simp + | @succ n S T U _ hTU ih => + have hP := measureReal_nonneg (μ := M T) (s := E) + have hQ := measureReal_nonneg (μ := M U) (s := E) + have h₁ := one_le_exp hε + have h₂ := (exp_pos (n * ε)).le + rw [Finset.sum_range_succ, Nat.cast_succ, add_mul, one_mul, exp_add] + rcases hTU with hTU | rfl + · have := h hTU E + have : exp (n * ε) * (M T).real E ≤ exp (n * ε) * (exp ε * (M U).real E + δ) := by gcongr + nlinarith + · nlinarith [mul_le_mul_of_nonneg_left h₁ (mul_nonneg h₂ hP)] + +/-- **Theorem 2.2 of Dwork–Roth** (group privacy): an `ε`-DP mechanism with `ε ≥ 0` is +`kε`-DP for groups of size `k`. Here "groups of size `k`" means datasets joined by a chain of `k` +neighbouring (or identical) steps (`IsKNeighbour Adj k`); the link with the book's +`‖x - y‖₁ ≤ k` on histograms (that such `x, y` are joined by such a chain for `Adj` given by +`‖x - y‖₁ ≤ 1`) is not formalized. -/ +theorem IsEpsDP.group (h : IsEpsDP M Adj ε) (hε : 0 ≤ ε) (k : ℕ) : + IsEpsDP M (IsKNeighbour Adj k) (k * ε) := by + simpa using IsEpsDeltaDP.group h hε le_rfl k + +/-- **Group privacy for `(ε, δ)`-DP with the constant of Dwork–Roth** (footnote 1 to +Theorem 2.2, Section 2.3): an `(ε, δ)`-DP mechanism with `ε, δ ≥ 0` is +`(kε, k e^{(k-1)ε} δ)`-DP for groups of size `k`, i.e. for datasets joined by a chain of `k` +neighbouring (or identical) steps (`IsKNeighbour Adj k`; the link with `‖x - y‖₁ ≤ k` is not +formalized). -/ +theorem IsEpsDeltaDP.group' (h : IsEpsDeltaDP M Adj ε δ) (hε : 0 ≤ ε) (hδ : 0 ≤ δ) (k : ℕ) : + IsEpsDeltaDP M (IsKNeighbour Adj k) (k * ε) (k * exp ((k - 1) * ε) * δ) := by + refine (h.group hε hδ k).mono le_rfl ?_ + have : ∑ i ∈ Finset.range k, exp (i * ε) ≤ k * exp ((k - 1) * ε) := by + have := Finset.sum_le_card_nsmul (Finset.range k) (fun i : ℕ ↦ exp (i * ε)) + (exp ((k - 1) * ε)) fun i hi ↦ by + have : (i : ℝ) + 1 ≤ k := by exact_mod_cast Finset.mem_range.1 hi + gcongr + linarith + simpa using this + nlinarith + +/-! ### Composition -/ + +/-- **Adaptive composition** (Theorem B.1 of Dwork–Roth, extending Theorem 3.16 to adaptively +chosen mechanisms; here with possibly different parameters): if `M₁` is `(ε₁, δ₁)`-DP and the +second mechanism `M₂`, which sees the dataset and the first output `y`, is `(ε₂, δ₂)`-DP in the +dataset for every fixed `y`, then the joint release `M₁ ⊗ₖ M₂` of both outputs is +`(ε₁ + ε₂, δ₁ + δ₂)`-DP. General measurable spaces; `M₁` and `M₂` are Markov kernels. -/ +theorem IsEpsDeltaDP.compProd {M₁ : Kernel 𝒟 Y} {M₂ : Kernel (𝒟 × Y) Z} [IsMarkovKernel M₁] + [IsMarkovKernel M₂] {ε₁ ε₂ δ₁ δ₂ : ℝ} (h₁ : IsEpsDeltaDP M₁ Adj ε₁ δ₁) + (h₂ : ∀ y, IsEpsDeltaDP (M₂.sectL y) Adj ε₂ δ₂) : + IsEpsDeltaDP (M₁ ⊗ₖ M₂) Adj (ε₁ + ε₂) (δ₁ + δ₂) := by + rw [isEpsDeltaDP_iff_forall_measurableSet] + intro S S' hS E hE + have hδ₁ := h₁.nonneg hS + have hδ₂ : 0 ≤ δ₂ := by + obtain ⟨y⟩ := nonempty_of_isProbabilityMeasure (M₁ S) + exact (h₂ y).nonneg hS + refine DP.measureReal_le_of_measure_le (add_nonneg hδ₁ hδ₂) ?_ + rw [Kernel.compProd_apply hE, Kernel.compProd_apply hE, ENNReal.ofReal_add hδ₁ hδ₂, exp_add, + ENNReal.ofReal_mul (exp_pos _).le] + exact DP.lintegral_le_mul_add (fun F _ ↦ h₁.measure_le hS F) + (Kernel.measurable_kernel_prodMk_left' hE S') (fun y ↦ prob_le_one) + fun y ↦ (h₂ y).measure_le hS _ + +/-- **Theorem 3.16 of Dwork–Roth for two mechanisms** (basic composition): if `M₁` is +`(ε₁, δ₁)`-DP and `M₂` is `(ε₂, δ₂)`-DP, then `S ↦ (M₁(S), M₂(S))` (independent runs) is +`(ε₁ + ε₂, δ₁ + δ₂)`-DP. General measurable spaces; `M₁` and `M₂` are Markov kernels. -/ +theorem IsEpsDeltaDP.prod {M₁ : Kernel 𝒟 Y} {M₂ : Kernel 𝒟 Z} [IsMarkovKernel M₁] + [IsMarkovKernel M₂] {ε₁ ε₂ δ₁ δ₂ : ℝ} (h₁ : IsEpsDeltaDP M₁ Adj ε₁ δ₁) + (h₂ : IsEpsDeltaDP M₂ Adj ε₂ δ₂) : + IsEpsDeltaDP (M₁ ×ₖ M₂) Adj (ε₁ + ε₂) (δ₁ + δ₂) := by + rw [isEpsDeltaDP_iff_forall_measurableSet] + intro S S' hS E hE + have hδ₁ := h₁.nonneg hS + have hδ₂ := h₂.nonneg hS + refine DP.measureReal_le_of_measure_le (add_nonneg hδ₁ hδ₂) ?_ + rw [Kernel.prod_apply' _ _ _ hE, Kernel.prod_apply' _ _ _ hE, ENNReal.ofReal_add hδ₁ hδ₂, + exp_add, ENNReal.ofReal_mul (exp_pos _).le] + exact DP.lintegral_le_mul_add (fun F _ ↦ h₁.measure_le hS F) + (measurable_measure_prodMk_left hE) (fun y ↦ prob_le_one) fun y ↦ h₂.measure_le hS _ + +/-- **Theorem 3.14 of Dwork–Roth** (composition of pure DP): if `M₁` is `ε₁`-DP and `M₂` is +`ε₂`-DP, then `S ↦ (M₁(S), M₂(S))` is `(ε₁ + ε₂)`-DP. -/ +theorem IsEpsDP.prod {M₁ : Kernel 𝒟 Y} {M₂ : Kernel 𝒟 Z} [IsMarkovKernel M₁] + [IsMarkovKernel M₂] {ε₁ ε₂ : ℝ} (h₁ : IsEpsDP M₁ Adj ε₁) (h₂ : IsEpsDP M₂ Adj ε₂) : + IsEpsDP (M₁ ×ₖ M₂) Adj (ε₁ + ε₂) := by + simpa using IsEpsDeltaDP.prod h₁ h₂ + +/-- Adaptive composition of pure DP (the case `δ₁ = δ₂ = 0` of `IsEpsDeltaDP.compProd`). -/ +theorem IsEpsDP.compProd {M₁ : Kernel 𝒟 Y} {M₂ : Kernel (𝒟 × Y) Z} [IsMarkovKernel M₁] + [IsMarkovKernel M₂] {ε₁ ε₂ : ℝ} (h₁ : IsEpsDP M₁ Adj ε₁) + (h₂ : ∀ y, IsEpsDP (M₂.sectL y) Adj ε₂) : IsEpsDP (M₁ ⊗ₖ M₂) Adj (ε₁ + ε₂) := by + simpa using IsEpsDeltaDP.compProd h₁ h₂ + +/-- The `k`-fold composition `S ↦ (M₀(S), …, M_{k-1}(S))` of independently run mechanisms +`M_i : Kernel 𝒟 (R i)`, with values in `(i : Fin k) → R i` (the mechanism `M_[k]` of +Corollary 3.15 and Theorem 3.16 of Dwork–Roth). It is defined by recursion on `k`, splitting off +the coordinate `0` with `MeasurableEquiv.piFinSuccAbove`. -/ +noncomputable def DP.composeFin : {k : ℕ} → {R : Fin k → Type*} → [∀ i, MeasurableSpace (R i)] → + ((i : Fin k) → Kernel 𝒟 (R i)) → Kernel 𝒟 ((i : Fin k) → R i) + | 0, _, _, _ => Kernel.const 𝒟 (Measure.dirac isEmptyElim) + | _ + 1, R, _, M => (M 0 ×ₖ DP.composeFin fun j ↦ M (Fin.succAbove 0 j)).map + (MeasurableEquiv.piFinSuccAbove R 0).symm + +instance DP.isMarkovKernel_composeFin {k : ℕ} {R : Fin k → Type*} [∀ i, MeasurableSpace (R i)] + (M : (i : Fin k) → Kernel 𝒟 (R i)) [∀ i, IsMarkovKernel (M i)] : + IsMarkovKernel (DP.composeFin M) := by + induction k with + | zero => rw [DP.composeFin]; infer_instance + | succ k ih => + have := ih fun j ↦ M (Fin.succAbove 0 j) + rw [DP.composeFin] + exact Kernel.IsMarkovKernel.map _ (MeasurableEquiv.measurable _) + +/-- **Theorem 3.16 of Dwork–Roth** (basic composition of `k` mechanisms): if +`M_i : Kernel 𝒟 (R i)` is `(ε_i, δ_i)`-DP for every `i < k`, then +`S ↦ (M₀(S), …, M_{k-1}(S))` is `(∑ ε_i, ∑ δ_i)`-DP. General measurable output spaces `R i`; +the `M_i` are Markov kernels. -/ +theorem DP.isEpsDeltaDP_composeFin {k : ℕ} {R : Fin k → Type*} [∀ i, MeasurableSpace (R i)] + {M : (i : Fin k) → Kernel 𝒟 (R i)} [∀ i, IsMarkovKernel (M i)] {ε δ : Fin k → ℝ} + (h : ∀ i, IsEpsDeltaDP (M i) Adj (ε i) (δ i)) : + IsEpsDeltaDP (DP.composeFin M) Adj (∑ i, ε i) (∑ i, δ i) := by + induction k with + | zero => simpa [DP.composeFin] using isEpsDeltaDP_const _ le_rfl le_rfl + | succ k ih => + rw [DP.composeFin, Fin.sum_univ_succAbove _ 0, Fin.sum_univ_succAbove _ 0] + exact ((h 0).prod (ih (M := fun j ↦ M (Fin.succAbove 0 j)) + (ε := fun j ↦ ε (Fin.succAbove 0 j)) (δ := fun j ↦ δ (Fin.succAbove 0 j)) + fun j ↦ h _)).map (MeasurableEquiv.measurable _) + +/-- **Corollary 3.15 of Dwork–Roth** (composition of `k` pure-DP mechanisms): if +`M_i : Kernel 𝒟 (R i)` is `ε_i`-DP for every `i < k`, then `S ↦ (M₀(S), …, M_{k-1}(S))` is +`(∑ ε_i)`-DP. -/ +theorem DP.isEpsDP_composeFin {k : ℕ} {R : Fin k → Type*} [∀ i, MeasurableSpace (R i)] + {M : (i : Fin k) → Kernel 𝒟 (R i)} [∀ i, IsMarkovKernel (M i)] {ε : Fin k → ℝ} + (h : ∀ i, IsEpsDP (M i) Adj (ε i)) : IsEpsDP (DP.composeFin M) Adj (∑ i, ε i) := by + simpa using DP.isEpsDeltaDP_composeFin (δ := fun _ ↦ 0) h + +/-! ### Change of dataset space -/ + +/-- `(ε, δ)`-DP transfers along a measurable map of datasets `g` that sends `Adj'`-neighbours to +`Adj`-neighbours: `S ↦ M(g S)` is then `(ε, δ)`-DP for `Adj'`. -/ +theorem IsEpsDeltaDP.comap {𝒟' : Type*} [MeasurableSpace 𝒟'] {Adj' : 𝒟' → 𝒟' → Prop} + (h : IsEpsDeltaDP M Adj ε δ) {g : 𝒟' → 𝒟} (hg : Measurable g) + (hAdj : ∀ ⦃S S'⦄, Adj' S S' → Adj (g S) (g S')) : IsEpsDeltaDP (M.comap g hg) Adj' ε δ := + fun _ _ hS E ↦ h (hAdj hS) E + +/-! ### Parallel composition + +F. McSherry, *Privacy integrated queries*, SIGMOD 2009, Theorem 4: if `M_i` is `ε_i`-DP and the +`D_i` are disjoint, then releasing `M_i(X ∩ D_i)` for all `i`, where each `M_i` may also see the +earlier outputs, is `max_i ε_i`-DP. We formalize the case of **two parts**. The dataset is +modelled as the pair of its parts, with the relation `DP.ParallelAdj` (one part changes by a +neighbouring step); `DP.parallelAdj_inter_compl` links this to one dataset `X : Set α` split as +`(X ∩ D, X ∩ Dᶜ)`, and `IsEpsDP.parallel_compProd_inter` is the resulting statement for one +dataset: the duplicate-free (sets rather than PINQ's multisets), adjacent-dataset (rather than +all-pairs symmetric-difference distance), complementary-bipartition version of Theorem 4. -/ + +/-- The neighbouring relation on datasets `(S₁, S₂)` made of two disjoint parts: two such +datasets are neighbours if they differ in exactly one part, by a neighbouring step there. -/ +def DP.ParallelAdj {𝒟₁ 𝒟₂ : Type*} (Adj₁ : 𝒟₁ → 𝒟₁ → Prop) (Adj₂ : 𝒟₂ → 𝒟₂ → Prop) + (S S' : 𝒟₁ × 𝒟₂) : Prop := + (Adj₁ S.1 S'.1 ∧ S.2 = S'.2) ∨ (S.1 = S'.1 ∧ Adj₂ S.2 S'.2) + +/-- Datasets `X, X' : Set α` of records differ in one record (one is obtained from the other by +adding or removing a record): their symmetric difference is a singleton. -/ +def DP.AddRemoveAdj {α : Type*} (X X' : Set α) : Prop := + ∃ a, symmDiff X X' = {a} + +/-- Splitting a dataset `X : Set α` along `D` and its complement turns datasets that differ in +one record into `DP.ParallelAdj`-neighbours. -/ +lemma DP.parallelAdj_inter_compl {α : Type*} (D : Set α) {X X' : Set α} + (h : DP.AddRemoveAdj X X') : + DP.ParallelAdj DP.AddRemoveAdj DP.AddRemoveAdj (X ∩ D, X ∩ Dᶜ) (X' ∩ D, X' ∩ Dᶜ) := by + obtain ⟨a, ha⟩ := h + have key (U : Set α) : symmDiff (X ∩ U) (X' ∩ U) = {a} ∩ U := by + rw [← inter_symmDiff_distrib_right, ha] + by_cases haD : a ∈ D + · refine Or.inl ⟨⟨a, by rw [key, singleton_inter_of_mem haD]⟩, ?_⟩ + rw [← symmDiff_eq_bot, key, singleton_inter_eq_empty.2 (notMem_compl_iff.2 haD)] + rfl + · refine Or.inr ⟨?_, ⟨a, by rw [key, singleton_inter_of_mem (mem_compl haD)]⟩⟩ + rw [← symmDiff_eq_bot, key, singleton_inter_eq_empty.2 haD] + rfl + +variable {𝒟₁ 𝒟₂ : Type*} {m𝒟₁ : MeasurableSpace 𝒟₁} {m𝒟₂ : MeasurableSpace 𝒟₂} + {Adj₁ : 𝒟₁ → 𝒟₁ → Prop} {Adj₂ : 𝒟₂ → 𝒟₂ → Prop} {ε₁ ε₂ δ₁ δ₂ : ℝ} + +/-- **Adaptive parallel composition, two parts** (McSherry, SIGMOD 2009, Theorem 4 for two parts, +here with the `(ε, δ)` extension): `M₁` runs on the first part `S₁` and `M₂` runs on the +second part `S₂` and also sees the first output `y`. If `M₁` is `(ε₁, δ₁)`-DP and `M₂` is +`(ε₂, δ₂)`-DP in the dataset for every fixed `y`, the joint release is +`(max ε₁ ε₂, max δ₁ δ₂)`-DP for `DP.ParallelAdj`. (PINQ treats pure DP only; the `δ` part is an +extension.) -/ +theorem IsEpsDeltaDP.parallel_compProd {M₁ : Kernel 𝒟₁ Y} {M₂ : Kernel (𝒟₂ × Y) Z} + [IsMarkovKernel M₁] [IsMarkovKernel M₂] (h₁ : IsEpsDeltaDP M₁ Adj₁ ε₁ δ₁) + (h₂ : ∀ y, IsEpsDeltaDP (M₂.sectL y) Adj₂ ε₂ δ₂) : + IsEpsDeltaDP (Kernel.prodMkRight 𝒟₂ M₁ ⊗ₖ + M₂.comap (fun p : (𝒟₁ × 𝒟₂) × Y ↦ (p.1.2, p.2)) (by fun_prop)) + (DP.ParallelAdj Adj₁ Adj₂) (max ε₁ ε₂) (max δ₁ δ₂) := by + rintro S S' (hS | hS) + · have A : IsEpsDeltaDP (Kernel.prodMkRight 𝒟₂ M₁) + (fun S S' ↦ Adj₁ S.1 S'.1 ∧ S.2 = S'.2) ε₁ δ₁ := fun _ _ h E ↦ h₁ h.1 E + have B (y : Y) : IsEpsDeltaDP ((M₂.comap (fun p : (𝒟₁ × 𝒟₂) × Y ↦ (p.1.2, p.2)) + (by fun_prop)).sectL y) (fun S S' ↦ Adj₁ S.1 S'.1 ∧ S.2 = S'.2) 0 0 := + fun _ _ h E ↦ by simp [h.2] + exact (A.compProd B).mono (by simp) (by simp) hS + · have A : IsEpsDeltaDP (Kernel.prodMkRight 𝒟₂ M₁) + (fun S S' ↦ S.1 = S'.1 ∧ Adj₂ S.2 S'.2) 0 0 := fun _ _ h E ↦ by + simp [Kernel.prodMkRight_apply, h.1] + have B (y : Y) : IsEpsDeltaDP ((M₂.comap (fun p : (𝒟₁ × 𝒟₂) × Y ↦ (p.1.2, p.2)) + (by fun_prop)).sectL y) (fun S S' ↦ S.1 = S'.1 ∧ Adj₂ S.2 S'.2) ε₂ δ₂ := + fun _ _ h E ↦ h₂ y h.2 E + exact (A.compProd B).mono (by simp) (by simp) hS + +/-- **Adaptive parallel composition for pure DP, two parts** (McSherry, SIGMOD 2009, Theorem 4 +for two parts): `ε₁`- and `ε₂`-DP mechanisms run on disjoint parts of the data, the second one +also seeing the first output, give a `max ε₁ ε₂`-DP release. -/ +theorem IsEpsDP.parallel_compProd {M₁ : Kernel 𝒟₁ Y} {M₂ : Kernel (𝒟₂ × Y) Z} + [IsMarkovKernel M₁] [IsMarkovKernel M₂] (h₁ : IsEpsDP M₁ Adj₁ ε₁) + (h₂ : ∀ y, IsEpsDP (M₂.sectL y) Adj₂ ε₂) : + IsEpsDP (Kernel.prodMkRight 𝒟₂ M₁ ⊗ₖ + M₂.comap (fun p : (𝒟₁ × 𝒟₂) × Y ↦ (p.1.2, p.2)) (by fun_prop)) + (DP.ParallelAdj Adj₁ Adj₂) (max ε₁ ε₂) := by + simpa using IsEpsDeltaDP.parallel_compProd h₁ h₂ + +/-- **McSherry's parallel composition for one dataset split in two** (SIGMOD 2009, Theorem 4: +the duplicate-free, adjacent-dataset, complementary-bipartition version): for datasets +`X : Set α` of records, neighbouring when they differ in one record (`DP.AddRemoveAdj`), and any +`D : Set α`, releasing `y ∼ M₁(X ∩ D)` and then `M₂(X ∩ Dᶜ, y)` is `max ε₁ ε₂`-DP if `M₁` is +`ε₁`-DP and each `M₂(·, y)` is `ε₂`-DP. Datasets carry Mathlib's σ-algebra +`Set.instMeasurableSpace` on `Set α`. PINQ's datasets are multisets and its guarantee covers all +pairs of datasets at a given symmetric-difference distance; neither bridge is formalized here. -/ +theorem IsEpsDP.parallel_compProd_inter {α : Type*} (D : Set α) + {M₁ : Kernel (Set α) Y} + {M₂ : Kernel (Set α × Y) Z} [IsMarkovKernel M₁] [IsMarkovKernel M₂] + (h₁ : IsEpsDP M₁ DP.AddRemoveAdj ε₁) (h₂ : ∀ y, IsEpsDP (M₂.sectL y) DP.AddRemoveAdj ε₂) : + IsEpsDP ((Kernel.prodMkRight (Set α) M₁ ⊗ₖ + M₂.comap (fun p : (Set α × Set α) × Y ↦ (p.1.2, p.2)) (by fun_prop)).comap + (fun X : Set α ↦ (X ∩ D, X ∩ Dᶜ)) + ((measurable_id.inter measurable_const).prodMk (measurable_id.inter measurable_const))) + DP.AddRemoveAdj (max ε₁ ε₂) := + (IsEpsDP.parallel_compProd h₁ h₂).comap _ fun _ _ h ↦ DP.parallelAdj_inter_compl D h + +/-- **Non-adaptive parallel composition, two parts** (the special case of +`IsEpsDeltaDP.parallel_compProd` in which `M₂` ignores the first output): running an +`(ε₁, δ₁)`-DP mechanism `M₁` on one part of the data and an `(ε₂, δ₂)`-DP mechanism `M₂` on the +other part gives a `(max ε₁ ε₂, max δ₁ δ₂)`-DP mechanism for `DP.ParallelAdj`. For `δ₁ = δ₂ = 0` +this is the non-adaptive two-part case of McSherry's Theorem 4; the `δ` part is not in PINQ. -/ +theorem IsEpsDeltaDP.parallel {M₁ : Kernel 𝒟₁ Y} {M₂ : Kernel 𝒟₂ Z} [IsMarkovKernel M₁] + [IsMarkovKernel M₂] (h₁ : IsEpsDeltaDP M₁ Adj₁ ε₁ δ₁) (h₂ : IsEpsDeltaDP M₂ Adj₂ ε₂ δ₂) : + IsEpsDeltaDP (Kernel.prodMkRight 𝒟₂ M₁ ×ₖ Kernel.prodMkLeft 𝒟₁ M₂) + (DP.ParallelAdj Adj₁ Adj₂) (max ε₁ ε₂) (max δ₁ δ₂) := by + rintro S S' (hS | hS) + · have A : IsEpsDeltaDP (Kernel.prodMkRight 𝒟₂ M₁) + (fun S S' ↦ Adj₁ S.1 S'.1 ∧ S.2 = S'.2) ε₁ δ₁ := fun _ _ h E ↦ h₁ h.1 E + have B : IsEpsDeltaDP (Kernel.prodMkLeft 𝒟₁ M₂) + (fun S S' ↦ Adj₁ S.1 S'.1 ∧ S.2 = S'.2) 0 0 := fun _ _ h E ↦ by + simp [Kernel.prodMkLeft_apply, h.2] + exact (A.prod B).mono (by simp) (by simp) hS + · have A : IsEpsDeltaDP (Kernel.prodMkRight 𝒟₂ M₁) + (fun S S' ↦ S.1 = S'.1 ∧ Adj₂ S.2 S'.2) 0 0 := fun _ _ h E ↦ by + simp [Kernel.prodMkRight_apply, h.1] + have B : IsEpsDeltaDP (Kernel.prodMkLeft 𝒟₁ M₂) + (fun S S' ↦ S.1 = S'.1 ∧ Adj₂ S.2 S'.2) ε₂ δ₂ := fun _ _ h E ↦ h₂ h.2 E + exact (A.prod B).mono (by simp) (by simp) hS + +/-- **Non-adaptive parallel composition for pure DP, two parts** (the non-adaptive two-part case +of McSherry, SIGMOD 2009, Theorem 4): `ε₁`- and `ε₂`-DP mechanisms run on disjoint parts of the +data give a `max ε₁ ε₂`-DP mechanism. -/ +theorem IsEpsDP.parallel {M₁ : Kernel 𝒟₁ Y} {M₂ : Kernel 𝒟₂ Z} [IsMarkovKernel M₁] + [IsMarkovKernel M₂] (h₁ : IsEpsDP M₁ Adj₁ ε₁) (h₂ : IsEpsDP M₂ Adj₂ ε₂) : + IsEpsDP (Kernel.prodMkRight 𝒟₂ M₁ ×ₖ Kernel.prodMkLeft 𝒟₁ M₂) + (DP.ParallelAdj Adj₁ Adj₂) (max ε₁ ε₂) := by + simpa using IsEpsDeltaDP.parallel h₁ h₂ + +end InformationTheory From 8339a38cf40b20b1332615def7aa7d60e7b26a18 Mon Sep 17 00:00:00 2001 From: Kousha Kalantari <25445970+KoushaKalantari@users.noreply.github.com> Date: Fri, 13 Nov 2026 10:19:15 -0800 Subject: [PATCH 2/5] feat(Privacy): randomized response, Laplace, geometric and exponential mechanisms Co-Authored-By: Claude Opus 5.5 --- Statlib.lean | 4 + Statlib/Privacy/DP/Exponential.lean | 334 +++++++++++++++ Statlib/Privacy/DP/Geometric.lean | 269 ++++++++++++ Statlib/Privacy/DP/Laplace.lean | 459 +++++++++++++++++++++ Statlib/Privacy/DP/RandomizedResponse.lean | 338 +++++++++++++++ 5 files changed, 1404 insertions(+) create mode 100644 Statlib/Privacy/DP/Exponential.lean create mode 100644 Statlib/Privacy/DP/Geometric.lean create mode 100644 Statlib/Privacy/DP/Laplace.lean create mode 100644 Statlib/Privacy/DP/RandomizedResponse.lean diff --git a/Statlib.lean b/Statlib.lean index 2f13f19..abafa48 100644 --- a/Statlib.lean +++ b/Statlib.lean @@ -139,6 +139,10 @@ import Statlib.ForMathlib.Topology.Order.MonotoneConvergence import Statlib.Inference.Inference import Statlib.Inference.QMD import Statlib.Privacy.DP.Basic +import Statlib.Privacy.DP.Exponential +import Statlib.Privacy.DP.Geometric +import Statlib.Privacy.DP.Laplace +import Statlib.Privacy.DP.RandomizedResponse import Statlib.Testing.Binary import Statlib.Testing.BinaryRenyi import Statlib.Testing.ChangeMeasure diff --git a/Statlib/Privacy/DP/Exponential.lean b/Statlib/Privacy/DP/Exponential.lean new file mode 100644 index 0000000..3ea405c --- /dev/null +++ b/Statlib/Privacy/DP/Exponential.lean @@ -0,0 +1,334 @@ +/- +Copyright (c) 2026 Kousha Kalantari. All rights reserved. +Released under Apache 2.0 license as described in the file LICENSE. +Authors: Kousha Kalantari +-/ +module + +public import Statlib.Privacy.DP.Basic +public import Mathlib.MeasureTheory.Measure.Count +public import Mathlib.Order.ConditionallyCompleteLattice.Finset +public import Mathlib.Probability.Kernel.WithDensity + +/-! +# The exponential mechanism + +The exponential mechanism privately selects an element of a set `R` of candidate outputs +according to a score (utility) function `u : 𝒟 → R → ℝ`. It is due to F. McSherry and +K. Talwar, *Mechanism design via differential privacy*, FOCS 2007, whose mechanism outputs `r` +with probability proportional to `exp (ε u(x, r)) μ(r)` for a base measure `μ` on a general range +and is `2εΔ`-differentially private when `u` has sensitivity `Δ`. We formalize the calibrated +version of C. Dwork and A. Roth, *The Algorithmic Foundations of Differential Privacy*, Found. +Trends Theor. Comput. Sci. 9(3–4), 2014, Section 3.4 (result numbers refer to this book): on a +finite set `R`, output `r ∈ R` on dataset `x` with probability proportional to +`exp (ε u(x, r) / (2Δ))`, which is `ε`-differentially private. + +The mechanism is a Markov kernel `𝒟 → R`, built as the measure with density +`exponentialMechanismProb u ε Δ x` with respect to the counting measure on the finite set `R`. +Privacy is measured with `IsEpsDeltaDP` from `Statlib.Privacy.DP.Basic` at `δ = 0` (pure +`ε`-differential privacy) for an arbitrary neighbouring relation `Adj` on datasets. The +sensitivity is a hypothesis: `Δ > 0` is any upper bound on `|u(x, r) - u(x', r)|` over +neighbouring `x, x'` and all `r` (Dwork–Roth use the exact sensitivity `Δu`, the smallest such +bound). The dataset space `𝒟` is an arbitrary measurable space, `R` is assumed nonempty, and +the scores `x ↦ u(x, r)` are assumed measurable, so that the construction is the mechanism of +Definition 3.4 (`exponentialMechanism_apply`, `isMarkovKernel_exponentialMechanism`). When the +density is not jointly measurable, or `R` is empty, the `Kernel.withDensity` construction +degenerates to the zero kernel (see `exponentialMechanism`). + +## Main definitions + +* `exponentialMechanismProb u ε Δ x r`: the probability + `exp (ε u(x, r) / (2Δ)) / ∑_{r'} exp (ε u(x, r') / (2Δ))` of outputting `r` on dataset `x`. +* `exponentialMechanism u ε Δ`: the exponential mechanism (Definition 3.4) as a kernel `𝒟 → R`. + +## Main statements (source result → Lean name) + +* The exponential mechanism is a Markov kernel when `R` is nonempty and the scores are + measurable in the dataset: `isMarkovKernel_exponentialMechanism`. +* Theorem 3.10 (privacy, for nonempty `R`): `isEpsDeltaDP_exponentialMechanism`, via the + pointwise likelihood-ratio bound `exponentialMechanismProb_le_exp_mul`. +* The key estimate in the proof of Theorem 3.11, + `P[u(x, M(x)) ≤ c] ≤ (|R| / |R_OPT|) exp (ε (c - OPT_u(x)) / (2Δ))`: + `exponentialMechanism_real_le`. +* Theorem 3.11 (utility, with `|R_OPT|`): `exponentialMechanism_utility`. +* Corollary 3.12 (utility, with `|R_OPT| ≥ 1`): `exponentialMechanism_utility_log_card`. + +## Not formalized + +* Infinite output ranges `R`: Dwork–Roth assume `R` finite only "for clarity" in the proof of + Theorem 3.10 and note it is not necessary, and McSherry–Talwar work with a base measure on a + general range. Here `R` is a finite type with the counting measure. +* General base measures `μ` in McSherry–Talwar's form. (With `μ` the counting measure on a + finite `R`, their mechanism `exp (ε u(x, r))` is `exponentialMechanism u (2 * ε * Δ) Δ` by + `exponentialMechanismProb_two_mul_mul`, and `isEpsDeltaDP_exponentialMechanism` gives their + `2εΔ` guarantee.) +-/ + +@[expose] public section + +open MeasureTheory ProbabilityTheory Real Finset +open scoped ENNReal + +namespace InformationTheory + +variable {𝒟 R : Type*} + +namespace ExponentialMechanism + +/-- A kernel `x ↦ μ.withDensity (f x)` with jointly measurable, normalizable densities +satisfying the pointwise bound `f x ≤ e^ε f x'` on neighbours is `ε`-differentially private. -/ +lemma isEpsDeltaDP_withDensity_const {𝒟 Y : Type*} [MeasurableSpace 𝒟] [MeasurableSpace Y] + {μ : Measure Y} [SFinite μ] {f : 𝒟 → Y → ℝ≥0∞} {Adj : 𝒟 → 𝒟 → Prop} {ε : ℝ} + (hf : Measurable (Function.uncurry f)) (hfin : ∀ x, ∫⁻ y, f x y ∂μ ≠ ∞) + (h : ∀ ⦃x x'⦄, Adj x x' → ∀ y, f x y ≤ ENNReal.ofReal (exp ε) * f x' y) : + IsEpsDeltaDP ((Kernel.const 𝒟 μ).withDensity f) Adj ε 0 := by + intro x x' hx E + simp only [Kernel.withDensity_apply _ hf, Kernel.const_apply, add_zero] + have hle : μ.withDensity (f x) E ≤ ENNReal.ofReal (exp ε) * μ.withDensity (f x') E := by + have hμ : μ.withDensity (f x) ≤ ENNReal.ofReal (exp ε) • μ.withDensity (f x') := by + rw [← withDensity_smul' _ _ ENNReal.ofReal_ne_top] + exact withDensity_mono (ae_of_all _ (h hx)) + simpa using hμ E + have hfin' : μ.withDensity (f x') E ≠ ∞ := by + refine ne_top_of_le_ne_top ?_ (measure_mono (Set.subset_univ E)) + rw [withDensity_apply _ MeasurableSet.univ, Measure.restrict_univ] + exact hfin x' + rw [measureReal_def, measureReal_def, ← ENNReal.toReal_ofReal (exp_pos ε).le, + ← ENNReal.toReal_mul] + exact ENNReal.toReal_mono (ENNReal.mul_ne_top ENNReal.ofReal_ne_top hfin') hle + +end ExponentialMechanism + +open ExponentialMechanism + +variable [Fintype R] {u : 𝒟 → R → ℝ} {ε Δ : ℝ} + +/-- The probability that the exponential mechanism with score `u`, privacy parameter `ε` and +sensitivity bound `Δ` outputs `r` on dataset `x`: +`exp (ε u(x, r) / (2Δ)) / ∑_{r'} exp (ε u(x, r') / (2Δ))`. -/ +noncomputable def exponentialMechanismProb (u : 𝒟 → R → ℝ) (ε Δ : ℝ) (x : 𝒟) (r : R) : ℝ := + exp (ε * u x r / (2 * Δ)) / ∑ r', exp (ε * u x r' / (2 * Δ)) + +namespace ExponentialMechanism + +lemma one_le_card_filter_eq_ciSup [Nonempty R] (u : 𝒟 → R → ℝ) (x : 𝒟) : + 1 ≤ #{r | u x r = ⨆ r', u x r'} := by + obtain ⟨r, hr⟩ := exists_eq_ciSup_of_finite (f := u x) + exact Finset.card_pos.2 ⟨r, Finset.mem_filter.2 ⟨Finset.mem_univ _, hr⟩⟩ + +lemma sum_exp_pos [Nonempty R] (u : 𝒟 → R → ℝ) (ε Δ : ℝ) (x : 𝒟) : + 0 < ∑ r, exp (ε * u x r / (2 * Δ)) := + Finset.sum_pos (fun _ _ ↦ exp_pos _) Finset.univ_nonempty + +end ExponentialMechanism + +lemma exponentialMechanismProb_nonneg (u : 𝒟 → R → ℝ) (ε Δ : ℝ) (x : 𝒟) (r : R) : + 0 ≤ exponentialMechanismProb u ε Δ x r := + div_nonneg (exp_pos _).le (Finset.sum_nonneg fun _ _ ↦ (exp_pos _).le) + +/-- The output probabilities of the exponential mechanism sum to one. -/ +lemma sum_exponentialMechanismProb [Nonempty R] (u : 𝒟 → R → ℝ) (ε Δ : ℝ) (x : 𝒟) : + ∑ r, exponentialMechanismProb u ε Δ x r = 1 := by + simp_rw [exponentialMechanismProb, ← Finset.sum_div] + exact div_self (sum_exp_pos u ε Δ x).ne' + +/-- McSherry–Talwar's uncalibrated mechanism with counting base measure, which outputs `r` with +probability proportional to `exp (ε u(x, r))`, is the calibrated mechanism with parameter +`2εΔ`. -/ +lemma exponentialMechanismProb_two_mul_mul (hΔ : Δ ≠ 0) (x : 𝒟) (r : R) : + exponentialMechanismProb u (2 * ε * Δ) Δ x r = + exp (ε * u x r) / ∑ r', exp (ε * u x r') := by + have h (r : R) : 2 * ε * Δ * u x r / (2 * Δ) = ε * u x r := by field_simp + simp only [exponentialMechanismProb, h] + +/-- **The likelihood ratio of the exponential mechanism is at most `e^ε`** (the computation in +the proof of Dwork–Roth, Theorem 3.10): if `|u(x, r') - u(x', r')| ≤ Δ` for all `r'`, then +`P[M(x) = r] ≤ e^ε P[M(x') = r]`. Half of the budget pays for the score of `r` and half for the +change of the normalizing constant. -/ +lemma exponentialMechanismProb_le_exp_mul (hε : 0 ≤ ε) (hΔ : 0 < Δ) {x x' : 𝒟} + (hu : ∀ r, |u x r - u x' r| ≤ Δ) (r : R) : + exponentialMechanismProb u ε Δ x r ≤ exp ε * exponentialMechanismProb u ε Δ x' r := by + have : Nonempty R := ⟨r⟩ + -- each unnormalized weight changes by a factor at most `e^{ε/2}` + have hw (x x' : 𝒟) (hu : ∀ r, |u x r - u x' r| ≤ Δ) (r : R) : + exp (ε * u x r / (2 * Δ)) ≤ exp (ε / 2) * exp (ε * u x' r / (2 * Δ)) := by + have key : ε * u x r ≤ ε * Δ + ε * u x' r := by + nlinarith [mul_le_mul_of_nonneg_left (abs_le.1 (hu r)).2 hε] + rw [← exp_add, exp_le_exp, show ε / 2 = ε * Δ / (2 * Δ) by field_simp, ← add_div] + gcongr + have hZ : ∑ r, exp (ε * u x' r / (2 * Δ)) ≤ exp (ε / 2) * ∑ r, exp (ε * u x r / (2 * Δ)) := by + rw [Finset.mul_sum] + exact Finset.sum_le_sum fun r _ ↦ hw x' x (fun r ↦ by rw [abs_sub_comm]; exact hu r) r + have hZx := sum_exp_pos u ε Δ x + have hZx' := sum_exp_pos u ε Δ x' + unfold exponentialMechanismProb + rw [mul_div_assoc', div_le_div_iff₀ hZx hZx'] + calc exp (ε * u x r / (2 * Δ)) * ∑ r, exp (ε * u x' r / (2 * Δ)) + ≤ (exp (ε / 2) * exp (ε * u x' r / (2 * Δ))) * + (exp (ε / 2) * ∑ r, exp (ε * u x r / (2 * Δ))) := by + gcongr + exact hw x x' hu r + _ = exp ε * exp (ε * u x' r / (2 * Δ)) * ∑ r, exp (ε * u x r / (2 * Δ)) := by + rw [show exp ε = exp (ε / 2) * exp (ε / 2) by rw [← exp_add, add_halves]] + ring + +variable [MeasurableSpace 𝒟] [MeasurableSpace R] [MeasurableSingletonClass R] + +/-- **The exponential mechanism** `M_E(x, u, R)` (Dwork–Roth, Definition 3.4, after +McSherry–Talwar): on dataset `x`, output `r ∈ R` with probability proportional to +`exp (ε u(x, r) / (2Δ))`. It is built as `Kernel.withDensity` of the constant counting-measure +kernel with density `(x, r) ↦ exponentialMechanismProb u ε Δ x r`. This describes the intended +distribution only under two conditions: +* the **density** is jointly measurable (`Kernel.withDensity` returns the **zero kernel** + otherwise). Measurability of each score `x ↦ u(x, r)` suffices (`exponentialMechanism_apply`), + but the test is on the density, not on the raw scores: a non-measurable offset added to all + scores cancels in the normalization; +* `R` is nonempty, so that the normalizing sum is positive and the masses sum to one (for empty + `R` the kernel is zero). + +Under both it is a Markov kernel (`isMarkovKernel_exponentialMechanism`, stated with the +sufficient condition that the scores are measurable). -/ +noncomputable def exponentialMechanism (u : 𝒟 → R → ℝ) (ε Δ : ℝ) : Kernel 𝒟 R := + (Kernel.const 𝒟 Measure.count).withDensity + fun x r ↦ ENNReal.ofReal (exponentialMechanismProb u ε Δ x r) + +namespace ExponentialMechanism + +lemma measurable_uncurry (hu : ∀ r, Measurable fun x ↦ u x r) : + Measurable (Function.uncurry fun x r ↦ ENNReal.ofReal (exponentialMechanismProb u ε Δ x r)) := + measurable_from_prod_countable_left fun r ↦ by + have hw (r : R) : Measurable fun x ↦ exp (ε * u x r / (2 * Δ)) := + (((hu r).const_mul ε).div_const _).exp + exact ((hw r).div (Finset.measurable_sum _ fun r _ ↦ hw r)).ennreal_ofReal + +end ExponentialMechanism + +lemma exponentialMechanism_apply (hu : ∀ r, Measurable fun x ↦ u x r) (x : 𝒟) : + exponentialMechanism u ε Δ x = + Measure.count.withDensity fun r ↦ ENNReal.ofReal (exponentialMechanismProb u ε Δ x r) := by + rw [exponentialMechanism, Kernel.withDensity_apply _ (measurable_uncurry hu), Kernel.const_apply] + +/-- The probability that the exponential mechanism outputs an element of `E` is +`∑_{r ∈ E} P[M(x) = r]`. -/ +lemma exponentialMechanism_real_apply (hu : ∀ r, Measurable fun x ↦ u x r) (x : 𝒟) (E : Set R) + [DecidablePred (· ∈ E)] : + (exponentialMechanism u ε Δ x).real E = ∑ r with r ∈ E, exponentialMechanismProb u ε Δ x r := by + have hE : E = ↑(Finset.univ.filter (· ∈ E)) := by ext; simp + rw [measureReal_def, exponentialMechanism_apply hu, withDensity_apply' _ E] + conv_lhs => rw [hE, lintegral_finset] + simp only [Measure.count_singleton, mul_one] + rw [ENNReal.toReal_sum fun r _ ↦ ENNReal.ofReal_ne_top] + exact Finset.sum_congr rfl fun r _ ↦ ENNReal.toReal_ofReal (exponentialMechanismProb_nonneg ..) + +/-- The exponential mechanism is a Markov kernel when `R` is nonempty and the scores are +measurable in the dataset. -/ +lemma isMarkovKernel_exponentialMechanism [Nonempty R] (hu : ∀ r, Measurable fun x ↦ u x r) : + IsMarkovKernel (exponentialMechanism u ε Δ) := by + refine ⟨fun x ↦ ⟨?_⟩⟩ + rw [exponentialMechanism_apply hu, withDensity_apply _ MeasurableSet.univ, Measure.restrict_univ, + lintegral_fintype, ← ENNReal.ofReal_one, ← sum_exponentialMechanismProb u ε Δ x, + ENNReal.ofReal_sum_of_nonneg fun r _ ↦ exponentialMechanismProb_nonneg u ε Δ x r] + simp + +/-- **Dwork–Roth, Theorem 3.10** (after McSherry–Talwar): the exponential mechanism on a +nonempty finite range `R` is `(ε, 0)`-differentially private, for `ε ≥ 0` and any bound `Δ > 0` +on the sensitivity of the score, `|u(x, r) - u(x', r)| ≤ Δ` for all neighbouring `x, x'` and all +`r ∈ R`. The scores are assumed measurable in the dataset, so that together with `[Nonempty R]` +the kernel is the randomized mechanism of Definition 3.4 (`isMarkovKernel_exponentialMechanism`); +for empty `R` the kernel is zero and the inequality would be vacuous. The proof does not use +`[Nonempty R]`; the general kernel inequality behind it is +`ExponentialMechanism.isEpsDeltaDP_withDensity_const`. -/ +@[nolint unusedArguments] +theorem isEpsDeltaDP_exponentialMechanism [Nonempty R] {Adj : 𝒟 → 𝒟 → Prop} + (hu : ∀ r, Measurable fun x ↦ u x r) (hε : 0 ≤ ε) (hΔ : 0 < Δ) + (hΔu : ∀ ⦃x x'⦄, Adj x x' → ∀ r, |u x r - u x' r| ≤ Δ) : + IsEpsDeltaDP (exponentialMechanism u ε Δ) Adj ε 0 := by + refine isEpsDeltaDP_withDensity_const (measurable_uncurry hu) (fun x ↦ ?_) + fun x x' hx r ↦ ?_ + · rw [lintegral_fintype] + exact ENNReal.sum_ne_top.2 fun y _ ↦ by simp + · rw [← ENNReal.ofReal_mul (exp_pos ε).le] + exact ENNReal.ofReal_le_ofReal (exponentialMechanismProb_le_exp_mul hε hΔ (hΔu hx) r) + +/-! ### Utility -/ + +/-- **The key estimate in the proof of Dwork–Roth, Theorem 3.11**: for every threshold `c`, +`P[u(x, M(x)) ≤ c] ≤ (|R| / |R_OPT|) exp (ε (c - OPT_u(x)) / (2Δ))`, where +`OPT_u(x) = max_r u(x, r)` and `R_OPT = {r : u(x, r) = OPT_u(x)}`. -/ +lemma exponentialMechanism_real_le [Nonempty R] (hu : ∀ r, Measurable fun x ↦ u x r) + (hε : 0 < ε) (hΔ : 0 < Δ) (x : 𝒟) (c : ℝ) : + (exponentialMechanism u ε Δ x).real {r | u x r ≤ c} ≤ + Fintype.card R / #{r | u x r = ⨆ r', u x r'} * + exp (ε * (c - ⨆ r', u x r') / (2 * Δ)) := by + classical + set OPT := ⨆ r', u x r' + set w : R → ℝ := fun r ↦ exp (ε * u x r / (2 * Δ)) with hw + have hZ : 0 < ∑ r, w r := ExponentialMechanism.sum_exp_pos u ε Δ x + have hK : (0 : ℝ) < #{r | u x r = OPT} := by + exact_mod_cast ExponentialMechanism.one_le_card_filter_eq_ciSup u x + -- the bad outputs have unnormalized mass at most `|R| exp (ε c / (2Δ))` + have h₁ : ∑ r with u x r ≤ c, w r ≤ Fintype.card R * exp (ε * c / (2 * Δ)) := + calc ∑ r with u x r ≤ c, w r ≤ ∑ r with u x r ≤ c, exp (ε * c / (2 * Δ)) := + Finset.sum_le_sum fun r hr ↦ by + simp only [hw, exp_le_exp] + gcongr + exact (Finset.mem_filter.1 hr).2 + _ ≤ Fintype.card R * exp (ε * c / (2 * Δ)) := by + rw [Finset.sum_const, nsmul_eq_mul] + gcongr + exact_mod_cast Finset.card_filter_le _ _ + -- the optimal outputs alone have unnormalized mass `|R_OPT| exp (ε OPT / (2Δ))` + have h₂ : #{r | u x r = OPT} * exp (ε * OPT / (2 * Δ)) ≤ ∑ r, w r := + calc (#{r | u x r = OPT} : ℝ) * exp (ε * OPT / (2 * Δ)) + = ∑ r with u x r = OPT, exp (ε * OPT / (2 * Δ)) := by rw [Finset.sum_const, nsmul_eq_mul] + _ = ∑ r with u x r = OPT, w r := Finset.sum_congr rfl fun r hr ↦ by + simp only [hw, (Finset.mem_filter.1 hr).2] + _ ≤ ∑ r, w r := Finset.sum_le_sum_of_subset_of_nonneg (Finset.filter_subset _ _) + fun r _ _ ↦ (exp_pos _).le + rw [exponentialMechanism_real_apply hu] + simp only [Set.mem_ofPred_eq, exponentialMechanismProb, ← Finset.sum_div] + calc (∑ r with u x r ≤ c, w r) / ∑ r, w r + ≤ Fintype.card R * exp (ε * c / (2 * Δ)) / (#{r | u x r = OPT} * exp (ε * OPT / (2 * Δ))) := + div_le_div₀ (by positivity) h₁ (by positivity) h₂ + _ = Fintype.card R / #{r | u x r = OPT} * exp (ε * (c - OPT) / (2 * Δ)) := by + rw [mul_sub, sub_div, exp_sub] + field_simp + +/-- **Dwork–Roth, Theorem 3.11** (utility of the exponential mechanism): for every dataset `x` +and every `t`, +`P[u(x, M(x)) ≤ OPT_u(x) - (2Δ/ε) (ln (|R| / |R_OPT|) + t)] ≤ e^{-t}`, +where `OPT_u(x) = max_r u(x, r)` and `R_OPT = {r : u(x, r) = OPT_u(x)}`. Here `R` is a nonempty +finite set, `ε > 0` and `Δ > 0`. -/ +theorem exponentialMechanism_utility [Nonempty R] (hu : ∀ r, Measurable fun x ↦ u x r) + (hε : 0 < ε) (hΔ : 0 < Δ) (x : 𝒟) (t : ℝ) : + (exponentialMechanism u ε Δ x).real {r | u x r ≤ (⨆ r', u x r') - + 2 * Δ / ε * (log (Fintype.card R / #{r | u x r = ⨆ r', u x r'}) + t)} ≤ exp (-t) := by + refine (exponentialMechanism_real_le hu hε hΔ x _).trans_eq ?_ + have hK : (0 : ℝ) < #{r | u x r = ⨆ r', u x r'} := by + exact_mod_cast ExponentialMechanism.one_le_card_filter_eq_ciSup u x + set K : ℝ := ((#{r | u x r = ⨆ r', u x r'} : ℕ) : ℝ) + have hN : (0 : ℝ) < Fintype.card R := by exact_mod_cast Fintype.card_pos + rw [show ε * ((⨆ r', u x r') - 2 * Δ / ε * (log (Fintype.card R / K) + t) - ⨆ r', u x r') / + (2 * Δ) = -log (Fintype.card R / K) + -t by field_simp; ring, + exp_add, exp_neg (log _), exp_log (by positivity)] + field_simp + +/-- **Dwork–Roth, Corollary 3.12** (utility of the exponential mechanism): for every dataset `x` +and every `t`, `P[u(x, M(x)) ≤ OPT_u(x) - (2Δ/ε) (ln |R| + t)] ≤ e^{-t}`, where +`OPT_u(x) = max_r u(x, r)`. Here `R` is a nonempty finite set, `ε > 0` and `Δ > 0`. -/ +theorem exponentialMechanism_utility_log_card [Nonempty R] (hu : ∀ r, Measurable fun x ↦ u x r) + (hε : 0 < ε) (hΔ : 0 < Δ) (x : 𝒟) (t : ℝ) : + (exponentialMechanism u ε Δ x).real + {r | u x r ≤ (⨆ r', u x r') - 2 * Δ / ε * (log (Fintype.card R) + t)} ≤ exp (-t) := by + refine (exponentialMechanism_real_le hu hε hΔ x _).trans ?_ + have hK : (1 : ℝ) ≤ #{r | u x r = ⨆ r', u x r'} := by + exact_mod_cast ExponentialMechanism.one_le_card_filter_eq_ciSup u x + set K : ℝ := ((#{r | u x r = ⨆ r', u x r'} : ℕ) : ℝ) + have hN : (0 : ℝ) < Fintype.card R := by exact_mod_cast Fintype.card_pos + rw [show ε * ((⨆ r', u x r') - 2 * Δ / ε * (log (Fintype.card R) + t) - ⨆ r', u x r') / + (2 * Δ) = -log (Fintype.card R) + -t by field_simp; ring, + exp_add, exp_neg (log _), exp_log hN, ← mul_assoc, div_mul_eq_mul_div, mul_inv_cancel₀ hN.ne', + one_div_mul_eq_div] + exact div_le_self (exp_pos _).le hK + +end InformationTheory diff --git a/Statlib/Privacy/DP/Geometric.lean b/Statlib/Privacy/DP/Geometric.lean new file mode 100644 index 0000000..e67bc95 --- /dev/null +++ b/Statlib/Privacy/DP/Geometric.lean @@ -0,0 +1,269 @@ +/- +Copyright (c) 2026 Kousha Kalantari. All rights reserved. +Released under Apache 2.0 license as described in the file LICENSE. +Authors: Kousha Kalantari +-/ +module + +public import Statlib.Privacy.DP.Basic +public import Mathlib.Analysis.SpecialFunctions.Log.Basic +public import Mathlib.Analysis.SpecificLimits.Basic +public import Mathlib.MeasureTheory.Measure.Count +public import Mathlib.Probability.Kernel.WithDensity +public import Mathlib.Topology.Algebra.InfiniteSum.NatInt + +/-! +# The geometric mechanism + +The geometric mechanism (the discrete analogue of the Laplace mechanism) of A. Ghosh, +T. Roughgarden and M. Sundararajan, *Universally utility-maximizing privacy mechanisms*, +STOC 2009 / SIAM J. Comput. 41(6), 2012 ([arXiv:0811.2841](https://arxiv.org/abs/0811.2841); +numbering follows arXiv v3), releases an integer-valued query `f(x) ∈ ℤ` with additive two-sided +geometric noise: for `α ∈ [0, 1)` the noise `Z` has law `P[Z = z] = (1 - α) / (1 + α) · α^{|z|}` +on `ℤ`, so the mechanism outputs `z ∈ ℤ` with probability proportional to `α^{|z - f(x)|}` +(Example 2.1). Ghosh–Roughgarden–Sundararajan measure privacy multiplicatively: a mechanism is +`α`-differentially private if all likelihood ratios on neighbouring databases lie in +`[α, 1/α]`, which is `(-ln α, 0)`-differential privacy in the usual additive convention. Example +2.1 states that for count queries (sensitivity `1`) the `α`-geometric mechanism is +`α`-differentially private in their sense. More generally, for an integer-valued query of +sensitivity at most `Δ`, the choice `α = e^{-ε/Δ}` gives `ε`-differential privacy. + +The mechanism is a Markov kernel `𝒟 → ℤ`, built as `Kernel.withDensity` of the constant kernel +`x ↦ count` (counting measure on `ℤ`); it is a Markov kernel as soon as `f` is measurable +(`isMarkovKernel_geometricMechanism`), and then it is the mechanism of Example 2.1 +(`geometricMechanism_apply`); the privacy theorems assume `f` measurable. +Privacy is measured with `IsEpsDeltaDP` from `Statlib.Privacy.DP.Basic` at `δ = 0` (pure +`ε`-differential privacy, over all events) for an arbitrary neighbouring relation `Adj` on +datasets. + +## Main definitions + +* `twoSidedGeometricPMFReal α μ z = (1 - α) / (1 + α) · α^{|z - μ|}`: the two-sided geometric + (discrete Laplace) probability mass function on `ℤ`, centred at `μ`. +* `twoSidedGeometricMeasure α μ`: the corresponding measure on `ℤ`. +* `geometricMechanism f α`: the `α`-geometric mechanism `x ↦ f(x) + Z` (Example 2.1). + +## Main statements (source result → Lean name) + +* The two-sided geometric masses sum to one: `hasSum_twoSidedGeometricPMFReal`; the measure is a + probability measure: `isProbabilityMeasure_twoSidedGeometricMeasure`; the mechanism is a + Markov kernel: `isMarkovKernel_geometricMechanism`. +* Example 2.1 (privacy of the `α`-geometric mechanism for queries of sensitivity `1`, `0 < α < 1`, + privacy level `-ln α`): `isEpsDeltaDP_geometricMechanism_neg_log`. +* Its generalization to sensitivity `Δ` with `α = e^{-ε/Δ}`: `isEpsDeltaDP_geometricMechanism`. + +## Remark on the range of `α` + +The paper states its results for every privacy level `α ∈ [0, 1]` and sets aside `α = 0, 1` as +degenerate cases at the start of Section 5 (arXiv v3). At `α = 1` the displayed untruncated mass +formula `(1 - α) / (1 + α) · α^{|z|}` of Example 2.1 is identically `0`, so this formula gives no +`1`-geometric mechanism to formalize (perfect privacy, `α = 1`, is achieved by data-independent +mechanisms, which are not geometric mechanisms of this form). At `α = 0` the mechanism releases +`f(x)` exactly; the additive privacy level `-log α` tends to `+∞` as `α ↓ 0`, and exact release +has no finite privacy bound when neighbouring answers differ. (In Lean, `Real.log 0 = 0`, so the +formula `-log α` must not be evaluated at `α = 0`.) This is a degenerate boundary convention, not +an error in the paper's privacy or optimality results. We therefore assume `0 ≤ α < 1` for the +distribution and `0 < α < 1` for the privacy statement. +-/ + +@[expose] public section + +open MeasureTheory ProbabilityTheory Real +open scoped ENNReal + +namespace InformationTheory + +variable {𝒟 : Type*} + +namespace Geometric + +/-- A kernel `x ↦ μ.withDensity (f x)` with jointly measurable, normalizable densities +satisfying the pointwise bound `f x ≤ e^ε f x'` on neighbours is `ε`-differentially private. -/ +lemma isEpsDeltaDP_withDensity_const {𝒟 Y : Type*} [MeasurableSpace 𝒟] [MeasurableSpace Y] + {μ : Measure Y} [SFinite μ] {f : 𝒟 → Y → ℝ≥0∞} {Adj : 𝒟 → 𝒟 → Prop} {ε : ℝ} + (hf : Measurable (Function.uncurry f)) (hfin : ∀ x, ∫⁻ y, f x y ∂μ ≠ ∞) + (h : ∀ ⦃x x'⦄, Adj x x' → ∀ y, f x y ≤ ENNReal.ofReal (exp ε) * f x' y) : + IsEpsDeltaDP ((Kernel.const 𝒟 μ).withDensity f) Adj ε 0 := by + intro x x' hx E + simp only [Kernel.withDensity_apply _ hf, Kernel.const_apply, add_zero] + have hle : μ.withDensity (f x) E ≤ ENNReal.ofReal (exp ε) * μ.withDensity (f x') E := by + have hμ : μ.withDensity (f x) ≤ ENNReal.ofReal (exp ε) • μ.withDensity (f x') := by + rw [← withDensity_smul' _ _ ENNReal.ofReal_ne_top] + exact withDensity_mono (ae_of_all _ (h hx)) + simpa using hμ E + have hfin' : μ.withDensity (f x') E ≠ ∞ := by + refine ne_top_of_le_ne_top ?_ (measure_mono (Set.subset_univ E)) + rw [withDensity_apply _ MeasurableSet.univ, Measure.restrict_univ] + exact hfin x' + rw [measureReal_def, measureReal_def, ← ENNReal.toReal_ofReal (exp_pos ε).le, + ← ENNReal.toReal_mul] + exact ENNReal.toReal_mono (ENNReal.mul_ne_top ENNReal.ofReal_ne_top hfin') hle + +end Geometric + +open Geometric + +/-! ### The two-sided geometric distribution -/ + +section PMF + +variable {α : ℝ} + +/-- The two-sided geometric (discrete Laplace) probability mass function on `ℤ` with parameter +`α` centred at `μ`: `z ↦ (1 - α) / (1 + α) · α^{|z - μ|}`. It is a probability mass function for +`0 ≤ α < 1` (`hasSum_twoSidedGeometricPMFReal`). -/ +noncomputable def twoSidedGeometricPMFReal (α : ℝ) (μ z : ℤ) : ℝ := + (1 - α) / (1 + α) * α ^ (z - μ).natAbs + +lemma twoSidedGeometricPMFReal_nonneg (hα₀ : 0 ≤ α) (hα₁ : α < 1) (μ z : ℤ) : + 0 ≤ twoSidedGeometricPMFReal α μ z := by + unfold twoSidedGeometricPMFReal + have : 0 ≤ 1 - α := by linarith + positivity + +/-- `∑_{z ∈ ℤ} α^{|z|} = (1 + α) / (1 - α)` for `0 ≤ α < 1`. -/ +lemma Geometric.hasSum_pow_natAbs (hα₀ : 0 ≤ α) (hα₁ : α < 1) : + HasSum (fun z : ℤ ↦ α ^ z.natAbs) ((1 + α) / (1 - α)) := by + have h₁ : HasSum (fun n : ℕ ↦ α ^ (n : ℤ).natAbs) (1 - α)⁻¹ := by + simp only [Int.natAbs_natCast] + exact hasSum_geometric_of_lt_one hα₀ hα₁ + have h₂ : HasSum (fun n : ℕ ↦ α ^ (-(n + 1 : ℤ)).natAbs) (α * (1 - α)⁻¹) := by + have : (fun n : ℕ ↦ α ^ (-(n + 1 : ℤ)).natAbs) = fun n : ℕ ↦ α * α ^ n := by + ext n + rw [Int.natAbs_neg, show ((n : ℤ) + 1) = ((n + 1 : ℕ) : ℤ) by push_cast; ring, + Int.natAbs_natCast, pow_succ'] + rw [this] + exact (hasSum_geometric_of_lt_one hα₀ hα₁).mul_left α + have h := HasSum.of_nat_of_neg_add_one (f := fun z : ℤ ↦ α ^ z.natAbs) h₁ h₂ + rwa [show (1 - α)⁻¹ + α * (1 - α)⁻¹ = (1 + α) / (1 - α) by rw [div_eq_mul_inv]; ring] at h + +/-- **The two-sided geometric masses sum to one** (for `0 ≤ α < 1`). -/ +lemma hasSum_twoSidedGeometricPMFReal (hα₀ : 0 ≤ α) (hα₁ : α < 1) (μ : ℤ) : + HasSum (twoSidedGeometricPMFReal α μ) 1 := by + have h := ((Equiv.subRight μ).hasSum_iff (f := fun z : ℤ ↦ α ^ z.natAbs)).2 + (Geometric.hasSum_pow_natAbs hα₀ hα₁) + have h1 : 1 - α ≠ 0 := by linarith + have h2 : 1 + α ≠ 0 := by linarith + convert h.mul_left ((1 - α) / (1 + α)) using 1 + · ext z + simp [twoSidedGeometricPMFReal] + · field_simp + +/-- The two-sided geometric distribution on `ℤ` with parameter `α` centred at `μ`: the measure +with density `twoSidedGeometricPMFReal α μ` with respect to the counting measure. It is a +probability measure for `0 ≤ α < 1` (`isProbabilityMeasure_twoSidedGeometricMeasure`). -/ +noncomputable def twoSidedGeometricMeasure (α : ℝ) (μ : ℤ) : Measure ℤ := + Measure.count.withDensity fun z ↦ ENNReal.ofReal (twoSidedGeometricPMFReal α μ z) + +lemma lintegral_twoSidedGeometricPMFReal (hα₀ : 0 ≤ α) (hα₁ : α < 1) (μ : ℤ) : + ∫⁻ z, ENNReal.ofReal (twoSidedGeometricPMFReal α μ z) ∂Measure.count = 1 := by + rw [lintegral_count, ← ENNReal.ofReal_tsum_of_nonneg + (twoSidedGeometricPMFReal_nonneg hα₀ hα₁ μ) + (hasSum_twoSidedGeometricPMFReal hα₀ hα₁ μ).summable, + (hasSum_twoSidedGeometricPMFReal hα₀ hα₁ μ).tsum_eq, ENNReal.ofReal_one] + +lemma isProbabilityMeasure_twoSidedGeometricMeasure (hα₀ : 0 ≤ α) (hα₁ : α < 1) (μ : ℤ) : + IsProbabilityMeasure (twoSidedGeometricMeasure α μ) := + ⟨by rw [twoSidedGeometricMeasure, withDensity_apply _ MeasurableSet.univ, Measure.restrict_univ, + lintegral_twoSidedGeometricPMFReal hα₀ hα₁]⟩ + +lemma twoSidedGeometricMeasure_real_singleton (hα₀ : 0 ≤ α) (hα₁ : α < 1) (μ z : ℤ) : + (twoSidedGeometricMeasure α μ).real {z} = twoSidedGeometricPMFReal α μ z := by + rw [measureReal_def, twoSidedGeometricMeasure, withDensity_apply _ (MeasurableSet.singleton z), + Measure.restrict_singleton, Measure.count_singleton, one_smul, lintegral_dirac, + ENNReal.toReal_ofReal (twoSidedGeometricPMFReal_nonneg hα₀ hα₁ μ z)] + +/-- **Likelihood ratio of shifted two-sided geometric masses**: if `α = e^{-ε/Δ}` with `ε ≥ 0`, +`Δ > 0` and `|μ - μ'| ≤ Δ`, then `P_μ(z) ≤ e^ε P_{μ'}(z)` for every `z ∈ ℤ`. -/ +lemma twoSidedGeometricPMFReal_le_exp_mul {ε : ℝ} (hε : 0 ≤ ε) {Δ : ℕ} (hΔ : 0 < Δ) {μ μ' : ℤ} + (h : (μ - μ').natAbs ≤ Δ) (z : ℤ) : + twoSidedGeometricPMFReal (exp (-ε / Δ)) μ z ≤ + exp ε * twoSidedGeometricPMFReal (exp (-ε / Δ)) μ' z := by + set α := exp (-ε / Δ) with hα + have hΔ' : (0 : ℝ) < Δ := by exact_mod_cast hΔ + have hα₀ : 0 ≤ α := (exp_pos _).le + have hα₁ : α ≤ 1 := exp_le_one_iff.2 (div_nonpos_of_nonpos_of_nonneg (neg_nonpos.2 hε) hΔ'.le) + have hαΔ : α ^ Δ = exp (-ε) := by + rw [hα, ← exp_nat_mul] + congr 1 + field_simp + have hab : (z - μ').natAbs ≤ (z - μ).natAbs + Δ := by + have := Int.natAbs_add_le (z - μ) (μ - μ') + rw [show z - μ + (μ - μ') = z - μ' by ring] at this + omega + have key : α ^ (z - μ).natAbs ≤ exp ε * α ^ (z - μ').natAbs := + calc α ^ (z - μ).natAbs = exp ε * α ^ ((z - μ).natAbs + Δ) := by + rw [pow_add, hαΔ, mul_left_comm, ← exp_add, add_neg_cancel, exp_zero, mul_one] + _ ≤ exp ε * α ^ (z - μ').natAbs := + mul_le_mul_of_nonneg_left (pow_le_pow_of_le_one hα₀ hα₁ hab) (exp_pos ε).le + have hc : 0 ≤ (1 - α) / (1 + α) := div_nonneg (by linarith) (by linarith) + unfold twoSidedGeometricPMFReal + calc (1 - α) / (1 + α) * α ^ (z - μ).natAbs + ≤ (1 - α) / (1 + α) * (exp ε * α ^ (z - μ').natAbs) := mul_le_mul_of_nonneg_left key hc + _ = exp ε * ((1 - α) / (1 + α) * α ^ (z - μ').natAbs) := by ring + +end PMF + +/-! ### The geometric mechanism -/ + +section Mechanism + +variable [MeasurableSpace 𝒟] {f : 𝒟 → ℤ} {α ε : ℝ} + +/-- **The `α`-geometric mechanism** (Ghosh–Roughgarden–Sundararajan, Example 2.1): on dataset +`x` it outputs `f(x) + Z` with `Z` two-sided geometric with parameter `α`, i.e. `z ∈ ℤ` with +probability `(1 - α) / (1 + α) · α^{|z - f(x)|}`. It is built as the kernel with density +`(x, z) ↦ twoSidedGeometricPMFReal α (f x) z` with respect to the constant kernel `x ↦ count`, +which is the intended kernel when `f` is measurable (`geometricMechanism_apply`) and the zero +kernel otherwise. -/ +noncomputable def geometricMechanism (f : 𝒟 → ℤ) (α : ℝ) : Kernel 𝒟 ℤ := + (Kernel.const 𝒟 Measure.count).withDensity + fun x z ↦ ENNReal.ofReal (twoSidedGeometricPMFReal α (f x) z) + +lemma measurable_uncurry_twoSidedGeometricPMFReal (hf : Measurable f) (α : ℝ) : + Measurable (Function.uncurry fun x z ↦ ENNReal.ofReal (twoSidedGeometricPMFReal α (f x) z)) := + measurable_from_prod_countable_left fun z ↦ (measurable_of_countable + fun μ ↦ ENNReal.ofReal (twoSidedGeometricPMFReal α μ z)).comp hf + +lemma geometricMechanism_apply (hf : Measurable f) (α : ℝ) (x : 𝒟) : + geometricMechanism f α x = twoSidedGeometricMeasure α (f x) := by + rw [geometricMechanism, + Kernel.withDensity_apply _ (measurable_uncurry_twoSidedGeometricPMFReal hf α), + Kernel.const_apply, twoSidedGeometricMeasure] + +/-- The geometric mechanism with a measurable query and `0 ≤ α < 1` is a Markov kernel. -/ +lemma isMarkovKernel_geometricMechanism (hf : Measurable f) (hα₀ : 0 ≤ α) (hα₁ : α < 1) : + IsMarkovKernel (geometricMechanism f α) := + ⟨fun x ↦ by + rw [geometricMechanism_apply hf] + exact isProbabilityMeasure_twoSidedGeometricMeasure hα₀ hα₁ _⟩ + +/-- **The geometric mechanism is `ε`-differentially private**: for a measurable integer-valued +query `f` whose sensitivity is at most `Δ > 0` (`|f(x) - f(x')| ≤ Δ` for all neighbouring +`x, x'`) and `ε > 0`, the geometric mechanism with `α = e^{-ε/Δ}` is +`(ε, 0)`-differentially private. -/ +theorem isEpsDeltaDP_geometricMechanism {Adj : 𝒟 → 𝒟 → Prop} {Δ : ℕ} (hf : Measurable f) + (hε : 0 < ε) (hΔ : 0 < Δ) (hΔf : ∀ ⦃x x'⦄, Adj x x' → (f x - f x').natAbs ≤ Δ) : + IsEpsDeltaDP (geometricMechanism f (exp (-ε / Δ))) Adj ε 0 := by + have hΔ' : (0 : ℝ) < Δ := by exact_mod_cast hΔ + have hα₁ : exp (-ε / Δ) < 1 := exp_lt_one_iff.2 (div_neg_of_neg_of_pos (neg_lt_zero.2 hε) hΔ') + refine isEpsDeltaDP_withDensity_const (measurable_uncurry_twoSidedGeometricPMFReal hf _) + (fun x ↦ ?_) fun x x' hx z ↦ ?_ + · rw [lintegral_twoSidedGeometricPMFReal (exp_pos _).le hα₁] + exact ENNReal.one_ne_top + · rw [← ENNReal.ofReal_mul (exp_pos ε).le] + exact ENNReal.ofReal_le_ofReal (twoSidedGeometricPMFReal_le_exp_mul hε.le hΔ (hΔf hx) z) + +/-- **Ghosh–Roughgarden–Sundararajan, Example 2.1**: for a measurable query `f` of sensitivity at +most `1` (for instance a count query) and `0 < α < 1`, the `α`-geometric mechanism is +`α`-differentially private in the multiplicative convention of the paper, i.e. +`(-ln α, 0)`-differentially private. -/ +theorem isEpsDeltaDP_geometricMechanism_neg_log {Adj : 𝒟 → 𝒟 → Prop} (hf : Measurable f) + (hα₀ : 0 < α) (hα₁ : α < 1) (hΔf : ∀ ⦃x x'⦄, Adj x x' → (f x - f x').natAbs ≤ 1) : + IsEpsDeltaDP (geometricMechanism f α) Adj (-log α) 0 := by + have h := isEpsDeltaDP_geometricMechanism hf (neg_pos.2 (log_neg hα₀ hα₁)) one_pos hΔf + rwa [Nat.cast_one, div_one, neg_neg, exp_log hα₀] at h + +end Mechanism + +end InformationTheory diff --git a/Statlib/Privacy/DP/Laplace.lean b/Statlib/Privacy/DP/Laplace.lean new file mode 100644 index 0000000..7e31477 --- /dev/null +++ b/Statlib/Privacy/DP/Laplace.lean @@ -0,0 +1,459 @@ +/- +Copyright (c) 2026 Kousha Kalantari. All rights reserved. +Released under Apache 2.0 license as described in the file LICENSE. +Authors: Kousha Kalantari +-/ +module + +public import Statlib.Privacy.DP.Basic +public import Mathlib.Analysis.SpecialFunctions.ImproperIntegrals +public import Mathlib.MeasureTheory.Group.Integral +public import Mathlib.MeasureTheory.Integral.Pi +public import Mathlib.MeasureTheory.Measure.Lebesgue.Integral +public import Mathlib.Probability.Kernel.WithDensity + +/-! +# The Laplace mechanism + +The Laplace mechanism of C. Dwork, F. McSherry, K. Nissim and A. Smith, *Calibrating noise to +sensitivity in private data analysis*, TCC 2006, releases a real-valued query `f(x)` with additive +Laplace noise whose scale is calibrated to the sensitivity of `f`. We follow C. Dwork and +A. Roth, *The Algorithmic Foundations of Differential Privacy*, Found. Trends Theor. Comput. Sci. +9(3–4), 2014, Section 3.3 (result numbers refer to this book). + +The Laplace distribution `Lap(μ, b)` with location `μ` and scale `b > 0` has density +`(2b)⁻¹ exp (-|y - μ| / b)` with respect to Lebesgue measure. The Laplace mechanism with scale `b` +is the kernel `x ↦ Lap(f(x), b)`, i.e. `f(x) + Y` with `Y ∼ Lap(b)`. It is built as +`Kernel.withDensity` of the constant kernel `x ↦ volume`; it is a Markov kernel as soon as `f` is +measurable (`isMarkovKernel_laplaceMechanism`), and then its output law at `x` is +`Lap(f(x), b)` (`laplaceMechanism_apply`). The privacy and accuracy theorems assume `f` +measurable. + +Privacy is measured with `IsEpsDeltaDP` from `Statlib.Privacy.DP.Basic` at `δ = 0` (pure +`ε`-differential privacy) for an arbitrary neighbouring relation `Adj` on datasets. The +sensitivity is a hypothesis: `Δ > 0` is any upper bound on `|f(x) - f(x')|` (respectively on the +`ℓ₁` distance `∑_i |f(x)_i - f(x')_i|` for vector-valued queries) over neighbouring `x, x'` +(Dwork–Roth use the exact `ℓ₁`-sensitivity `Δf`, Definition 3.1, the smallest such bound). +Vector-valued queries `f : 𝒟 → (ι → ℝ)` (Dwork–Roth's `ℝ^k`, with `k = |ι|`) get i.i.d. noise in +each coordinate: the output law is the measure with the product Laplace density on `ι → ℝ`, which +is the product measure `∏_i Lap(f(x)_i, b)` (`laplaceMechanismPi_apply`). + +## Main definitions + +* `laplacePDFReal μ b`, `laplacePDF μ b`: the Laplace density (Definition 3.2), real- and + `ℝ≥0∞`-valued. +* `laplaceMeasure μ b`: the Laplace distribution `Lap(μ, b)` on `ℝ`. +* `laplaceMechanism f b`: the Laplace mechanism `x ↦ f(x) + Lap(b)` for a real-valued query + (Definition 3.3 with `k = 1`, the scale `b = Δf / ε` left as a parameter). +* `laplacePiPDFReal μ b`: the product Laplace density on `ι → ℝ`. +* `laplaceMechanismPi f b`: the Laplace mechanism `x ↦ f(x) + (Y_i)_{i ∈ ι}`, `Y_i` i.i.d. + `Lap(b)`, for a vector-valued query (Definition 3.3); its output law is + `Measure.pi fun i ↦ laplaceMeasure (f x i) b` (`laplaceMechanismPi_apply`). + +## Main statements (source result → Lean name) + +* The Laplace density integrates to one: `integral_laplacePDFReal_eq_one`, + `lintegral_laplacePDF_eq_one`; `laplaceMeasure μ b` is a probability measure: + `isProbabilityMeasure_laplaceMeasure`; the mechanisms are Markov kernels: + `isMarkovKernel_laplaceMechanism`, `isMarkovKernel_laplaceMechanismPi`. +* Theorem 3.6 (privacy): `isEpsDeltaDP_laplaceMechanismPi` (vector-valued queries, `ℓ₁` + sensitivity) and its scalar case `isEpsDeltaDP_laplaceMechanism`; the latter also in the form + `isEpsDeltaDP_laplaceMechanism_of_le` for an arbitrary scale `b ≥ Δ / ε`. +* Fact 3.7 (tail of the Laplace distribution): `laplaceMeasure_real_le_abs_sub`. +* Theorem 3.8 (accuracy): `laplaceMechanismPi_real_le_norm_sub` (vector-valued queries, sup + norm, `k = |ι| ≥ 1`), and its scalar case `laplaceMechanism_real_le_abs_sub`. +-/ + +@[expose] public section + +open MeasureTheory ProbabilityTheory Real Set +open scoped ENNReal + +namespace InformationTheory + +variable {𝒟 : Type*} + +namespace Laplace + +/-- A kernel `x ↦ μ.withDensity (f x)` with jointly measurable, normalizable densities +satisfying the pointwise bound `f x ≤ e^ε f x'` on neighbours is `ε`-differentially private. -/ +lemma isEpsDeltaDP_withDensity_const {𝒟 Y : Type*} [MeasurableSpace 𝒟] [MeasurableSpace Y] + {μ : Measure Y} [SFinite μ] {f : 𝒟 → Y → ℝ≥0∞} {Adj : 𝒟 → 𝒟 → Prop} {ε : ℝ} + (hf : Measurable (Function.uncurry f)) (hfin : ∀ x, ∫⁻ y, f x y ∂μ ≠ ∞) + (h : ∀ ⦃x x'⦄, Adj x x' → ∀ y, f x y ≤ ENNReal.ofReal (exp ε) * f x' y) : + IsEpsDeltaDP ((Kernel.const 𝒟 μ).withDensity f) Adj ε 0 := by + intro x x' hx E + simp only [Kernel.withDensity_apply _ hf, Kernel.const_apply, add_zero] + have hle : μ.withDensity (f x) E ≤ ENNReal.ofReal (exp ε) * μ.withDensity (f x') E := by + have hμ : μ.withDensity (f x) ≤ ENNReal.ofReal (exp ε) • μ.withDensity (f x') := by + rw [← withDensity_smul' _ _ ENNReal.ofReal_ne_top] + exact withDensity_mono (ae_of_all _ (h hx)) + simpa using hμ E + have hfin' : μ.withDensity (f x') E ≠ ∞ := by + refine ne_top_of_le_ne_top ?_ (measure_mono (Set.subset_univ E)) + rw [withDensity_apply _ MeasurableSet.univ, Measure.restrict_univ] + exact hfin x' + rw [measureReal_def, measureReal_def, ← ENNReal.toReal_ofReal (exp_pos ε).le, + ← ENNReal.toReal_mul] + exact ENNReal.toReal_mono (ENNReal.mul_ne_top ENNReal.ofReal_ne_top hfin') hle + +end Laplace + +open Laplace + +/-! ### The Laplace distribution -/ + +section LaplacePDF + +variable {μ b : ℝ} + +/-- The density `(2b)⁻¹ exp (-|y - μ| / b)` of the Laplace distribution with location `μ` and +scale `b` (Dwork–Roth, Definition 3.2, there with `μ = 0`). It is a probability density for +`b > 0`. -/ +noncomputable def laplacePDFReal (μ b y : ℝ) : ℝ := + (2 * b)⁻¹ * exp (-|y - μ| / b) + +/-- The Laplace density as an `ℝ≥0∞`-valued function. -/ +noncomputable def laplacePDF (μ b y : ℝ) : ℝ≥0∞ := + ENNReal.ofReal (laplacePDFReal μ b y) + +lemma laplacePDFReal_pos (hb : 0 < b) (y : ℝ) : 0 < laplacePDFReal μ b y := by + unfold laplacePDFReal + positivity + +@[fun_prop] +lemma measurable_laplacePDFReal (μ b : ℝ) : Measurable (laplacePDFReal μ b) := by + unfold laplacePDFReal + fun_prop + +@[fun_prop] +lemma measurable_laplacePDF (μ b : ℝ) : Measurable (laplacePDF μ b) := by + unfold laplacePDF + fun_prop + +/-- The Laplace density integrates to one. -/ +lemma integral_laplacePDFReal_eq_one (μ : ℝ) (hb : 0 < b) : ∫ y, laplacePDFReal μ b y = 1 := by + have hI : ∫ y in Ioi (0 : ℝ), exp (-b⁻¹ * y) = b := by + rw [integral_exp_mul_Ioi (neg_lt_zero.2 (inv_pos.2 hb)), mul_zero, exp_zero] + field_simp + unfold laplacePDFReal + rw [integral_const_mul, integral_sub_right_eq_self (fun y ↦ exp (-|y| / b)) μ, + integral_comp_abs (f := fun y ↦ exp (-y / b))] + simp_rw [neg_div, div_eq_inv_mul, ← neg_mul] + rw [hI] + field_simp + +lemma integrable_laplacePDFReal (μ : ℝ) (hb : 0 < b) : Integrable (laplacePDFReal μ b) := + Integrable.of_integral_ne_zero (by simp [integral_laplacePDFReal_eq_one μ hb]) + +/-- The Laplace density integrates to one. -/ +lemma lintegral_laplacePDF_eq_one (μ : ℝ) (hb : 0 < b) : ∫⁻ y, laplacePDF μ b y = 1 := by + unfold laplacePDF + rw [← ENNReal.toReal_eq_one_iff, ← integral_eq_lintegral_of_nonneg_ae + (ae_of_all _ fun y ↦ (laplacePDFReal_pos hb y).le) (by fun_prop), + integral_laplacePDFReal_eq_one μ hb] + +/-- The Laplace distribution `Lap(μ, b)` with location `μ` and scale `b`: the measure on `ℝ` +with density `laplacePDF μ b` with respect to Lebesgue measure. It is a probability measure for +`b > 0` (`isProbabilityMeasure_laplaceMeasure`). -/ +noncomputable def laplaceMeasure (μ b : ℝ) : Measure ℝ := + volume.withDensity (laplacePDF μ b) + +lemma isProbabilityMeasure_laplaceMeasure (μ : ℝ) (hb : 0 < b) : + IsProbabilityMeasure (laplaceMeasure μ b) := + ⟨by rw [laplaceMeasure, withDensity_apply _ MeasurableSet.univ, Measure.restrict_univ, + lintegral_laplacePDF_eq_one μ hb]⟩ + +/-- **Likelihood ratio of shifted Laplace densities**: if `|μ - μ'| ≤ ε b`, then +`Lap(y | μ, b) ≤ e^ε Lap(y | μ', b)` for every `y` (the computation in the proof of Dwork–Roth, +Theorem 3.6). -/ +lemma laplacePDFReal_le_exp_mul {μ' ε : ℝ} (hb : 0 < b) (h : |μ - μ'| ≤ ε * b) (y : ℝ) : + laplacePDFReal μ b y ≤ exp ε * laplacePDFReal μ' b y := by + unfold laplacePDFReal + rw [mul_left_comm, ← exp_add] + gcongr + have h₁ : |y - μ'| ≤ |y - μ| + |μ - μ'| := abs_sub_le y μ μ' + rw [← sub_nonneg, show ε + -|y - μ'| / b - -|y - μ| / b = (ε * b - |y - μ'| + |y - μ|) / b by + field_simp; ring] + exact div_nonneg (by linarith) hb.le + +/-- **Dwork–Roth, Fact 3.7**: if `Y ∼ Lap(μ, b)` with `b > 0`, then `P[|Y - μ| ≥ t b] = e^{-t}` +for every `t ≥ 0`. -/ +theorem laplaceMeasure_real_le_abs_sub (μ : ℝ) (hb : 0 < b) {t : ℝ} (ht : 0 ≤ t) : + (laplaceMeasure μ b).real {y | t * b ≤ |y - μ|} = exp (-t) := by + have := isProbabilityMeasure_laplaceMeasure μ hb + rcases ht.eq_or_lt with rfl | ht + · simp + set c := t * b with hc_def + have hE : MeasurableSet {y : ℝ | c ≤ |y - μ|} := measurableSet_le measurable_const (by fun_prop) + -- pass to a Bochner integral of `k (|y - μ|)` + set k : ℝ → ℝ := fun s ↦ if c ≤ s then (2 * b)⁻¹ * exp (-b⁻¹ * s) else 0 with hk + have h₁ : (laplaceMeasure μ b).real {y | c ≤ |y - μ|} = ∫ y, k |y - μ| := by + rw [measureReal_def, laplaceMeasure, withDensity_apply _ hE] + unfold laplacePDF + rw [← integral_eq_lintegral_of_nonneg_ae (ae_of_all _ fun y ↦ (laplacePDFReal_pos hb y).le) + (by fun_prop), ← integral_indicator hE] + congr 1 + ext y + simp only [Set.indicator, Set.mem_ofPred_eq, k, laplacePDFReal, div_eq_inv_mul, mul_neg, + neg_mul] + -- on `(0, ∞)`, `k` is the restriction of the exponential density to `[c, ∞)` + have h₂ : ∫ s in Ioi 0, k s = ∫ s in Ici c, (2 * b)⁻¹ * exp (-b⁻¹ * s) := by + rw [← integral_indicator measurableSet_Ioi, ← integral_indicator measurableSet_Ici] + congr 1 + ext s + have : 0 < c := mul_pos ht hb + simp only [Set.indicator, mem_Ioi, mem_Ici, k] + split_ifs <;> first | rfl | linarith + rw [h₁, integral_sub_right_eq_self (fun y ↦ k |y|) μ, integral_comp_abs (f := k), h₂, + integral_Ici_eq_integral_Ioi, integral_const_mul, + integral_exp_mul_Ioi (neg_lt_zero.2 (inv_pos.2 hb)), hc_def] + field_simp + +end LaplacePDF + +/-! ### The Laplace mechanism -/ + +section Mechanism + +variable [MeasurableSpace 𝒟] {f : 𝒟 → ℝ} {b ε Δ : ℝ} + +/-- **The Laplace mechanism** `M_L(x, f, ε) = f(x) + Lap(b)` (Dwork–Roth, Definition 3.3, case +`k = 1`), with the noise scale `b` as a parameter (Dwork–Roth take `b = Δf / ε`): on dataset `x` +it outputs a sample of `Lap(f(x), b)`. It is built as the kernel with density +`(x, y) ↦ laplacePDF (f x) b y` with respect to the constant kernel `x ↦ volume`, which is the +intended kernel when `f` is measurable (`laplaceMechanism_apply`) and the zero kernel otherwise. -/ +noncomputable def laplaceMechanism (f : 𝒟 → ℝ) (b : ℝ) : Kernel 𝒟 ℝ := + (Kernel.const 𝒟 volume).withDensity fun x ↦ laplacePDF (f x) b + +lemma measurable_uncurry_laplacePDF (hf : Measurable f) (b : ℝ) : + Measurable (Function.uncurry fun x ↦ laplacePDF (f x) b) := by + unfold laplacePDF laplacePDFReal + fun_prop + +lemma laplaceMechanism_apply (hf : Measurable f) (b : ℝ) (x : 𝒟) : + laplaceMechanism f b x = laplaceMeasure (f x) b := by + rw [laplaceMechanism, Kernel.withDensity_apply _ (measurable_uncurry_laplacePDF hf b), + Kernel.const_apply, laplaceMeasure] + +/-- The Laplace mechanism with a measurable query and positive scale is a Markov kernel. -/ +lemma isMarkovKernel_laplaceMechanism (hf : Measurable f) (hb : 0 < b) : + IsMarkovKernel (laplaceMechanism f b) := + ⟨fun x ↦ by rw [laplaceMechanism_apply hf]; exact isProbabilityMeasure_laplaceMeasure _ hb⟩ + +/-- The Laplace mechanism with scale `b > 0` is `ε`-differentially private as soon as +`|f(x) - f(x')| ≤ ε b` for all neighbouring `x, x'` (a form of Dwork–Roth, Theorem 3.6, `k = 1`, +for any scale `b ≥ Δ / ε`). -/ +theorem isEpsDeltaDP_laplaceMechanism_of_le {Adj : 𝒟 → 𝒟 → Prop} (hf : Measurable f) + (hb : 0 < b) (hΔf : ∀ ⦃x x'⦄, Adj x x' → |f x - f x'| ≤ ε * b) : + IsEpsDeltaDP (laplaceMechanism f b) Adj ε 0 := by + refine isEpsDeltaDP_withDensity_const (measurable_uncurry_laplacePDF hf b) (fun x ↦ ?_) + fun x x' hx y ↦ ?_ + · rw [lintegral_laplacePDF_eq_one _ hb] + exact ENNReal.one_ne_top + · rw [laplacePDF, laplacePDF, ← ENNReal.ofReal_mul (exp_pos ε).le] + exact ENNReal.ofReal_le_ofReal (laplacePDFReal_le_exp_mul hb (hΔf hx) y) + +/-- **Dwork–Roth, Theorem 3.6** (Dwork–McSherry–Nissim–Smith), for real-valued queries (`k = 1`): +the Laplace mechanism with noise scale `Δ / ε` is `(ε, 0)`-differentially private, for a +measurable query `f`, `ε > 0` and any bound `Δ > 0` on the sensitivity of `f`, +`|f(x) - f(x')| ≤ Δ` for all neighbouring `x, x'`. -/ +theorem isEpsDeltaDP_laplaceMechanism {Adj : 𝒟 → 𝒟 → Prop} (hf : Measurable f) (hε : 0 < ε) + (hΔ : 0 < Δ) (hΔf : ∀ ⦃x x'⦄, Adj x x' → |f x - f x'| ≤ Δ) : + IsEpsDeltaDP (laplaceMechanism f (Δ / ε)) Adj ε 0 := + isEpsDeltaDP_laplaceMechanism_of_le hf (by positivity) fun x x' hx ↦ by + rw [mul_div_cancel₀ _ hε.ne'] + exact hΔf hx + +/-- **Dwork–Roth, Theorem 3.8**, case `k = 1` (accuracy of the Laplace mechanism): for a +measurable query `f`, every dataset `x` and `δ ∈ (0, 1]`, the output `y` of the Laplace mechanism +with scale `b > 0` satisfies `P[|f(x) - y| ≥ ln (1/δ) · b] ≤ δ` (with equality). Dwork–Roth take +`b = Δf / ε`. -/ +theorem laplaceMechanism_real_le_abs_sub (hf : Measurable f) (hb : 0 < b) (x : 𝒟) {δ : ℝ} + (hδ₀ : 0 < δ) (hδ₁ : δ ≤ 1) : + (laplaceMechanism f b x).real {y | log (1 / δ) * b ≤ |f x - y|} ≤ δ := by + rw [laplaceMechanism_apply hf] + simp_rw [abs_sub_comm (f x)] + rw [laplaceMeasure_real_le_abs_sub _ hb (log_nonneg (one_le_one_div hδ₀ hδ₁)), + log_div one_ne_zero hδ₀.ne', log_one, zero_sub, neg_neg, exp_log hδ₀] + +end Mechanism + +/-! ### The `ℓ₁` Laplace mechanism for vector-valued queries -/ + +section MechanismPi + +variable {ι : Type*} [Fintype ι] {b ε Δ : ℝ} + +/-- The density `y ↦ ∏_i Lap(y_i | μ_i, b)` on `ι → ℝ` of a vector of independent Laplace +variables with locations `μ_i` and common scale `b`. -/ +noncomputable def laplacePiPDFReal (μ : ι → ℝ) (b : ℝ) (y : ι → ℝ) : ℝ := + ∏ i, laplacePDFReal (μ i) b (y i) + +lemma laplacePiPDFReal_pos (μ : ι → ℝ) (hb : 0 < b) (y : ι → ℝ) : + 0 < laplacePiPDFReal μ b y := + Finset.prod_pos fun i _ ↦ laplacePDFReal_pos hb (y i) + +/-- The product Laplace density integrates to one. -/ +lemma integral_laplacePiPDFReal_eq_one (μ : ι → ℝ) (hb : 0 < b) : + ∫ y, laplacePiPDFReal μ b y = 1 := by + unfold laplacePiPDFReal + rw [integral_fintype_prod_volume_eq_prod (fun i ↦ laplacePDFReal (μ i) b)] + exact Finset.prod_eq_one fun i _ ↦ integral_laplacePDFReal_eq_one (μ i) hb + +lemma integrable_laplacePiPDFReal (μ : ι → ℝ) (hb : 0 < b) : + Integrable (laplacePiPDFReal μ b) := + Integrable.of_integral_ne_zero (by simp [integral_laplacePiPDFReal_eq_one μ hb]) + +/-- The product Laplace density integrates to one. -/ +lemma lintegral_laplacePiPDFReal_eq_one (μ : ι → ℝ) (hb : 0 < b) : + ∫⁻ y, ENNReal.ofReal (laplacePiPDFReal μ b y) = 1 := by + have hm : Measurable (laplacePiPDFReal μ b) := by + unfold laplacePiPDFReal laplacePDFReal + fun_prop + rw [← ENNReal.toReal_eq_one_iff, ← integral_eq_lintegral_of_nonneg_ae + (ae_of_all _ fun y ↦ (laplacePiPDFReal_pos μ hb y).le) hm.aestronglyMeasurable, + integral_laplacePiPDFReal_eq_one μ hb] + +/-- **Likelihood ratio of shifted product Laplace densities**: if `‖μ - μ'‖₁ ≤ ε b`, then +`p_μ(y) ≤ e^ε p_{μ'}(y)` for every `y` (the computation in the proof of Dwork–Roth, +Theorem 3.6). -/ +lemma laplacePiPDFReal_le_exp_mul {μ μ' : ι → ℝ} (hb : 0 < b) (h : ∑ i, |μ i - μ' i| ≤ ε * b) + (y : ι → ℝ) : laplacePiPDFReal μ b y ≤ exp ε * laplacePiPDFReal μ' b y := by + have h₁ (i : ι) : laplacePDFReal (μ i) b (y i) ≤ + exp (|μ i - μ' i| / b) * laplacePDFReal (μ' i) b (y i) := + laplacePDFReal_le_exp_mul hb (by rw [div_mul_cancel₀ _ hb.ne']) (y i) + unfold laplacePiPDFReal + calc ∏ i, laplacePDFReal (μ i) b (y i) + ≤ ∏ i, (exp (|μ i - μ' i| / b) * laplacePDFReal (μ' i) b (y i)) := + Finset.prod_le_prod₀ (fun i _ ↦ (laplacePDFReal_pos hb _).le) fun i _ ↦ h₁ i + _ = exp (∑ i, |μ i - μ' i| / b) * ∏ i, laplacePDFReal (μ' i) b (y i) := by + rw [Finset.prod_mul_distrib, exp_sum] + _ ≤ exp ε * ∏ i, laplacePDFReal (μ' i) b (y i) := by + gcongr + · exact Finset.prod_nonneg fun i _ ↦ (laplacePDFReal_pos hb _).le + · rwa [← Finset.sum_div, div_le_iff₀ hb] + +variable [MeasurableSpace 𝒟] {f : 𝒟 → ι → ℝ} + +/-- **The Laplace mechanism for vector-valued queries** `M_L(x, f, ε) = f(x) + (Y_1, …, Y_k)` +with `Y_i` i.i.d. `Lap(b)` (Dwork–Roth, Definition 3.3, with the coordinates indexed by a finite +type `ι` and the noise scale `b` as a parameter; Dwork–Roth take `b = Δf / ε`). It is built as +`Kernel.withDensity` of the constant kernel `x ↦ volume` on `ι → ℝ` with density +`(x, y) ↦ ∏_i Lap(y_i | f(x)_i, b)`. This is the intended mechanism, with output law +`∏_i Lap(f(x)_i, b)`, when that density is jointly measurable (for instance when `f` is +measurable, `laplaceMechanismPi_apply`) and `b > 0` (so that it is normalized, +`isMarkovKernel_laplaceMechanismPi`); `Kernel.withDensity` returns the **zero kernel** when the +density is not jointly measurable. -/ +noncomputable def laplaceMechanismPi (f : 𝒟 → ι → ℝ) (b : ℝ) : Kernel 𝒟 (ι → ℝ) := + (Kernel.const 𝒟 volume).withDensity fun x y ↦ ENNReal.ofReal (laplacePiPDFReal (f x) b y) + +lemma measurable_uncurry_laplacePiPDFReal (hf : Measurable f) (b : ℝ) : + Measurable (Function.uncurry fun x y ↦ ENNReal.ofReal (laplacePiPDFReal (f x) b y)) := by + unfold laplacePiPDFReal laplacePDFReal + fun_prop + +/-- For a measurable query and `b > 0`, the output law of the vector Laplace mechanism is the +product of the Laplace laws `Lap(f(x)_i, b)`: the noise coordinates are i.i.d. `Lap(b)`. -/ +lemma laplaceMechanismPi_apply (hf : Measurable f) (hb : 0 < b) (x : 𝒟) : + laplaceMechanismPi f b x = Measure.pi fun i ↦ laplaceMeasure (f x i) b := by + have := fun i ↦ isProbabilityMeasure_laplaceMeasure (f x i) hb + set g : ι → ℝ → ℝ := fun i ↦ laplacePDFReal (f x i) b + have hg (i : ι) : Integrable (g i) := integrable_laplacePDFReal _ hb + have hg₀ (i : ι) (y : ℝ) : 0 ≤ g i y := (laplacePDFReal_pos hb y).le + rw [laplaceMechanismPi, Kernel.withDensity_apply _ (measurable_uncurry_laplacePiPDFReal hf b), + Kernel.const_apply] + refine (Measure.pi_eq fun s hs ↦ ?_).symm + -- each factor is `∫_{s i} g i` + have hfac (i : ι) : + laplaceMeasure (f x i) b (s i) = ENNReal.ofReal (∫ y, (s i).indicator (g i) y) := by + rw [laplaceMeasure, withDensity_apply _ (hs i), integral_indicator (hs i), + ofReal_integral_eq_lintegral_ofReal (hg i).integrableOn (ae_of_all _ fun y ↦ hg₀ i y)] + rfl + -- the indicator of the box is the product of the coordinate indicators + have hind (y : ι → ℝ) : (Set.univ.pi s).indicator (laplacePiPDFReal (f x) b) y = + ∏ i, (s i).indicator (g i) (y i) := by + by_cases hy : y ∈ Set.univ.pi s + · rw [indicator_of_mem hy] + exact Finset.prod_congr rfl fun i _ ↦ (indicator_of_mem (hy i (mem_univ i)) _).symm + · rw [indicator_of_notMem hy] + obtain ⟨i, hi⟩ : ∃ i, y i ∉ s i := by simpa [Set.mem_univ_pi] using hy + exact (Finset.prod_eq_zero (Finset.mem_univ i) (indicator_of_notMem hi _)).symm + have hint : Integrable (fun y : ι → ℝ ↦ ∏ i, (s i).indicator (g i) (y i)) := + Integrable.fintype_prod fun i ↦ (hg i).indicator (hs i) + rw [withDensity_apply _ (MeasurableSet.univ_pi hs), ← ofReal_integral_eq_lintegral_ofReal + ((integrable_laplacePiPDFReal (f x) hb).integrableOn) + (ae_of_all _ fun y ↦ (laplacePiPDFReal_pos _ hb y).le), + ← integral_indicator (MeasurableSet.univ_pi hs)] + simp_rw [hind, hfac] + rw [integral_fintype_prod_volume_eq_prod (fun i ↦ (s i).indicator (g i)), + ENNReal.ofReal_prod_of_nonneg fun i _ ↦ integral_nonneg fun y ↦ indicator_nonneg + (fun y _ ↦ hg₀ i y) y] + +/-- The vector Laplace mechanism with a measurable query and positive scale is a Markov +kernel. -/ +lemma isMarkovKernel_laplaceMechanismPi (hf : Measurable f) (hb : 0 < b) : + IsMarkovKernel (laplaceMechanismPi f b) := by + refine ⟨fun x ↦ ⟨?_⟩⟩ + rw [laplaceMechanismPi, Kernel.withDensity_apply _ (measurable_uncurry_laplacePiPDFReal hf b), + Kernel.const_apply, withDensity_apply _ MeasurableSet.univ, Measure.restrict_univ, + lintegral_laplacePiPDFReal_eq_one _ hb] + +/-- **Dwork–Roth, Theorem 3.6** (Dwork–McSherry–Nissim–Smith): the Laplace mechanism for a +measurable vector-valued query `f : 𝒟 → (ι → ℝ)` with noise scale `Δ / ε` is +`(ε, 0)`-differentially private, for `ε > 0` and any bound `Δ > 0` on the `ℓ₁`-sensitivity of +`f`, `∑_i |f(x)_i - f(x')_i| ≤ Δ` for all neighbouring `x, x'`. -/ +theorem isEpsDeltaDP_laplaceMechanismPi {Adj : 𝒟 → 𝒟 → Prop} (hf : Measurable f) (hε : 0 < ε) + (hΔ : 0 < Δ) (hΔf : ∀ ⦃x x'⦄, Adj x x' → ∑ i, |f x i - f x' i| ≤ Δ) : + IsEpsDeltaDP (laplaceMechanismPi f (Δ / ε)) Adj ε 0 := by + have hb : 0 < Δ / ε := div_pos hΔ hε + refine isEpsDeltaDP_withDensity_const (measurable_uncurry_laplacePiPDFReal hf _) + (fun x ↦ ?_) fun x x' hx y ↦ ?_ + · rw [lintegral_laplacePiPDFReal_eq_one _ hb] + exact ENNReal.one_ne_top + · rw [← ENNReal.ofReal_mul (exp_pos ε).le] + refine ENNReal.ofReal_le_ofReal (laplacePiPDFReal_le_exp_mul hb ?_ y) + rw [mul_div_cancel₀ _ hε.ne'] + exact hΔf hx + +/-- **Dwork–Roth, Theorem 3.8** (accuracy of the Laplace mechanism): for a measurable query +`f : 𝒟 → (ι → ℝ)` with `k = |ι| ≥ 1` coordinates, every dataset `x` and `δ ∈ (0, 1]`, the output +`y` of the Laplace mechanism with scale `b > 0` satisfies +`P[‖f(x) - y‖_∞ ≥ ln (k/δ) · b] ≤ δ`, where `‖·‖` is the sup norm on `ι → ℝ`. Dwork–Roth take +`b = Δf / ε`. -/ +theorem laplaceMechanismPi_real_le_norm_sub [Nonempty ι] (hf : Measurable f) (hb : 0 < b) + (x : 𝒟) {δ : ℝ} (hδ₀ : 0 < δ) (hδ₁ : δ ≤ 1) : + (laplaceMechanismPi f b x).real + {y | log (Fintype.card ι / δ) * b ≤ ‖f x - y‖} ≤ δ := by + have := fun i ↦ isProbabilityMeasure_laplaceMeasure (f x i) hb + have hk : (1 : ℝ) ≤ Fintype.card ι := by exact_mod_cast Fintype.card_pos + set t := log (Fintype.card ι / δ) + have ht : 0 ≤ t := log_nonneg (by rw [le_div_iff₀ hδ₀]; linarith) + -- the event is covered by the coordinate events `|y_i - f(x)_i| ≥ t b` + have hsub : {y : ι → ℝ | t * b ≤ ‖f x - y‖} ⊆ + ⋃ i, (fun y ↦ y i) ⁻¹' {z | t * b ≤ |z - f x i|} := by + intro y hy + by_contra h + simp only [mem_iUnion, mem_preimage, Set.mem_ofPred_eq, not_exists, not_le] at h + have hpos : 0 < t * b := (abs_nonneg _).trans_lt (h (Classical.arbitrary ι)) + have : ‖f x - y‖ < t * b := (pi_norm_lt_iff hpos).2 fun i ↦ by + rw [Pi.sub_apply, Real.norm_eq_abs, abs_sub_comm] + exact h i + exact absurd hy (not_le.2 this) + -- each coordinate event has probability `e^{-t} = δ / k` (Fact 3.7) + have hcoord (i : ι) : (Measure.pi fun i ↦ laplaceMeasure (f x i) b).real + ((fun y ↦ y i) ⁻¹' {z | t * b ≤ |z - f x i|}) = δ / Fintype.card ι := by + rw [(measurePreserving_eval _ i).measureReal_preimage + (measurableSet_le measurable_const (by fun_prop)).nullMeasurableSet, + laplaceMeasure_real_le_abs_sub _ hb ht, exp_neg, exp_log (by positivity), inv_div] + rw [laplaceMechanismPi_apply hf hb] + calc _ ≤ _ := measureReal_mono hsub + _ ≤ _ := measureReal_iUnion_fintype_le _ + _ = δ := by + simp_rw [hcoord] + rw [Finset.sum_const, Finset.card_univ, nsmul_eq_mul] + field_simp + +end MechanismPi + +end InformationTheory diff --git a/Statlib/Privacy/DP/RandomizedResponse.lean b/Statlib/Privacy/DP/RandomizedResponse.lean new file mode 100644 index 0000000..a49303b --- /dev/null +++ b/Statlib/Privacy/DP/RandomizedResponse.lean @@ -0,0 +1,338 @@ +/- +Copyright (c) 2026 Kousha Kalantari. All rights reserved. +Released under Apache 2.0 license as described in the file LICENSE. +Authors: Kousha Kalantari +-/ +module + +public import Statlib.Privacy.DP.Basic +public import Mathlib.MeasureTheory.Measure.Count +public import Mathlib.Probability.Kernel.WithDensity + +/-! +# Randomized response + +Randomized response is the oldest differentially private mechanism: each respondent reports the +true answer with some probability and a different answer otherwise. + +* Binary randomized response goes back to S. L. Warner, *Randomized response: a survey technique + for eliminating evasive answer bias*, J. Amer. Statist. Assoc. 60(309), 1965: the respondent + reports the truth with a design probability `p ≠ 1/2`. Its two-coin special case (`p = 3/4`) is + the first mechanism of C. Dwork and A. Roth, *The Algorithmic Foundations of Differential + Privacy*, Found. Trends Theor. Comput. Sci. 9(3–4), 2014, Section 3.2. +* The `k`-ary randomized response with privacy parameter `ε` on a finite alphabet `α` of size + `k = |α|` is from P. Kairouz, S. Oh and P. Viswanath, *Extremal mechanisms for local + differential privacy*, NeurIPS 2014 / J. Mach. Learn. Res. 17, 2016: on input `x` it outputs + `x` with probability `e^ε / (e^ε + k - 1)` and each of the other `k - 1` symbols with + probability `1 / (e^ε + k - 1)`. For `k = 2` (on `Bool`) it keeps the true bit with + probability `e^ε / (1 + e^ε)`, which is Warner's scheme with `p = e^ε / (1 + e^ε)`, i.e. + `ε = ln (p / (1 - p))`. + +Randomized response is a *local* mechanism: it is applied to a single respondent's datum, so +every pair of inputs is neighbouring (`Adj = fun _ _ ↦ True`). Privacy is measured with +`IsEpsDeltaDP` from `Statlib.Privacy.DP.Basic` at `δ = 0` (pure `ε`-differential privacy). The +parameter `ε` may be negative (Warner's `p < 1/2`); the exact privacy level is then `|ε|`. + +## Main definitions + +* `randomizedResponseProb ε x y`: the probability that `k`-ary randomized response outputs `y` + on input `x`. +* `randomizedResponse ε`: `k`-ary randomized response as a Markov kernel `α → α`. +* `binaryRandomizedResponse ε`: binary randomized response, the case `α = Bool`. +* `twoCoinRandomizedResponse`: the two-coin procedure of Dwork–Roth, Section 3.2, as the kernel + `b ↦ ½ δ_b + ½ uniform` on `Bool`. + +## Main statements (source result → Lean name) + +* `randomizedResponse ε` is a Markov kernel: `isMarkovKernel_randomizedResponse`. +* `k`-ary randomized response is `|ε|`-locally differentially private: + `isEpsDeltaDP_randomizedResponse_abs` (and `ε`-LDP for `ε ≥ 0`: + `isEpsDeltaDP_randomizedResponse`); `|ε|` is its exact privacy level, i.e. it is + `ε'`-differentially private iff `|ε| ≤ ε'` (as soon as `k ≥ 2`): + `isEpsDeltaDP_randomizedResponse_iff`. +* Binary randomized response keeps the true bit with probability `e^ε / (1 + e^ε)` + (`binaryRandomizedResponse_real_self`) and is exactly `|ε|`-LDP: + `isEpsDeltaDP_binaryRandomizedResponse_iff`. Warner's scheme with design probability + `p ∈ (0, 1)` is the case `ε = ln (p / (1 - p))` (`binaryRandomizedResponse_log_div_real_self`) + and is exactly `|ln (p / (1 - p))|`-LDP: `isEpsDeltaDP_binaryRandomizedResponse_log_div_iff`. +* Dwork–Roth, Claim 3.5: the two-coin procedure is binary randomized response with `ε = ln 3` + (`twoCoinRandomizedResponse_eq`; it keeps the true answer with probability `3/4`: + `binaryRandomizedResponse_log_three`), and it is `(ln 3, 0)`-differentially private, with + `ln 3` exact: `isEpsDeltaDP_twoCoinRandomizedResponse_iff`. +-/ + +@[expose] public section + +open MeasureTheory ProbabilityTheory Real +open scoped ENNReal + +namespace InformationTheory + +variable {α : Type*} + +namespace RandomizedResponse + +/-- A kernel `x ↦ μ.withDensity (f x)` with jointly measurable, normalizable densities +satisfying the pointwise bound `f x ≤ e^ε f x'` on neighbours is `ε`-differentially private. -/ +lemma isEpsDeltaDP_withDensity_const {𝒟 Y : Type*} [MeasurableSpace 𝒟] [MeasurableSpace Y] + {μ : Measure Y} [SFinite μ] {f : 𝒟 → Y → ℝ≥0∞} {Adj : 𝒟 → 𝒟 → Prop} {ε : ℝ} + (hf : Measurable (Function.uncurry f)) (hfin : ∀ x, ∫⁻ y, f x y ∂μ ≠ ∞) + (h : ∀ ⦃x x'⦄, Adj x x' → ∀ y, f x y ≤ ENNReal.ofReal (exp ε) * f x' y) : + IsEpsDeltaDP ((Kernel.const 𝒟 μ).withDensity f) Adj ε 0 := by + intro x x' hx E + simp only [Kernel.withDensity_apply _ hf, Kernel.const_apply, add_zero] + have hle : μ.withDensity (f x) E ≤ ENNReal.ofReal (exp ε) * μ.withDensity (f x') E := by + have hμ : μ.withDensity (f x) ≤ ENNReal.ofReal (exp ε) • μ.withDensity (f x') := by + rw [← withDensity_smul' _ _ ENNReal.ofReal_ne_top] + exact withDensity_mono (ae_of_all _ (h hx)) + simpa using hμ E + have hfin' : μ.withDensity (f x') E ≠ ∞ := by + refine ne_top_of_le_ne_top ?_ (measure_mono (Set.subset_univ E)) + rw [withDensity_apply _ MeasurableSet.univ, Measure.restrict_univ] + exact hfin x' + rw [measureReal_def, measureReal_def, ← ENNReal.toReal_ofReal (exp_pos ε).le, + ← ENNReal.toReal_mul] + exact ENNReal.toReal_mono (ENNReal.mul_ne_top ENNReal.ofReal_ne_top hfin') hle + +end RandomizedResponse + +section KAry + +variable [Fintype α] {ε : ℝ} + +namespace RandomizedResponse + +lemma card_sub_one_nonneg (x : α) : (0 : ℝ) ≤ Fintype.card α - 1 := by + have : 1 ≤ Fintype.card α := Fintype.card_pos_iff.2 ⟨x⟩ + rw [sub_nonneg] + exact_mod_cast this + +lemma denom_pos (ε : ℝ) (x : α) : 0 < exp ε + Fintype.card α - 1 := by + linarith [card_sub_one_nonneg x, exp_pos ε] + +end RandomizedResponse + +open RandomizedResponse + +variable [DecidableEq α] + +/-- The probability that `k`-ary randomized response with privacy parameter `ε` outputs `y` on +input `x`, where `k = |α|`: `e^ε / (e^ε + k - 1)` if `y = x` and `1 / (e^ε + k - 1)` otherwise. -/ +noncomputable def randomizedResponseProb (ε : ℝ) (x y : α) : ℝ := + (if x = y then exp ε else 1) / (exp ε + Fintype.card α - 1) + +lemma randomizedResponseProb_nonneg (ε : ℝ) (x y : α) : 0 ≤ randomizedResponseProb ε x y := by + have := denom_pos ε x + unfold randomizedResponseProb + split_ifs <;> positivity + +lemma randomizedResponseProb_self (ε : ℝ) (x : α) : + randomizedResponseProb ε x x = exp ε / (exp ε + Fintype.card α - 1) := by + simp [randomizedResponseProb] + +lemma randomizedResponseProb_of_ne (ε : ℝ) {x y : α} (h : x ≠ y) : + randomizedResponseProb ε x y = 1 / (exp ε + Fintype.card α - 1) := by + simp [randomizedResponseProb, h] + +/-- The output probabilities of randomized response sum to one. -/ +lemma sum_randomizedResponseProb (ε : ℝ) (x : α) : ∑ y, randomizedResponseProb ε x y = 1 := by + have hd := (denom_pos ε x).ne' + have h (y : α) : randomizedResponseProb ε x y = + (1 + if x = y then exp ε - 1 else 0) / (exp ε + Fintype.card α - 1) := by + unfold randomizedResponseProb; split_ifs <;> ring + simp_rw [h, ← Finset.sum_div, Finset.sum_add_distrib, Finset.sum_ite_eq, Finset.sum_const, + Finset.card_univ, nsmul_eq_mul, mul_one] + simp only [Finset.mem_univ, ↓reduceIte] + field_simp + ring + +/-- The likelihood ratio of randomized response is at most `e^{|ε|}`: +`RR(y | x) ≤ e^{|ε|} RR(y | x')` for all inputs `x, x'` and outputs `y`. -/ +lemma randomizedResponseProb_le_exp_abs_mul (ε : ℝ) (x x' y : α) : + randomizedResponseProb ε x y ≤ exp |ε| * randomizedResponseProb ε x' y := by + have hd := denom_pos ε x + have h₁ : 1 ≤ exp |ε| := one_le_exp (abs_nonneg ε) + have h₂ : exp ε ≤ exp |ε| := exp_le_exp.2 (le_abs_self ε) + have h₃ : 1 ≤ exp |ε| * exp ε := by + rw [← exp_add] + exact one_le_exp (by linarith [neg_abs_le ε]) + have h₄ := exp_pos ε + unfold randomizedResponseProb + rw [mul_div_assoc'] + gcongr + split_ifs <;> nlinarith + +/-- The likelihood ratio of randomized response is at most `e^ε`: `RR(y | x) ≤ e^ε RR(y | x')` +for all inputs `x, x'` and outputs `y` (for `ε ≥ 0`). -/ +lemma randomizedResponseProb_le_exp_mul (hε : 0 ≤ ε) (x x' y : α) : + randomizedResponseProb ε x y ≤ exp ε * randomizedResponseProb ε x' y := by + simpa [abs_of_nonneg hε] using randomizedResponseProb_le_exp_abs_mul ε x x' y + +variable [MeasurableSpace α] [MeasurableSingletonClass α] + +/-- **`k`-ary randomized response** (Kairouz–Oh–Viswanath) with privacy parameter `ε` as a Markov +kernel on the finite alphabet `α` (`k = |α|`): on input `x`, output `x` with probability +`e^ε / (e^ε + k - 1)` and each other symbol with probability `1 / (e^ε + k - 1)`. It is the +measure with density `randomizedResponseProb ε x` with respect to the counting measure. -/ +noncomputable def randomizedResponse (ε : ℝ) : Kernel α α := + (Kernel.const α Measure.count).withDensity fun x y ↦ ENNReal.ofReal (randomizedResponseProb ε x y) + +lemma randomizedResponse_apply (ε : ℝ) (x : α) : + randomizedResponse ε x = + Measure.count.withDensity fun y ↦ ENNReal.ofReal (randomizedResponseProb ε x y) := by + rw [randomizedResponse, Kernel.withDensity_apply _ (measurable_of_finite _), Kernel.const_apply] + +lemma randomizedResponse_apply_singleton (ε : ℝ) (x y : α) : + randomizedResponse ε x {y} = ENNReal.ofReal (randomizedResponseProb ε x y) := by + rw [randomizedResponse_apply, withDensity_apply _ (MeasurableSet.singleton y), + Measure.restrict_singleton, Measure.count_singleton, one_smul, lintegral_dirac] + +lemma randomizedResponse_real_singleton (ε : ℝ) (x y : α) : + (randomizedResponse ε x).real {y} = randomizedResponseProb ε x y := by + rw [measureReal_def, randomizedResponse_apply_singleton, + ENNReal.toReal_ofReal (randomizedResponseProb_nonneg ε x y)] + +/-- Randomized response is a Markov kernel. -/ +instance isMarkovKernel_randomizedResponse (ε : ℝ) : + IsMarkovKernel (randomizedResponse (α := α) ε) := by + refine ⟨fun x ↦ ⟨?_⟩⟩ + rw [randomizedResponse_apply, withDensity_apply _ MeasurableSet.univ, Measure.restrict_univ, + lintegral_fintype, ← ENNReal.ofReal_one, ← sum_randomizedResponseProb ε x, + ENNReal.ofReal_sum_of_nonneg fun y _ ↦ randomizedResponseProb_nonneg ε x y] + simp + +/-- **`k`-ary randomized response is `|ε|`-locally differentially private**: for all inputs +`x, x'` and every event `E`, `P[RR(x) ∈ E] ≤ e^{|ε|} P[RR(x') ∈ E]`. -/ +theorem isEpsDeltaDP_randomizedResponse_abs (ε : ℝ) : + IsEpsDeltaDP (randomizedResponse (α := α) ε) (fun _ _ ↦ True) |ε| 0 := by + refine isEpsDeltaDP_withDensity_const (measurable_of_finite _) (fun x ↦ ?_) + fun x x' _ y ↦ ?_ + · rw [lintegral_fintype] + exact ENNReal.sum_ne_top.2 fun y _ ↦ by simp + · rw [← ENNReal.ofReal_mul (exp_pos _).le] + exact ENNReal.ofReal_le_ofReal (randomizedResponseProb_le_exp_abs_mul ε x x' y) + +/-- **`k`-ary randomized response is `ε`-locally differentially private** (for `ε ≥ 0`): for all +inputs `x, x'` and every event `E`, `P[RR(x) ∈ E] ≤ e^ε P[RR(x') ∈ E]`. -/ +theorem isEpsDeltaDP_randomizedResponse (hε : 0 ≤ ε) : + IsEpsDeltaDP (randomizedResponse (α := α) ε) (fun _ _ ↦ True) ε 0 := by + simpa [abs_of_nonneg hε] using isEpsDeltaDP_randomizedResponse_abs (α := α) ε + +/-- **`|ε|` is the exact privacy level of `k`-ary randomized response** (for `k ≥ 2`): it is +`ε'`-locally differentially private if and only if `|ε| ≤ ε'`. For `x ≠ x'`, the ratios +`P[RR(x) = x] / P[RR(x') = x] = e^ε` and `P[RR(x) = x'] / P[RR(x') = x'] = e^{-ε}` show that no +smaller `ε'` works. -/ +theorem isEpsDeltaDP_randomizedResponse_iff [Nontrivial α] (ε : ℝ) {ε' : ℝ} : + IsEpsDeltaDP (randomizedResponse (α := α) ε) (fun _ _ ↦ True) ε' 0 ↔ |ε| ≤ ε' := by + refine ⟨fun h ↦ ?_, fun h S S' _ E ↦ ?_⟩ + · obtain ⟨x, x', hxx'⟩ := exists_pair_ne α + have hd := denom_pos ε x + have h₁ := h (S := x) (S' := x') trivial {x} + have h₂ := h (S := x) (S' := x') trivial {x'} + rw [randomizedResponse_real_singleton, randomizedResponse_real_singleton, + randomizedResponseProb_self, randomizedResponseProb_of_ne _ hxx'.symm, add_zero, mul_one_div, + div_le_div_iff_of_pos_right hd, exp_le_exp] at h₁ + rw [randomizedResponse_real_singleton, randomizedResponse_real_singleton, + randomizedResponseProb_self, randomizedResponseProb_of_ne _ hxx', add_zero, mul_div_assoc', + div_le_div_iff_of_pos_right hd, ← exp_add, one_le_exp_iff] at h₂ + exact abs_le.2 ⟨by linarith, h₁⟩ + · calc (randomizedResponse ε S).real E ≤ exp |ε| * (randomizedResponse ε S').real E + 0 := + isEpsDeltaDP_randomizedResponse_abs ε trivial E + _ ≤ exp ε' * (randomizedResponse ε S').real E + 0 := by gcongr + +end KAry + +section Binary + +variable {ε : ℝ} + +/-- **Binary randomized response** (Warner) with privacy parameter `ε`: `k`-ary randomized +response on `Bool`, which reports the true bit with probability `e^ε / (1 + e^ε)` and flips it +with probability `1 / (1 + e^ε)` (`binaryRandomizedResponse_real_self`, +`binaryRandomizedResponse_real_not`). -/ +noncomputable abbrev binaryRandomizedResponse (ε : ℝ) : Kernel Bool Bool := + randomizedResponse ε + +/-- Binary randomized response reports the true bit with probability `e^ε / (1 + e^ε)`. -/ +lemma binaryRandomizedResponse_real_self (ε : ℝ) (b : Bool) : + (binaryRandomizedResponse ε b).real {b} = exp ε / (1 + exp ε) := by + rw [randomizedResponse_real_singleton, randomizedResponseProb_self, Fintype.card_bool] + congr 1 + push_cast + ring + +/-- Binary randomized response flips the bit with probability `1 / (1 + e^ε)`. -/ +lemma binaryRandomizedResponse_real_not (ε : ℝ) (b : Bool) : + (binaryRandomizedResponse ε b).real {!b} = 1 / (1 + exp ε) := by + rw [randomizedResponse_real_singleton, randomizedResponseProb_of_ne _ (Bool.not_ne_self b).symm, + Fintype.card_bool] + congr 1 + push_cast + ring + +/-- **Binary randomized response is exactly `|ε|`-locally differentially private**: it is +`ε'`-differentially private if and only if `|ε| ≤ ε'`. -/ +theorem isEpsDeltaDP_binaryRandomizedResponse_iff (ε : ℝ) {ε' : ℝ} : + IsEpsDeltaDP (binaryRandomizedResponse ε) (fun _ _ ↦ True) ε' 0 ↔ |ε| ≤ ε' := + isEpsDeltaDP_randomizedResponse_iff ε + +/-- Warner's randomized response with design probability `p ∈ (0, 1)`, which reports the truth +with probability `p`, is binary randomized response with `ε = ln (p / (1 - p))` (negative when +`p < 1/2`). -/ +lemma binaryRandomizedResponse_log_div_real_self {p : ℝ} (hp₀ : 0 < p) (hp₁ : p < 1) + (b : Bool) : (binaryRandomizedResponse (log (p / (1 - p))) b).real {b} = p := by + have : 0 < 1 - p := by linarith + rw [binaryRandomizedResponse_real_self, exp_log (by positivity)] + field_simp + ring + +/-- **Privacy of Warner's randomized response**: with design probability `p ∈ (0, 1)` +(`binaryRandomizedResponse_log_div_real_self`), the scheme is `ε'`-locally differentially private +if and only if `|ln (p / (1 - p))| ≤ ε'`. -/ +theorem isEpsDeltaDP_binaryRandomizedResponse_log_div_iff (p : ℝ) {ε' : ℝ} : + IsEpsDeltaDP (binaryRandomizedResponse (log (p / (1 - p)))) (fun _ _ ↦ True) ε' 0 ↔ + |log (p / (1 - p))| ≤ ε' := + isEpsDeltaDP_binaryRandomizedResponse_iff _ + +/-- The two-coin randomized response of Dwork–Roth, Section 3.2: flip a fair coin; on tails +answer truthfully, on heads answer according to a second fair coin. As a kernel on `Bool` it is +the mixture `½ · δ_b + ½ · uniform`. -/ +noncomputable def twoCoinRandomizedResponse : Kernel Bool Bool := + Kernel.ofFunOfCountable fun b ↦ + (2⁻¹ : ℝ≥0∞) • Measure.dirac b + (2⁻¹ : ℝ≥0∞) • ((2⁻¹ : ℝ≥0∞) • Measure.count) + +/-- The two-coin randomized response of Dwork–Roth, Section 3.2, is binary randomized response +with `ε = ln 3`: it reports the truth with probability `3/4` and lies with probability `1/4` +(`binaryRandomizedResponse_log_three`). -/ +theorem twoCoinRandomizedResponse_eq : twoCoinRandomizedResponse = + binaryRandomizedResponse (log 3) := by + have h2 : (2⁻¹ : ℝ≥0∞) = ENNReal.ofReal 2⁻¹ := by + rw [ENNReal.ofReal_inv_of_pos two_pos, ENNReal.ofReal_ofNat] + ext b : 1 + refine Measure.ext_iff_singleton.2 fun c ↦ ?_ + rw [randomizedResponse_apply_singleton] + simp [twoCoinRandomizedResponse, Kernel.ofFunOfCountable] + cases b <;> cases c <;> + simp only [randomizedResponseProb, exp_log (show (0 : ℝ) < 3 by norm_num), h2, + ← ENNReal.ofReal_mul (show (0 : ℝ) ≤ 2⁻¹ by norm_num)] <;> + norm_num [← ENNReal.ofReal_add] + +/-- The two-coin randomized response reports the truth with probability `3/4` and lies with +probability `1/4`. -/ +lemma binaryRandomizedResponse_log_three (b : Bool) : + (binaryRandomizedResponse (log 3) b).real {b} = 3 / 4 ∧ + (binaryRandomizedResponse (log 3) b).real {!b} = 1 / 4 := by + rw [binaryRandomizedResponse_real_self, binaryRandomizedResponse_real_not, + exp_log (by norm_num)] + norm_num + +/-- **Dwork–Roth, Claim 3.5**: the two-coin randomized response of Section 3.2 is +`(ln 3, 0)`-differentially private, and `ln 3` is its exact privacy level. -/ +theorem isEpsDeltaDP_twoCoinRandomizedResponse_iff {ε' : ℝ} : + IsEpsDeltaDP twoCoinRandomizedResponse (fun _ _ ↦ True) ε' 0 ↔ log 3 ≤ ε' := by + rw [twoCoinRandomizedResponse_eq, isEpsDeltaDP_binaryRandomizedResponse_iff, + abs_of_nonneg (log_nonneg (by norm_num))] + +end Binary + +end InformationTheory From bc8e55225476eafed25ae24e26c981e63621e680 Mon Sep 17 00:00:00 2001 From: Kousha Kalantari <25445970+KoushaKalantari@users.noreply.github.com> Date: Fri, 13 Nov 2026 10:21:05 -0800 Subject: [PATCH 3/5] =?UTF-8?q?feat(Privacy):=20R=C3=A9nyi=20differential?= =?UTF-8?q?=20privacy=20(Mironov)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- Statlib.lean | 1 + Statlib/Privacy/RenyiDP/Basic.lean | 718 +++++++++++++++++++++++++++++ 2 files changed, 719 insertions(+) create mode 100644 Statlib/Privacy/RenyiDP/Basic.lean diff --git a/Statlib.lean b/Statlib.lean index abafa48..9fb07bc 100644 --- a/Statlib.lean +++ b/Statlib.lean @@ -143,6 +143,7 @@ import Statlib.Privacy.DP.Exponential import Statlib.Privacy.DP.Geometric import Statlib.Privacy.DP.Laplace import Statlib.Privacy.DP.RandomizedResponse +import Statlib.Privacy.RenyiDP.Basic import Statlib.Testing.Binary import Statlib.Testing.BinaryRenyi import Statlib.Testing.ChangeMeasure diff --git a/Statlib/Privacy/RenyiDP/Basic.lean b/Statlib/Privacy/RenyiDP/Basic.lean new file mode 100644 index 0000000..0de4643 --- /dev/null +++ b/Statlib/Privacy/RenyiDP/Basic.lean @@ -0,0 +1,718 @@ +/- +Copyright (c) 2026 Kousha Kalantari. All rights reserved. +Released under Apache 2.0 license as described in the file LICENSE. +Authors: Kousha Kalantari +-/ +module + +public import Statlib.Privacy.DP.Basic +public import Mathlib.MeasureTheory.Integral.MeanInequalities +public import Statlib.Divergence.Hellinger.CondHellinger +public import Statlib.Divergence.Renyi.ChainRule +public import Statlib.Divergence.Renyi.Prod + +/-! +# Rényi differential privacy + +This file formalizes I. Mironov, *Rényi differential privacy*, IEEE 30th Computer Security +Foundations Symposium (CSF), 2017 ([arXiv:1702.07476](https://arxiv.org/abs/1702.07476)); result +numbers follow arXiv v3. + +A randomized mechanism `M`, given as a Markov kernel from datasets to outputs, has `ε`-Rényi +differential privacy of order `α` (is `(α, ε)`-RDP) if `D_α(M(S) ‖ M(S')) ≤ ε` for all +neighbouring datasets `S, S'`, where `D_α(P ‖ Q) = (α - 1)⁻¹ log E_Q[(P/Q)^α]` is the Rényi +divergence. The paper shows that RDP shares the basic properties of differential privacy +(post-processing, adaptive composition with additive budgets, group privacy), that pure `ε`-DP is +`(∞, ε)`-RDP, and that `(α, ε)`-RDP implies `(ε + log (1/δ) / (α - 1), δ)`-DP. + +**Scope.** Except for randomized response, everything is proved for **general measurable output +spaces**: mechanisms are Markov kernels `M : Kernel 𝒟 Y`, the Rényi divergence is Statlib's +`renyiDiv` (in nats, with values in `ℝ≥0∞`), `(ε, δ)`-DP is `IsEpsDeltaDP` from +`Statlib.Privacy.DP.Basic` (pure `ε`-DP is `δ = 0`) and the neighbouring relation `Adj` is +arbitrary (symmetric where stated). Adaptive composition assumes +`CountableOrCountablyGenerated Y Z` (the first output alphabet is countable or the second one is +countably generated, e.g. standard Borel), which is what disintegrating the joint divergence +needs. Randomized response is on `Bool`, and the point-mass formula for the divergence is on +finite alphabets. + +## Main definitions + +* `IsRenyiDP M Adj α ε`: `(α, ε)`-Rényi differential privacy (Definition 4). +* `RenyiDP.randomizedResponseOfProb p`: randomized response with parameter `p` (Section VI.A). + +## Main statements (paper result → Lean name) + +* Definition 4 → `IsRenyiDP`. +* Post-processing (Section III) → `IsRenyiDP.comp` (Markov kernels), `IsRenyiDP.map`. +* Proposition 1 (adaptive sequential composition) → `IsRenyiDP.compProd`, for `α > 1`; the + nonadaptive case, for every `α ≥ 0`, is `IsRenyiDP.prod`. +* Proposition 2 (group privacy) → `IsRenyiDP.renyiDiv_le_of_isKNeighbour` (inputs joined by + `2 ^ c` neighbouring steps) and `IsRenyiDP.comap` (pre-processing by a `2 ^ c`-stable map). +* Proposition 3 (from RDP to `(ε, δ)`-DP) → `IsRenyiDP.isEpsDeltaDP`, and the stronger + `P[M(S) ∈ E] ≤ max (e^{ε'} P[M(S') ∈ E], δ)` proved in the paper → + `IsRenyiDP.measureReal_le_max`. +* Section IV, "`ε`-DP is `(∞, ε)`-RDP, hence `(α, ε)`-RDP for all finite `α`" → + `IsEpsDeltaDP.isRenyiDP` (every finite order `α ≥ 0`), and the characterization + `isEpsDeltaDP_zero_iff_forall_isRenyiDP` (`ε`-DP iff `(α, ε)`-RDP for every `α > 1`). +* Proposition 5 and Table II (randomized response, orders `α > 1`) → + `isRenyiDP_randomizedResponseOfProb_comap`, with the exact divergence + `RenyiDP.renyiDiv_randomizedResponseOfProb`. +* Proposition 9 (monotonicity in the order) → Statlib's `renyiDiv_le_renyiDiv_of_le`; for + mechanisms, `IsRenyiDP.of_order_le`. +* Proposition 10 (probability preservation), in its **finite-divergence, bounded-budget** + version (`D_α(P ‖ Q) ≤ c` with `c` real) → `RenyiDP.measure_le_rpow_of_renyiDiv_le`; for + mechanisms (Table I, change in the probability of an outcome) `IsRenyiDP.measureReal_le_rpow` + and `IsRenyiDP.le_measureReal`. +* Proposition 11 (weak triangle inequality) → `RenyiDP.renyiDiv_le_mul_renyiDiv_add_renyiDiv`; + Corollary 4, item 1 → `RenyiDP.renyiDiv_le_mul_renyiDiv_two_mul_add`. +* Definition 3 with point masses on a finite alphabet → `RenyiDP.renyiDiv_eq_ofReal_log_sum`. + +## Implementation notes + +* The order is a real number `α` and the definition is stated for every `α`; the paper uses + `α > 1`, extended by continuity to `α = 1` (the Kullback-Leibler divergence, which is also what + `renyiDiv 1` is) and to `α = ∞`, which is pure DP and is handled through + `isEpsDeltaDP_zero_iff_forall_isRenyiDP` (Statlib has no Rényi divergence of order `∞`). +* The privacy parameter is `ε : ℝ≥0`. The divergence takes values in `[0, ∞]`, a parameter + `ε = ∞` would be vacuous, and with `ℝ≥0` budgets add in `ℝ≥0` and coerce both to `ℝ≥0∞` (to + compare with `renyiDiv`) and to `ℝ` (for `IsEpsDeltaDP`) without side conditions. +* The proofs go through two characterizations, for probability measures and `α > 1`, of + `D_α(P ‖ Q) ≤ c`: by the Hellinger integral, `P ≪ Q ∧ ∫⁻ (∂P/∂Q) ^ α ∂Q ≤ e^{(α - 1) c}` + (`RenyiDP.renyiDiv_le_ofReal_iff_lintegral`), and by the Hellinger divergence + (`RenyiDP.renyiDiv_le_ofReal_iff_hellingerDiv`); the latter is affine in the joint law and + gives adaptive composition from Statlib's chain rule `renyiDiv_compProd_eq_add_condRenyiDiv`. +* In Proposition 1 the second mechanism is a kernel `N : Kernel (𝒟 × Y) Z` (Mathlib's convention + for `M ⊗ₖ N`), and its privacy for a fixed first output `y` is that of `N.sectL y`. +* Group privacy uses `IsKNeighbour` from `Statlib.Privacy.DP.Basic` (chains in which consecutive + datasets are neighbouring or equal). + +## Notes on the paper + +* Section III, paragraph *Group privacy* (arXiv v3): the definition of a `c`-stable + transformation, read literally, has the implication reversed ("`g(A)` and `g(B)` are adjacent + in `D′` implies that there exists a sequence ... `D_0 = A, ..., D_c = B` ... adjacent in `D`", + for `g : D → D′`), whereas the statement and proof of Proposition 2 use the standard notion of + McSherry for `g : D′ → D`: adjacent inputs `D, D′` are mapped to `g(D), g(D′)` joined by a + chain of `c` adjacent steps. `IsRenyiDP.comap` uses the latter; Proposition 2 is true for it. +* Proof of Proposition 2: the middle terms `D_α(h(D) ‖ h(A))` and `D_{α-1}(h(A) ‖ h(D′))` should + read `f(A)` instead of `h(A)`, since `A` is an input of `f`, not of `h = f ∘ g`. The argument + is otherwise correct. + +## Not formalized + +* The Bayesian interpretation (inequality (1) and the Bayes-factor rows of Table I), + Proposition 8 (nonnegativity; Statlib's `renyiDiv` is `ℝ≥0∞`-valued), Corollary 4, items 2–3 + (they involve `D_∞`) and item 4 (a finite-order `β` statement, omitted separately), adaptive + composition at `α = 1`. +* The extended-valued form of Proposition 10 (`D_α(P ‖ Q) = ∞` allowed, with infinite-divergence + and zero-probability events treated separately). +* Lemma 1, Proposition 4 and Corollary 1 (advanced composition), Proposition 5 at `α = 1`, + Proposition 6 and Corollary 2 (Laplace mechanism), Proposition 7 and Corollary 3 (Gaussian + mechanism), the differential-privacy column of Table II (e.g. randomized response with + parameter `0 < p < 1` is `|log (p / (1 - p))|`-DP), and the numerical comparisons of + Sections VI–VII. +-/ + +@[expose] public section + +open MeasureTheory ProbabilityTheory Real Set Filter +open scoped ENNReal NNReal Topology + +namespace InformationTheory + +variable {𝒟 Y Z : Type*} {m𝒟 : MeasurableSpace 𝒟} {mY : MeasurableSpace Y} + {mZ : MeasurableSpace Z} + +/-! ### The Rényi divergence of an order `α > 1` between probability measures -/ + +namespace RenyiDP + +variable {P Q : Measure Y} [IsProbabilityMeasure P] [IsProbabilityMeasure Q] {α c : ℝ} + +/-- For probability measures and `α > 1`, `D_α(P ‖ Q) ≤ c` iff the Hellinger divergence is finite +and the Hellinger integral `∫ (∂P/∂Q) ^ α ∂Q` is at most `exp ((α - 1) c)`. -/ +lemma renyiDiv_le_ofReal_iff_integral (hα : 1 < α) (hc : 0 ≤ c) : + renyiDiv α P Q ≤ ENNReal.ofReal c ↔ + hellingerDiv α P Q ≠ ∞ ∧ ∫ y, ((∂P/∂Q) y).toReal ^ α ∂Q ≤ exp ((α - 1) * c) := by + rw [renyiDiv_of_one_lt hα] + split_ifs with h + · simp [h] + · have hR : renyiDiv α P Q ≠ ∞ := by + rwa [ne_eq, renyiDiv_eq_top_iff_hellingerDiv_eq_top_of_one_lt hα] + have hpos := integral_rpow_rnDeriv_pos_of_renyiDiv_ne_top (by linarith) hR + simp only [measure_univ, ENNReal.toReal_one, Real.log_one, mul_zero, add_zero, sub_zero] + rw [ENNReal.ofReal_le_ofReal_iff hc, inv_mul_le_iff₀ (by linarith), + Real.log_le_iff_le_exp hpos] + simp [h] + +/-- For probability measures and `α > 1`, `D_α(P ‖ Q) ≤ c` iff `P ≪ Q` and the Hellinger +integral `∫⁻ (∂P/∂Q) ^ α ∂Q` is at most `exp ((α - 1) c)`. -/ +lemma renyiDiv_le_ofReal_iff_lintegral (hα : 1 < α) (hc : 0 ≤ c) : + renyiDiv α P Q ≤ ENNReal.ofReal c ↔ + P ≪ Q ∧ ∫⁻ y, (∂P/∂Q) y ^ α ∂Q ≤ ENNReal.ofReal (exp ((α - 1) * c)) := by + rw [renyiDiv_le_ofReal_iff_integral hα hc, hellingerDiv_ne_top_iff_of_one_lt hα, + integrable_rpow_rnDeriv_iff_lintegral_ne_top (by linarith), + integral_rpow_rnDeriv_eq_toReal_lintegral (by linarith)] + constructor + · rintro ⟨⟨hfin, hac⟩, hle⟩ + exact ⟨hac, (ENNReal.le_ofReal_iff_toReal_le hfin (exp_pos _).le).2 hle⟩ + · rintro ⟨hac, hle⟩ + have hfin : ∫⁻ y, (∂P/∂Q) y ^ α ∂Q ≠ ∞ := ne_top_of_le_ne_top ENNReal.ofReal_ne_top hle + exact ⟨⟨hfin, hac⟩, (ENNReal.le_ofReal_iff_toReal_le hfin (exp_pos _).le).1 hle⟩ + +/-- For probability measures and `α > 1`, `D_α(P ‖ Q) ≤ c` iff the Hellinger divergence of +order `α` is at most `(exp ((α - 1) c) - 1) / (α - 1)`. -/ +lemma renyiDiv_le_ofReal_iff_hellingerDiv (hα : 1 < α) (hc : 0 ≤ c) : + renyiDiv α P Q ≤ ENNReal.ofReal c ↔ + hellingerDiv α P Q ≤ ENNReal.ofReal ((exp ((α - 1) * c) - 1) / (α - 1)) := by + rw [renyiDiv_le_ofReal_iff_integral hα hc] + by_cases h : hellingerDiv α P Q = ∞ + · simp [h] + have hI := mul_hellingerDiv_add_meas_eq_integral (μ := P) (ν := Q) (by linarith) hα.ne' h + simp only [measure_univ, ENNReal.toReal_one, mul_one] at hI + have hα' : 0 < α - 1 := by linarith + have h1 : 1 ≤ exp ((α - 1) * c) := one_le_exp (by positivity) + rw [ENNReal.le_ofReal_iff_toReal_le h (div_nonneg (by linarith) hα'.le), ← hI, + le_div_iff₀ hα'] + simp only [ne_eq, h, not_false_eq_true, true_and] + constructor <;> intro h' <;> linarith + +/-- If `P(s) ≤ e^c Q(s)` for every set `s`, then `D_α(P ‖ Q) ≤ c` for every `α > 1`. -/ +lemma renyiDiv_le_of_forall_le (hα : 1 < α) (hc : 0 ≤ c) + (h : ∀ s, P s ≤ ENNReal.ofReal (exp c) * Q s) : renyiDiv α P Q ≤ ENNReal.ofReal c := by + set C := ENNReal.ofReal (exp c) + have hac : P ≪ Q := fun s hs ↦ le_antisymm ((h s).trans (by simp [hs])) zero_le + have hle : ∀ᵐ y ∂Q, (∂P/∂Q) y ≤ C := + ae_le_of_forall_setLIntegral_le_of_sigmaFinite (Measure.measurable_rnDeriv P Q) + fun s _ _ ↦ by rw [Measure.setLIntegral_rnDeriv hac, setLIntegral_const]; exact h s + refine (renyiDiv_le_ofReal_iff_lintegral hα hc).2 ⟨hac, ?_⟩ + calc ∫⁻ y, (∂P/∂Q) y ^ α ∂Q ≤ ∫⁻ y, C ^ (α - 1) * (∂P/∂Q) y ∂Q := by + refine lintegral_mono_ae ?_ + filter_upwards [hle, Measure.rnDeriv_lt_top P Q] with y hy hy' + rcases eq_or_ne ((∂P/∂Q) y) 0 with h0 | h0 + · rw [h0, ENNReal.zero_rpow_of_pos (by linarith), mul_zero] + calc (∂P/∂Q) y ^ α = (∂P/∂Q) y ^ (α - 1) * (∂P/∂Q) y := by + conv_lhs => rw [show α = (α - 1) + 1 by ring] + rw [ENNReal.rpow_add _ _ h0 hy'.ne, ENNReal.rpow_one] + _ ≤ C ^ (α - 1) * (∂P/∂Q) y := by gcongr + _ = C ^ (α - 1) := by + rw [lintegral_const_mul _ (Measure.measurable_rnDeriv _ _), Measure.lintegral_rnDeriv hac, + measure_univ, mul_one] + _ = ENNReal.ofReal (exp ((α - 1) * c)) := by + rw [ENNReal.ofReal_rpow_of_nonneg (exp_pos _).le (by linarith), ← Real.exp_mul, mul_comm] + +/-- **Proposition 10** of Mironov (probability preservation), **finite-divergence, +bounded-budget version**: for probability measures, `α > 1` and `D_α(P ‖ Q) ≤ c` with `c` a real +number, every set `s` satisfies `P(s) ≤ (e^c Q(s)) ^ ((α - 1) / α)`. This is not the complete +extended-valued statement (which would also allow `D_α(P ‖ Q) = ∞`). The paper assumes that `P` +and `Q` have the same support; that assumption is not needed here, because +`D_α(P ‖ Q) ≤ c < ∞` already implies `P ≪ Q` (common support alone would not make the divergence +finite). -/ +theorem measure_le_rpow_of_renyiDiv_le (hα : 1 < α) (hc : 0 ≤ c) + (h : renyiDiv α P Q ≤ ENNReal.ofReal c) (s : Set Y) : + P s ≤ (ENNReal.ofReal (exp c) * Q s) ^ ((α - 1) / α) := by + obtain ⟨hac, hle⟩ := (renyiDiv_le_ofReal_iff_lintegral hα hc).1 h + have hα0 : 0 < α := by linarith + have hpq := Real.HolderConjugate.conjExponent hα + calc P s = ∫⁻ y in s, ((P.rnDeriv Q * fun _ ↦ 1 : Y → ℝ≥0∞)) y ∂Q := by + simp [Measure.setLIntegral_rnDeriv hac] + _ ≤ (∫⁻ y in s, (∂P/∂Q) y ^ α ∂Q) ^ (1 / α) + * (∫⁻ y in s, (fun _ ↦ (1 : ℝ≥0∞)) y ^ Real.conjExponent α ∂Q) + ^ (1 / Real.conjExponent α) := + ENNReal.lintegral_mul_le_Lp_mul_Lq _ hpq (Measure.measurable_rnDeriv _ _).aemeasurable + aemeasurable_const + _ ≤ ENNReal.ofReal (exp ((α - 1) * c)) ^ (1 / α) * Q s ^ ((α - 1) / α) := by + gcongr + · exact (setLIntegral_le_lintegral _ _).trans hle + · rw [Real.conjExponent, one_div_div] + simp + _ = (ENNReal.ofReal (exp c) * Q s) ^ ((α - 1) / α) := by + rw [ENNReal.mul_rpow_of_nonneg _ _ (div_nonneg (by linarith) hα0.le), + ENNReal.ofReal_rpow_of_nonneg (exp_pos _).le (by positivity), + ENNReal.ofReal_rpow_of_nonneg (exp_pos _).le (div_nonneg (by linarith) hα0.le), + ← Real.exp_mul, ← Real.exp_mul] + congr 3 + field_simp + +/-- The real inequality behind Proposition 3 of Mironov: for `α > 1`, `δ > 0` and `x ≥ 0`, +`x ^ ((α - 1) / α) ≤ max (δ ^ (-(α - 1)⁻¹) x, δ)`. -/ +lemma rpow_le_max (hα : 1 < α) {δ x : ℝ} (hδ : 0 < δ) (hx : 0 ≤ x) : + x ^ ((α - 1) / α) ≤ max (δ ^ (-(α - 1)⁻¹) * x) δ := by + have hα0 : 0 < α := by linarith + have hα1 : 0 < α - 1 := by linarith + rcases le_or_gt (x ^ ((α - 1) / α)) δ with h | h + · exact le_max_of_le_right h + refine le_max_of_le_left ?_ + have hx0 : 0 < x := by + rcases hx.eq_or_lt with rfl | hx0 + · rw [Real.zero_rpow (by positivity)] at h + linarith + · exact hx0 + calc x ^ ((α - 1) / α) = x ^ (1 + (α - 1) / α * (-(α - 1)⁻¹)) := by + congr 1 + field_simp + ring + _ = x * (x ^ ((α - 1) / α)) ^ (-(α - 1)⁻¹) := by + rw [Real.rpow_add hx0, Real.rpow_one, Real.rpow_mul hx] + _ ≤ x * δ ^ (-(α - 1)⁻¹) := mul_le_mul_of_nonneg_left + (Real.rpow_le_rpow_of_nonpos hδ h.le (neg_nonpos.2 (inv_nonneg.2 hα1.le))) hx + _ = δ ^ (-(α - 1)⁻¹) * x := mul_comm _ _ + +/-- **Proposition 11** of Mironov (weak triangle inequality): for probability measures `P, Q, R`, +`α > 1` and Hölder conjugate exponents `p, q > 1`, +`D_α(P ‖ Q) ≤ (α - 1 / p) / (α - 1) · D_{pα}(P ‖ R) + D_{q (α - 1 / p)}(R ‖ Q)`. -/ +theorem renyiDiv_le_mul_renyiDiv_add_renyiDiv {R : Measure Y} [IsProbabilityMeasure R] {p q : ℝ} + (hα : 1 < α) (hpq : p.HolderConjugate q) : + renyiDiv α P Q ≤ ENNReal.ofReal ((α - 1 / p) / (α - 1)) * renyiDiv (p * α) P R + + renyiDiv (q * (α - 1 / p)) R Q := by + have hp : 1 < p := hpq.lt + have hq_eq : q = p / (p - 1) := hpq.conjugate_eq + have hp0 : 0 < 1 / p := one_div_pos.2 (by linarith) + have hq0 : 0 < 1 / q := one_div_pos.2 (by linarith [hpq.symm.lt]) + have hα1 : 0 < α - 1 := by linarith + have hαp : 0 < α - 1 / p := by + have : 1 / p < 1 := (div_lt_one (by linarith)).2 hp + linarith + have hcoef : 0 < (α - 1 / p) / (α - 1) := div_pos hαp hα1 + have hpα : 1 < p * α := by nlinarith + have hqα' : q * (α - 1 / p) = q * (α - 1) + 1 := by + rw [hq_eq] + field_simp + ring + have hqα : 1 < q * (α - 1 / p) := by + rw [hqα'] + nlinarith [hpq.symm.lt] + set a := renyiDiv (p * α) P R + set b := renyiDiv (q * (α - 1 / p)) R Q + rcases eq_or_ne a ∞ with ha | ha + · rw [ha, ENNReal.mul_top (by simpa using hcoef), top_add] + exact le_top + rcases eq_or_ne b ∞ with hb | hb + · rw [hb, add_top] + exact le_top + obtain ⟨hPR, hIa⟩ := (renyiDiv_le_ofReal_iff_lintegral hpα ENNReal.toReal_nonneg).1 + (ENNReal.ofReal_toReal ha).ge + obtain ⟨hRQ, hIb⟩ := (renyiDiv_le_ofReal_iff_lintegral hqα ENNReal.toReal_nonneg).1 + (ENNReal.ofReal_toReal hb).ge + rw [← ENNReal.ofReal_toReal ha, ← ENNReal.ofReal_toReal hb, ← ENNReal.ofReal_mul hcoef.le, + ← ENNReal.ofReal_add (by positivity) ENNReal.toReal_nonneg] + refine (renyiDiv_le_ofReal_iff_lintegral hα (by positivity)).2 ⟨hPR.trans hRQ, ?_⟩ + have hf := Measure.measurable_rnDeriv P R + have hg := Measure.measurable_rnDeriv R Q + calc ∫⁻ y, (∂P/∂Q) y ^ α ∂Q + = ∫⁻ y, ((fun y ↦ (∂P/∂R) y ^ α * (∂R/∂Q) y ^ (1 / p)) + * fun y ↦ (∂R/∂Q) y ^ (α - 1 / p)) y ∂Q := by + refine lintegral_congr_ae ?_ + filter_upwards [Measure.rnDeriv_mul_rnDeriv hPR (κ := Q)] with y hy + rw [← hy, Pi.mul_apply, Pi.mul_apply, ENNReal.mul_rpow_of_nonneg _ _ (by linarith), + mul_assoc, ← ENNReal.rpow_add_of_nonneg _ _ hp0.le hαp.le, add_sub_cancel] + _ ≤ (∫⁻ y, ((∂P/∂R) y ^ α * (∂R/∂Q) y ^ (1 / p)) ^ p ∂Q) ^ (1 / p) + * (∫⁻ y, ((∂R/∂Q) y ^ (α - 1 / p)) ^ q ∂Q) ^ (1 / q) := + ENNReal.lintegral_mul_le_Lp_mul_Lq Q hpq + ((hf.pow_const _).mul (hg.pow_const _)).aemeasurable (hg.pow_const _).aemeasurable + _ = (∫⁻ y, (∂P/∂R) y ^ (p * α) ∂R) ^ (1 / p) + * (∫⁻ y, (∂R/∂Q) y ^ (q * (α - 1 / p)) ∂Q) ^ (1 / q) := by + congr 2 + · rw [← lintegral_rnDeriv_mul hRQ (hf.pow_const _).aemeasurable] + refine lintegral_congr fun y ↦ ?_ + rw [ENNReal.mul_rpow_of_nonneg _ _ (by linarith), ← ENNReal.rpow_mul, ← ENNReal.rpow_mul, + one_div_mul_cancel (by linarith), ENNReal.rpow_one, mul_comm α p, mul_comm] + · refine lintegral_congr fun y ↦ ?_ + rw [← ENNReal.rpow_mul, mul_comm] + _ ≤ ENNReal.ofReal (exp ((p * α - 1) * a.toReal)) ^ (1 / p) + * ENNReal.ofReal (exp ((q * (α - 1 / p) - 1) * b.toReal)) ^ (1 / q) := by + gcongr + _ = ENNReal.ofReal (exp ((α - 1) * ((α - 1 / p) / (α - 1) * a.toReal + b.toReal))) := by + rw [ENNReal.ofReal_rpow_of_nonneg (exp_pos _).le hp0.le, + ENNReal.ofReal_rpow_of_nonneg (exp_pos _).le hq0.le, ← Real.exp_mul, ← Real.exp_mul, + ← ENNReal.ofReal_mul (exp_pos _).le, ← Real.exp_add, hqα', hq_eq] + congr 2 + field_simp + ring + +/-- **Corollary 4.1** of Mironov (the weak triangle inequality with `p = q = 2`): for probability +measures and `α > 1`, `D_α(P ‖ Q) ≤ (α - 1/2) / (α - 1) · D_{2α}(P ‖ R) + D_{2α - 1}(R ‖ Q)`. The +paper assumes a common support; this is not needed. -/ +theorem renyiDiv_le_mul_renyiDiv_two_mul_add {R : Measure Y} [IsProbabilityMeasure R] + (hα : 1 < α) : + renyiDiv α P Q ≤ ENNReal.ofReal ((α - 1 / 2) / (α - 1)) * renyiDiv (2 * α) P R + + renyiDiv (2 * α - 1) R Q := by + have := renyiDiv_le_mul_renyiDiv_add_renyiDiv (P := P) (Q := Q) (R := R) hα + Real.HolderConjugate.two_two + rwa [show (2 : ℝ) * (α - 1 / 2) = 2 * α - 1 by ring] at this + +end RenyiDP + +/-! ### Rényi differential privacy -/ + +/-- **Rényi differential privacy** (Definition 4 of Mironov): a mechanism `M`, given as a kernel +from datasets to outputs, is `(α, ε)`-RDP for the neighbouring relation `Adj` if +`D_α(M(S) ‖ M(S')) ≤ ε` for all neighbouring `S, S'`. + +The divergence is Statlib's `renyiDiv`, in nats with values in `ℝ≥0∞`. The paper uses orders +`α > 1`, extended to `α = 1` by continuity (the Kullback-Leibler divergence, which is also what +`renyiDiv 1` is) and to `α = ∞` (pure differential privacy, see +`isEpsDeltaDP_zero_iff_forall_isRenyiDP`); the definition is stated for every real order and the +results below assume the range of orders they need. The definition is only meaningful for +`α ≥ 0`: for `α < 0` Statlib's Hellinger divergence vanishes and the condition carries no +information (the paper does not use orders below `1`, Remark 2). It is also only meaningful for +Markov kernels, which every result below assumes. For kernels whose output measures are nonzero +and finite, `renyiDiv` compares the normalized measures `M(S) / M(S)(univ)`; at zero measures it +takes degenerate values, and for non-finite output measures even this interpretation fails +(inverting an infinite mass gives `0`, and `renyiDiv` need not equal a divergence of normalized +measures), so such kernels are excluded. The privacy +parameter is `ε : ℝ≥0`, since the divergence is nonnegative and a parameter `ε = ∞` would be +vacuous. -/ +def IsRenyiDP (M : Kernel 𝒟 Y) (Adj : 𝒟 → 𝒟 → Prop) (α : ℝ) (ε : ℝ≥0) : Prop := + ∀ ⦃S S'⦄, Adj S S' → renyiDiv α (M S) (M S') ≤ ε + +variable {M : Kernel 𝒟 Y} {Adj : 𝒟 → 𝒟 → Prop} {α β : ℝ} {ε ε₁ ε₂ : ℝ≥0} + +/-- An `(α, ε₁)`-RDP mechanism is `(α, ε₂)`-RDP for every `ε₂ ≥ ε₁`. -/ +lemma IsRenyiDP.mono (h : IsRenyiDP M Adj α ε₁) (hε : ε₁ ≤ ε₂) : IsRenyiDP M Adj α ε₂ := + fun _ _ hS ↦ (h hS).trans (by exact_mod_cast hε) + +/-- **Proposition 9** (monotonicity in the order) for mechanisms: an `(β, ε)`-RDP mechanism is +`(α, ε)`-RDP for every `0 ≤ α ≤ β`. The paper states the monotonicity of the divergence for +`1 ≤ α < β`; it is Statlib's `renyiDiv_le_renyiDiv_of_le`, valid for `0 ≤ α ≤ β`. -/ +theorem IsRenyiDP.of_order_le [IsMarkovKernel M] (h : IsRenyiDP M Adj β ε) (hα : 0 ≤ α) + (hαβ : α ≤ β) : IsRenyiDP M Adj α ε := + fun _ _ hS ↦ (renyiDiv_le_renyiDiv_of_le hα hαβ).trans (h hS) + +/-- **Post-processing** (Section III of Mironov): if `M` is `(α, ε)`-RDP, so is `Proc ∘ M` for +every Markov kernel `Proc`, for every order `α ≥ 0`. -/ +theorem IsRenyiDP.comp [IsMarkovKernel M] (h : IsRenyiDP M Adj α ε) (hα : 0 ≤ α) + (Proc : Kernel Y Z) [IsMarkovKernel Proc] : IsRenyiDP (Proc ∘ₖ M) Adj α ε := + fun S S' hS ↦ by + rw [Kernel.comp_apply, Kernel.comp_apply] + exact (renyiDiv_comp_right_le hα _ _ Proc).trans (h hS) + +/-- **Post-processing** by a measurable map: if `M` is `(α, ε)`-RDP, so is `g ∘ M`. -/ +theorem IsRenyiDP.map [IsMarkovKernel M] (h : IsRenyiDP M Adj α ε) (hα : 0 ≤ α) {g : Y → Z} + (hg : Measurable g) : IsRenyiDP (M.map g) Adj α ε := + fun S S' hS ↦ by + rw [Kernel.map_apply _ hg, Kernel.map_apply _ hg] + exact (renyiDiv_map_le hα _ _ hg).trans (h hS) + +/-- **Composition of independent mechanisms** (Proposition 1, nonadaptive case): if `M₁` is +`(α, ε₁)`-RDP and `M₂` is `(α, ε₂)`-RDP, then releasing both outputs, computed with independent +randomness, is `(α, ε₁ + ε₂)`-RDP, for every order `α ≥ 0`. -/ +theorem IsRenyiDP.prod {M₁ : Kernel 𝒟 Y} {M₂ : Kernel 𝒟 Z} [IsMarkovKernel M₁] + [IsMarkovKernel M₂] (h₁ : IsRenyiDP M₁ Adj α ε₁) (h₂ : IsRenyiDP M₂ Adj α ε₂) + (hα : 0 ≤ α) : IsRenyiDP (M₁ ×ₖ M₂) Adj α (ε₁ + ε₂) := + fun S S' hS ↦ by + rw [Kernel.prod_apply, Kernel.prod_apply, renyiDiv_prod hα, ENNReal.coe_add] + exact add_le_add (h₁ hS) (h₂ hS) + +namespace RenyiDP + +/-- For `α > 1`, the conditional Rényi divergence `D_α(ρ ⊗ κ ‖ ρ ⊗ η)` with respect to a +probability measure `ρ` is at most any uniform bound on `D_α(κ y ‖ η y)`. -/ +lemma condRenyiDiv_le [MeasurableSpace.CountableOrCountablyGenerated Y Z] {κ η : Kernel Y Z} + [IsMarkovKernel κ] [IsMarkovKernel η] {ρ : Measure Y} [IsProbabilityMeasure ρ] (hα : 1 < α) + (h : ∀ y, renyiDiv α (κ y) (η y) ≤ ε) : condRenyiDiv α κ η ρ ≤ ε := by + have hε : (ε : ℝ≥0∞) = ENNReal.ofReal (ε : ℝ) := by simp + rw [condRenyiDiv, hε, renyiDiv_le_ofReal_iff_hellingerDiv hα ε.coe_nonneg] + change condHellingerDiv α κ η ρ ≤ _ + rw [condHellingerDiv_eq_lintegral] + calc ∫⁻ y, hellingerDiv α (κ y) (η y) ∂ρ + ≤ ∫⁻ _, ENNReal.ofReal ((exp ((α - 1) * ε) - 1) / (α - 1)) ∂ρ := by + refine lintegral_mono fun y ↦ ?_ + rw [← renyiDiv_le_ofReal_iff_hellingerDiv hα ε.coe_nonneg, ← hε] + exact h y + _ = _ := by simp + +end RenyiDP + +/-- **Proposition 1** of Mironov (adaptive sequential composition): if `M` is `(α, ε₁)`-RDP and, +for every first output `y`, the mechanism `S ↦ N(S, y)` is `(α, ε₂)`-RDP, then the mechanism +`S ↦ (X, Y)` with `X ∼ M(S)` and `Y ∼ N(S, X)` (the kernel `M ⊗ₖ N`) is `(α, ε₁ + ε₂)`-RDP. +Stated for `α > 1` (the paper's range; its case `α = 1` is by continuity), with the measurability +assumption `CountableOrCountablyGenerated Y Z` (`Y` countable or `Z` countably generated, e.g. a +standard Borel space) needed to disintegrate the joint divergence. -/ +theorem IsRenyiDP.compProd [MeasurableSpace.CountableOrCountablyGenerated Y Z] + {N : Kernel (𝒟 × Y) Z} [IsMarkovKernel M] [IsMarkovKernel N] (hM : IsRenyiDP M Adj α ε₁) + (hN : ∀ y, IsRenyiDP (N.sectL y) Adj α ε₂) (hα : 1 < α) : + IsRenyiDP (M ⊗ₖ N) Adj α (ε₁ + ε₂) := by + intro S S' hS + rw [Kernel.compProd_apply_eq_compProd_sectR, Kernel.compProd_apply_eq_compProd_sectR, + renyiDiv_compProd_eq_add_condRenyiDiv (by linarith), ENNReal.coe_add] + refine add_le_add (hM hS) ?_ + have hR : renyiDiv α (M S) (M S') ≠ ∞ := ne_top_of_le_ne_top ENNReal.coe_ne_top (hM hS) + have := isProbabilityMeasure_renyiMeasure_of_renyiDiv_ne_top (by linarith) hR + exact RenyiDP.condRenyiDiv_le hα fun y ↦ hN y hS + +/-! ### Rényi and `(ε, δ)`-differential privacy -/ + +/-- **Pure differential privacy implies RDP** (Section IV of Mironov: `ε`-DP is `(∞, ε)`-RDP, +hence `(α, ε)`-RDP for every finite `α` by monotonicity): an `ε`-DP mechanism is `(α, ε)`-RDP +for every finite order `α ≥ 0`. -/ +theorem IsEpsDeltaDP.isRenyiDP [IsMarkovKernel M] (h : IsEpsDeltaDP M Adj ε 0) (hα : 0 ≤ α) : + IsRenyiDP M Adj α ε := by + have key (β : ℝ) (hβ : 1 < β) : IsRenyiDP M Adj β ε := by + intro S S' hS + have hε : (ε : ℝ≥0∞) = ENNReal.ofReal (ε : ℝ) := by simp + rw [hε] + refine RenyiDP.renyiDiv_le_of_forall_le hβ ε.coe_nonneg fun s ↦ ?_ + rw [← ofReal_measureReal, ← ofReal_measureReal, ← ENNReal.ofReal_mul (exp_pos _).le] + exact ENNReal.ofReal_le_ofReal ((h hS s).trans_eq (add_zero _)) + rcases le_or_gt α 1 with hα1 | hα1 + · exact (key 2 one_lt_two).of_order_le hα (by linarith) + · exact key α hα1 + +/-- The "bad outcomes" guarantee of RDP (Section III and Table I of Mironov, via +Proposition 10): if `M` is `(α, ε)`-RDP with `α > 1`, then for neighbouring `S, S'` and every +event `E`, `P[M(S) ∈ E] ≤ (e^ε P[M(S') ∈ E]) ^ ((α - 1) / α)`. -/ +theorem IsRenyiDP.measureReal_le_rpow [IsMarkovKernel M] (h : IsRenyiDP M Adj α ε) + (hα : 1 < α) ⦃S S' : 𝒟⦄ (hS : Adj S S') (E : Set Y) : + (M S).real E ≤ (exp ε * (M S').real E) ^ ((α - 1) / α) := by + have hR : renyiDiv α (M S) (M S') ≤ ENNReal.ofReal ε := by simpa using h hS + have := ENNReal.toReal_mono (by finiteness) + (RenyiDP.measure_le_rpow_of_renyiDiv_le hα ε.coe_nonneg hR E) + rwa [← ENNReal.toReal_rpow, ENNReal.toReal_mul, ENNReal.toReal_ofReal (exp_pos _).le] at this + +/-- The lower half of the "bad outcomes" guarantee of RDP (Table I of Mironov): for a symmetric +neighbouring relation, if `M` is `(α, ε)`-RDP with `α > 1`, then for neighbouring `S, S'` and every +event `E`, `e^{-ε} P[M(S') ∈ E] ^ (α / (α - 1)) ≤ P[M(S) ∈ E]`. -/ +theorem IsRenyiDP.le_measureReal [IsMarkovKernel M] [Std.Symm Adj] (h : IsRenyiDP M Adj α ε) + (hα : 1 < α) ⦃S S' : 𝒟⦄ (hS : Adj S S') (E : Set Y) : + exp (-ε) * (M S').real E ^ (α / (α - 1)) ≤ (M S).real E := by + have hα1 : 0 < α - 1 := by linarith + have h1 := h.measureReal_le_rpow hα (Std.Symm.symm _ _ hS) E + have h2 : (M S').real E ^ (α / (α - 1)) ≤ exp ε * (M S).real E := by + calc (M S').real E ^ (α / (α - 1)) + ≤ ((exp ε * (M S).real E) ^ ((α - 1) / α)) ^ (α / (α - 1)) := by gcongr + _ = exp ε * (M S).real E := by + rw [← Real.rpow_mul (by positivity), div_mul_div_comm, mul_comm (α - 1), + div_self (by positivity), Real.rpow_one] + rw [Real.exp_neg, inv_mul_le_iff₀ (exp_pos _)] + exact h2 + +/-- **Proposition 3** of Mironov, in the stronger form proved in the paper: if `M` is +`(α, ε)`-RDP with `α > 1`, then for every `δ > 0`, neighbouring `S, S'` and every event `E`, +`P[M(S) ∈ E] ≤ max (e^{ε'} P[M(S') ∈ E], δ)` with `ε' = ε + log (1 / δ) / (α - 1)`. -/ +theorem IsRenyiDP.measureReal_le_max [IsMarkovKernel M] (h : IsRenyiDP M Adj α ε) (hα : 1 < α) + {δ : ℝ} (hδ : 0 < δ) ⦃S S' : 𝒟⦄ (hS : Adj S S') (E : Set Y) : + (M S).real E ≤ max (exp (ε + log (1 / δ) / (α - 1)) * (M S').real E) δ := by + have h3 : δ ^ (-(α - 1)⁻¹) * (exp ε * (M S').real E) + = exp (ε + log (1 / δ) / (α - 1)) * (M S').real E := by + rw [Real.rpow_def_of_pos hδ, ← mul_assoc, ← Real.exp_add, one_div, Real.log_inv] + ring_nf + rw [← h3] + exact (h.measureReal_le_rpow hα hS E).trans (RenyiDP.rpow_le_max hα hδ (by positivity)) + +/-- **Proposition 3** of Mironov (from RDP to `(ε, δ)`-DP): an `(α, ε)`-RDP mechanism with +`α > 1` is `(ε + log (1 / δ) / (α - 1), δ)`-DP for every `δ > 0`. The paper states it for +`0 < δ < 1`. -/ +theorem IsRenyiDP.isEpsDeltaDP [IsMarkovKernel M] (h : IsRenyiDP M Adj α ε) (hα : 1 < α) + {δ : ℝ} (hδ : 0 < δ) : IsEpsDeltaDP M Adj (ε + log (1 / δ) / (α - 1)) δ := + fun _ _ hS E ↦ (h.measureReal_le_max hα hδ hS E).trans + (max_le_add_of_nonneg (by positivity) hδ.le) + +/-- **Pure differential privacy is RDP of order `∞`** (Section IV of Mironov: `ε`-DP coincides +with `(∞, ε)`-RDP, where `D_∞ = lim_{α → ∞} D_α`): a mechanism is `ε`-DP iff it is +`(α, ε)`-RDP for every `α > 1`. -/ +theorem isEpsDeltaDP_zero_iff_forall_isRenyiDP [IsMarkovKernel M] : + IsEpsDeltaDP M Adj ε 0 ↔ ∀ α, 1 < α → IsRenyiDP M Adj α ε := by + refine ⟨fun h α hα ↦ h.isRenyiDP (by linarith), fun h S S' hS E ↦ ?_⟩ + rw [add_zero] + set x := exp ε * (M S').real E + have hx : 0 ≤ x := by positivity + have h1 : Tendsto (fun α : ℝ ↦ (α - 1) / α) atTop (𝓝 1) := by + have : Tendsto (fun α : ℝ ↦ 1 - α⁻¹) atTop (𝓝 (1 - 0)) := + tendsto_const_nhds.sub tendsto_inv_atTop_zero + rw [sub_zero] at this + refine this.congr' ?_ + filter_upwards [eventually_gt_atTop 0] with α hα + field_simp + have hlim : Tendsto (fun α : ℝ ↦ x ^ ((α - 1) / α)) atTop (𝓝 x) := by + rcases hx.eq_or_lt with hx0 | hx0 + · rw [← hx0] + refine tendsto_const_nhds.congr' ?_ + filter_upwards [eventually_gt_atTop 1] with α hα + rw [Real.zero_rpow (div_pos (by linarith) (by linarith)).ne'] + · have := ((Real.continuousAt_const_rpow hx0.ne').tendsto).comp h1 + rwa [Real.rpow_one] at this + exact ge_of_tendsto hlim + ((eventually_gt_atTop 1).mono fun α hα ↦ (h α hα).measureReal_le_rpow hα hS E) + +/-! ### Group privacy -/ + +/-- A chain of `k + m` neighbouring steps splits into a chain of `k` steps followed by a chain of +`m` steps. -/ +private lemma exists_isKNeighbour_of_add {k m : ℕ} {S U : 𝒟} + (h : IsKNeighbour Adj (k + m) S U) : + ∃ T, IsKNeighbour Adj k S T ∧ IsKNeighbour Adj m T U := by + induction m generalizing U with + | zero => exact ⟨U, h, .zero U⟩ + | succ m ih => + cases h with + | succ h' hTU => + obtain ⟨T, h₁, h₂⟩ := ih h' + exact ⟨T, h₁, .succ h₂ hTU⟩ + +/-- **Proposition 2** of Mironov (group privacy), for inputs at distance `2 ^ c`: if `M` is +`(α, ε)`-RDP and `α ≥ 2 ^ (c + 1)`, then `D_{α / 2^c}(M(S) ‖ M(S')) ≤ 3^c ε` whenever `S` and +`S'` are joined by a chain of `2 ^ c` neighbouring (or equal) steps. -/ +theorem IsRenyiDP.renyiDiv_le_of_isKNeighbour [IsMarkovKernel M] (h : IsRenyiDP M Adj α ε) + (c : ℕ) (hα : 2 ^ (c + 1) ≤ α) {S S' : 𝒟} (hSS' : IsKNeighbour Adj (2 ^ c) S S') : + renyiDiv (α / 2 ^ c) (M S) (M S') ≤ 3 ^ c * ε := by + induction c generalizing S S' with + | zero => + simp only [pow_zero, div_one, one_mul] at hSS' ⊢ + cases hSS' with + | succ h₀ hstep => + cases h₀ + rcases hstep with hS | rfl + · exact h hS + · simp + | succ c ih => + have hα' : (2 : ℝ) ^ (c + 1) ≤ α := + le_trans (pow_le_pow_right₀ (by norm_num) (by omega)) hα + rw [pow_succ, mul_two] at hSS' + obtain ⟨T, h₁, h₂⟩ := exists_isKNeighbour_of_add hSS' + set γ := α / 2 ^ (c + 1) with hγ + have hγ2 : 2 ≤ γ := by + rw [hγ, le_div_iff₀ (by positivity), ← pow_succ'] + exact hα + have h2γ : 2 * γ = α / 2 ^ c := by + rw [hγ, pow_succ] + field_simp + have hcoef : ENNReal.ofReal ((γ - 1 / 2) / (γ - 1)) ≤ 2 := by + have hle : (γ - 1 / 2) / (γ - 1) ≤ 2 := by + rw [div_le_iff₀ (by linarith)] + linarith + exact (ENNReal.ofReal_le_ofReal hle).trans_eq (by simp) + calc renyiDiv γ (M S) (M S') + ≤ ENNReal.ofReal ((γ - 1 / 2) / (γ - 1)) * renyiDiv (2 * γ) (M S) (M T) + + renyiDiv (2 * γ - 1) (M T) (M S') := + RenyiDP.renyiDiv_le_mul_renyiDiv_two_mul_add (by linarith) + _ ≤ 2 * (3 ^ c * ε) + 3 ^ c * ε := by + gcongr + · rw [h2γ] + exact ih hα' h₁ + · calc renyiDiv (2 * γ - 1) (M T) (M S') ≤ renyiDiv (2 * γ) (M T) (M S') := + renyiDiv_le_renyiDiv_of_le (by linarith) (by linarith) + _ ≤ 3 ^ c * ε := by + rw [h2γ] + exact ih hα' h₂ + _ = 3 ^ (c + 1) * ε := by ring + +/-- **Proposition 2** of Mironov (group privacy): if `M` is `(α, ε)`-RDP, `g` is `2 ^ c`-stable +(neighbouring inputs `D, D'` are mapped to datasets `g D, g D'` joined by a chain of `2 ^ c` +neighbouring or equal steps) and `α ≥ 2 ^ (c + 1)`, then `M ∘ g` is `(α / 2^c, 3^c ε)`-RDP. -/ +theorem IsRenyiDP.comap [IsMarkovKernel M] (h : IsRenyiDP M Adj α ε) {𝒟' : Type*} + {m𝒟' : MeasurableSpace 𝒟'} {Adj' : 𝒟' → 𝒟' → Prop} {g : 𝒟' → 𝒟} (hg : Measurable g) (c : ℕ) + (hstab : ∀ ⦃D D'⦄, Adj' D D' → IsKNeighbour Adj (2 ^ c) (g D) (g D')) + (hα : 2 ^ (c + 1) ≤ α) : IsRenyiDP (M.comap g hg) Adj' (α / 2 ^ c) (3 ^ c * ε) := + fun D D' hD ↦ by + rw [Kernel.comap_apply, Kernel.comap_apply, ENNReal.coe_mul, ENNReal.coe_pow] + exact h.renyiDiv_le_of_isKNeighbour c hα (hstab hD) + +/-! ### Finite alphabets and randomized response -/ + +namespace RenyiDP + +/-- On a finite alphabet, for probability measures with `P ≪ Q` and `α > 1`, the Rényi divergence +is given by Definition 3 of Mironov with point masses: +`D_α(P ‖ Q) = (α - 1)⁻¹ log ∑_y P(y) ^ α Q(y) ^ (1 - α)`. -/ +theorem renyiDiv_eq_ofReal_log_sum [Fintype Y] [MeasurableSingletonClass Y] {P Q : Measure Y} + [IsProbabilityMeasure P] [IsProbabilityMeasure Q] (hα : 1 < α) (hac : P ≪ Q) : + renyiDiv α P Q + = ENNReal.ofReal ((α - 1)⁻¹ * log (∑ y, P.real {y} ^ α * Q.real {y} ^ (1 - α))) := by + rw [renyiDiv_eq_log_integral_of_one_lt hα Integrable.of_finite hac, + integral_fintype Integrable.of_finite] + simp only [measure_univ, ENNReal.toReal_one, Real.log_one, mul_zero, add_zero, sub_zero, + smul_eq_mul] + congr 3 + refine Finset.sum_congr rfl fun y _ ↦ ?_ + rcases eq_or_ne (Q {y}) 0 with hQ | hQ + · have hP : P {y} = 0 := hac hQ + simp [measureReal_def, hQ, hP, Real.zero_rpow (by linarith : α ≠ 0)] + have hy : P {y} = (∂P/∂Q) y * Q {y} := by + rw [← Measure.setLIntegral_rnDeriv hac, lintegral_singleton] + have hQr : 0 < Q.real {y} := ENNReal.toReal_pos hQ (measure_ne_top _ _) + have hr : ((∂P/∂Q) y).toReal = P.real {y} / Q.real {y} := by + rw [measureReal_def P, hy, ENNReal.toReal_mul, ← measureReal_def, mul_div_assoc, + div_self hQr.ne', mul_one] + rw [hr, Real.div_rpow measureReal_nonneg hQr.le, Real.rpow_sub hQr, Real.rpow_one] + field_simp + +open unitInterval in +/-- **Binary randomized response** with parameter `p` (Section VI.A of Mironov): on input `b`, +output `b` with probability `p` and `!b` with probability `1 - p`. Applied to a predicate `f` on +datasets, `(randomizedResponseOfProb p).comap f hf` is the mechanism `RR_p f` of the paper. It is +parametrized, as in the paper, by the probability `p` of a truthful answer. -/ +noncomputable def randomizedResponseOfProb (p : I) : Kernel Bool Bool := + Kernel.ofFunOfCountable fun b ↦ + ENNReal.ofReal p • Measure.dirac b + ENNReal.ofReal (1 - p) • Measure.dirac (!b) + +open unitInterval + +lemma randomizedResponseOfProb_apply (p : I) (b : Bool) : + randomizedResponseOfProb p b + = ENNReal.ofReal p • Measure.dirac b + ENNReal.ofReal (1 - p) • Measure.dirac (!b) := + rfl + +lemma randomizedResponseOfProb_apply_singleton (p : I) (b c : Bool) : + randomizedResponseOfProb p b {c} + = if c = b then ENNReal.ofReal p else ENNReal.ofReal (1 - p) := by + cases b <;> cases c <;> simp [randomizedResponseOfProb_apply] + +instance (p : I) : IsMarkovKernel (randomizedResponseOfProb p) := by + refine ⟨fun b ↦ ⟨?_⟩⟩ + rw [randomizedResponseOfProb_apply, ← ENNReal.ofReal_one] + simp only [FunLike.coe_add, FunLike.coe_smul, Pi.add_apply, Pi.smul_apply, measure_univ, + smul_eq_mul, mul_one] + rw [← ENNReal.ofReal_add p.2.1 (sub_nonneg.2 p.2.2), add_sub_cancel] + +/-- **Proposition 5** of Mironov (randomized response, order `α > 1`), the exact value behind +Table II: for `p ∈ (0, 1)`, +`D_α(RR_p(b) ‖ RR_p(!b)) = (α - 1)⁻¹ log (p ^ α (1 - p) ^ (1 - α) + (1 - p) ^ α p ^ (1 - α))`. -/ +theorem renyiDiv_randomizedResponseOfProb {p : I} (hp0 : 0 < (p : ℝ)) (hp1 : (p : ℝ) < 1) + (hα : 1 < α) (b : Bool) : + renyiDiv α (randomizedResponseOfProb p b) (randomizedResponseOfProb p (!b)) + = ENNReal.ofReal ((α - 1)⁻¹ + * log ((p : ℝ) ^ α * (1 - p) ^ (1 - α) + (1 - p) ^ α * (p : ℝ) ^ (1 - α))) := by + have hreal (b c : Bool) : (randomizedResponseOfProb p b).real {c} + = if c = b then (p : ℝ) else 1 - p := by + rw [measureReal_def, randomizedResponseOfProb_apply_singleton] + split_ifs + · exact ENNReal.toReal_ofReal hp0.le + · exact ENNReal.toReal_ofReal (by linarith) + have hac : randomizedResponseOfProb p b ≪ randomizedResponseOfProb p (!b) := fun s hs ↦ by + rcases s.eq_empty_or_nonempty with rfl | ⟨c, hc⟩ + · simp + · exfalso + have h1 := measure_mono (μ := randomizedResponseOfProb p (!b)) + (Set.singleton_subset_iff.2 hc) + rw [hs, randomizedResponseOfProb_apply_singleton] at h1 + split_ifs at h1 + · simp only [nonpos_iff_eq_zero, ENNReal.ofReal_eq_zero] at h1 + linarith + · simp only [nonpos_iff_eq_zero, ENNReal.ofReal_eq_zero] at h1 + linarith + rw [renyiDiv_eq_ofReal_log_sum hα hac, Fintype.sum_bool, hreal, hreal, hreal, hreal] + cases b <;> simp [add_comm] + +end RenyiDP + +open RenyiDP unitInterval in +/-- **Proposition 5** of Mironov (randomized response, order `α > 1`): for a measurable predicate +`f` on datasets and `p ∈ (0, 1)`, the randomized response mechanism `RR_p f` is `(α, ε)`-RDP with +`ε = (α - 1)⁻¹ log (p ^ α (1 - p) ^ (1 - α) + (1 - p) ^ α p ^ (1 - α))` (Table II), for every +neighbouring relation. The bound is attained by every pair with `f(S) ≠ f(S')` +(`RenyiDP.renyiDiv_randomizedResponseOfProb`). -/ +theorem isRenyiDP_randomizedResponseOfProb_comap {f : 𝒟 → Bool} (hf : Measurable f) {p : I} + (hp0 : 0 < (p : ℝ)) (hp1 : (p : ℝ) < 1) (hα : 1 < α) : + IsRenyiDP ((randomizedResponseOfProb p).comap f hf) Adj α (Real.toNNReal ((α - 1)⁻¹ + * log ((p : ℝ) ^ α * (1 - p) ^ (1 - α) + (1 - p) ^ α * (p : ℝ) ^ (1 - α)))) := + fun S S' _ ↦ by + rw [Kernel.comap_apply, Kernel.comap_apply] + change _ ≤ ENNReal.ofReal _ + rcases Bool.eq_or_eq_not (f S') (f S) with h | h + · simp [h] + · rw [h] + exact (renyiDiv_randomizedResponseOfProb hp0 hp1 hα (f S)).le + +end InformationTheory From 865524d961835058d33b844c034cac0d4d421faa Mon Sep 17 00:00:00 2001 From: Kousha Kalantari <25445970+KoushaKalantari@users.noreply.github.com> Date: Fri, 13 Nov 2026 10:23:20 -0800 Subject: [PATCH 4/5] =?UTF-8?q?feat(Privacy):=20zero-concentrated=20differ?= =?UTF-8?q?ential=20privacy=20(Bun=E2=80=93Steinke)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- Statlib.lean | 2 + Statlib/Privacy/ZCDP/Basic.lean | 825 ++++++++++++++++++++++++++++++ Statlib/Privacy/ZCDP/RenyiDP.lean | 64 +++ 3 files changed, 891 insertions(+) create mode 100644 Statlib/Privacy/ZCDP/Basic.lean create mode 100644 Statlib/Privacy/ZCDP/RenyiDP.lean diff --git a/Statlib.lean b/Statlib.lean index 9fb07bc..6e45456 100644 --- a/Statlib.lean +++ b/Statlib.lean @@ -144,6 +144,8 @@ import Statlib.Privacy.DP.Geometric import Statlib.Privacy.DP.Laplace import Statlib.Privacy.DP.RandomizedResponse import Statlib.Privacy.RenyiDP.Basic +import Statlib.Privacy.ZCDP.Basic +import Statlib.Privacy.ZCDP.RenyiDP import Statlib.Testing.Binary import Statlib.Testing.BinaryRenyi import Statlib.Testing.ChangeMeasure diff --git a/Statlib/Privacy/ZCDP/Basic.lean b/Statlib/Privacy/ZCDP/Basic.lean new file mode 100644 index 0000000..b2518f4 --- /dev/null +++ b/Statlib/Privacy/ZCDP/Basic.lean @@ -0,0 +1,825 @@ +/- +Copyright (c) 2026 Kousha Kalantari. All rights reserved. +Released under Apache 2.0 license as described in the file LICENSE. +Authors: Kousha Kalantari +-/ +module + +public import Statlib.Privacy.DP.Basic +public import Mathlib.MeasureTheory.Function.AEEqOfLIntegral +public import Mathlib.MeasureTheory.Integral.MeanInequalities +public import Mathlib.Probability.Distributions.Gaussian.Real +public import Mathlib.Probability.Moments.SubGaussian +public import Statlib.Divergence.Renyi.ChainRule +public import Statlib.Divergence.Renyi.Prod + +/-! +# Zero-concentrated differential privacy + +This file formalizes the basic theory of zero-concentrated differential privacy from M. Bun and +T. Steinke, *Concentrated differential privacy: simplifications, extensions, and lower bounds*, +TCC 2016 ([arXiv:1605.02065](https://arxiv.org/abs/1605.02065), v1; result numbers follow it). + +A mechanism is a kernel `M : Kernel 𝒟 Y` from datasets to outputs, and `Adj : 𝒟 → 𝒟 → Prop` is +an arbitrary neighbouring relation (the paper's "differ in a single entry"). The Rényi divergence +`D_α` is Statlib's `renyiDiv`, which for probability measures is the paper's Definition 2.1 (and +is `∞` when `M S` is not absolutely continuous with respect to `M S'`, as in footnote 10). +`(ε, δ)`-differential privacy is `IsEpsDeltaDP` from `Statlib.Privacy.DP.Basic`. + +**Scope.** Nothing here assumes finite alphabets: all results hold for arbitrary measurable +output spaces, except that adaptive composition assumes the first output space countable or the +second countably generated (`CountableOrCountablyGenerated`). + +## Main definitions + +* `IsZCDP M Adj ξ ρ`: `(ξ, ρ)`-zCDP (Definition 1.1); the paper's `ρ`-zCDP is `IsZCDP M Adj 0 ρ`. +* `ZCDP.piKernel M`: the product `S ↦ ∏ i, M i S` of finitely many Markov kernels. +* `ZCDP.gaussianKernel q hq v`: the Gaussian mechanism `S ↦ N(q S, v)`. + +## Main statements (paper result → Lean name) + +* Lemma 1.8 (post-processing): `IsZCDP.comp` (Markov kernels), `IsZCDP.map` (measurable maps). +* Lemma 1.7 (composition): `IsZCDP.prod`; iterated, `IsZCDP.pi` and `IsZCDP.piKernel` (for the + product kernel `ZCDP.piKernel`). +* Lemma 2.3 (adaptive composition): `IsZCDP.snd_compProd`, and `IsZCDP.compProd` for the + mechanism that releases both outputs. It combines Statlib's chain rule with + `ZCDP.condRenyiDiv_le`, a bound on the conditional term. +* Proposition 1.3: `IsZCDP.isEpsDeltaDP'`; Lemma 3.5 (its `(ξ, ρ)` form): `IsZCDP.isEpsDeltaDP`. +* Lemma 3.2 (`ε`-DP iff `(ε, 0)`-zCDP): `isZCDP_iff_isEpsDeltaDP`. +* Proposition 1.4 (`ε`-DP implies `(ε² / 2)`-zCDP): `IsEpsDeltaDP.isZCDP`; Proposition 3.3 (its + form for two measures): `ZCDP.renyiDiv_le_sq_of_le_smul`. +* Composition of `k` pure-DP mechanisms is `(k ε² / 2)`-zCDP (Proposition 1.4 with Lemma 1.7): + `isZCDP_pi_of_isEpsDeltaDP`; hence `(k ε² / 2 + ε √(2 k log (1 / δ)), δ)`-DP, a bound not + stated in the paper that follows with Proposition 1.3: `isEpsDeltaDP_pi_of_isEpsDeltaDP`. +* Lemma 2.4 (Rényi divergence between Gaussians, orders `α > 1`): `ZCDP.renyiDiv_gaussianReal`; + Proposition 1.6 (Gaussian mechanism): `isZCDP_of_gaussianReal`, `ZCDP.isZCDP_gaussianKernel`. + +## Implementation notes + +* The bound `D_α ≤ ξ + ρ α` of `IsZCDP` is compared in `EReal`, so that it is exactly the paper's + condition for all real `ξ`, `ρ` (an infinite divergence or a negative bound fails it); + `isZCDP_iff_toReal` and `isZCDP_iff_le_ofReal` give the `ℝ` and `ℝ≥0∞` forms. +* Proofs that differ from the paper's: Lemma 3.5 uses the Hölder bound + `μ(T) ^ α ≤ exp ((α - 1) D_α(μ ‖ ν)) ν(T) ^ (α - 1)` (`ZCDP.measureReal_rpow_le`) instead of + the privacy-loss/Markov argument, at the paper's order `α = 1 + √(log (1 / δ) / ρ)`; + Proposition 3.3 applies Hoeffding's lemma (Mathlib's `hasSubgaussianMGF_of_mem_Icc`) twice to + the privacy loss instead of the `sinh` inequality of Lemma B.1; Lemma 2.4 uses the Gaussian + moment generating function instead of completing the square. +* Mathlib has no product of a finite family of kernels; `ZCDP.piKernel` constructs it. +* `IsZCDP` is meant for Markov kernels: `renyiDiv` normalizes finite measures, so for other + kernels the condition is not the paper's (e.g. the zero kernel is `(0, 0)`-zCDP). All results + below assume Markov (or finite) kernels. +* `IsEpsDeltaDP` quantifies over all sets `E`, with `(M S).real E` an outer measure for + non-measurable `E`. For probability measures this is equivalent to quantifying over measurable + sets, by taking a measurable hull (`isEpsDeltaDP_iff_forall_measurableSet` in + `Statlib.Privacy.DP.Basic`); `IsZCDP.isEpsDeltaDP` handles arbitrary `E` directly through + `toMeasurable`. +* Lemma 3.5 and Proposition 1.3 (remark): the paper's proof chooses the order + `α = (ε - ξ + ρ) / 2ρ`, which excludes the edge cases `ρ = 0` and `δ > 1` (where the argument + of `√(4 ρ log (1 / δ))` is negative; in Lean `√` of a negative number is `0`). Both are + immediate: `δ ≥ 1` is trivial, and `ρ = 0` is the limit `ρ ↓ 0`. +* Proposition 1.4 assumes the neighbouring relation symmetric (`[Std.Symm Adj]`), as the paper's + is; the other results need no symmetry. + +## Not formalized + +The privacy-loss formulation (Definition 1.2, equation (2)), the `(ε, δ)` refinements and the +converse (Lemmas 3.6 and 3.7), mean-concentrated DP (Section 4), group privacy (Section 5), the +lower bounds (Sections 6 and 7), approximate zCDP (Section 8), the multivariate Gaussian +(Lemma 2.5), and Lemma 2.4 at order `α = 1`. +-/ + +@[expose] public section + +open MeasureTheory ProbabilityTheory Real + +open scoped ENNReal NNReal + +namespace InformationTheory + +variable {𝒟 Y Z : Type*} {m𝒟 : MeasurableSpace 𝒟} {mY : MeasurableSpace Y} + {mZ : MeasurableSpace Z} + +/-- **Zero-concentrated differential privacy** (Definition 1.1 of Bun–Steinke): a mechanism +`M : Kernel 𝒟 Y` is `(ξ, ρ)`-zCDP for the neighbouring relation `Adj` if +`D_α(M S ‖ M S') ≤ ξ + ρ α` for all neighbours `Adj S S'` and all orders `α ∈ (1, ∞)`, where +`D_α` is Statlib's Rényi divergence `renyiDiv`. The comparison is made in `EReal`, so that, as in +the paper, an infinite divergence violates the bound and a negative right-hand side cannot be +met. The paper's `ρ`-zCDP is `IsZCDP M Adj 0 ρ`. + +The definition is meant for Markov kernels: `renyiDiv` normalizes finite measures, so for other +kernels it does not express the paper's condition (e.g. the zero kernel is `(0, 0)`-zCDP). -/ +def IsZCDP (M : Kernel 𝒟 Y) (Adj : 𝒟 → 𝒟 → Prop) (ξ ρ : ℝ) : Prop := + ∀ ⦃S S'⦄, Adj S S' → ∀ α : ℝ, 1 < α → + (renyiDiv α (M S) (M S') : EReal) ≤ ((ξ + ρ * α : ℝ) : EReal) + +namespace ZCDP + +/-- An extended nonnegative real is at most a real number `r`, as elements of `EReal`, iff it is +finite and its real value is at most `r`. -/ +lemma coe_ennreal_le_coe_iff {R : ℝ≥0∞} {r : ℝ} : + (R : EReal) ≤ (r : EReal) ↔ R ≠ ∞ ∧ R.toReal ≤ r := by + rcases eq_or_ne R ∞ with rfl | hR + · simp + · rw [← EReal.coe_ennreal_toReal hR, EReal.coe_le_coe_iff] + simp [hR] + +/-- For `r ≥ 0`, the `EReal` comparison `R ≤ r` is the `ℝ≥0∞` comparison `R ≤ ofReal r`. -/ +lemma coe_ennreal_le_coe_iff_le_ofReal {R : ℝ≥0∞} {r : ℝ} (hr : 0 ≤ r) : + (R : EReal) ≤ (r : EReal) ↔ R ≤ ENNReal.ofReal r := by + rw [coe_ennreal_le_coe_iff] + constructor + · rintro ⟨hR, h⟩ + exact (ENNReal.le_ofReal_iff_toReal_le hR hr).2 h + · intro h + exact ⟨ne_top_of_le_ne_top ENNReal.ofReal_ne_top h, ENNReal.toReal_le_of_le_ofReal hr h⟩ + +end ZCDP + +variable {M : Kernel 𝒟 Y} {Adj : 𝒟 → 𝒟 → Prop} {ξ ρ ξ' ρ' : ℝ} + +/-- `(ξ, ρ)`-zCDP in real form: each divergence is finite with real value at most `ξ + ρ α`. -/ +lemma isZCDP_iff_toReal : IsZCDP M Adj ξ ρ ↔ ∀ ⦃S S'⦄, Adj S S' → ∀ α : ℝ, 1 < α → + renyiDiv α (M S) (M S') ≠ ∞ ∧ (renyiDiv α (M S) (M S')).toReal ≤ ξ + ρ * α := by + simp_rw [IsZCDP, ZCDP.coe_ennreal_le_coe_iff] + +/-- For nonnegative parameters, `(ξ, ρ)`-zCDP is the bound `D_α(M S ‖ M S') ≤ ξ + ρ α` in +`ℝ≥0∞`. -/ +lemma isZCDP_iff_le_ofReal (hξ : 0 ≤ ξ) (hρ : 0 ≤ ρ) : IsZCDP M Adj ξ ρ ↔ ∀ ⦃S S'⦄, Adj S S' → + ∀ α : ℝ, 1 < α → renyiDiv α (M S) (M S') ≤ ENNReal.ofReal (ξ + ρ * α) := by + refine forall₂_congr fun S S' ↦ forall_congr' fun _ ↦ forall₂_congr fun α hα ↦ ?_ + exact ZCDP.coe_ennreal_le_coe_iff_le_ofReal (by nlinarith) + +namespace IsZCDP + +/-- Under zCDP, the Rényi divergences between neighbours are finite. -/ +lemma renyiDiv_ne_top (h : IsZCDP M Adj ξ ρ) {S S' : 𝒟} (hS : Adj S S') {α : ℝ} (hα : 1 < α) : + renyiDiv α (M S) (M S') ≠ ∞ := + (isZCDP_iff_toReal.1 h hS α hα).1 + +/-- Under `(ξ, ρ)`-zCDP, `D_α(M S ‖ M S') ≤ ξ + ρ α` in `ℝ`. -/ +lemma toReal_renyiDiv_le (h : IsZCDP M Adj ξ ρ) {S S' : 𝒟} (hS : Adj S S') {α : ℝ} + (hα : 1 < α) : (renyiDiv α (M S) (M S')).toReal ≤ ξ + ρ * α := + (isZCDP_iff_toReal.1 h hS α hα).2 + +/-- Under `(ξ, ρ)`-zCDP, `D_α(M S ‖ M S') ≤ ξ + ρ α` in `ℝ≥0∞`. -/ +lemma renyiDiv_le_ofReal (h : IsZCDP M Adj ξ ρ) {S S' : 𝒟} (hS : Adj S S') {α : ℝ} + (hα : 1 < α) : renyiDiv α (M S) (M S') ≤ ENNReal.ofReal (ξ + ρ * α) := + (ENNReal.le_ofReal_iff_toReal_le (h.renyiDiv_ne_top hS hα) + (ENNReal.toReal_nonneg.trans (h.toReal_renyiDiv_le hS hα))).2 (h.toReal_renyiDiv_le hS hα) + +/-- `(ξ, ρ)`-zCDP is monotone in both parameters. -/ +lemma mono (h : IsZCDP M Adj ξ ρ) (hξ : ξ ≤ ξ') (hρ : ρ ≤ ρ') : IsZCDP M Adj ξ' ρ' := + isZCDP_iff_toReal.2 fun _ _ hS α hα ↦ ⟨h.renyiDiv_ne_top hS hα, + (h.toReal_renyiDiv_le hS hα).trans (by nlinarith)⟩ + +/-- A divergence bound for a modified mechanism by the divergence of `M` transfers zCDP. -/ +lemma of_renyiDiv_le {N : Kernel 𝒟 Z} (h : IsZCDP M Adj ξ ρ) + (hle : ∀ ⦃S S'⦄, Adj S S' → ∀ α : ℝ, 1 < α → + renyiDiv α (N S) (N S') ≤ renyiDiv α (M S) (M S')) : + IsZCDP N Adj ξ ρ := fun _ _ hS α hα ↦ + (EReal.coe_ennreal_le_coe_ennreal_iff.2 (hle hS α hα)).trans (h hS α hα) + +/-- **Post-processing** (Lemma 1.8 of Bun–Steinke, randomized form): if `M` is `(ξ, ρ)`-zCDP and +`κ` is a Markov kernel, then `κ ∘ₖ M` is `(ξ, ρ)`-zCDP. -/ +theorem comp [IsFiniteKernel M] (h : IsZCDP M Adj ξ ρ) (κ : Kernel Y Z) [IsMarkovKernel κ] : + IsZCDP (κ ∘ₖ M) Adj ξ ρ := + h.of_renyiDiv_le fun S S' _ α hα ↦ by + simpa only [Kernel.comp_apply] using renyiDiv_comp_right_le (by linarith) (M S) (M S') κ + +/-- **Post-processing** (Lemma 1.8 of Bun–Steinke) by a measurable map `f`. -/ +theorem map [IsFiniteKernel M] (h : IsZCDP M Adj ξ ρ) {f : Y → Z} (hf : Measurable f) : + IsZCDP (M.map f) Adj ξ ρ := + h.of_renyiDiv_le fun S S' _ α hα ↦ by + simpa only [Kernel.map_apply _ hf] using renyiDiv_map_le (by linarith) (M S) (M S') hf + +/-! ### Composition -/ + +/-- **Composition** (Lemma 1.7 of Bun–Steinke, with the `(ξ, ρ)` parameters of Lemma 2.3): if `M` +is `(ξ, ρ)`-zCDP and `M₂` is `(ξ', ρ')`-zCDP, then releasing both outputs, `S ↦ M S × M₂ S` +(`M ×ₖ M₂`), is `(ξ + ξ', ρ + ρ')`-zCDP. -/ +theorem prod [IsMarkovKernel M] {M₂ : Kernel 𝒟 Z} [IsMarkovKernel M₂] (h : IsZCDP M Adj ξ ρ) + (h₂ : IsZCDP M₂ Adj ξ' ρ') : IsZCDP (M ×ₖ M₂) Adj (ξ + ξ') (ρ + ρ') := by + refine isZCDP_iff_toReal.2 fun S S' hS α hα ↦ ?_ + rw [Kernel.prod_apply, Kernel.prod_apply, renyiDiv_prod (by linarith)] + have h₁ := h.renyiDiv_ne_top hS hα + have h₂' := h₂.renyiDiv_ne_top hS hα + refine ⟨ENNReal.add_ne_top.2 ⟨h₁, h₂'⟩, ?_⟩ + rw [ENNReal.toReal_add h₁ h₂'] + linarith [h.toReal_renyiDiv_le hS hα, h₂.toReal_renyiDiv_le hS hα] + +end IsZCDP + +namespace ZCDP + +/-- The product `S ↦ ∏ i, M i S` of finitely many Markov kernels, which releases independent +outputs of all of them. Its measurability is checked on the π-system of boxes. -/ +noncomputable def piKernel {ι : Type*} [Fintype ι] {Y : ι → Type*} [∀ i, MeasurableSpace (Y i)] + (M : ∀ i, Kernel 𝒟 (Y i)) [∀ i, IsMarkovKernel (M i)] : Kernel 𝒟 (∀ i, Y i) where + toFun S := Measure.pi fun i ↦ M i S + measurable' := by + refine Measurable.measure_of_isPiSystem_of_isProbabilityMeasure generateFrom_pi.symm + isPiSystem_pi ?_ + rintro _ ⟨s, hs, rfl⟩ + simp_rw [Measure.pi_pi] + exact Finset.measurable_prod _ fun i _ ↦ Kernel.measurable_coe (M i) (hs i (Set.mem_univ i)) + +/-- The value of the product kernel at a dataset. -/ +lemma piKernel_apply {ι : Type*} [Fintype ι] {Y : ι → Type*} [∀ i, MeasurableSpace (Y i)] + (M : ∀ i, Kernel 𝒟 (Y i)) [∀ i, IsMarkovKernel (M i)] (S : 𝒟) : + piKernel M S = Measure.pi fun i ↦ M i S := rfl + +instance {ι : Type*} [Fintype ι] {Y : ι → Type*} [∀ i, MeasurableSpace (Y i)] + (M : ∀ i, Kernel 𝒟 (Y i)) [∀ i, IsMarkovKernel (M i)] : IsMarkovKernel (piKernel M) := + ⟨fun S ↦ by rw [piKernel_apply]; infer_instance⟩ + +end ZCDP + +namespace IsZCDP + +/-- **Composition of finitely many mechanisms** (Lemma 1.7 of Bun–Steinke, iterated as the paper +indicates after Lemma 2.3, non-adaptive form): if each `M i` is `(ξ i, ρ i)`-zCDP, then a +mechanism `K` releasing independent outputs of all of them, `K S = ∏ i, M i S`, is +`(∑ ξ i, ∑ ρ i)`-zCDP. Such a `K` is `ZCDP.piKernel M` (see `IsZCDP.piKernel`). -/ +theorem pi {ι : Type*} [Fintype ι] {Y : ι → Type*} [∀ i, MeasurableSpace (Y i)] + {M : ∀ i, Kernel 𝒟 (Y i)} [∀ i, IsMarkovKernel (M i)] {K : Kernel 𝒟 (∀ i, Y i)} + (hK : ∀ S, K S = Measure.pi fun i ↦ M i S) {ξ ρ : ι → ℝ} + (h : ∀ i, IsZCDP (M i) Adj (ξ i) (ρ i)) : IsZCDP K Adj (∑ i, ξ i) (∑ i, ρ i) := by + refine isZCDP_iff_toReal.2 fun S S' hS α hα ↦ ?_ + rw [hK, hK, renyiDiv_pi (by linarith)] + have hne : ∀ i ∈ Finset.univ, renyiDiv α (M i S) (M i S') ≠ ∞ := + fun i _ ↦ (h i).renyiDiv_ne_top hS hα + refine ⟨ENNReal.sum_ne_top.2 hne, ?_⟩ + rw [ENNReal.toReal_sum hne, Finset.sum_mul, ← Finset.sum_add_distrib] + exact Finset.sum_le_sum fun i _ ↦ (h i).toReal_renyiDiv_le hS hα + +/-- `IsZCDP.pi` for the product kernel `ZCDP.piKernel M`. -/ +theorem piKernel {ι : Type*} [Fintype ι] {Y : ι → Type*} [∀ i, MeasurableSpace (Y i)] + {M : ∀ i, Kernel 𝒟 (Y i)} [∀ i, IsMarkovKernel (M i)] {ξ ρ : ι → ℝ} + (h : ∀ i, IsZCDP (M i) Adj (ξ i) (ρ i)) : + IsZCDP (ZCDP.piKernel M) Adj (∑ i, ξ i) (∑ i, ρ i) := + pi (fun _ ↦ rfl) h + +end IsZCDP + +namespace ZCDP + +/-- For probability measures and `a > 1`, a finite Rényi divergence and the Hellinger divergence +are related by `exp ((a - 1) Rₐ(μ, ν)) = 1 + (a - 1) Hₐ(μ, ν)`. -/ +lemma exp_mul_toReal_renyiDiv {μ ν : Measure Y} [IsProbabilityMeasure μ] + [IsProbabilityMeasure ν] {a : ℝ} (ha : 1 < a) (h : renyiDiv a μ ν ≠ ∞) : + exp ((a - 1) * (renyiDiv a μ ν).toReal) = 1 + (a - 1) * (hellingerDiv a μ ν).toReal := by + have hH : hellingerDiv a μ ν ≠ ∞ := + fun hH ↦ h ((renyiDiv_eq_top_iff_hellingerDiv_eq_top_of_one_lt ha).2 hH) + have h1 := mul_hellingerDiv_add_meas_eq_integral (μ := μ) (ν := ν) (by linarith) ha.ne' hH + have hpos := integral_rpow_rnDeriv_pos_of_renyiDiv_ne_top (by linarith) h + simp only [measure_univ, ENNReal.toReal_one, mul_one] at h1 + rw [toReal_renyiDiv_of_ne_top_of_isProbabilityMeasure (by linarith) ha.ne' h, ← mul_assoc, + mul_inv_cancel₀ (by linarith : a - 1 ≠ 0), one_mul, exp_log hpos] + linarith + +/-- If `Rₐ(κ y, η y) ≤ r` for every `y`, then the conditional Rényi divergence +`Rₐ(P ⊗ₘ κ, P ⊗ₘ η)` is at most `r`, for a probability measure `P` and `a > 1`. This is the +bound on the conditional term of the chain rule `renyiDiv_compProd_eq_add_condRenyiDiv`; with +it, the chain rule gives the upper bound `Rₐ(μ ⊗ₘ κ, ν ⊗ₘ η) ≤ Rₐ(μ, ν) + sup_y Rₐ(κ y, η y)` of +the composition property in Lemma 2.2 of Bun–Steinke. The proof goes through the Hellinger +divergence, which is an integral over `y` (`condHellingerDiv_eq_lintegral`). -/ +lemma condRenyiDiv_le {κ η : Kernel Y Z} [IsMarkovKernel κ] [IsMarkovKernel η] + [MeasurableSpace.CountableOrCountablyGenerated Y Z] (P : Measure Y) [IsProbabilityMeasure P] + {a r : ℝ} (ha : 1 < a) + (h : ∀ y, renyiDiv a (κ y) (η y) ≠ ∞ ∧ (renyiDiv a (κ y) (η y)).toReal ≤ r) : + condRenyiDiv a κ η P ≠ ∞ ∧ (condRenyiDiv a κ η P).toReal ≤ r := by + have ha1 : 0 < a - 1 := by linarith + have hr : 0 ≤ r := by + obtain ⟨y⟩ := nonempty_of_isProbabilityMeasure P + exact ENNReal.toReal_nonneg.trans (h y).2 + set c := (exp ((a - 1) * r) - 1) / (a - 1) with hc_def + have hc : 0 ≤ c := div_nonneg (by linarith [one_le_exp (mul_nonneg ha1.le hr)]) ha1.le + have hH : ∀ y, hellingerDiv a (κ y) (η y) ≤ ENNReal.ofReal c := by + intro y + obtain ⟨hR, hRr⟩ := h y + have hH' : hellingerDiv a (κ y) (η y) ≠ ∞ := + fun hH ↦ hR ((renyiDiv_eq_top_iff_hellingerDiv_eq_top_of_one_lt ha).2 hH) + rw [ENNReal.le_ofReal_iff_toReal_le hH' hc, hc_def, le_div_iff₀ ha1] + have h1 := exp_mul_toReal_renyiDiv ha hR + have h2 : exp ((a - 1) * (renyiDiv a (κ y) (η y)).toReal) ≤ exp ((a - 1) * r) := + exp_le_exp.2 (mul_le_mul_of_nonneg_left hRr ha1.le) + linarith + have hcond : hellingerDiv a (P ⊗ₘ κ) (P ⊗ₘ η) ≤ ENNReal.ofReal c := by + rw [← condHellingerDiv, condHellingerDiv_eq_lintegral] + calc ∫⁻ y, hellingerDiv a (κ y) (η y) ∂P ≤ ∫⁻ _, ENNReal.ofReal c ∂P := lintegral_mono hH + _ = ENNReal.ofReal c := by simp + have hHne : hellingerDiv a (P ⊗ₘ κ) (P ⊗ₘ η) ≠ ∞ := + ne_top_of_le_ne_top ENNReal.ofReal_ne_top hcond + have hRne : renyiDiv a (P ⊗ₘ κ) (P ⊗ₘ η) ≠ ∞ := + fun hR ↦ hHne ((renyiDiv_eq_top_iff_hellingerDiv_eq_top_of_one_lt ha).1 hR) + refine ⟨hRne, ?_⟩ + have he := exp_mul_toReal_renyiDiv ha hRne + have hHc := ENNReal.toReal_le_of_le_ofReal hc hcond + have hle : exp ((a - 1) * (renyiDiv a (P ⊗ₘ κ) (P ⊗ₘ η)).toReal) ≤ exp ((a - 1) * r) := by + rw [he] + have : (a - 1) * (hellingerDiv a (P ⊗ₘ κ) (P ⊗ₘ η)).toReal ≤ (a - 1) * c := + mul_le_mul_of_nonneg_left hHc ha1.le + rw [hc_def, mul_div_cancel₀ _ ha1.ne'] at this + linarith + exact le_of_mul_le_mul_left (exp_le_exp.1 hle) ha1 + +end ZCDP + +namespace IsZCDP + +/-- **Adaptive composition** (Lemma 2.3 of Bun–Steinke, releasing both outputs): let `M` be +`(ξ, ρ)`-zCDP and let `M' : Kernel (𝒟 × Y) Z` be `(ξ', ρ')`-zCDP as a function of its first +argument, for every fixed second argument `y`. Then the mechanism that draws `y ∼ M S` and then +`z ∼ M' (S, y)` and releases `(y, z)` (`M ⊗ₖ M'`) is `(ξ + ξ', ρ + ρ')`-zCDP. The proof uses the +chain rule `renyiDiv_compProd_eq_add_condRenyiDiv`; it assumes `Y` countable or `Z` countably +generated (e.g. standard Borel). -/ +theorem compProd [IsMarkovKernel M] {M' : Kernel (𝒟 × Y) Z} [IsMarkovKernel M'] + [MeasurableSpace.CountableOrCountablyGenerated Y Z] (h : IsZCDP M Adj ξ ρ) + (h' : ∀ y, IsZCDP (M'.sectL y) Adj ξ' ρ') : IsZCDP (M ⊗ₖ M') Adj (ξ + ξ') (ρ + ρ') := by + refine isZCDP_iff_toReal.2 fun S S' hS α hα ↦ ?_ + rw [Kernel.compProd_apply_eq_compProd_sectR, Kernel.compProd_apply_eq_compProd_sectR, + renyiDiv_compProd_eq_add_condRenyiDiv (by linarith)] + have hR := h.renyiDiv_ne_top hS hα + have := isProbabilityMeasure_renyiMeasure_of_renyiDiv_ne_top (by linarith) hR + obtain ⟨hC, hCr⟩ := ZCDP.condRenyiDiv_le (κ := M'.sectR S) (η := M'.sectR S') + (renyiMeasure α (M S) (M S')) (r := ξ' + ρ' * α) hα fun y ↦ isZCDP_iff_toReal.1 (h' y) hS α hα + refine ⟨ENNReal.add_ne_top.2 ⟨hR, hC⟩, ?_⟩ + rw [ENNReal.toReal_add hR hC] + linarith [h.toReal_renyiDiv_le hS hα] + +/-- **Adaptive composition** (Lemma 2.3 of Bun–Steinke, as stated): with `M` and `M'` as in +`IsZCDP.compProd`, the mechanism `S ↦ M' (S, M S)` that releases only the second output +(`(M ⊗ₖ M').snd`) is `(ξ + ξ', ρ + ρ')`-zCDP. -/ +theorem snd_compProd [IsMarkovKernel M] {M' : Kernel (𝒟 × Y) Z} [IsMarkovKernel M'] + [MeasurableSpace.CountableOrCountablyGenerated Y Z] (h : IsZCDP M Adj ξ ρ) + (h' : ∀ y, IsZCDP (M'.sectL y) Adj ξ' ρ') : IsZCDP (M ⊗ₖ M').snd Adj (ξ + ξ') (ρ + ρ') := + (h.compProd h').of_renyiDiv_le fun S S' _ α hα ↦ by + simpa only [Kernel.snd_apply] using + renyiDiv_map_le (by linarith) ((M ⊗ₖ M') S) ((M ⊗ₖ M') S') measurable_snd + +end IsZCDP + +/-! ### From zCDP to approximate differential privacy -/ + +namespace ZCDP + +/-- For probability measures, `a > 1` and finite `Rₐ(μ, ν)`, the Hellinger integral is +`∫⁻ (∂μ/∂ν) ^ a ∂ν = exp ((a - 1) Rₐ(μ, ν))`. -/ +lemma lintegral_rpow_rnDeriv_eq {μ ν : Measure Y} [IsProbabilityMeasure μ] + [IsProbabilityMeasure ν] {a : ℝ} (ha : 1 < a) (h : renyiDiv a μ ν ≠ ∞) : + ∫⁻ x, (∂μ/∂ν) x ^ a ∂ν = ENNReal.ofReal (exp ((a - 1) * (renyiDiv a μ ν).toReal)) := by + have hint := integrable_rpow_rnDeriv_of_renyiDiv_ne_top (by linarith : (0 : ℝ) ≤ a) h + have hpos := integral_rpow_rnDeriv_pos_of_renyiDiv_ne_top (by linarith) h + rw [toReal_renyiDiv_of_ne_top_of_isProbabilityMeasure (by linarith) ha.ne' h, ← mul_assoc, + mul_inv_cancel₀ (by linarith : a - 1 ≠ 0), one_mul, exp_log hpos, + integral_rpow_rnDeriv_eq_toReal_lintegral (by linarith), ENNReal.ofReal_toReal + ((integrable_rpow_rnDeriv_iff_lintegral_ne_top (by linarith)).1 hint)] + +/-- **Probability preservation** of the Rényi divergence, by Hölder's inequality: for +probability measures, `a > 1`, finite `Rₐ(μ, ν)` and a measurable set `T`, +`μ(T) ^ a ≤ exp ((a - 1) Rₐ(μ, ν)) ν(T) ^ (a - 1)`. -/ +lemma measureReal_rpow_le {μ ν : Measure Y} [IsProbabilityMeasure μ] + [IsProbabilityMeasure ν] {a : ℝ} (ha : 1 < a) (h : renyiDiv a μ ν ≠ ∞) {T : Set Y} + (hT : MeasurableSet T) : + μ.real T ^ a ≤ exp ((a - 1) * (renyiDiv a μ ν).toReal) * ν.real T ^ (a - 1) := by + have hac : μ ≪ ν := absolutelyContinuous_of_renyiDiv_ne_top ha.le h + have hpq : a.HolderConjugate (a / (a - 1)) := + (Real.holderConjugate_iff_eq_conjExponent ha).2 rfl + have ha0 : 0 < a := by linarith + have hq0 : 0 < a / (a - 1) := div_pos ha0 (by linarith) + have hH := ENNReal.lintegral_mul_le_Lp_mul_Lq ν hpq (f := fun x ↦ (∂μ/∂ν) x) + (g := T.indicator 1) (Measure.measurable_rnDeriv μ ν).aemeasurable + ((measurable_one.indicator hT).aemeasurable) + have h1 : ∫⁻ x, ((fun x ↦ (∂μ/∂ν) x) * T.indicator (1 : Y → ℝ≥0∞)) x ∂ν = μ T := by + rw [← Measure.setLIntegral_rnDeriv hac T, ← lintegral_indicator hT] + congr 1 with x + by_cases hx : x ∈ T <;> simp [hx] + have h2 : ∫⁻ x, (T.indicator (1 : Y → ℝ≥0∞) x) ^ (a / (a - 1)) ∂ν = ν T := by + rw [← lintegral_indicator_one hT] + congr 1 with x + by_cases hx : x ∈ T <;> simp [hx, hq0] + rw [h1, h2, lintegral_rpow_rnDeriv_eq ha h] at hH + have hH' := ENNReal.rpow_le_rpow hH ha0.le + rw [ENNReal.mul_rpow_of_nonneg _ _ ha0.le, ← ENNReal.rpow_mul, ← ENNReal.rpow_mul, + one_div_mul_cancel ha0.ne', ENNReal.rpow_one, + show 1 / (a / (a - 1)) * a = a - 1 by field_simp] at hH' + have := ENNReal.toReal_mono (ENNReal.mul_ne_top ENNReal.ofReal_ne_top + (ENNReal.rpow_ne_top_of_nonneg (by linarith) (measure_ne_top _ _))) hH' + rwa [ENNReal.toReal_mul, ENNReal.toReal_ofReal (exp_pos _).le, ← ENNReal.toReal_rpow, + ← ENNReal.toReal_rpow] at this + +/-- The numerical step of the conversion: if `p ^ a ≤ exp ((a - 1) r) q ^ (a - 1)` with `a > 1`, +`q ≥ 0` and `δ > 0`, then `p ≤ exp (r + log (1 / δ) / (a - 1)) q + δ`. -/ +lemma le_exp_mul_add {a r δ p q : ℝ} (ha : 1 < a) (hδ : 0 < δ) (hq : 0 ≤ q) + (h : p ^ a ≤ exp ((a - 1) * r) * q ^ (a - 1)) : + p ≤ exp (r + log (1 / δ) / (a - 1)) * q + δ := by + set ε := r + log (1 / δ) / (a - 1) + by_contra hcon + rw [not_le] at hcon + have ha1 : 0 < a - 1 := by linarith + have hlt : q < p * exp (-ε) := by + have : exp ε * q < p := by nlinarith [(exp_pos ε).le, mul_nonneg (exp_pos ε).le hq] + rw [exp_neg, ← div_eq_mul_inv, lt_div_iff₀ (exp_pos ε)] + linarith + have hpp : 0 < p := by nlinarith [mul_nonneg (exp_pos ε).le hq] + have h2 : q ^ (a - 1) ≤ (p * exp (-ε)) ^ (a - 1) := rpow_le_rpow hq hlt.le ha1.le + rw [mul_rpow hpp.le (exp_pos _).le, ← exp_mul] at h2 + have h3 : p ^ a = p * p ^ (a - 1) := by + rw [rpow_sub hpp, rpow_one]; field_simp + have h4 : exp ((a - 1) * r) * exp (-ε * (a - 1)) = δ := by + rw [← exp_add] + have : (a - 1) * r + -ε * (a - 1) = log δ := by + simp only [ε, one_div, log_inv]; field_simp; ring + rw [this, exp_log hδ] + have h5 : p * p ^ (a - 1) ≤ δ * p ^ (a - 1) := by + calc p * p ^ (a - 1) = p ^ a := h3.symm + _ ≤ exp ((a - 1) * r) * q ^ (a - 1) := h + _ ≤ exp ((a - 1) * r) * (p ^ (a - 1) * exp (-ε * (a - 1))) := + mul_le_mul_of_nonneg_left h2 (exp_pos _).le + _ = δ * p ^ (a - 1) := by rw [← h4]; ring + have := le_of_mul_le_mul_right h5 (rpow_pos_of_pos hpp _) + nlinarith [mul_nonneg (exp_pos ε).le hq] + +/-- If `Rₐ(μ, ν) ≤ r` for some `a > 1`, then `μ(T) ≤ exp (r + log (1 / δ) / (a - 1)) ν(T) + δ` +for every set `T` and `δ > 0`. -/ +lemma measureReal_le_exp_mul_add {μ ν : Measure Y} [IsProbabilityMeasure μ] + [IsProbabilityMeasure ν] {a r δ : ℝ} (ha : 1 < a) (hδ : 0 < δ) (h : renyiDiv a μ ν ≠ ∞) + (hr : (renyiDiv a μ ν).toReal ≤ r) (T : Set Y) : + μ.real T ≤ exp (r + log (1 / δ) / (a - 1)) * ν.real T + δ := by + have hT' : ν.real (toMeasurable ν T) = ν.real T := by + simp [measureReal_def, measure_toMeasurable] + rw [← hT'] + refine (measureReal_mono (subset_toMeasurable ν T)).trans (le_exp_mul_add ha hδ + measureReal_nonneg ((measureReal_rpow_le ha h (measurableSet_toMeasurable ν T)).trans ?_)) + exact mul_le_mul_of_nonneg_right (exp_le_exp.2 (mul_le_mul_of_nonneg_left hr (by linarith))) + (rpow_nonneg measureReal_nonneg _) + +/-- `IsZCDP.isEpsDeltaDP` for `ρ > 0` and `δ < 1`, with the order `α = 1 + √(log (1 / δ) / ρ)`. -/ +lemma isEpsDeltaDP_of_pos [IsMarkovKernel M] (h : IsZCDP M Adj ξ ρ) (hρ : 0 < ρ) {δ : ℝ} + (hδ : 0 < δ) (hδ1 : δ < 1) : IsEpsDeltaDP M Adj (ξ + ρ + 2 * √(ρ * log (1 / δ))) δ := by + intro S S' hS E + have hL : 0 < log (1 / δ) := log_pos (one_lt_one_div hδ hδ1) + set L := log (1 / δ) + have hLρ : 0 < L / ρ := div_pos hL hρ + set s := √(L / ρ) with hs_def + have hs : 0 < s := sqrt_pos.2 hLρ + have hs2 : L = ρ * s ^ 2 := by + rw [hs_def, sq_sqrt hLρ.le]; field_simp + have hα : 1 < 1 + s := by linarith + have key := measureReal_le_exp_mul_add (μ := M S) (ν := M S') hα hδ (h.renyiDiv_ne_top hS hα) + (h.toReal_renyiDiv_le hS hα) E + have hsq : √(ρ * L) = ρ * s := by + rw [hs2, show ρ * (ρ * s ^ 2) = (ρ * s) ^ 2 by ring] + exact sqrt_sq (by positivity) + have heq : ξ + ρ + 2 * √(ρ * L) = ξ + ρ * (1 + s) + L / (1 + s - 1) := by + rw [hsq, add_sub_cancel_left, hs2] + field_simp + ring + rwa [heq] + +end ZCDP + +/-- **Lemma 3.5** of Bun–Steinke (zCDP implies approximate DP): if `M` is `(ξ, ρ)`-zCDP, then it +is `(ξ + ρ + √(4 ρ log (1 / δ)), δ)`-DP for every `δ > 0`; here `√(4 ρ log (1 / δ))` is written +`2 √(ρ log (1 / δ))`. No sign conditions are needed: `ρ < 0` is incompatible with zCDP (for +neighbouring datasets), and for `δ ≥ 1` the claim is trivial. The proof is not the paper's +privacy-loss/Markov argument but the probability-preservation bound +`μ(T) ^ α ≤ exp ((α - 1) D_α(μ ‖ ν)) ν(T) ^ (α - 1)` (Hölder's inequality, +`ZCDP.measureReal_rpow_le`), at the paper's order `α = 1 + √(log (1 / δ) / ρ)`; the case `ρ = 0`, +where that order is undefined, follows by letting `ρ ↓ 0`. -/ +theorem IsZCDP.isEpsDeltaDP [IsMarkovKernel M] (h : IsZCDP M Adj ξ ρ) {δ : ℝ} (hδ : 0 < δ) : + IsEpsDeltaDP M Adj (ξ + ρ + 2 * √(ρ * log (1 / δ))) δ := by + rcases le_or_gt 1 δ with hδ1 | hδ1 + · intro S S' hS E + exact measureReal_le_one.trans (hδ1.trans + (le_add_of_nonneg_left (mul_nonneg (exp_pos _).le measureReal_nonneg))) + rcases lt_trichotomy ρ 0 with hρ | rfl | hρ + · intro S S' hS E + exfalso + have hα : 1 < 2 + |ξ| / -ρ := by + have : 0 ≤ |ξ| / -ρ := div_nonneg (abs_nonneg ξ) (by linarith) + linarith + have h1 := h.toReal_renyiDiv_le hS hα + have h2 : ρ * (|ξ| / -ρ) = -|ξ| := by + rw [mul_div_assoc', div_neg, mul_div_cancel_left₀ _ hρ.ne] + rw [mul_add, h2] at h1 + linarith [ENNReal.toReal_nonneg (a := renyiDiv (2 + |ξ| / -ρ) (M S) (M S')), le_abs_self ξ] + · intro S S' hS E + have key : ∀ t : ℝ, 0 < t → (M S).real E ≤ + exp (ξ + t + 2 * √(t * log (1 / δ))) * (M S').real E + δ := + fun t ht ↦ ZCDP.isEpsDeltaDP_of_pos (h.mono le_rfl ht.le) ht hδ hδ1 hS E + have hcont : Filter.Tendsto + (fun t : ℝ ↦ exp (ξ + t + 2 * √(t * log (1 / δ))) * (M S').real E + δ) + (nhdsWithin 0 (Set.Ioi 0)) + (nhds (exp (ξ + 0 + 2 * √(0 * log (1 / δ))) * (M S').real E + δ)) := + ((by fun_prop : Continuous fun t : ℝ ↦ + exp (ξ + t + 2 * √(t * log (1 / δ))) * (M S').real E + δ).tendsto 0).mono_left + nhdsWithin_le_nhds + exact ge_of_tendsto hcont (eventually_nhdsWithin_of_forall key) + · exact ZCDP.isEpsDeltaDP_of_pos h hρ hδ hδ1 + +/-- **Proposition 1.3** of Bun–Steinke: if `M` is `ρ`-zCDP, then it is +`(ρ + 2 √(ρ log (1 / δ)), δ)`-DP for every `δ > 0`. -/ +theorem IsZCDP.isEpsDeltaDP' [IsMarkovKernel M] (h : IsZCDP M Adj 0 ρ) {δ : ℝ} (hδ : 0 < δ) : + IsEpsDeltaDP M Adj (ρ + 2 * √(ρ * log (1 / δ))) δ := by + simpa using h.isEpsDeltaDP hδ + +namespace ZCDP + +/-- If `μ ≤ e^ε ν` (that is, `D_∞(μ ‖ ν) ≤ ε`), then `D_a(μ ‖ ν) ≤ ε` for every `a > 1` +(monotonicity of the Rényi divergence in the order, used in Lemma 3.2 of Bun–Steinke). -/ +lemma renyiDiv_le_of_le_smul {μ ν : Measure Y} [IsProbabilityMeasure μ] + [IsProbabilityMeasure ν] {ε : ℝ} + (h1 : ∀ E, MeasurableSet E → μ E ≤ ENNReal.ofReal (exp ε) * ν E) {a : ℝ} (ha : 1 < a) : + renyiDiv a μ ν ≠ ∞ ∧ (renyiDiv a μ ν).toReal ≤ ε := by + set c := ENNReal.ofReal (exp ε) with hc_def + have hac : μ ≪ ν := Measure.AbsolutelyContinuous.mk fun s hs hνs ↦ + le_antisymm ((h1 s hs).trans (by simp [hνs])) zero_le + have hup : ∀ᵐ x ∂ν, μ.rnDeriv ν x ≤ c := + ae_le_of_forall_setLIntegral_le_of_sigmaFinite (Measure.measurable_rnDeriv μ ν) + fun s hs _ ↦ by + rw [Measure.setLIntegral_rnDeriv hac, setLIntegral_const] + exact h1 s hs + have h_int : Integrable (fun x ↦ (μ.rnDeriv ν x).toReal ^ a) ν := by + refine Integrable.of_bound + ((Measure.measurable_rnDeriv μ ν).ennreal_toReal.pow_const a).aestronglyMeasurable + (exp ε ^ a) ?_ + filter_upwards [hup] with x hu + rw [Real.norm_eq_abs, abs_of_nonneg (rpow_nonneg ENNReal.toReal_nonneg _)] + exact rpow_le_rpow ENNReal.toReal_nonneg (ENNReal.toReal_le_of_le_ofReal (exp_pos ε).le hu) + (by linarith) + have hR : renyiDiv a μ ν ≠ ∞ := (renyiDiv_ne_top_iff_of_one_lt ha).2 ⟨h_int, hac⟩ + refine ⟨hR, ?_⟩ + have hle : ∫⁻ x, (∂μ/∂ν) x ^ a ∂ν ≤ ∫⁻ x, c ^ (a - 1) * (∂μ/∂ν) x ∂ν := by + refine lintegral_mono_ae ?_ + filter_upwards [hup] with x hx + rw [show a = (a - 1) + 1 by ring, ENNReal.rpow_add_of_nonneg _ _ (by linarith) zero_le_one, + ENNReal.rpow_one, add_sub_cancel_right] + exact mul_le_mul_left (ENNReal.rpow_le_rpow hx (by linarith)) _ + rw [lintegral_const_mul _ (Measure.measurable_rnDeriv μ ν), Measure.lintegral_rnDeriv hac, + measure_univ, mul_one, lintegral_rpow_rnDeriv_eq ha hR, hc_def, + ENNReal.ofReal_rpow_of_nonneg (exp_pos ε).le (by linarith), ← exp_mul, + ENNReal.ofReal_le_ofReal_iff (exp_pos _).le, exp_le_exp] at hle + nlinarith + +/-- **Proposition 3.3** of Bun–Steinke, for measures: if `μ ≤ e^ε ν` and `ν ≤ e^ε μ` (that is, +`D_∞(μ ‖ ν) ≤ ε` and `D_∞(ν ‖ μ) ≤ ε`), then `D_a(μ ‖ ν) ≤ ε² a / 2` for every `a > 1`. The proof +applies Hoeffding's lemma (`hasSubgaussianMGF_of_mem_Icc`) to the privacy loss +`Z = log (dμ/dν) ∈ [-ε, ε]` under `μ` twice: at `t = -1`, where `E_μ[e^{-Z}] = 1` gives +`E_μ[Z] ≤ ε² / 2`, and at `t = a - 1`, where `E_μ[e^{(a-1)Z}] = exp ((a - 1) D_a(μ ‖ ν))`. -/ +lemma renyiDiv_le_sq_of_le_smul {μ ν : Measure Y} [IsProbabilityMeasure μ] + [IsProbabilityMeasure ν] {ε : ℝ} (hε : 0 ≤ ε) + (h1 : ∀ E, MeasurableSet E → μ E ≤ ENNReal.ofReal (exp ε) * ν E) + (h2 : ∀ E, MeasurableSet E → ν E ≤ ENNReal.ofReal (exp ε) * μ E) {a : ℝ} (ha : 1 < a) : + renyiDiv a μ ν ≠ ∞ ∧ (renyiDiv a μ ν).toReal ≤ ε ^ 2 / 2 * a := by + set c := ENNReal.ofReal (exp ε) with hc_def + have hc0 : c ≠ 0 := by simp [c, exp_pos] + have hcinv : c⁻¹ = ENNReal.ofReal (exp (-ε)) := by + rw [hc_def, exp_neg, ENNReal.ofReal_inv_of_pos (exp_pos ε)] + have hac : μ ≪ ν := Measure.AbsolutelyContinuous.mk fun s hs hνs ↦ + le_antisymm ((h1 s hs).trans (by simp [hνs])) zero_le + have hca : ν ≪ μ := Measure.AbsolutelyContinuous.mk fun s hs hμs ↦ + le_antisymm ((h2 s hs).trans (by simp [hμs])) zero_le + have hup : ∀ᵐ x ∂ν, μ.rnDeriv ν x ≤ c := + ae_le_of_forall_setLIntegral_le_of_sigmaFinite (Measure.measurable_rnDeriv μ ν) + fun s hs _ ↦ by + rw [Measure.setLIntegral_rnDeriv hac, setLIntegral_const] + exact h1 s hs + have hlow : ∀ᵐ x ∂ν, c⁻¹ ≤ μ.rnDeriv ν x := + ae_le_of_forall_setLIntegral_le_of_sigmaFinite measurable_const fun s hs _ ↦ by + rw [Measure.setLIntegral_rnDeriv hac, setLIntegral_const, + ENNReal.inv_mul_le_iff hc0 ENNReal.ofReal_ne_top] + exact h2 s hs + set Z : Y → ℝ := fun x ↦ log (μ.rnDeriv ν x).toReal with hZ_def + have hbd : ∀ᵐ x ∂μ, exp (-ε) ≤ (μ.rnDeriv ν x).toReal ∧ (μ.rnDeriv ν x).toReal ≤ exp ε := by + refine hac.ae_le ?_ + filter_upwards [hup, hlow, Measure.rnDeriv_lt_top μ ν] with x hu hl ht + refine ⟨?_, ENNReal.toReal_le_of_le_ofReal (exp_pos ε).le hu⟩ + rw [hcinv] at hl + exact (ENNReal.ofReal_le_iff_le_toReal ht.ne).1 hl + have hZ : ∀ᵐ x ∂μ, Z x ∈ Set.Icc (-ε) ε := by + filter_upwards [hbd] with x ⟨hl, hu⟩ + have hpos : 0 < (μ.rnDeriv ν x).toReal := (exp_pos _).trans_le hl + exact ⟨(le_log_iff_exp_le hpos).2 hl, (log_le_iff_le_exp hpos).2 hu⟩ + have hmeas : AEMeasurable Z μ := + ((Measure.measurable_rnDeriv μ ν).ennreal_toReal.log).aemeasurable + have hsub := hasSubgaussianMGF_of_mem_Icc hmeas hZ + set m := ∫ x, Z x ∂μ + have hc2 : (((‖ε - -ε‖₊ / 2) ^ 2 : ℝ≥0) : ℝ) = ε ^ 2 := by + simp only [NNReal.coe_pow, NNReal.coe_div, coe_nnNorm, Real.norm_eq_abs, sub_neg_eq_add] + rw [abs_of_nonneg (by linarith)] + norm_num + have hmgf : ∀ t, mgf Z μ t ≤ exp (t * m + ε ^ 2 * t ^ 2 / 2) := by + intro t + have h := hsub.mgf_le t + rw [hc2] at h + have : mgf (fun x ↦ Z x - m) μ t = mgf Z μ t * exp (-(t * m)) := by + simp only [sub_eq_add_neg] + rw [mgf_add_const, mul_neg] + rw [this, ← le_div_iff₀ (exp_pos _), ← exp_sub] at h + convert h using 2 + ring + -- `t = -1`: the mean of the privacy loss is at most `ε ^ 2 / 2` + have hmgf1 : mgf Z μ (-1) = 1 := by + have h_ae : (fun x ↦ exp (-1 * Z x)) =ᵐ[μ] fun x ↦ (ν.rnDeriv μ x).toReal := by + filter_upwards [hbd, Measure.inv_rnDeriv hac] with x ⟨hl, _⟩ hinv + have hpos : 0 < (μ.rnDeriv ν x).toReal := (exp_pos _).trans_le hl + rw [← hinv, Pi.inv_apply, ENNReal.toReal_inv, neg_one_mul, exp_neg, hZ_def, + exp_log hpos] + rw [mgf, integral_congr_ae h_ae, Measure.integral_toReal_rnDeriv hca, probReal_univ] + have hm : m ≤ ε ^ 2 / 2 := by + have h := hmgf (-1) + rw [hmgf1] at h + have h' : exp 0 ≤ exp (-1 * m + ε ^ 2 * (-1) ^ 2 / 2) := by rwa [exp_zero] + rw [exp_le_exp] at h' + nlinarith + -- `t = a - 1` + have ha1 : 0 < a - 1 := by linarith + have hJ : mgf Z μ (a - 1) ≤ exp ((a - 1) * (ε ^ 2 / 2 * a)) := by + refine (hmgf (a - 1)).trans (exp_le_exp.2 ?_) + nlinarith [mul_le_mul_of_nonneg_left hm ha1.le] + have hJpos : 0 < mgf Z μ (a - 1) := mgf_pos (integrable_exp_mul_of_mem_Icc hmeas hZ) + have hJeq : mgf Z μ (a - 1) = ∫ x, (μ.rnDeriv ν x).toReal ^ (a - 1) ∂μ := by + refine integral_congr_ae ?_ + filter_upwards [hbd] with x ⟨hl, _⟩ + have hpos : 0 < (μ.rnDeriv ν x).toReal := (exp_pos _).trans_le hl + rw [rpow_def_of_pos hpos, hZ_def, mul_comm] + have h_int : Integrable (fun x ↦ (μ.rnDeriv ν x).toReal ^ a) ν := by + refine Integrable.of_bound + ((Measure.measurable_rnDeriv μ ν).ennreal_toReal.pow_const a).aestronglyMeasurable + (exp ε ^ a) ?_ + filter_upwards [hup] with x hu + rw [Real.norm_eq_abs, abs_of_nonneg (rpow_nonneg ENNReal.toReal_nonneg _)] + exact rpow_le_rpow ENNReal.toReal_nonneg (ENNReal.toReal_le_of_le_ofReal (exp_pos ε).le hu) + (by linarith) + rw [renyiDiv_eq_log_integral_of_one_lt' ha h_int hac] + refine ⟨ENNReal.ofReal_ne_top, ?_⟩ + rw [ENNReal.toReal_ofReal', ← hJeq] + simp only [measure_univ, ENNReal.toReal_one, log_one, mul_zero, add_zero, sub_zero] + refine max_le ?_ (by positivity) + rw [inv_mul_le_iff₀ ha1] + exact (log_le_iff_le_exp hJpos).2 hJ +end ZCDP + +/-- **Lemma 3.2** of Bun–Steinke: a mechanism is `ε`-DP iff it is `(ε, 0)`-zCDP. -/ +theorem isZCDP_iff_isEpsDeltaDP [IsMarkovKernel M] {ε : ℝ} : + IsZCDP M Adj ε 0 ↔ IsEpsDeltaDP M Adj ε 0 := by + refine ⟨fun h S S' hS E ↦ ?_, fun h ↦ isZCDP_iff_toReal.2 fun S S' hS α hα ↦ ?_⟩ + · refine le_of_forall_sub_le fun δ hδ ↦ ?_ + have := h.isEpsDeltaDP hδ hS E + simp only [add_zero, zero_mul, sqrt_zero, mul_zero] at this + linarith + · simp only [zero_mul, add_zero] + refine ZCDP.renyiDiv_le_of_le_smul (fun E hE ↦ ?_) hα + rw [← ofReal_measureReal, ← ofReal_measureReal, ← ENNReal.ofReal_mul (exp_pos ε).le] + exact ENNReal.ofReal_le_ofReal (by simpa using h hS E) + +/-- **Proposition 1.4** of Bun–Steinke (proved as Proposition 3.3): an `ε`-DP mechanism is +`(ε² / 2)`-zCDP, for a symmetric neighbouring relation. The proof uses Hoeffding's lemma rather +than the paper's `sinh` inequality (Lemma B.1); see `ZCDP.renyiDiv_le_sq_of_le_smul`. -/ +theorem IsEpsDeltaDP.isZCDP [IsMarkovKernel M] [Std.Symm Adj] {ε : ℝ} + (h : IsEpsDeltaDP M Adj ε 0) : IsZCDP M Adj 0 (ε ^ 2 / 2) := by + refine isZCDP_iff_toReal.2 fun S S' hS α hα ↦ ?_ + have hle : ∀ ⦃S S'⦄, Adj S S' → ∀ E, MeasurableSet E → + M S E ≤ ENNReal.ofReal (exp ε) * M S' E := fun S S' hS E _ ↦ by + rw [← ofReal_measureReal, ← ofReal_measureReal, ← ENNReal.ofReal_mul (exp_pos ε).le] + exact ENNReal.ofReal_le_ofReal (by simpa using h hS E) + have hε : 0 ≤ ε := by + have := h hS Set.univ + simp only [probReal_univ, mul_one, add_zero] at this + exact one_le_exp_iff.1 this + rw [zero_add] + exact ZCDP.renyiDiv_le_sq_of_le_smul hε (hle hS) (hle (Std.Symm.symm _ _ hS)) hα + +/-- **Composition of pure DP via zCDP** (Proposition 1.4 combined with Lemma 1.7 of Bun–Steinke): +if `M i` is `ε`-DP for each of `k = |ι|` indices, the product mechanism `ZCDP.piKernel M`, +releasing independent outputs of all of them, is `(k ε² / 2)`-zCDP. -/ +theorem isZCDP_pi_of_isEpsDeltaDP {ι : Type*} [Fintype ι] {Y : ι → Type*} + [∀ i, MeasurableSpace (Y i)] {M : ∀ i, Kernel 𝒟 (Y i)} [∀ i, IsMarkovKernel (M i)] + [Std.Symm Adj] {ε : ℝ} (h : ∀ i, IsEpsDeltaDP (M i) Adj ε 0) : + IsZCDP (ZCDP.piKernel M) Adj 0 (Fintype.card ι * ε ^ 2 / 2) := by + have := IsZCDP.piKernel fun i ↦ (h i).isZCDP + simp only [Finset.sum_const_zero, Finset.sum_const, Finset.card_univ, nsmul_eq_mul] at this + rwa [mul_div_assoc] + +/-- **Composition of pure DP, approximate-DP form.** This bound is not stated in the paper; it +follows from Proposition 1.4, Lemma 1.7 and Proposition 1.3 of Bun–Steinke (it is a zCDP form +of the advanced composition theorem discussed in §1.3; it is not comparable with the paper's +Corollary 8.11, whose logarithmic term is `log (√(π / 2) ε √k / δ)`). The product mechanism +`ZCDP.piKernel M` of `k = |ι|` mechanisms, each `ε`-DP with `ε ≥ 0`, is +`(k ε² / 2 + ε √(2 k log (1 / δ)), δ)`-DP for every `δ > 0`. -/ +theorem isEpsDeltaDP_pi_of_isEpsDeltaDP {ι : Type*} [Fintype ι] {Y : ι → Type*} + [∀ i, MeasurableSpace (Y i)] {M : ∀ i, Kernel 𝒟 (Y i)} [∀ i, IsMarkovKernel (M i)] + [Std.Symm Adj] {ε : ℝ} (hε : 0 ≤ ε) + (h : ∀ i, IsEpsDeltaDP (M i) Adj ε 0) {δ : ℝ} (hδ : 0 < δ) : + IsEpsDeltaDP (ZCDP.piKernel M) Adj + (Fintype.card ι * ε ^ 2 / 2 + ε * √(2 * Fintype.card ι * log (1 / δ))) δ := by + have := (isZCDP_pi_of_isEpsDeltaDP h).isEpsDeltaDP' hδ + rwa [show (Fintype.card ι : ℝ) * ε ^ 2 / 2 * log (1 / δ) + = (ε / 2) ^ 2 * (2 * Fintype.card ι * log (1 / δ)) by ring, + sqrt_mul (sq_nonneg _), sqrt_sq (by linarith), + show 2 * (ε / 2 * √(2 * Fintype.card ι * log (1 / δ))) + = ε * √(2 * Fintype.card ι * log (1 / δ)) by ring] at this + +/-! ### The Gaussian mechanism -/ + +namespace ZCDP + +/-- The Radon–Nikodym derivative between Gaussians with the same variance `v ≠ 0`: +`dN(μ₁, v)/dN(μ₂, v) (x) = exp ((μ₁ - μ₂) x / v - (μ₁² - μ₂²) / (2 v))`. -/ +lemma rnDeriv_gaussianReal_gaussianReal (μ₁ μ₂ : ℝ) {v : ℝ≥0} (hv : v ≠ 0) : + (gaussianReal μ₁ v).rnDeriv (gaussianReal μ₂ v) =ᵐ[gaussianReal μ₂ v] + fun x ↦ ENNReal.ofReal (exp ((μ₁ - μ₂) / v * x - (μ₁ ^ 2 - μ₂ ^ 2) / (2 * v))) := by + have h1 := Measure.rnDeriv_eq_div (μ := gaussianReal μ₁ v) (ν := gaussianReal μ₂ v) + (gaussianReal_absolutelyContinuous μ₁ hv) (gaussianReal_absolutelyContinuous μ₂ hv) + filter_upwards [h1, (gaussianReal_absolutelyContinuous μ₂ hv).ae_le (rnDeriv_gaussianReal μ₁ v), + (gaussianReal_absolutelyContinuous μ₂ hv).ae_le (rnDeriv_gaussianReal μ₂ v)] with x h1 h2 h3 + rw [h1, h2, h3, gaussianPDF, gaussianPDF, + ← ENNReal.ofReal_div_of_pos (gaussianPDFReal_pos _ _ _ hv), gaussianPDFReal_def, + gaussianPDFReal_def] + congr 1 + have hs : 0 < (√(2 * π * v))⁻¹ := inv_pos.2 (sqrt_pos.2 (by positivity)) + rw [mul_div_mul_left _ _ hs.ne', ← exp_sub] + congr 1 + have : (v : ℝ) ≠ 0 := by exact_mod_cast hv + field_simp + ring + +/-- **Lemma 2.4** of Bun–Steinke, for orders `a > 1`: the Rényi divergence between Gaussians with +the same variance `v = σ² ≠ 0` is `D_a(N(μ₁, σ²) ‖ N(μ₂, σ²)) = a (μ₁ - μ₂)² / (2 σ²)`. The proof +writes `D_a` as `(a - 1)⁻¹ log E_{N(μ₁, σ²)}[(dN(μ₁, σ²)/dN(μ₂, σ²))^(a - 1)]` and evaluates the +expectation with the Gaussian moment generating function, instead of completing the square. -/ +lemma renyiDiv_gaussianReal (μ₁ μ₂ : ℝ) {v : ℝ≥0} (hv : v ≠ 0) {a : ℝ} (ha : 1 < a) : + renyiDiv a (gaussianReal μ₁ v) (gaussianReal μ₂ v) + = ENNReal.ofReal (a * (μ₁ - μ₂) ^ 2 / (2 * v)) := by + set P := gaussianReal μ₁ v + set Q := gaussianReal μ₂ v + have hPQ : P ≪ Q := (gaussianReal_absolutelyContinuous μ₁ hv).trans + (gaussianReal_absolutelyContinuous' μ₂ hv) + have hrn := rnDeriv_gaussianReal_gaussianReal μ₁ μ₂ hv + set s := (μ₁ - μ₂) / v + set c := (μ₁ ^ 2 - μ₂ ^ 2) / (2 * v) + have h_int : Integrable (fun x ↦ (P.rnDeriv Q x).toReal ^ a) Q := by + refine ((integrable_exp_mul_gaussianReal (a * s)).mul_const (exp (-(a * c)))).congr ?_ + filter_upwards [hrn] with x hx + rw [hx, ENNReal.toReal_ofReal (exp_pos _).le, ← exp_mul, ← exp_add] + congr 1 + ring + have hJ : ∫ x, (P.rnDeriv Q x).toReal ^ (a - 1) ∂P + = exp (-((a - 1) * c)) * exp (μ₁ * ((a - 1) * s) + v * ((a - 1) * s) ^ 2 / 2) := by + have hm := congrFun (mgf_fun_id_gaussianReal (μ := μ₁) (v := v)) ((a - 1) * s) + rw [← hm, mgf, ← integral_const_mul] + refine integral_congr_ae ?_ + filter_upwards [hPQ.ae_le hrn] with x hx + rw [hx, ENNReal.toReal_ofReal (exp_pos _).le, ← exp_mul, ← exp_add] + congr 1 + ring + rw [renyiDiv_eq_log_integral_of_one_lt' ha h_int hPQ, hJ, ← exp_add, log_exp] + simp only [measure_univ, ENNReal.toReal_one, log_one, mul_zero, add_zero, sub_zero] + congr 1 + have hv' : (v : ℝ) ≠ 0 := by exact_mod_cast hv + have ha' : a - 1 ≠ 0 := by linarith + simp only [s, c] + field_simp + ring +/-- The Gaussian mechanism `S ↦ N(q S, v)` as a Markov kernel, for a measurable query `q`. -/ +noncomputable def gaussianKernel (q : 𝒟 → ℝ) (hq : Measurable q) (v : ℝ≥0) : Kernel 𝒟 ℝ where + toFun S := gaussianReal (q S) v + measurable' := measurable_gaussianReal.comp (hq.prodMk measurable_const) + +/-- The value of the Gaussian mechanism at a dataset. -/ +lemma gaussianKernel_apply (q : 𝒟 → ℝ) (hq : Measurable q) (v : ℝ≥0) (S : 𝒟) : + gaussianKernel q hq v S = gaussianReal (q S) v := rfl + +instance (q : 𝒟 → ℝ) (hq : Measurable q) (v : ℝ≥0) : IsMarkovKernel (gaussianKernel q hq v) := + ⟨fun S ↦ by rw [gaussianKernel_apply]; infer_instance⟩ + +end ZCDP + +/-- **Proposition 1.6** of Bun–Steinke (Gaussian mechanism): if the query `q : 𝒟 → ℝ` has +sensitivity `Δ` (`|q S - q S'| ≤ Δ` for neighbours) and `M S = N(q S, σ²)` with variance +`v = σ² ≠ 0`, then `M` is `(Δ² / (2 σ²))`-zCDP. Such a kernel is `ZCDP.gaussianKernel` when `q` +is measurable (`ZCDP.isZCDP_gaussianKernel`). -/ +theorem isZCDP_of_gaussianReal {q : 𝒟 → ℝ} {Δ : ℝ} (hq : ∀ ⦃S S'⦄, Adj S S' → |q S - q S'| ≤ Δ) + {v : ℝ≥0} (hv : v ≠ 0) {M : Kernel 𝒟 ℝ} (hM : ∀ S, M S = gaussianReal (q S) v) : + IsZCDP M Adj 0 (Δ ^ 2 / (2 * v)) := by + refine isZCDP_iff_toReal.2 fun S S' hS α hα ↦ ?_ + rw [hM, hM, ZCDP.renyiDiv_gaussianReal _ _ hv hα] + have hv' : (0 : ℝ) < 2 * v := mul_pos two_pos (NNReal.coe_pos.2 (pos_iff_ne_zero.2 hv)) + refine ⟨ENNReal.ofReal_ne_top, ?_⟩ + rw [ENNReal.toReal_ofReal (div_nonneg (mul_nonneg (by linarith) (sq_nonneg _)) hv'.le), + zero_add, div_mul_eq_mul_div, mul_comm _ α] + have : (q S - q S') ^ 2 ≤ Δ ^ 2 := by + rw [← sq_abs] + exact pow_le_pow_left₀ (abs_nonneg _) (hq hS) 2 + exact div_le_div_of_nonneg_right (mul_le_mul_of_nonneg_left this (by linarith)) hv'.le + +/-- The Gaussian mechanism `ZCDP.gaussianKernel` is `(Δ² / (2 σ²))`-zCDP for a measurable query +of sensitivity `Δ` (Proposition 1.6 of Bun–Steinke). -/ +theorem ZCDP.isZCDP_gaussianKernel {q : 𝒟 → ℝ} (hq_meas : Measurable q) {Δ : ℝ} + (hq : ∀ ⦃S S'⦄, Adj S S' → |q S - q S'| ≤ Δ) {v : ℝ≥0} (hv : v ≠ 0) : + IsZCDP (gaussianKernel q hq_meas v) Adj 0 (Δ ^ 2 / (2 * v)) := + isZCDP_of_gaussianReal hq hv fun _ ↦ rfl + +end InformationTheory diff --git a/Statlib/Privacy/ZCDP/RenyiDP.lean b/Statlib/Privacy/ZCDP/RenyiDP.lean new file mode 100644 index 0000000..ab14cc8 --- /dev/null +++ b/Statlib/Privacy/ZCDP/RenyiDP.lean @@ -0,0 +1,64 @@ +/- +Copyright (c) 2026 Kousha Kalantari. All rights reserved. +Released under Apache 2.0 license as described in the file LICENSE. +Authors: Kousha Kalantari +-/ +module + +public import Statlib.Privacy.RenyiDP.Basic +public import Statlib.Privacy.ZCDP.Basic + +/-! +# Zero-concentrated DP as Rényi DP at every order + +Bun and Steinke (*Concentrated differential privacy: simplifications, extensions, and lower +bounds*, TCC 2016, arXiv:1605.02065, Definition 1.1) define `(ξ, ρ)`-zCDP by the bound +`D_α(M S ‖ M S') ≤ ξ + ρ α` for every order `α > 1`, which is Mironov's `(α, ξ + ρ α)`-Rényi +differential privacy (*Rényi differential privacy*, CSF 2017, arXiv:1702.07476, Definition 4) +holding simultaneously at every order `α > 1`. Both papers note this correspondence; this file +records it as an equivalence between the library's `IsZCDP` and `IsRenyiDP`. + +## Main statements + +* `isZCDP_iff_forall_isRenyiDP`: for `ξ, ρ ≥ 0`, `(ξ, ρ)`-zCDP holds iff the mechanism is + `(α, ξ + ρ α)`-RDP for every `α > 1`. +* `isZCDP_zero_iff_forall_isRenyiDP`: the case `ξ = 0` (`ρ`-zCDP). +-/ + +@[expose] public section + +open MeasureTheory ProbabilityTheory +open scoped ENNReal NNReal + +namespace InformationTheory + +variable {𝒟 Y : Type*} {m𝒟 : MeasurableSpace 𝒟} {mY : MeasurableSpace Y} + {M : Kernel 𝒟 Y} {Adj : 𝒟 → 𝒟 → Prop} + +/-- **zCDP is RDP at every order**: for nonnegative `ξ, ρ`, the mechanism is `(ξ, ρ)`-zCDP +(Bun–Steinke, Definition 1.1) iff it is `(α, ξ + ρ α)`-Rényi DP (Mironov, Definition 4) for every +order `α > 1`. -/ +theorem isZCDP_iff_forall_isRenyiDP (ξ ρ : ℝ≥0) : + IsZCDP M Adj ξ ρ ↔ ∀ α : ℝ, 1 < α → IsRenyiDP M Adj α (ξ + ρ * α.toNNReal) := by + rw [isZCDP_iff_le_ofReal ξ.coe_nonneg ρ.coe_nonneg] + have key : ∀ α : ℝ, 1 < α → + ENNReal.ofReal ((ξ : ℝ) + ρ * α) = ((ξ + ρ * α.toNNReal : ℝ≥0) : ℝ≥0∞) := by + intro α hα + rw [← ENNReal.ofReal_coe_nnreal] + congr 1 + push_cast + rw [Real.coe_toNNReal _ (by linarith)] + constructor + · intro h α hα S S' hS + rw [← key α hα] + exact h hS α hα + · intro h S S' hS α hα + rw [key α hα] + exact h α hα hS + +/-- `ρ`-zCDP iff `(α, ρ α)`-Rényi DP at every order `α > 1`. -/ +theorem isZCDP_zero_iff_forall_isRenyiDP (ρ : ℝ≥0) : + IsZCDP M Adj 0 ρ ↔ ∀ α : ℝ, 1 < α → IsRenyiDP M Adj α (ρ * α.toNNReal) := by + simpa using isZCDP_iff_forall_isRenyiDP (M := M) (Adj := Adj) 0 ρ + +end InformationTheory From e7a303fb4ee903b0901bc733a9d5705a8c34101d Mon Sep 17 00:00:00 2001 From: Kousha Kalantari <25445970+KoushaKalantari@users.noreply.github.com> Date: Fri, 13 Nov 2026 11:31:18 -0800 Subject: [PATCH 5/5] =?UTF-8?q?feat(Privacy):=20advanced=20composition=20(?= =?UTF-8?q?Dwork=E2=80=93Rothblum=E2=80=93Vadhan)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- Statlib.lean | 1 + .../Privacy/AdvancedComposition/Basic.lean | 541 ++++++++++++++++++ 2 files changed, 542 insertions(+) create mode 100644 Statlib/Privacy/AdvancedComposition/Basic.lean diff --git a/Statlib.lean b/Statlib.lean index 6e45456..e2dfd54 100644 --- a/Statlib.lean +++ b/Statlib.lean @@ -138,6 +138,7 @@ import Statlib.ForMathlib.Probability.StrongLaw import Statlib.ForMathlib.Topology.Order.MonotoneConvergence import Statlib.Inference.Inference import Statlib.Inference.QMD +import Statlib.Privacy.AdvancedComposition.Basic import Statlib.Privacy.DP.Basic import Statlib.Privacy.DP.Exponential import Statlib.Privacy.DP.Geometric diff --git a/Statlib/Privacy/AdvancedComposition/Basic.lean b/Statlib/Privacy/AdvancedComposition/Basic.lean new file mode 100644 index 0000000..a1b61b0 --- /dev/null +++ b/Statlib/Privacy/AdvancedComposition/Basic.lean @@ -0,0 +1,541 @@ +/- +Copyright (c) 2026 Kousha Kalantari. All rights reserved. +Released under Apache 2.0 license as described in the file LICENSE. +Authors: Kousha Kalantari +-/ +module + +public import Statlib.Privacy.DP.Basic +public import Statlib.Privacy.DP.RandomizedResponse +public import Statlib.Privacy.ZCDP.Basic +public import Mathlib.Analysis.Complex.ExponentialBounds + +/-! +# The advanced composition theorem + +This file formalizes the advanced composition theorem of differential privacy for `ε`-DP +mechanisms, in its adaptive `k`-fold form, from + +* C. Dwork, G. N. Rothblum and S. Vadhan, *Boosting and differential privacy*, FOCS 2010 + (DRV), Theorem III.3, and +* C. Dwork and A. Roth, *The Algorithmic Foundations of Differential Privacy*, Found. Trends + Theor. Comput. Sci. 9(3–4), 2014, Theorem 3.20 and Corollary 3.21 (page numbers below are the + printed ones of the PDF at https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf): + +the `k`-fold adaptive composition of `ε`-DP mechanisms is `(ε', δ')`-DP for every `δ' > 0`, where +`ε' = √(2 k ln (1 / δ')) ε + k ε (e^ε - 1)`. + +A mechanism is a kernel from datasets to outputs, `Adj` is a neighbouring relation on datasets, +assumed symmetric (as the book's `‖x - y‖₁ ≤ 1` is), and `(ε, δ)`-DP is `IsEpsDeltaDP` (pure +`ε`-DP is `IsEpsDP`), both from `Statlib.Privacy.DP.Basic`. Output spaces are +**arbitrary measurable spaces**; the adaptive results assume the common output space `Y` of the +rounds countably generated (e.g. countable, or standard Borel). + +## Route + +The proof is not the book's (privacy loss random variable, the bound `D(Y ‖ Z) ≤ ε (e^ε - 1)` of +Lemma 3.18 and Azuma's inequality). It follows the zero-concentrated DP route of M. Bun and +T. Steinke, *Concentrated differential privacy: simplifications, extensions, and lower bounds*, +TCC 2016 ([arXiv:1605.02065](https://arxiv.org/abs/1605.02065), v1), with the results of +`Statlib.Privacy.ZCDP.Basic`: + +1. an `ε`-DP mechanism is `(ε² / 2)`-zCDP (Proposition 1.4, `IsEpsDeltaDP.isZCDP`); +2. zCDP adds up under adaptive composition (Lemma 2.3, `IsZCDP.compProd`, iterated here as + `AdvancedComposition.isZCDP_adaptiveCompose`), so the `k`-fold composition is + `(k ε² / 2)`-zCDP; +3. `ρ`-zCDP implies `(ρ + 2 √(ρ ln (1 / δ')), δ')`-DP (Proposition 1.3, `IsZCDP.isEpsDeltaDP'`), + which for `ρ = k ε² / 2` is `(k ε² / 2 + ε √(2 k ln (1 / δ')), δ')`-DP; +4. `k ε² / 2 ≤ k ε (e^ε - 1)`. + +This route gives the sharper second term `k ε² / 2` in place of `k ε (e^ε - 1)`. Bun and Steinke +note (Section 1.2.1, after Proposition 1.4) that Dwork and Rothblum's version of Proposition 1.4 +gives `(½ ε (e^ε - 1))`-zCDP, a factor `½` better than the earlier bound of DRV, and that +Proposition 1.4 is slightly stronger still (Remark 3.4 compares the KL bounds `½ ε²` and +`½ ε (e^ε - 1)`); after their Corollary 8.11 (Section 8.3), whose first term `½ ‖ε‖₂²` is +`k ε² / 2` for equal `ε_i`, they note that it saves a constant factor in the first term over the +composition theorem of DRV. The exact bound `k ε² / 2 + ε √(2 k ln (1 / δ'))` is not displayed in +the paper. + +## Main definitions + +* `AdvancedComposition.adaptiveCompose M k`: the `k`-fold adaptive composition of mechanisms + `M n : Kernel (𝒟 × (Fin n → Y)) Y`, the `n`-th of which sees the dataset and the `n` earlier + outputs. + +## Main statements (paper result → Lean name, in namespace `AdvancedComposition`) + +* Theorem 3.20 of Dwork–Roth for `(ε, 0)`-DP mechanisms, adaptive (for a fixed neighbouring + relation and history-indexed mechanisms, see the implementation notes): + `isEpsDeltaDP_adaptiveCompose`; sharper zCDP form `isEpsDeltaDP_adaptiveCompose_sq`; with + per-round parameters `ε n`, zCDP form: `isEpsDeltaDP_adaptiveCompose_sum_sq`. Theorem III.3 of + DRV assumes `ε > 0` and `δ, δ' > 0`, so it gives the case `δ = 0` only in the limit `δ → 0`; + here `δ = 0` is proved directly. +* The same for `k` mechanisms run independently on the same dataset (non-adaptive): + `isEpsDeltaDP_composeFin` and `isEpsDeltaDP_composeFin_sq` (for `DP.composeFin`, with + possibly different output spaces), `isEpsDeltaDP_piKernel` (for `ZCDP.piKernel`). +* Composition of zCDP (Lemmas 1.7 and 2.3 of Bun–Steinke, `k`-fold): `isZCDP_adaptiveCompose`, + `isZCDP_composeFin`; for `ε`-DP rounds, `isZCDP_adaptiveCompose_of_isEpsDP` (`k ε² / 2`). +* Corollary 3.21 of Dwork–Roth for `(ε, 0)`-DP mechanisms, under the sufficient additional + hypothesis `δ' ≤ e^{-ε'/8}` (see the notes below): + `isEpsDeltaDP_adaptiveCompose_of_le_eight_mul_log` (stated as `ε' ≤ 8 ln (1 / δ')`), + `isEpsDeltaDP_adaptiveCompose_of_le_one` (the book's `ε' ≤ 1` and `δ' ≤ e^{-1/8}`); a + counterexample to the printed statement with `δ'` close to `1`, outside the book's intended + regime of negligible `δ'`: `exists_not_isEpsDeltaDP_adaptiveCompose`. +* A `(1, e^{-32})`-DP guarantee in the setting of Example 3.7 of Dwork–Roth: + `isEpsDeltaDP_adaptiveCompose_ten_thousand`. + +## Implementation notes + +* **The composition experiment.** The results here are stated for one fixed neighbouring + relation `Adj` and history-indexed mechanisms, which is a special case of the book's + adversary experiment; the encoding of the experiment below is informal (not formalized). + Dwork–Roth (Section 3.5.2, pp. 48–49) and DRV (Section III.A) + define `k`-fold adaptive composition through an experiment: in round `i` an adversary chooses, + depending on the earlier outputs, two adjacent databases `x_i^0, x_i^1` and a mechanism `M_i` + from the family, and receives an output of `M_i(x_i^b)`; the family is `(ε', δ')`-DP under + `k`-fold adaptive composition if the adversary's views for `b = 0, 1` are + `(ε', δ')`-indistinguishable. Here round `n` is a kernel `M n : Kernel (𝒟 × (Fin n → Y)) Y` + that must be `ε`-DP in the dataset for every history `x` + (`∀ x, IsEpsDP ((M n).sectL x) Adj ε`), and the conclusion is `(ε', δ')`-DP of the composed + kernel. The experiment is the instance `𝒟 = Bool` (the secret bit `b`) with + `Adj = fun _ _ ↦ True` and `M n (b, x)` the law of the round-`n` output in experiment `b` after + history `x`: the hypothesis then says that the mechanism chosen after `x`, run on the two + databases chosen after `x`, gives `ε`-indistinguishable outputs in both directions, which holds + because that mechanism is `ε`-DP and the databases are adjacent. The adversary's choices must be + measurable in the history (this is part of `M n` being a kernel), all rounds share the output + space `Y`, and the adversary's coins, which the book includes in the view, can be released by + an extra first round that does not depend on `b`, with parameter `0` + (`isEpsDeltaDP_adaptiveCompose_sum_sq` allows per-round parameters). +* Countable generation of `Y` is the hypothesis of Statlib's chain rule for the Rényi divergence, + on which `IsZCDP.compProd` rests. The non-adaptive results need no such hypothesis. +* The book allows `δ' = 0` (for which `ε' = ∞`); here `δ' > 0`. For `δ' ≥ 1`, Lean's + `√(2 k ln (1 / δ'))` is `0`, and the conclusion is then the true statement that a Markov kernel + is `(ε', δ')`-DP whenever `δ' ≥ 1`. As in the book, `ε ≥ 0`; DRV assume `ε > 0` and + `δ, δ' > 0`. +* The Corollary 3.21 statements are totalized extensions of the book's, whose formula + presupposes `k ≥ 1` and whose hypotheses include `0 < ε' < 1`: both allow `k = 0`, and + `isEpsDeltaDP_adaptiveCompose_of_le_eight_mul_log` also allows `ε' = 0` with `δ' = 1`. There + the prescribed `ε = ε' / (2 √(2 k ln (1 / δ')))` has denominator `0`, so Lean's `ε` is `0`. + These cases are harmless: for `k = 0` the hypothesis on the rounds is vacuous and the + conclusion is that the constant `0`-fold composition is `(ε', δ')`-DP, and for `δ' = 1` the + conclusion holds for every Markov kernel. + +## Notes on the paper + +* Corollary 3.21 of Dwork–Roth (Section 3.5.2, p. 52), read literally, fails for `δ'` close to + `1`, outside the regime it is meant for. The corollary asserts that, for `0 < ε' < 1` and + `δ' > 0`, `k` mechanisms that are each `(ε, δ)`-DP with `ε = ε' / (2 √(2 k ln (1 / δ')))` + compose to an `(ε', k δ + δ')`-DP mechanism; it places no further restriction on `δ'` or `k`. + For `k = 1`, `δ = 0`, `ε' = 9/10` and `δ' = e^{-1/800} ≈ 0.99875`, this `ε` is `9`, and binary + randomized response with parameter `9` is `9`-DP but not `(9/10, e^{-1/800})`-DP: it is + `(9/10, δ')`-DP only for `δ' ≥ (e⁹ - e^{9/10}) / (1 + e⁹) ≈ 0.99957` + (`exists_not_isEpsDeltaDP_adaptiveCompose`). The book states (Section 2.3, p. 25) that + "typically, and throughout this monograph, we require that `δ` be a negligible function" of a + security parameter, so the corollary is meant for cryptographically small `δ'`; the + counterexample is a failure of the unrestricted printed formula outside that intended regime, + not of the corollary as used in the book. A sufficient additional hypothesis is + `δ' ≤ e^{-ε'/8}` (`isEpsDeltaDP_adaptiveCompose_of_le_eight_mul_log`); it is not necessary + (for `k = 1` and `ε' = δ' = 9/10` it fails, but the prescribed `ε ≈ 0.9803`, and an `ε`-DP + mechanism is `(9/10, 1 - e^{9/10 - ε})`-DP with `1 - e^{9/10 - ε} ≈ 0.077 < 9/10`; not + formalized). Every counterexample with `δ = 0` has `δ' > e^{-ε'/8} > 0.88` for + `ε' < 1`; only `δ = 0` is examined here. The proof has two gaps. + It takes the bound of Theorem 3.20 to be `ε* = √(2 k ln (1 / δ')) ε + k ε²`, whereas + Theorem 3.20 gives `k ε (e^ε - 1) ≥ k ε²` in the second term (for `δ = 0` the zCDP route does + give `k ε² / 2`, so this step is repairable there). And "when `ε' < 1`, we have that + `ε* ≤ ε'`" holds if and only if `ε' ≤ 4 ln (1 / δ')`, which fails for `δ'` close to `1`. For + `δ = 0` the corollary holds when `δ' ≤ e^{-ε'/8}`, i.e. `ε' ≤ 8 ln (1 / δ')`, for instance when + `ε' ≤ 1` and `δ' ≤ e^{-1/8}` (`isEpsDeltaDP_adaptiveCompose_of_le_eight_mul_log`, + `isEpsDeltaDP_adaptiveCompose_of_le_one`). +* Example 3.7 of Dwork–Roth (p. 52) states that for `k = 10 000` mechanisms, "Theorem 3.20 says + that, taking `δ' = e^{-32}` it suffices to have `ε₀ ≤ 1/801`" for the cumulative privacy loss to + be at most `1` with probability at least `1 - e^{-32}`. The numerical conclusion appears to be + true: the exact optimal composition (Kairouz–Oh–Viswanath, ICML 2015) gives + `δ(1) ≈ e^{-38.8}` at `ε₀ = 1/801` (checked numerically only, not formalized). Only the + attribution to Theorem 3.20 is wrong: it accounts only for the first term + `√(2 k ln (1 / δ')) ε₀ = 800 ε₀`, and at `ε₀ = 1/801` the bound of Theorem 3.20 is + `ε' ≈ 1.014` (it gives `ε' ≤ 1` only for `ε₀ ≤ 1/812.3`, approximately). The sharper bound + `isEpsDeltaDP_adaptiveCompose_sq` gives `ε' ≈ 1.007` at `ε₀ = 1/801` and `ε' ≤ 1` for + `ε₀ ≤ 1/806.2` (approximately), in particular at `ε₀ = 1/807` + (`isEpsDeltaDP_adaptiveCompose_ten_thousand`). The book states a tail bound on the privacy + loss, which is stronger than the `(1, e^{-32})`-DP proved here. +* In the proof of Theorem 3.20 (pp. 50–51), Azuma's inequality is applied with "`β = ε · ε₀`", + where `ε₀` is not defined in the book (it is DRV's `ε₀ = e^ε - 1`), so `β = ε (e^ε - 1)` is + meant; the pure case writes `δ` for `δ'` (in `Pr[V⁰ ∈ B] ≤ δ` and `z = √(2 ln (1 / δ))`); and the + last line should read `D_∞^{δ' + kδ}(V⁰ ‖ V¹) ≤ ε'` rather than `(V⁰ ‖ W)`. These are typos; the + theorem is true for `(ε, 0)`-DP mechanisms (`isEpsDeltaDP_adaptiveCompose`). + +## Not formalized + +* Theorem 3.20 (Theorem III.3 of DRV) and Corollary 3.21 for `(ε, δ)`-DP mechanisms with `δ > 0` + (total failure probability `k δ + δ'`), whose proof uses the characterization of approximate + max-divergence (Lemma 3.17 of Dwork–Roth). +* The book's proof through the privacy loss, Lemma 3.18 and Azuma's inequality (Lemma 3.19), and + the adversary-view formulation of the experiment as such (see the implementation notes). +-/ + +@[expose] public section + +open MeasureTheory ProbabilityTheory Real +open scoped ENNReal + +namespace InformationTheory + +namespace AdvancedComposition + +variable {𝒟 Y : Type*} {m𝒟 : MeasurableSpace 𝒟} {mY : MeasurableSpace Y} + {Adj : 𝒟 → 𝒟 → Prop} + +/-! ### Adaptive composition -/ + +/-- Appending an element to a tuple, `(x, y) ↦ (x₀, …, x_{n-1}, y)`, is measurable. -/ +lemma measurable_snoc (n : ℕ) : + Measurable fun p : (Fin n → Y) × Y ↦ (Fin.snoc p.1 p.2 : Fin (n + 1) → Y) := by + refine measurable_pi_iff.2 fun i ↦ ?_ + induction i using Fin.lastCases with + | last => simpa only [Fin.snoc_last] using measurable_snd + | cast j => + simp only [Fin.snoc_castSucc] + exact (measurable_pi_apply j).comp measurable_fst + +/-- The **`k`-fold adaptive composition** of the mechanisms `M n`, `n < k`: on a dataset `S`, it +draws `y₀ ∼ M 0 (S, ())`, then `y₁ ∼ M 1 (S, (y₀))`, …, `y_{k-1} ∼ M (k-1) (S, (y₀, …, y_{k-2}))`, +and releases `(y₀, …, y_{k-1})`. The `n`-th mechanism `M n : Kernel (𝒟 × (Fin n → Y)) Y` sees the +dataset and the outputs of the `n` earlier ones (its *history*), so that it may be chosen +adaptively. It is defined by recursion on `k` with `Kernel.compProd`. -/ +noncomputable def adaptiveCompose (M : (n : ℕ) → Kernel (𝒟 × (Fin n → Y)) Y) : + (k : ℕ) → Kernel 𝒟 (Fin k → Y) + | 0 => Kernel.const 𝒟 (Measure.dirac isEmptyElim) + | k + 1 => (adaptiveCompose M k ⊗ₖ M k).map fun p ↦ (Fin.snoc p.1 p.2 : Fin (k + 1) → Y) + +variable {M : (n : ℕ) → Kernel (𝒟 × (Fin n → Y)) Y} + +lemma adaptiveCompose_zero (M : (n : ℕ) → Kernel (𝒟 × (Fin n → Y)) Y) : + adaptiveCompose M 0 = Kernel.const 𝒟 (Measure.dirac isEmptyElim) := rfl + +lemma adaptiveCompose_succ (M : (n : ℕ) → Kernel (𝒟 × (Fin n → Y)) Y) (k : ℕ) : + adaptiveCompose M (k + 1) = + (adaptiveCompose M k ⊗ₖ M k).map fun p ↦ (Fin.snoc p.1 p.2 : Fin (k + 1) → Y) := rfl + +instance isMarkovKernel_adaptiveCompose (M : (n : ℕ) → Kernel (𝒟 × (Fin n → Y)) Y) + [∀ n, IsMarkovKernel (M n)] (k : ℕ) : IsMarkovKernel (adaptiveCompose M k) := by + induction k with + | zero => rw [adaptiveCompose_zero]; infer_instance + | succ k ih => rw [adaptiveCompose_succ]; exact Kernel.IsMarkovKernel.map _ (measurable_snoc k) + +/-- The first output of the adaptive composition is the output of `M 0` (with the empty +history). -/ +lemma map_adaptiveCompose_one (M : (n : ℕ) → Kernel (𝒟 × (Fin n → Y)) Y) + [∀ n, IsMarkovKernel (M n)] : + (adaptiveCompose M 1).map (fun y ↦ y 0) = (M 0).sectL isEmptyElim := by + ext S E hE + rw [Kernel.map_apply' _ (measurable_pi_apply 0) _ hE, adaptiveCompose_succ, + Kernel.map_apply' _ (measurable_snoc 0) _ (measurable_pi_apply 0 hE), + Kernel.compProd_apply ((measurable_snoc 0) (measurable_pi_apply 0 hE)), adaptiveCompose_zero, + Kernel.const_apply, lintegral_dirac' _ Subsingleton.measurable, Kernel.sectL_apply] + rfl + +/-! ### Composition of zCDP mechanisms -/ + +/-- A constant mechanism with a probability distribution as output is `(0, 0)`-zCDP. -/ +lemma isZCDP_const (μ : Measure Y) [IsProbabilityMeasure μ] : + IsZCDP (Kernel.const 𝒟 μ) Adj 0 0 := + isZCDP_iff_toReal.2 fun S S' _ α _ ↦ by simp [renyiDiv_self] + +/-- **Adaptive composition of `k` zCDP mechanisms** (Lemma 2.3 of Bun–Steinke, iterated): if, for +every `n < k` and every history `x`, the mechanism `S ↦ M n (S, x)` is `(ξ n, ρ n)`-zCDP, then +the `k`-fold adaptive composition is `(∑_{n simpa [adaptiveCompose_zero] using isZCDP_const (𝒟 := 𝒟) (Adj := Adj) _ + | succ k ih => + rw [adaptiveCompose_succ, Finset.sum_range_succ, Finset.sum_range_succ] + exact ((ih fun n hn ↦ h n (hn.trans k.lt_succ_self)).compProd + (h k k.lt_succ_self)).map (measurable_snoc k) + +/-- **Composition of `k` zCDP mechanisms**, non-adaptive form for the `k`-fold composition +`DP.composeFin` of `Statlib.Privacy.DP.Basic` (Lemma 1.7 of Bun–Steinke, iterated): if `M i` is +`(ξ i, ρ i)`-zCDP for every `i < k`, then `S ↦ (M₀(S), …, M_{k-1}(S))` is +`(∑ ξ i, ∑ ρ i)`-zCDP. Arbitrary measurable output spaces `R i`. -/ +theorem isZCDP_composeFin {k : ℕ} {R : Fin k → Type*} [∀ i, MeasurableSpace (R i)] + {M : (i : Fin k) → Kernel 𝒟 (R i)} [∀ i, IsMarkovKernel (M i)] {ξ ρ : Fin k → ℝ} + (h : ∀ i, IsZCDP (M i) Adj (ξ i) (ρ i)) : + IsZCDP (DP.composeFin M) Adj (∑ i, ξ i) (∑ i, ρ i) := by + induction k with + | zero => simpa [DP.composeFin] using isZCDP_const (𝒟 := 𝒟) (Adj := Adj) _ + | succ k ih => + rw [DP.composeFin, Fin.sum_univ_succAbove _ 0, Fin.sum_univ_succAbove _ 0] + exact ((h 0).prod (ih (M := fun j ↦ M (Fin.succAbove 0 j)) + (ξ := fun j ↦ ξ (Fin.succAbove 0 j)) (ρ := fun j ↦ ρ (Fin.succAbove 0 j)) + fun j ↦ h _)).map (MeasurableEquiv.measurable _) + +/-! ### Advanced composition of pure-DP mechanisms -/ + +/-- `ε² / 2 ≤ ε (e^ε - 1)` for `ε ≥ 0`: the per-mechanism term of the zCDP route is at most the +term `ε (e^ε - 1)` of Dwork–Roth's Theorem 3.20. -/ +lemma sq_div_two_le_mul_exp_sub_one {ε : ℝ} (hε : 0 ≤ ε) : ε ^ 2 / 2 ≤ ε * (exp ε - 1) := by + nlinarith [add_one_le_exp ε] + +/-- `2 √((k ε² / 2) L) = ε √(2 k L)` for `ε ≥ 0` (for every real `k` and `L`, as `√` of a negative +number is `0`). -/ +lemma two_mul_sqrt_eq {k ε L : ℝ} (hε : 0 ≤ ε) : + 2 * √(k * ε ^ 2 / 2 * L) = ε * √(2 * k * L) := by + rw [show k * ε ^ 2 / 2 * L = (ε / 2) ^ 2 * (2 * k * L) by ring, sqrt_mul (sq_nonneg _), + sqrt_sq (by linarith)] + ring + +/-- **Adaptive composition of `k` pure-DP mechanisms is `(k ε² / 2)`-zCDP** (Proposition 1.4 and +Lemma 2.3 of Bun–Steinke): if, for every `n < k` and every history `x`, the mechanism +`S ↦ M n (S, x)` is `ε`-DP for the symmetric neighbouring relation `Adj`, then the `k`-fold +adaptive composition is `(k ε² / 2)`-zCDP. `Y` is countably generated. -/ +theorem isZCDP_adaptiveCompose_of_isEpsDP [MeasurableSpace.CountablyGenerated Y] [Std.Symm Adj] + [∀ n, IsMarkovKernel (M n)] {ε : ℝ} {k : ℕ} + (h : ∀ n < k, ∀ x, IsEpsDP ((M n).sectL x) Adj ε) : + IsZCDP (adaptiveCompose M k) Adj 0 (k * ε ^ 2 / 2) := by + simpa [mul_div_assoc] using isZCDP_adaptiveCompose (ξ := fun _ ↦ 0) (ρ := fun _ ↦ ε ^ 2 / 2) + fun n hn x ↦ (h n hn x).isZCDP + +/-- **Advanced composition, zCDP form** (the bound obtained from Proposition 1.4, Lemma 2.3 and +Proposition 1.3 of Bun–Steinke): if, for every `n < k` and every history `x`, the mechanism +`S ↦ M n (S, x)` is `ε`-DP (`ε ≥ 0`) for the symmetric neighbouring relation `Adj`, then the +`k`-fold adaptive composition is `(k ε² / 2 + ε √(2 k ln (1 / δ)), δ)`-DP for every `δ > 0`. +This is Theorem 3.20 of Dwork–Roth for pure-DP mechanisms with the sharper second term `k ε² / 2` +in place of `k ε (e^ε - 1)` (see `isEpsDeltaDP_adaptiveCompose`). `Y` is countably generated. + +The hypothesis fixes one neighbouring relation `Adj` and lets the mechanisms depend on the +history; this is a special case of the adversary experiment of Dwork–Roth (Section 3.5.2), into +which that experiment can be encoded only informally (with `𝒟 = Bool`, see the module +documentation). Releasing the adversary's coins needs an extra round with parameter `0`, which +only `isEpsDeltaDP_adaptiveCompose_sum_sq` allows. -/ +theorem isEpsDeltaDP_adaptiveCompose_sq [MeasurableSpace.CountablyGenerated Y] [Std.Symm Adj] + [∀ n, IsMarkovKernel (M n)] {ε : ℝ} (hε : 0 ≤ ε) {k : ℕ} + (h : ∀ n < k, ∀ x, IsEpsDP ((M n).sectL x) Adj ε) {δ : ℝ} (hδ : 0 < δ) : + IsEpsDeltaDP (adaptiveCompose M k) Adj (k * ε ^ 2 / 2 + ε * √(2 * k * log (1 / δ))) δ := by + have := (isZCDP_adaptiveCompose_of_isEpsDP h).isEpsDeltaDP' hδ + rwa [two_mul_sqrt_eq hε] at this + +/-- **Advanced composition with per-round parameters, zCDP form** (Proposition 1.4, Lemma 2.3 and +Proposition 1.3 of Bun–Steinke): if, for every `n < k` and every history `x`, the mechanism +`S ↦ M n (S, x)` is `ε n`-DP for the symmetric neighbouring relation `Adj`, then, with +`ρ = ∑_{n 0`. `Y` is countably generated. -/ +theorem isEpsDeltaDP_adaptiveCompose_sum_sq [MeasurableSpace.CountablyGenerated Y] + [Std.Symm Adj] [∀ n, IsMarkovKernel (M n)] {ε : ℕ → ℝ} {k : ℕ} + (h : ∀ n < k, ∀ x, IsEpsDP ((M n).sectL x) Adj (ε n)) {δ : ℝ} (hδ : 0 < δ) : + IsEpsDeltaDP (adaptiveCompose M k) Adj ((∑ n ∈ Finset.range k, ε n ^ 2 / 2) + + 2 * √((∑ n ∈ Finset.range k, ε n ^ 2 / 2) * log (1 / δ))) δ := by + have h0 := isZCDP_adaptiveCompose (ξ := fun _ ↦ 0) (ρ := fun n ↦ ε n ^ 2 / 2) + fun n hn x ↦ (h n hn x).isZCDP + rw [Finset.sum_const_zero] at h0 + exact h0.isEpsDeltaDP' hδ + +/-- **Advanced composition theorem** (Theorem 3.20 of Dwork–Roth for `(ε, 0)`-DP mechanisms): +if, for every `n < k` and every history `x`, the mechanism `S ↦ M n (S, x)` is `ε`-DP (`ε ≥ 0`) +for the symmetric neighbouring relation `Adj`, then the `k`-fold adaptive composition is +`(ε', δ')`-DP for every `δ' > 0`, where `ε' = √(2 k ln (1 / δ')) ε + k ε (e^ε - 1)`. `Y` is +countably generated. + +Theorem III.3 of Dwork–Rothblum–Vadhan states the same bound for `ε > 0` and `δ, δ' > 0` (total +failure probability `k δ + δ'`); the case `δ = 0` proved here follows from it only by letting +`δ → 0`. + +The hypothesis fixes one neighbouring relation `Adj` and lets the mechanisms depend on the +history; this is a special case of the adversary experiment of Dwork–Roth (Section 3.5.2), into +which that experiment can be encoded only informally (with `𝒟 = Bool`, see the module +documentation). Releasing the adversary's coins needs an extra round with parameter `0`, which +only `isEpsDeltaDP_adaptiveCompose_sum_sq` allows. -/ +theorem isEpsDeltaDP_adaptiveCompose [MeasurableSpace.CountablyGenerated Y] [Std.Symm Adj] + [∀ n, IsMarkovKernel (M n)] {ε : ℝ} (hε : 0 ≤ ε) {k : ℕ} + (h : ∀ n < k, ∀ x, IsEpsDP ((M n).sectL x) Adj ε) {δ' : ℝ} (hδ' : 0 < δ') : + IsEpsDeltaDP (adaptiveCompose M k) Adj + (√(2 * k * log (1 / δ')) * ε + k * ε * (exp ε - 1)) δ' := + (isEpsDeltaDP_adaptiveCompose_sq hε h hδ').mono (by + nlinarith [mul_le_mul_of_nonneg_left (sq_div_two_le_mul_exp_sub_one hε) + (Nat.cast_nonneg (α := ℝ) k)]) le_rfl + +/-- **Advanced composition, non-adaptive form, zCDP bound**: if `M i` is `ε`-DP (`ε ≥ 0`) for +every `i < k`, for the symmetric neighbouring relation `Adj`, then the `k`-fold composition +`S ↦ (M₀(S), …, M_{k-1}(S))` (`DP.composeFin`) is `(k ε² / 2 + ε √(2 k ln (1 / δ)), δ)`-DP for +every `δ > 0`. Arbitrary measurable output spaces `R i`. -/ +theorem isEpsDeltaDP_composeFin_sq [Std.Symm Adj] {k : ℕ} {R : Fin k → Type*} + [∀ i, MeasurableSpace (R i)] {M : (i : Fin k) → Kernel 𝒟 (R i)} [∀ i, IsMarkovKernel (M i)] + {ε : ℝ} (hε : 0 ≤ ε) (h : ∀ i, IsEpsDP (M i) Adj ε) {δ : ℝ} (hδ : 0 < δ) : + IsEpsDeltaDP (DP.composeFin M) Adj (k * ε ^ 2 / 2 + ε * √(2 * k * log (1 / δ))) δ := by + have h0 := isZCDP_composeFin (ξ := fun _ ↦ 0) (ρ := fun _ ↦ ε ^ 2 / 2) fun i ↦ (h i).isZCDP + simp only [Finset.sum_const_zero, Finset.sum_const, Finset.card_univ, Fintype.card_fin, + nsmul_eq_mul, ← mul_div_assoc] at h0 + have := h0.isEpsDeltaDP' hδ + rwa [two_mul_sqrt_eq hε] at this + +/-- **Advanced composition theorem, non-adaptive form** (Theorem 3.20 of Dwork–Roth for `k` +mechanisms run independently on the same dataset, `(ε, 0)`-DP inputs): if `M i` is `ε`-DP +(`ε ≥ 0`) for every `i < k`, for the symmetric neighbouring relation `Adj`, then +`S ↦ (M₀(S), …, M_{k-1}(S))` (`DP.composeFin`) is +`(√(2 k ln (1 / δ')) ε + k ε (e^ε - 1), δ')`-DP for every `δ' > 0`. Arbitrary measurable output +spaces `R i`. -/ +theorem isEpsDeltaDP_composeFin [Std.Symm Adj] {k : ℕ} {R : Fin k → Type*} + [∀ i, MeasurableSpace (R i)] {M : (i : Fin k) → Kernel 𝒟 (R i)} [∀ i, IsMarkovKernel (M i)] + {ε : ℝ} (hε : 0 ≤ ε) (h : ∀ i, IsEpsDP (M i) Adj ε) {δ' : ℝ} (hδ' : 0 < δ') : + IsEpsDeltaDP (DP.composeFin M) Adj + (√(2 * k * log (1 / δ')) * ε + k * ε * (exp ε - 1)) δ' := + (isEpsDeltaDP_composeFin_sq hε h hδ').mono (by + nlinarith [mul_le_mul_of_nonneg_left (sq_div_two_le_mul_exp_sub_one hε) + (Nat.cast_nonneg (α := ℝ) k)]) le_rfl + +/-- **Advanced composition theorem for the product kernel** `ZCDP.piKernel` (Theorem 3.20 of +Dwork–Roth, non-adaptive, `(ε, 0)`-DP inputs): if each of the `k = |ι|` mechanisms `M i` is `ε`-DP +(`ε ≥ 0`) for the symmetric neighbouring relation `Adj`, then `S ↦ ∏ i, M i S` is +`(√(2 k ln (1 / δ')) ε + k ε (e^ε - 1), δ')`-DP for every `δ' > 0`. It follows from the sharper +`isEpsDeltaDP_pi_of_isEpsDeltaDP` of `Statlib.Privacy.ZCDP.Basic`. -/ +theorem isEpsDeltaDP_piKernel {ι : Type*} [Fintype ι] {R : ι → Type*} + [∀ i, MeasurableSpace (R i)] {M : ∀ i, Kernel 𝒟 (R i)} [∀ i, IsMarkovKernel (M i)] + [Std.Symm Adj] {ε : ℝ} (hε : 0 ≤ ε) (h : ∀ i, IsEpsDP (M i) Adj ε) {δ' : ℝ} (hδ' : 0 < δ') : + IsEpsDeltaDP (ZCDP.piKernel M) Adj + (√(2 * Fintype.card ι * log (1 / δ')) * ε + Fintype.card ι * ε * (exp ε - 1)) δ' := + (isEpsDeltaDP_pi_of_isEpsDeltaDP hε h hδ').mono (by + nlinarith [mul_le_mul_of_nonneg_left (sq_div_two_le_mul_exp_sub_one hε) + (Nat.cast_nonneg (α := ℝ) (Fintype.card ι))]) le_rfl + +/-! ### Corollary 3.21: choosing the per-mechanism `ε` -/ + +/-- **Corollary 3.21 of Dwork–Roth with the additional hypothesis `δ' ≤ e^{-ε'/8}`, for +`(ε, 0)`-DP mechanisms**: let `ε' ≥ 0` and `δ' > 0` with `ε' ≤ 8 ln (1 / δ')`, i.e. +`δ' ≤ e^{-ε'/8}`. If, for every `n < k` and every history `x`, the mechanism `S ↦ M n (S, x)` is +`ε`-DP for the symmetric neighbouring relation `Adj`, where `ε = ε' / (2 √(2 k ln (1 / δ')))`, +then the `k`-fold adaptive composition is `(ε', δ')`-DP. + +The book assumes `0 < ε' < 1` and `δ' > 0`, with `δ'` negligible by its global convention +(p. 25); a sufficient additional hypothesis is `δ' ≤ e^{-ε'/8}`. Some restriction is needed: the +printed statement fails for some `δ'` close to `1` (`exists_not_isEpsDeltaDP_adaptiveCompose`), +and every counterexample with `δ = 0` has `δ' > e^{-ε'/8}` (`> 0.88` for `ε' < 1`). The +hypothesis holds for instance when `ε' ≤ 1` and `δ' ≤ e^{-1/8}` +(`isEpsDeltaDP_adaptiveCompose_of_le_one`). `Y` is countably generated. + +This is a totalized extension of the corollary: it also allows `k = 0`, and `ε' = 0` with +`δ' = 1`, where the denominator `2 √(2 k ln (1 / δ'))` is `0` and Lean's `ε` is `0`; these cases +are harmless (the `0`-fold composition is a constant kernel, and every Markov kernel is +`(ε', 1)`-DP). -/ +theorem isEpsDeltaDP_adaptiveCompose_of_le_eight_mul_log [MeasurableSpace.CountablyGenerated Y] + [Std.Symm Adj] [∀ n, IsMarkovKernel (M n)] {k : ℕ} {ε' δ' : ℝ} (hε' : 0 ≤ ε') + (hδ' : 0 < δ') (hε'δ' : ε' ≤ 8 * log (1 / δ')) + (h : ∀ n < k, ∀ x, IsEpsDP ((M n).sectL x) Adj (ε' / (2 * √(2 * k * log (1 / δ'))))) : + IsEpsDeltaDP (adaptiveCompose M k) Adj ε' δ' := by + set L := log (1 / δ') + set s := √(2 * k * L) with hs + refine (isEpsDeltaDP_adaptiveCompose_sq (div_nonneg hε' (by positivity)) h hδ').mono ?_ le_rfl + rcases (sqrt_nonneg (2 * k * L)).eq_or_lt with hs0 | hspos + · rw [hs, ← hs0] + simpa using hε' + have hL : 0 < 2 * k * L := sqrt_pos.1 hspos + have hs2 : s ^ 2 = 2 * k * L := sq_sqrt hL.le + have hk : (k : ℝ) ≠ 0 := by + rintro hk + simp [hk] at hL + have hLpos : 0 < L := by + by_contra hcon + nlinarith [Nat.cast_nonneg (α := ℝ) k] + have h1 : k * (ε' / (2 * s)) ^ 2 / 2 = ε' ^ 2 / (16 * L) := by + rw [div_pow, mul_pow, hs2] + field_simp + ring + have hsne : s ≠ 0 := hspos.ne' + have h2 : ε' / (2 * s) * s = ε' / 2 := by + field_simp + rw [h1, h2] + have : ε' ^ 2 / (16 * L) ≤ ε' / 2 := by + rw [div_le_iff₀ (by positivity)] + nlinarith + linarith + +/-- **Corollary 3.21 of Dwork–Roth with the book's hypotheses plus `δ' ≤ e^{-1/8}`**, for +`(ε, 0)`-DP mechanisms: for `0 ≤ ε' ≤ 1` and `0 < δ' ≤ e^{-1/8}`, if every round is `ε`-DP with +`ε = ε' / (2 √(2 k ln (1 / δ')))`, then the `k`-fold adaptive composition is `(ε', δ')`-DP. For a +given `ε' < 1`, the weaker `δ' ≤ e^{-ε'/8}` is already sufficient +(`isEpsDeltaDP_adaptiveCompose_of_le_eight_mul_log`); `δ' ≤ e^{-1/8}` is the version uniform in +`ε' ≤ 1`. Some restriction on `δ'` is needed (`exists_not_isEpsDeltaDP_adaptiveCompose`). +The statement also allows `k = 0` (a totalized extension: the prescribed `ε` is then `ε' / 0 = 0` +in Lean, the hypothesis on the rounds is vacuous, and the `0`-fold composition is a constant +kernel); here `δ' < 1` holds automatically. -/ +theorem isEpsDeltaDP_adaptiveCompose_of_le_one [MeasurableSpace.CountablyGenerated Y] + [Std.Symm Adj] [∀ n, IsMarkovKernel (M n)] {k : ℕ} {ε' δ' : ℝ} (hε' : 0 ≤ ε') (hε'1 : ε' ≤ 1) + (hδ' : 0 < δ') (hδ'e : δ' ≤ exp (-(1 / 8))) + (h : ∀ n < k, ∀ x, IsEpsDP ((M n).sectL x) Adj (ε' / (2 * √(2 * k * log (1 / δ'))))) : + IsEpsDeltaDP (adaptiveCompose M k) Adj ε' δ' := by + refine isEpsDeltaDP_adaptiveCompose_of_le_eight_mul_log hε' hδ' ?_ h + have := log_le_log hδ' hδ'e + rw [log_exp] at this + rw [one_div, log_inv] + linarith + +/-- **The printed Corollary 3.21 of Dwork–Roth fails without a restriction on `δ'`**, for `δ'` +close to `1`, outside the book's intended regime: the book requires `δ` to be negligible +"throughout this monograph" (Section 2.3, p. 25), while the corollary's hypotheses (p. 52) are only +`0 < ε' < 1` and `δ' > 0`. With `k = 1`, +`ε' = 9/10 ∈ (0, 1)`, `δ' = e^{-1/800} > 0` and `δ = 0`, the corollary asserts that the `1`-fold +composition of an `ε`-DP mechanism, `ε = ε' / (2 √(2 k ln (1 / δ'))) = 9`, is `(ε', δ')`-DP. +Binary randomized response with parameter `9` (on `Bool`, where any two inputs are neighbours) is +`9`-DP, but its `1`-fold composition is not `(9/10, e^{-1/800})`-DP: it reports the true bit with +probability `e⁹ / (1 + e⁹) > e^{9/10} / (1 + e⁹) + e^{-1/800}` (it is `(9/10, δ')`-DP only for +`δ' ≥ (e⁹ - e^{9/10}) / (1 + e⁹) ≈ 0.99957`, while `e^{-1/800} ≈ 0.99875`). Every counterexample +with `δ = 0` has `δ' > e^{-ε'/8}`, which exceeds `0.88` for `ε' < 1` +(`isEpsDeltaDP_adaptiveCompose_of_le_eight_mul_log`, a sufficient additional hypothesis); only +`δ = 0` is examined. -/ +theorem exists_not_isEpsDeltaDP_adaptiveCompose : + ∃ M : (n : ℕ) → Kernel (Bool × (Fin n → Bool)) Bool, (∀ n, IsMarkovKernel (M n)) ∧ + (∀ n x, IsEpsDP ((M n).sectL x) (fun _ _ ↦ True) + (9 / 10 / (2 * √(2 * (1 : ℕ) * log (1 / exp (-(1 / 800))))))) ∧ + ¬ IsEpsDeltaDP (adaptiveCompose M 1) (fun _ _ ↦ True) (9 / 10) (exp (-(1 / 800))) := by + have hε : 9 / 10 / (2 * √(2 * (1 : ℕ) * log (1 / exp (-(1 / 800))))) = (9 : ℝ) := by + rw [one_div (exp _), ← exp_neg, neg_neg, log_exp, Nat.cast_one, + show (2 : ℝ) * 1 * (1 / 800) = (1 / 20) ^ 2 by norm_num, sqrt_sq (by norm_num)] + norm_num + have hRR : IsEpsDP (binaryRandomizedResponse 9) (fun _ _ ↦ True) 9 := + (isEpsDeltaDP_binaryRandomizedResponse_iff 9).2 (by norm_num) + refine ⟨fun n ↦ (binaryRandomizedResponse 9).comap Prod.fst measurable_fst, + fun n ↦ inferInstance, fun n x ↦ by rw [hε]; exact fun S S' hS E ↦ hRR hS E, fun hDP ↦ ?_⟩ + have h := (hDP.map (measurable_pi_apply 0)) (S := true) (S' := false) trivial {true} + rw [map_adaptiveCompose_one] at h + change (binaryRandomizedResponse 9 true).real {true} ≤ + exp (9 / 10) * (binaryRandomizedResponse 9 false).real {!false} + exp (-(1 / 800)) at h + rw [binaryRandomizedResponse_real_self, binaryRandomizedResponse_real_not] at h + have hA : 7000 < exp 9 := by + have h1 : (2.7 : ℝ) ^ 9 < exp 1 ^ 9 := + pow_lt_pow_left₀ (by linarith [exp_one_gt_d9]) (by norm_num) (by norm_num) + rw [← exp_nat_mul] at h1 + norm_num at h1 + linarith + have hB : exp (9 / 10) < 3 := by + linarith [exp_le_exp.2 (show (9 / 10 : ℝ) ≤ 1 by norm_num), exp_one_lt_d9] + have hC : exp (-(1 / 800)) ≤ 800 / 801 := by + rw [exp_neg, show (800 / 801 : ℝ) = (1 / 800 + 1)⁻¹ by norm_num] + exact inv_anti₀ (by norm_num) (add_one_le_exp _) + have hpos : 0 < 1 + exp 9 := by positivity + rw [div_le_iff₀ hpos, add_mul, mul_one_div, div_mul_cancel₀ _ hpos.ne'] at h + nlinarith [mul_le_mul_of_nonneg_right hC (exp_pos 9).le] + +/-- **The setting of Example 3.7 of Dwork–Roth**: `k = 10 000` adaptively composed `ε₀`-DP +mechanisms with `ε₀ = 1/807` are `(1, e^{-32})`-DP (for the symmetric neighbouring relation `Adj`). +The book attributes `ε₀ ≤ 1/801` to Theorem 3.20; only that attribution is wrong: at +`ε₀ = 1/801` the bound of Theorem 3.20 is `ε' ≈ 1.014 > 1`, and the zCDP bound +`isEpsDeltaDP_adaptiveCompose_sq` is `ε' ≈ 1.007`, which is at most `1` at `ε₀ = 1/807`. The +book's numerical conclusion itself appears to be true: the exact optimal composition +(Kairouz–Oh–Viswanath, ICML 2015) gives `δ(1) ≈ e^{-38.8}` at `ε₀ = 1/801` (checked numerically +only, not formalized). The book states a tail bound on the privacy loss, which is stronger than +the `(1, e^{-32})`-DP proved here. `Y` is countably generated. -/ +theorem isEpsDeltaDP_adaptiveCompose_ten_thousand [MeasurableSpace.CountablyGenerated Y] + [Std.Symm Adj] [∀ n, IsMarkovKernel (M n)] + (h : ∀ n < 10000, ∀ x, IsEpsDP ((M n).sectL x) Adj (1 / 807)) : + IsEpsDeltaDP (adaptiveCompose M 10000) Adj 1 (exp (-32)) := by + refine (isEpsDeltaDP_adaptiveCompose_sq (by norm_num) h (exp_pos _)).mono ?_ le_rfl + rw [one_div (exp _), ← exp_neg, neg_neg, log_exp, Nat.cast_ofNat, + show (2 : ℝ) * 10000 * 32 = 800 ^ 2 by norm_num, sqrt_sq (by norm_num)] + norm_num + +end AdvancedComposition + +end InformationTheory