The Limits of Differential Privacy in Online Learning

Differential privacy (DP) is a formal notion that restricts the privacy leakage of an algorithm when running on sensitive data, in which privacy-utility trade-off is one of the central problems in private data analysis. In this work, we investigate the fundamental limits of differential privacy in online learning algorithms and present evidence that separates three types of constraints: no DP, pure DP, and approximate DP. We first describe a hypothesis class that is online learnable under approximate DP but not online learnable under pure DP under the adaptive adversarial setting. This indicates that approximate DP must be adopted when dealing with adaptive adversaries. We then prove that any private online learner must make an infinite number of mistakes for almost all hypothesis classes. This essentially generalizes previous results and shows a strong separation between private and non-private settings since a finite mistake bound is always attainable (as long as the class is online learnable) when there is no privacy requirement.

Paper

Similar papers

Peer review

Reviewer kW4d6/10 · confidence 3/52024-06-23

Summary

The paper studies online learning of concept classes under DP (differential privacy) constraint. The paper makes progress towards understanding mistake bounds (mostly) in the realizable case in a few settings. Concretely, The paper shows that: 1. If the adversary is oblivious, then PAC pure DP learnability implies online pure DP learnability. 2. On the other hand, if the adversary is adaptive, then PAC pure DP learnability does not imply online pure DP learnability. 3. In contrast with the standard online learning model, for every non-trivial hypothesis class, the mistake bound depends on $T$.

Strengths

1. Assuming that the results are correct, the paper contains a significant improvement towards understanding mistake bounds in online learning under DP constraint. 2. The questions and results are interesting and in the scope of NeurIPS. 3. The proof techniques are explained in detail.

Weaknesses

1.It is a bit hard to digest the results and understand the remaining gaps, because there are many settings considered in the paper (DP/non-DP, oblivious/adaptive, approximate/pure...), and no figure/meta-theorem that neatly explains all the relationships. Such a figure/meta-theorem would significantly improve the presentation of the paper. 2. As a consequence of the above, it is not exactly clear how tight the results are. If you prove a lower bound (as in Section 4), I think it is better to formally state, right after it, the best known lower bound (and the dual statement for proving a lower bound).

Questions

Questions: 1. I think that the type of classes you require in Section 4 is related (or equivalent?) to the notion of "non-trivial classes" which is used in lower bounds for learning with data poisoning. See, e.g., [1,2]. This setting is also related to some notions of "stability", like DP (see [2]). 2. What is the best known upper bound for the result of Section 4? Suggestions: 1. Line 84: you wrote $\epsilon$ before defining DP. 2. If I understand correctly, the sentence in line 190 is supposed to say: "Our result reveals that, against oblivious adversaries, pure private online learnability is equivalent to pure private offline learnability in both realizable and agnostic settings." If so, I would recommend clarifying it. 3. The whole paragraph that begins at line 192 is unclear.  It involves not yet stated results and definitions. Either remove it or move it to a suitable place. 4. It could be helpful to add a figure of the landscape of separations between the many settings discussed in the paper (DP/non-DP, oblivious/adaptive, approximate/pure...). Typos/english: 1. line 146: "not known by" --> "unknown to" 2. line 264: "an lower" --> "a lower" 3. line 272: "inpupt" [1] Nader H Bshouty, Nadav Eiron, and Eyal Kushilevitz. Pac learning with nasty noise. TheoreticalComputer Science, 288(2):255–275, 2002. [2] S. Hanneke, A. Karbasi, M. Mahmoody, I. Mehalel, and S. Moran. On optimal learningunder targeted data poisoning. In Advances in Neural Information Processing Systems36, 2022.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Reviewer sPMY7/10 · confidence 3/52024-07-12

Summary

The paper demonstrates that any function class that is offline PAC learnable with pure DP is also online learnable with pure DP against an oblivious adversary. In this context, a hypothesis class is considered online learnable in the realizable setting if there exists an algorithm with a sublinear mistake bound. The paper also establishes a distinction between online learning with pure privacy for oblivious and adaptive adversaries. Specifically, it shows that the hypothesis class $point_N$ is privately online learnable against an oblivious adversary but not against adaptive adversaries. This finding also indicates a separation between pure and approximate private online learnability, as $point_N$ is online learnable with approximate DP against an adaptive adversary. Additionally, the paper presents a general lower bound on DP online learning against an oblivious adversary for non-complementary function classes.

Strengths

- The paper offers general results on private online learnability, identifying conditions under which a hypothesis class is online learnable with pure DP against an oblivious adversary. This connection to Representation Dimension links to existing results on DP PAC learnability. - It also explores different layers of separation using the function class $point_N$, contributing to a deeper understanding of the cost of privacy in online learning.

Weaknesses

The proof of Theorem 4.3 is not clear

Questions

It looks like Lemma E.2 shows the concentration assumption in in Dmitriev et al. Can you give some intuition on how the proof of Theorem 4.3 get rid of this assumption? Also, are there examples of DP online learning algorithms that do not satisfy their concentration assumption?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

No significant limitations.

Reviewer YzYS6/10 · confidence 3/52024-07-12

Summary

This paper studies limits of pure DP and approximate DP in the context of online learning (with oblivious and adaptive adversaries).

Strengths

The research questions are interesting and the results may have fundamental value. I'm not an expert in all topics covered by the paper, but the contributions seem novel to me. The text is in general well-written and understandable.

Weaknesses

The paper has a lot of theorems and lemmas, which is interesting. Still, the paper has no conclusions, discussion, further work, description of limitations or illustrative experiments or extensive examples. This puts the task of understanding the value and applicability of the work to a large extent to the reader.

Questions

.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

4

Limitations

The authors don't discuss limitations or societal impact. I believe there are no ethical concerns with this theoretical work.

Reviewer ud1R8/10 · confidence 1/52024-07-14

Summary

Various results are proved about online learning of private learning algorithms, contrasting no DP, pure DP and approximate DP.

Strengths

The paper present some interesting new results on online learning with DP. I read up to section 3 and the writing is very clear and the results are important and purportedly novel. (I don't have background or recent experience in the area of online learning so I cannot independently confirm their novelty.)

Weaknesses

I can't point to any weaknesses, but this paper is outside of my area, and I was only able to follow up to Section 3, so it is certainly possible there is something I missed. I am basically taking the paper at its word on the claims made in Sections 1 and 2.

Questions

No questions.

Rating

8

Confidence

1

Soundness

4

Presentation

4

Contribution

3

Limitations

No potential negative societal impact.

Reviewer sPMY2024-08-12

Thanks for your answers. Though the algorithm that predicts randomly at each time step does not satisfy the $\beta$-concentration assumption, it has poor utility. However, I appreciate the explanation that clarifies the proof strategy, and I think the paper has great contribution in providing an algorithm-independent lower bound for DP online learning. I raised my score accordingly.

© 2026 NYSGPT2525 LLC