Adaptive Data Analysis in a Balanced Adversarial Model

In adaptive data analysis, a mechanism gets $n$ i.i.d. samples from an unknown distribution $D$, and is required to provide accurate estimations to a sequence of adaptively chosen statistical queries with respect to $D$. Hardt and Ullman (FOCS 2014) and Steinke and Ullman (COLT 2015) showed that in general, it is computationally hard to answer more than $Θ(n^2)$ adaptive queries, assuming the existence of one-way functions. However, these negative results strongly rely on an adversarial model that significantly advantages the adversarial analyst over the mechanism, as the analyst, who chooses the adaptive queries, also chooses the underlying distribution $D$. This imbalance raises questions with respect to the applicability of the obtained hardness results -- an analyst who has complete knowledge of the underlying distribution $D$ would have little need, if at all, to issue statistical queries to a mechanism which only holds a finite number of samples from $D$. We consider more restricted adversaries, called \emph{balanced}, where each such adversary consists of two separated algorithms: The \emph{sampler} who is the entity that chooses the distribution and provides the samples to the mechanism, and the \emph{analyst} who chooses the adaptive queries, but has no prior knowledge of the underlying distribution (and hence has no a priori advantage with respect to the mechanism). We improve the quality of previous lower bounds by revisiting them using an efficient \emph{balanced} adversary, under standard public-key cryptography assumptions. We show that these stronger hardness assumptions are unavoidable in the sense that any computationally bounded \emph{balanced} adversary that has the structure of all known attacks, implies the existence of public-key cryptography.

Paper

References (47)

Scroll for more · 35 remaining

Similar papers

Peer review

Reviewer Ejfp6/10 · confidence 2/52023-07-04

Summary

The paper considers balanced adversaries that do not have access to the underlying distribution and instead interact with an oracle adversary that chooses the distribution and samples points from it. The authors show that with this more realistic adversarial setting, they can improve the previous known lower bounds for adaptive data analysis.

Strengths

The authors consider a more realistic balanced adversary that does not have access to the underlying distribution. This adversarial setting is novel and allows for better bounds on the adaptive data analysis.

Weaknesses

While considering this balanced adversary is more practical, I would suggest the authors to include a summary table comparing their setting, assumptions and their bounds with respect to the prior works. This would better highlight the differences and focus more on the key contribution of the paper.

Questions

I am confused why the authors have the value 2000n in Theorem 2.4. Is there something particular about this value 2000, and if so how do they arrive at this? Can this be a parameter k where k does not take large values?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

I don't feel there is any direct or evident negative societal impact of this work.

Reviewer CEqx9/10 · confidence 5/52023-07-04

Summary

Adaptive data analysis considers the setting where a mechanism and an analyst play a game against each other, where for $k$ sequential rounds, the analyst issues queries to a mechanism which holds a dataset of size $n$ sampled from a distribution, the mechanism returns a response, and the analyst chooses the next query accordingly. The mechanism wins the game if all responses are close to the expected value of the queries on the dataset. Using tools developed in the context of differential privacy, it was proven that a mechanism can respond to up to $k = O(n^{2})$ queries, while winning the game with high probability. At the same time, it was proven that if the number of queries exceeds $O(n^{2})$, there exists an analyst that can win the game. This lower bound assumes the analyst can choose the distribution, and is based on the cryptographic assumption of the existence of fingerprinting codes. This imposes an asymmetry between the power and knowledge of the analyst and mechanism which is unnatural, since an analyst that knows the distribution has no reason to query the mechanism. The current work solves this problem, by proving a similar lower bound for a balanced setting, where the distribution is chosen by a third player. This proof is based on the stronger cryptographic assumption of the existence of public-key cryptography. The authors also prove that the lower bound does not hold, without this assumption, proving a separation between the balanced and unbalanced settings.

Strengths

The deep asymmetry that was used in the proof the previous lower bounds was undesired, and left an open question whether it is essential, since it is not justified in most natural settings. Solving this open question is an important step in understanding the dynamics of adaptive data analysis. They are based on a novel addition to the existing lower bound framework, and are presented in a clear and formal way. The distinction between the two cryptographic assumptions required in both cases, might hold implications for cryptography as well.

Weaknesses

I have only one minor comment to add. It seems like the authors use another implicit assumption throughout the paper, that the mechanism knows the analyst it is playing against. If this was not the case, the separation of the analyst into two analysts (one choosing the distribution and the other issuing the queries) will be meaningless, as we could always consider the case where the first analyst always chooses the same distribution, and so the second one can use that information while the mechanism cannot. I do not bring this up as a limitation of the work, but as a recommendation to clarify the setting, so to avoid confusion by the reader. ======= **Edit after rebuttal discussion:** I have no further concerns.

Questions

In section 2.4 the authors mention the sample size, number of queries and dimensions are provided in unary representation. I would appreciate if the authors can clarify how this affects the computational complexity, and how they justify this decision.

Rating

9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

N/A

Reviewer Mqg25/10 · confidence 2/52023-07-05

Summary

The paper proposes to consider balanced adversaries for ADA. Under the public-key assumptions, the paper proves that

Strengths

1. The paper assumes a more practical threat model for ADA. 2. The theoretical results in this paper seem to be well-supported. I am not an expert in this research direction, so I am not sure about my judgement.

Weaknesses

1. There is no discussion about the real-world machine learning applications of the results. 2. There is no conclusion section to summarize the paper. The paper ends after Section 4 and makes me feel that the paper is not complete. 3. The theoretical results are based on the assumption that public-key cryptography exists. 4. I am not an expert in this research direction, and I feel that this paper is not easy to follow because there are many terms. I suggest that the paper could add a table to explain the terms used in the paper to improve the readability.

Questions

1. Is there any real-world machine learning application of the results?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Reviewer vTGp6/10 · confidence 4/52023-07-06

Summary

The paper studies limitations of "adaptive data analysis" with a focus on "computational assumptions" that are used for proving such limitations (i.e., lower bounds). A mechanism $M$ answers $m=m(n)$ queries (almost) correctly, if it succeeds in the following game for every distribution $D$ and PPT adversary $A$. The mechanism is given n iid samples from D. Then the adversary $A$ asks $m$ adaptive statistical queries $q_1,\dots,q_m$ from the mechanism $M$. The mechanism succeeds if all of its answers are within $\pm 1/10$ of the true answers with respect to distribution $D$. Previous work has shown that at most $m=\Theta(n^2)$ queries can be answered adaptively by computationally bounded mechanisms, assuming one-way functions exist. Their proof uses private-key encryption schemes (whose existence is equivalent to one-way functions) and a reduction from a class of mechanisms, called "natural". The previous lower bound also implicitly assumes a rather powerful adversary who both chooses the distribution D as well as the queries $q_1,...q_m$. This paper starts by making the (interesting) observation that if an adversary already knows the distribution $D$, then it won't have a meaningful interest in making the queries. So, in some sense, the previous lower bounds are not as meaningful as one wishes in real world scenarios; this raises the hope for getting positive results that go around the lower bounds by leveraging this observation. In summary, the main result of the paper is bootstrap the previous lower bound to a model to a weaker adversarial model. In more detail, the contributions of this paper are as follow: 1. A new formal model of security called the "balanced model" is defined, in which the adversary is split into two entities that do not communicate. The first adversary $A_1$ picks the distribution $D$ and the iid samples. The second adversary $A_2$ picks the adaptive queries. 2. It is shows that assuming the stronger primitive of secure public-key encryption (as opposed to just OWFs) one can still extend the previous lower to the balanced model. 3. The assumption of public-key encryption is somehow justified, by showing any lower bound with minimal properties (that hold for current lower bound proofs) would lead to key agreement protocols (which is a close primitive/assumption to public key encryption).

Strengths

The main strengths of the paper: * excellent writing quality. my only criticism is that the definition used in *previous* work is not presented formally, and one cannot determine the exact details from the informal text. this is needed for making a fully detailed comparison between the new definition and the old one, because this is a major part of this paper. * revisiting a previous lower bound for an important problem from a natural practical perspective.

Weaknesses

My criticisms of the paper are the following: 1. About the model (Item 1 above): I am not sure if the new split (balanced) model is adding much value actually. suppose we fix a mechanism $M$ and look for a balanced 2-part polynomial-size adversary $(A_1,A_2)$ against $M$. In this setting, it seems meaningless to say that only $A_1$ knows the distribution D, because one can fix the distribution $D$ that $A_1$ picks to its "best possible" distribution (against $M$). This way, $A_2$ also would be aware of this distribution implicitly, as it can depend on it. In other words, one can see this argument as giving shared coin to $A_1,A_2$, and then fixing it to its best value (which leads to no communication between $A_1$ and $A_2$). I shall add that, from a game theoretic perspective, it would seems a preferred and stronger lower bound if we could use *one* fixed adversary strategy that bounds the power of every computationally limited mechanism. However, for the sake of proving lower bounds, it is fine if we'd pick the adversary $A_M$ based on the mechanism $M$. 2. About the technical depth of the lower bound (Item 2 above). This proof seems like a rather straightforward adaptation of the previous proof. but using IBE (or a bunch of public key encryption schemes) instead of Secret Key Encryption schemes. This is a mild criticism, as I am in general in favor of simpler proofs. But this proof seems like a simple proof for a new question rather than a simple proof for a question that existed before, so I am a bit torn here. 3. About necessity of Key Agreement (Item 3 above): I am not a fan of investing in weakening assumptions that are used for *lower bounds*. as I believe there is a fundamental difference between weakening assumptions behind positive results vs those of negative results. When we weaken the assumption behind, say, public key encryption from an assumption $P_1$ to a weaker assumption $P_2$, it means that the schemes that would be deployed *right now* would not break at an unknown time (with potential catastrophic consequences) even if $P_1$ turns to be false. However, when we weaken the (still widely believed) computational assumption behind a a *lower bound*, is is a completely different story, because even using an assumption as strong as IO to prove an impossibility has the same practical consequence as using OWFs. The two scenarios would diverge (and a weaker assumption would start to look interesting) after breaking every IO scheme, which is most likely never gonna happen, and the weakening assumption changes nothing right now. So, in summary, when it comes to lower bounds I am totally fine with using something as strong as any assumption that seems strong enough for the foreseeable future. update: the issues above were discussed and I am happy with author(s)' response.

Questions

My main question is about the meaningfulness of the balanced attack model which is described above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The limitations are also elaborated in the "weakness" section of the review above.

Reviewer CEqx2023-08-15

If I may add to the authors response. The issue raised by reviewer vTGp resembles the one I raised in my review, and I had to spend some time convincing myself what is the right way to think of it, so it might be the case that a fellow reviewer might find my perspective helpful. To the best of my understanding does not point to a limitation in the current work, but to a subtle lack of formalism in previous works on this subject. Here is how I have been thinking of it. Even in the setting where the analyst is all powerful, and has access to and control over everything but the sampled dataset, if the analyst always chooses the same distribution and the mechanism knows which analyst does it compete against, it can provide perfectly accurate private responses, since it effectively has access to the underlying distribution. This limitation can be avoided by adding an assumption about the mechanism's knowledge about the attacker, but this is a dangerous assumption which results in brittle guarantees. Instead, the implicit chosen assumption is that the mechanism knows the analyst, but the analyst uses randomness which prevents the mechanism from leveraging this knowledge.

Reviewer vTGp2023-08-18

thanks

yes, we raised the same point, and thanks for the clarification.

Reviewer CEqx2023-08-15

I thank the reviewers for the clarification, and for the opportunity to learn about this exciting result.

Reviewer Ejfp2023-08-16

Thank you for the clarifications.

Reviewer vTGp2023-08-18

Ack

Thanks for the responses. I found them very useful, particularly the one about the use of improving assumptions for lower bounds. It would be very helpful if the authors please do include these discussions in the paper (basically about all the raised points). I'd increase my score.

Reviewer Mqg22023-08-19

Thank you for the rebuttal

Thank you for the clarifications.

Program Chairsdecision2023-09-21

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC