Bridging Multicalibration and Out-of-distribution Generalization Beyond Covariate Shift

We establish a new model-agnostic optimization framework for out-of-distribution generalization via multicalibration, a criterion that ensures a predictor is calibrated across a family of overlapping groups. Multicalibration is shown to be associated with robustness of statistical inference under covariate shift. We further establish a link between multicalibration and robustness for prediction tasks both under and beyond covariate shift. We accomplish this by extending multicalibration to incorporate grouping functions that consider covariates and labels jointly. This leads to an equivalence of the extended multicalibration and invariance, an objective for robust learning in existence of concept shift. We show a linear structure of the grouping function class spanned by density ratios, resulting in a unifying framework for robust learning by designing specific grouping functions. We propose MC-Pseudolabel, a post-processing algorithm to achieve both extended multicalibration and out-of-distribution generalization. The algorithm, with lightweight hyperparameters and optimization through a series of supervised regression steps, achieves superior performance on real-world datasets with distribution shift.

Paper

Similar papers

Peer review

Reviewer ezjY5/10 · confidence 3/52024-06-19

Summary

The authors propose utilizing multicalibration algorithms to achieve distributional robustness w.r.t. both concept and covariate shift. They do this by allowing subgroups to not only be a function of the features X (as is standard in multigroup fairness), but also a function of the label Y. There are numerous theoretical and experimental results. The first result (thm 3.2) states that under closure properties of the hypothesis class $\mathcal{H}$ w.r.t. possible distribution shifts, and a weak-learnability assumption on $\mathcal{H}$, multicalibration w.r.t. a particular set of subgroups $\mathcal{H}_1$ guarantees that the resulting predictor will achieve good error (relative to the bayes optimal predictor) under any possible covariate shifted distribution $P_T$. In theorem 3.1., the authors show that an L2-multicalibrated predictor f cannot be significantly improved by any post-processing function in order to achieve robustness to concept shift. The authors then show an equivalence between approximate multicalibration and a notion called approximate invariance from prior work (theorem 3.4). Invariance is a property of a representation $\Phi$ which says that there exists a predictor $g^*$ based only on $\Phi$ which achieves good error across all environments. The equivalence between the two notions is up to considering subgroup functions defined by the density ratios between the target environments and the source distribution, similar to the result in theorem 3.1. Next, the authors demonstrate that (joint) multicalibration cannot always be achieved for all jointly defined subgroups. Thus, they are interested in the maximal feasible set of jointly defined groups for which we can achieve multicalibration towards. These groups turn out to lie in a linear space (Prop. 4.1.) and have a well-defined spanning set which looks similar to an invariant predictor (Thm. 4.2). Finally, they show that the subgroup function space can be decomposed into an invariant and multicalibrated part (Thm. 4.3). Lastly, experiments are conducted which demonstrate the usefulness of multicalibration in achieving distributional robustness for simple neural networks on the PovertyMap and ACSIncome datasets.

Strengths

Originality: To my knowledge, the work is very original, especially the consideration of the maximal grouping function space. While extending subgroups to be defined on the label as well as feature is a natural generalization, the authors have unique function space decomposition results (section 4), which are interesting in their own right and whose proofs utilize classical results from linear algebra. Substantial theory is developed through most of the paper, mostly surrounding the inclusion of particular density ratios in the subgroup function classes. I think the results taken together are very original and interesting! Significance: understanding what is possible with families of multicalibration post-processing algorithms is a very important and interesting direction, and combining this with out of distribution generalization is an area which is quite interesting.

Weaknesses

I have not checked the proofs thoroughly. I believe the paper has much room for improvement in terms of clarity, organization, and discussion. In isolation, these may be minor, but taken together, the writing poses a major limitation to the greater community understanding the work (as well as its position in the literature). Throughout, I have asked explicit questions which I would like the authors to address, and labeled them as Q1, Q2, etc. W1: Firstly, I believe the introduction requires substantial re-writing. Currently, it makes sense to someone who is familiar with both the multicalibration and distributional robustness literature (most people are not). For example, the first sentence of the paper mentions both out-of-distribution generalization and multicalibration, without motivating either of these communities. (Q1) Why might we want to connect these notions? What do we stand to gain from applying multicalibration, and what does multicalibration address that the (substantial) work on OOD generalization has not considered? Perhaps these questions are answered implicitly elsewhere in the work (and I missed them), but I believe that the authors should do a better job of framing the problem they are trying to solve in the context of the literature. W2: I believe that an explicit discussion of related work in the introduction can be very helpful. I understand that the authors defer all related work discussion to the appendix, but as someone familiar with the multicalibration literature but not distributional robustness, the main paper was hard to follow. W3: Little interpretation of results. As this is predominantly a theoretical paper, I understand that it can be challenging to concisely state and provide intuitive explanations of each result. However, I believe that the paper can be substantially improved with some additional intuition. Even understanding the theorem statements in section 4 took substantial effort from my part. As another example, in Section 2.2: line 101-106: it seems implied that these assumptions are from previous work, but it would be good to state explicitly. (Assuming that is the case). Further, any further discussion on the assumptions would be appreciated. I believe it could be useful to state the assumptions in 2.2 as two separate assumptions, in order to make references to them in the following theorem statement more clear. There is no discussion after the statement of the theorem 2.3. What is the class H_1, and why does it help us achieve robustness to distribution shift? How does it relate to the two assumptions stated previously? Also, how does Kim et al. [22] relate to theorem 2.3? These should all be explicitly discussed. As a positive example, the authors do provide some discussion in lines 175-178. Expanding and including similar discussion throughout the paper can help make the ideas more parseable to the unfamiliar reader. W4: Section 4 is extremely difficult to parse. This also connects to W3, as there is little interpretation of the results. What are the implications of the grouping functions being a linear space? Why is the decomposition in theorem 4.3 a Minkowski sum? Also, how do these results connect to the algorithm developed in section 5? Minor comments 35: The authors use the term “joint” grouping functions, but have only implicitly described them in the previous paragraph. It would be clearer to concretely mention that you call these subgroups which depend on both X and Y joint grouping functions. 41-43: The definition of invariance / IRM is not very clear here. I think it is ok to defer a formal description to later, and just mention that you provide an equivalence between the invariant learning framework and multicalibration. In line 120, there is some motivation for covariate shift and concept shift. I believe this discussion should go far earlier, perhaps in the introduction.

Questions

Q1. How does section 5 relate to the rest of the work? Why do we need another multicalibration algorithm? What problem is MC-PseudoLabel solving which existing OOD / domain adaptation algorithms do not already solve? I believe that these are critical and should be stated explicitly. Q2. Line 83: “We say that f is $\alpha$-approximately calibrated if $h \equiv 1$.” What does this mean? I think you mean if $\mathcal{H}$ includes the constant function $h\equiv 1$, then we can say that f is $\alpha$ approximately calibrated. Q3. Line 18: “Multicalibration is a strengthening of calibration, which requires a predictor f to be correct on average within each level set:” For clarity, perhaps you could introduce calibration first in an isolated way, without mentioning multicalibration? The discussion here could potentially be misconstrued by the reader. Q4. Line 80: Definition 2.1. Outside of extending to joint subgroups h(X) -> h(X, Y), how does this definition relate to existing multicalibration definitions? Q5. Theorem 3.1: In the appendix proof of this theorem, the authors state a similarity to Blasiok et al. [5] w.r.t. the theorem statement. How similar is the proof, and what complications arise from needing to consider joint grouping functions? This should be in the main paper, especially since others may notice a resemblance between the results as well. Q6. Line 180: “Section 3 inspires one to construct richer grouping function classes for stronger generalizability”. Why is this the case, or can you expand on what this means exactly? Is the idea that due to theorem 3.4, we understand that multicalibration w.r.t. Sufficiently rich joint grouping functions gives us a nice property, but we want to efficiently achieve this which may be difficult in practice?

Rating

5

Confidence

3

Soundness

3

Presentation

1

Contribution

3

Limitations

The authors have stated the main limitation of the work, which is that most OOD generalization papers deal with classification, not regression. However, I see this as a feature, since regression can also be a useful consideration in practice.

Reviewer HkL58/10 · confidence 4/52024-07-08

Summary

The authors explore an extension of multicalibration which includes joint grouping functions; groups that depend on both x and y. They show that multicalibration confers robustness to distribution shift problems. The authors then develop an optimization framework that post-processes a model to multicalibrate it. Finally, empirical demonstration of the robustness properties are given through experimentation.

Strengths

The authors provide compelling reasons why multicalibration should be considered in modeling and model evaluation frameworks. In doing so, they provide a calibration framework that confers robustness on post-processed models. The proposed method appears significantly easier to use than methods that leverage actor-critic, domain confusion, and other modeling tricks to achieve robustness. Section 4 shows that the condition "the grouping function class must include all density ratios between target and source measures" is achievable in practice. The paper is well written and easy to follow.

Weaknesses

A few questions are given below. I have nothing else to add.

Questions

Line 9 of Algorithm 1 is unclear to me. I had thought A finds low risk models f(x) on an empirical dataset {(x,y)}. What does it mean for a grouping function to have low risk on a dataset? What is risk here? With regard to Algorithm 1, the authors state "The prediction of grouping functions rectify the uncalibrated model and serves as pseudolabels for model updates." Can this statement be expanded, and details be provided?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have adequately addressed limitations.

Reviewer TrUb5/10 · confidence 2/52024-07-12

Summary

The paper explores multicalibration in the context of concept shifts, theoretically demonstrating the equivalence of multicalibration and invariance while providing a structural analysis of multicalibration. It introduces a novel algorithm that simplifies model selection and improves performance on real-world datasets.

Strengths

- The paper provides rigorous and solid theoretical work for its claims. - MC-Pseudolabel algorithm offers a practical tool that appears to integrate well with their frameworks.

Weaknesses

- The empirical work is limited and not that convincing. - Although technical solid, some assumptions might be too strong to achieve in reality. - The complexity and detailed mathematical theories/assumptions would make the paper challenging for a broader audience to easily digest.

Questions

- It seems that the setting you called "out-of-distribution generalization beyond covariate shift" is basically concept drift. I wonder why you choose to call that. - In theorem 4.2 and 4.3, you consider an "absolutely continuous probability measure" without stating what measure it is with respect to.

Rating

5

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

See weakness.

Reviewer 7ztk8/10 · confidence 4/52024-07-14

Summary

This work studies an extension of multicalibration to include grouping functions of both covariates and labels. They show that just as multicalibration with respect to covariate density functions guarantees robustness to covariate shift, that extended multicalibration can imply robustness to concept shift. They further show a connection between extended multicalibrated predictors and invariant predictors. They study necessary and sufficient conditions on (covariate-dependent) grouping functions to achieve approximately Bayes optimal predictions on all target distributions, and introduce a boosting style algorithm that converges to an extended multicalibrated predictor under certain distributional assumptions. The usefulness of this new algorithm is empirically validated on PovertyMap, ACSIncome, and VesselPower.

Strengths

This is a very interesting work that as far as I know if the first to considering multicalibration with respect to grouping functions that depend on labels. This gives a new approach to handling concept shift, and a new perspective on invariant predictors. I'm personally looking forward to follow-up work. I found the paper very well-structured and easy to follow.

Weaknesses

There are some questions regarding the feasibility of the MC-PseudoLabel that I did not see addressed in the work that would be helpful in understanding its usefulness. First, there's a known correspondence between multicalibration with respect to a hypothesis class H and weak agnostic learning for H, which unfortunately limits the grouping function classes for which we can efficiently obtain multicalibrated predictors. Does a similar correspondence also hold for extended multicalibration, or is a stronger weak learning condition required (alongside distributional assumptions)? I'm also curious about the necessary assumptions on the data for convergence of MC-PseudoLabel. On what kinds of distributions will the algorithm fail to convergence for H_{2, \Phi}? Typos/suggested edits: Abstract “in existence” -> “in the presence” Section 1 “flexibly designed to incorporates” -> “incorporate” “Simultaneoulsy producing invariant predictors” -> “produce” “porverty estimation” Section 2 “which is learned in the source distribution” -> “which are learned” 2.2 “simultaneous approaches” -> “simultaneously approach” “We show that multicalibration notion w.r.t.” -> “We show that our/this multicalibration notion” Section 5 M(Sigma) is defined in the Appendix, but not in Theorem 5.2 where is first appears

Questions

Repeated from weaknesses: 1. Is there a correspondence between weak agnostic learning and extended multicalibration? 2. On what kinds of distributions will the algorithm fail to convergence for H_{2, \Phi}?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors largely addressed the limitations (with the exception of possible computational limitations mentioned in Questions).

Reviewer ezjY2024-08-07

I acknowledge and thank the authors for the response. In hindsight, and taking into account other reviewers insights, I believe my given score (3: reject) to be overly pessimistic. In my review, I agree that the technical contribution is quite strong. I also believe that the experiments are impressive and very comprehensive for a mainly theoretical paper. I have therefore increased my score to a 5. I still believe that the paper has much room for improvement in terms of presentation. In particular, I spent quite a long time understanding the main theorems / propositions from the paper (mainly section 4). I would appreciate either informal statements of the theorems, or expanded discussion before and after theorem statements. At the moment, this discussion seems to occur in different sections and not immediately preceding or succeeding the statements.

Reviewer 7ztk2024-08-12

Thanks to the authors for the helpful response to my questions! I will keep my score.

Reviewer TrUb2024-08-13

Thank you for the rebuttal. It addresses most of my concerns. I'll maintain my score for now.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC