Summary
This paper explores the question of learning most likely functions vs most likely parameters in the context of MAP estimation, a common setting including minimizing log likelihood with L1 or L2 regularization, as these regularizations correspond to imposing a prior. Differences in parameterization affect the MAP through change of measure for the prior, and potentially in very unhelpful ways. Because of these potential problems with parameterizations, authors explore doing MAP not on parameters as we usually do (referred to as PS-MAP), but on function outputs for a set of inputs, an idea from Wolpert (1993). Authors refer to this as function-space MAP (FS-MAP).
The contributions are as follows:
- While Wolpert considered function outputs on a finite set of inputs, authors generalize to function outputs on a measure on input space. Authors emphasize that this measure of input space can be specified according to the task at hand (e.g. restricting to natural images, rather than all images).
- Authors empirically compare FS-MAP and PS-MAP on a simple example and empirically find that FS-MAP finds flatter minima, has better generalization in some settings, and produces less overfitting. Authors also discuss how FS-MAP and PS-MAP compare with the posterior mean in variations on this simple setting. Also, FS-MAP’s test performance depends on the measure of inputs that are considered important.
- Authors discuss problems with FS-MAP: FS-MAP can be computationally very expensive, and can behave in pathological ways under certain conditions. Authors also provide conditions under which it is well-behaved, as well as a scalable approximation to FS-MAP, called L-MAP, that also behaves ok even when FS-MAP would behave pathologically. Authors empirically compare FS-MAP and PS-MAP on neural networks for image classification. L-MAP performs similarly to PS-MAP, but is better calibrated in one experimental setting, and also finds flatter minima.
Strengths
1. Authors raise and discuss an interesting question that I had not thought about.
2. There is a substantive discussion of PS-MAP and FS-MAP, including weaknesses of both approaches.
3. Authors provide a computationally feasible, well-behaved approximation to a computationally too-expensive and poorly-behaved objective (L-MAP, vs FS-MAP).
Weaknesses
1. There is a tension between statements like “parameters have no meaning independent of the functional form of the models they parameterize” (lines 22-23) and the fact that the difference between best parameters vs best functions is a matter of parameterization of the prior, and the prior is a prior on the parameters (Equation 1). This tension is central to the question of parameterization for MAP and is not addressed.
2. This tension continues into the experiments, many of which emphasize flat minima. Authors cite Dinh’s “Sharp minima can generalize for deep nets”, which points out that the flatness / sharpness of minima is a function of parameterization.
3. Function-space MAP and priors on functions are defined much later in the paper than ideal. For example, in Figure 1, I do not know how FS-MAP is calculated. p_X (important to FS-MAP) is also not specified.
4. The terminology of “most likely functions” and “more probable functions” obscures the dependence of these terms on the choice of p_X (e.g. line 37, 184).
5. The connection to variational inference does not seem particularly strong. MAP as VI would involve choosing the best q, while the construction in Section B.9 does not do that. It’s also unclear to me why we care about these constructions; e.g. why do we care about q, besides that it ultimately leads to (13) and (B.13)?
6. Also, (B.8) looks like a KL divergence, not a variational lower bound / ELBO. Maybe I’m missing something here.
7. The experiments and discussion around experiments could be more thorough. See questions.
8. If someone told me they were looking at most likely functions, rather than most likely parameters for MAP, I would be curious about priors on functions in place of priors on parameters. (This isn't a weakness / question / limitation; it's more of a comment.)
Questions
1. On line 135-136, “the distance between two function evaluations is measured locally by [...]” : what two function evaluations are being referred to?
2. Why is it important to have infinitely many evaluation points? I assume that’s not the appeal but that’s how it was advertised. (Lines 127-128)
3. What is p_X=N/A in Figure 3b?
4. Can you provide intuition for why FS-MAP archives better generalization, in the settings where it does achieve better generalization? Can you also provide intuition for why it is less prone to overfitting? Can you discuss when this wouldn’t be the case? (Section 3.3)
5. The numbers in Table 1 are all very similar. What do you think is the reason for this? Should we be surprised by the similarities between the different choices of p_X, given how much p_X mattered in Section 3.3?
6. “For example, while it is 331 natural to assume that FS-MAP more closely approximates a Bayesian model average”: Why is it natural to make this assumption? (lines 330-331)
7. Could you briefly discuss why the posterior mean is particularly important (line 254)? The citation that follows is a textbook.
8. Why do we want to constrain eigenvalues of the Jacobian? I understand that you can run into problems if they are quite large. Is that the only reason? Is there a risk of over-constraining?
9. Why is calibration better with FS-MAP for CIFAR-10 and p_X=CIFAR-100 compared to PS-MAP, but not other settings?
10. Can you explain further why L-MAP is more robust to training with label noise? The explanation in line 325 is Section 3.3, which appears to be about input space, not about label noise
11. Should I expect these comparisons between FS-MAP and PS-MAP in other settings? In what settings should these results hold vs not (beyond satisfying the assumptions)?
12. While Wolpert’s evaluation on finite sets of points is a special case of the authors’ formulation, are they not effectively the same in practical settings?
13. Authors discuss a measure on which to consider function outputs. I am curious about the connections to distribution shift.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The experiments and discussion around experiments could be more thorough. See questions. It's not clear to me which experimental results should generalize, and under what conditions.