Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing

Summary

This paper simultaneously addresses the label imbalance problem and uncertainty qualification capability in regression. The authors propose to enhance the reweighting technique dealing with the imbalance problem in (Yang et al., 2021) to be applicable to VAE and combine the method with the output distribution and the corresponding loss in (Amini et al., 2020), which provides uncertainty qualification capability. Experimental results on several real-world datasets demonstrate that the proposed method performs better than state-of-the-art imbalanced regression methods in terms of both accuracy and uncertainty estimation.

Strengths

### Originality: - The authors simultaneously address the label imbalance problem and uncertainty estimation capability in regression, which is a novel problem setting. ### Quality: - The combination of the reweighting technique in (Yang et al., 2021) and the output distribution and the corresponding loss in (Amini et al., 2020) is non-trivial and works well. - The authors showed the superiority of the proposed method experimentally in terms of both accuracy and uncertainty estimation, where they used multiple public datasets. ### Clarity: - The presentation is clear.

Weaknesses

- Good combination of the SOTA, but the originality can be limited because of that.

Questions

NA

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

They discussed that the exact computation of variance of the variances is challenging in Section 5.3.

Summary

The authors propose a variational regression model for imbalanced data, which (1) borrows data with similar regression labels for variational distribution (neighboring and identically distributed: N.I.D.) and (2) utilize the conjugate distributions to impose probabilistic reweighting on the imbalanced data to give better uncertainty estimation. Experiments show the proposed model achieves the SOTA on several datasets.

Strengths

- Imbalanced regression is a research area that is yet to be explored, although its real-world application is important. - In addition, the proposed model can provide uncertainty of the prediction, which is important for real-world applications. - The uncertainty estimation on imbalanced datasets is also an interesting field but is yet to be explored. - Overall, the present topic is very relevant in the community. I encourage the authors to develop and explore this direction more, in view of the nice performance of the model. - The performance of the proposed model is excellent. Both of the accuracy and uncertainty estimation outperform the SOTA models. - The present paper is easy to follow and well-written. I enjoyed reading it. I found some nice "road signs" for the readers to follow the logic and story.

Weaknesses

- The code for reproduction is not available. I would like to see and use your code. I could not find any other major weakness. - See Questions below.

Questions

Here are my questions and comments. - [Question] Just out of curiosity. What is the optimal prior for VIR? For the standard VAE, the optimal prior (that maximizes ELBO) is known to be the aggregated posterior. This might lead to an interesting future work. - [Comment (minor)] "NIG" on L.137 should be spelled out here, rather than on L. 213, or define NIG on L. 40. - [Comment (super-minor)] (1), (2), ... in the main text might sometimes be confused with the number of equations. Possible alternatives are: (i), (ii),..., (I), (II), ..., (a), (b), ..., (A), (B), ..., 1), 2), ..., etc. - [Question] Where does the idea of the hierarchical structure of the statistics (L. 166--195) (statistics of statistics for smoothing) come from? What motivated this idea? This is very interesting. - [Comment (minor)] L. 191: cross -> across? - [Comment (minor)] L. 212: correspond -> corresponding? - [Question (major)] At first glance, I could not fully understand why Eqn. 5 alleviate the imbalance problem. Could you clarify the underlying mechanism more? - [Question (major)] Is the performance improvement (accuracy and uncertainty) "significant"? If so, in what sense? (Sorry for the vague question, but this is an important point when evaluating model's performance and paper's quality because a marginal improvement is sometimes meaningless.)

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Limitations are included in Section 5.3.

Summary

This paper proposes a variational imbalanced regression model by taking the Neighboring and Identically Distributed (N.I.D.) assumption to solve both imbalanced regression and uncertainty estimation problems. Experiments on four imbalanced datasets demonstrate the effectiveness of the proposed method.

Strengths

1) Compared with imbalanced classification, imbalanced regression is underexplored and also an important topic. 2) The Neighboring and Identically Distributed (N.I.D.) assumption seems more reasonable than the Independent and Identically Distributed (I.I.D.) assumption. 3) The proposed method improves not only the performance of few-shot region, but also the performance of many-shot region.

Weaknesses

1) The description of some parts of the paper is not clear, e.g., how to define the neighboring labels? What is the detailed formulation of the importance weights in line 237? 2) Some statements in the paper are somewhat subjective and lack support, e.g., in line 42-43, the authors claim that "This allows the negative log likelihood to naturally put more focus on the minority data", but I do not find any "naturally" thing, the important weights are mainly determined by the kernel functions, which needs manually selection; in line 273, the authors claim that "Such dual representation is more robust to noise", but they neither list any references nor conduct any experiments to support it.

Questions

1) What are the new challenges of imbalanced regression compared with imbalanced classification? 2) Why do you partition the label space into B equal-interval bins? Does the choice of B affect the performance of VIR? 3) The choice of the kernel functions and experiments for the corresponding parameters selection should be given since it determines the importance weights. 4) In section 4.1, the authors claim that N.I.D. will lead to a lower generalization error just by simple text description, without any rigorous computation or derivation, this is not convincing for me.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes, the authors have adequately addressed the limitations.

Summary

In this work, authors recognize that although the existing regression models for the imbalanced datasets have been mainly developed to improve the prediction accuracy, they overlooked the quality of the uncertainty estimation. In this context, authors propose a deep probabilistic regression framework, to improve the uncertainty estimation performance as well by combining the idea of [1] and [2]. Specifically, authors first consider multiple bins, splitted on the range of labels to get statistic of the labels. Then, they revise the latent statistic of VAE for the imbalanced datasets, by smoothing these features based on the statistic on each bins and then applying probabilistic whitening and recoloring procedure. Next, they use the revised latent features to get the posterior distribution of NIG distribution, which acts as the pseudo counts that alleviates the issue of imbalanced sets. Last, they employ these parameters for prediction and training. Empirically, authors demonstrate that the proposed approach can improve the performance of the prediction accuracy and uncertainty estimation on various datasets. [1] Delving into Deep Imbalanced Regression - ICML 21 [2] Deep Evidential Regression - NeurIPS 20

Strengths

* This work extends the DIR of [1], as the probabilistic model, to estimate the uncertainty. * This work considers to use the NIG posterior distribution, updated by the statistic of the stochastic latent feature. This seems to help balance the latent features of the imbalanced labels and thus yield the credible uncertainty for the imbalanced datasets. I believe that this is a novel part of this work as comparing [1] and [2]. [1] Delving into Deep Imbalanced Regression - ICML 21 [2] Deep Evidential Regression - NeurIPS 20

Weaknesses

* Absence of ablation study > In experiment section, it seems that the proposed method improves the prediction accuracy and uncertainty estimation. However, the current work does not investigate (1) whether each trick of the proposed method, such as the use of the stochastic latent feature (VAE) and use of the posterior distribution (NIG), is effective and (2) whether the proposed method is consistent up to the number of bins. * Less explanation on why the evidential regression model is used along with DIR, instead of using other BNNs. > In general, BNNs is widely used to estimate the uncertainty. I believe that applying the BNNs with DIR could be a direct way to solve the targeted problem. However, author takes the evidential regression approach, without explaining the reason or its motivation. This seems to less persuade why the proposed method is reasonable. If authors provide the motivation of this approach or can demonstrate that the proposed method could outperform the results of the DIR, obtained by BNNs or other approach for uncertainty estimation, I believe that the contribution of this work would be more clear and strong.

Questions

* Is the performance of the proposed method consistent up to the number of bins? How can performance vary with the number of bins? * Why did the authors employ the stochastic latent feature through VAE, that is different approach to the [1] ? Does the stochastic feature improve the performance as comparing when the deterministic statistic is used to update the parameters of the NIG posterior distribution ? * Did the authors consider the BNNs with DIR or the Deep ensemble approach [2] with DIR to solve the imbalanced regression problem ? [1] Delving into Deep Imbalanced Regression - ICML 21 [2] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles - NeurIPS 17

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See above Weaknesses and Questions.

Reply by Reviewer 9XGJ

Thank you for the reply. I carefully read all the reviews and responses. - All of my questions and concerns are addressed. - The global response and the discussion with Reviewer cFEB are interesting and will make the paper more convincing. - I think the fact that no other BNN models are not included in the paper is not a major probolem because the proof-of-concept is already done in the main text and the result is convincing. - As for the number of bins, it is a common problem when we use the quantization- (= binning-) based approach to regression problems; it is a common hyperparameter. - The global response shows that the proposed model is relatively robust to the number of bins. Overall, I strongly support the acceptance of the present paper and keep the score as is: - The topic is relevant and important for real-world applications. - The topic is interesting because this is an interdisciplinary research between imbalanced regression and Bayesina inference. - I could not find major technical flaws. - Author's responses are insightful and make sense, which will make the paper better after revision. - The code will be published. - The experimental results are convincing. Excellent work!

Thank You

Thank you for your prompt and encouraging response as well as for increasing the score! We are glad that our response has been helpful and convincing.