Leave No Stone Unturned: Mine Extra Knowledge for Imbalanced Facial Expression Recognition

Summary

This paper deals with the topic of facial expression recognition (FER). In particular, authors point out the problem of imbalanced class data since most FER data sets will have many more neutral or happy face images than images with other facial expressions. Authors propose an approach to address this problem building on two ideas. First is that there may be information to be learnt about the minor classes even from the samples from the major classes. Based on this observation, authors propose a novel attention map rebalancing to regularize the model. As the second idea, they introduce a label smoothing approach that weights the minor classes more than the major classes. Results on RAF-DB and FERPlus facial expression image data sets show that the proposed approach is able to outperform state of the art FER approaches.

Strengths

The main idea of learning about the minor classes from training samples in both major and minor classes appears to be novel and is one of the strengths of this paper. The idea of weighting the minor classes more heavily or using smoothed labels have been used in other applications before, but perhaps not in FER. Another strength of this paper is the set of numerical results provided. Both data sets being used are state of the art data sets and results clearly indicate that the proposed method outperforms competing methods for FER. Another strength of the paper is that it is relatively well-written although authors should have more clearly explained the difference between "overall accuracy" and "mean accuracy".

Weaknesses

In Fig. 1, flipping of the images is shown to introduce the notion of re-balanced attention consistency. It is not clear if flipping is the only transformation that makes sense for the FER or if some other transformations (e.g., scaling, intensity attenuation or gain) make sense. That needs to be discussed in more detail. Another weakness is that the proposed method is somewhat similar to the EAC approach although authors do a good job of clarifying the similarities and differences between their approach and EAC. The main assertion that there is information to be gained about minor classes from samples in major classes seems to mainly backed up by pointing to the similarities in corresponding attention maps (e.g., open mouth regions for fear and surprise categories). This evidence is subjective and not entirely convincing.

Questions

1. What is the difference between mean accuracy and overall accuracy? 2. Line 106: How do we know that the two types of information are "orthogonal"? Perhaps, characterizing them as "complementary" might be more appropriate. 3. Fig. 1 caption: "--- while do not degrade the high accuracy on major classes" should be "--- while not degrading the high accuracy on major classes" 4. Fig. 2: I assume that the bar corresponding to 0.6 probability is for neutral images --- that needs to be clearly indicated in the figure. 5. Eq. (7): Is the tilde on y() here being used to denote the flipped samples or something else? 6. Shouldn't there be bold-face numbers in every column of Table 1 and other tables?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

There is no discussion of limitations in this submission. Authors may want to add some discussion of the performances of the proposed approaches based on gender, race and/or age considerations.

Summary

This paper mainly focuses on solving the imbalanced problem in Facial Expression Recognition (FER). The goal of this paper is to enhance the performance on minor classes without compromising the performance on major classes. The contribution is two fold. Re-balanced attention consistency (RAC) module is proposed to mines extra knowledge pertaining to minor classes from both major and minor samples. Re-balanced smooth labels (RSL) module is proposed to regulate the classification loss and promote balanced learning. Extensive experiments on different datasets and backbones validate the effectiveness of the proposed method.

Strengths

(1) This paper adapts attention map consistency to solve the imbalanced learning problem of FER for the first time, which proves that solving imbalanced FER through re-balanced strategy is promising. (2) The experiment including various backbones, imbalance and attention visualization is sufficient.

Weaknesses

(1) The purpose of re-balanced attention consistency (RAC) is to extract balanced and transformation invariant knowledge of minor classes from all training samples. The word "knowledge" is unclear and should be more specific. Does it refer to high response values for specific regions on the facial feature map or something else? (2) The authors' explanation about the role of re-balanced smooth labels (RSL) is to improve performance to regulate the classification loss and promote balanced learning. This explanation is too abstract and difficult for readers to understand. Why is it designed as Eq.(8)? What do the two terms of Eq.(8) represent and why fusing these two terms works? I think the author should provide a more detailed explanation. (3) The proposed re-balanced attention consistency (RAC) is very similar to the attention consistency used in EAC. The difference is that an additional balance weight is used to re-weight the attention maps. I would like to know how much performance has been improved due to the addition of balance weight. I have looked at the ablation study in Table 5 and can only know the performance improvement of the entire RAC module. (4) The comparison in Table 1 is questionable. The overall accuracy of EAC is 88.01%, which does not match the number of 89.99% in the original EAC paper (See Table 6 in [46]). (5) In line 283, the authors think non-overlapping attention maps is better than overlapping ones. Are there any references supporting this viewpoint? Overall I think the paper is tackling the right problem. However, the unclear description of the proposed modules weakens the contribution of the paper.

Questions

Same as the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did not discuss the limitation in this paper. I suggest that the authors provide the "maximum capability" of the proposed method. In other words, what degree of imbalance will cause the proposed method to fail.

Summary

This paper focuses on the imbalanced learning problem in facial expression recognition (FER). Unlike existing works in imbalanced learning for image classification, the proposed method addresses imbalanced FER from a novel perspective of label distribution learning. Specifically, the proposed method is motivated by the observation that certain major classes in FER may contain valuable features for the minor classes. Building upon this observation, a re-balanced attention map consistency module is introduced, which encourages the model to extract useful information about the minor classes from both minor and major-class samples through attention map consistency. Additionally, a re-balanced label smooth module is incorporated to leverage prior knowledge of the training data distribution and regularize the classification loss. Experimental results on FER datasets with varying levels of imbalance and different backbone architectures demonstrate the effectiveness of the proposed method, particularly in improving the performance of the minor classes.

Strengths

1. This paper provides an interesting label distribution learning perspective to deal with the imbalanced FER, which is novel to me. It proposes two modules to mine extra information of minor-class samples from both major and minor-class samples. From my understanding, this method is specifically designed for FER and technical novelties are sound. 2. The experimental demonstration well supports the claims. The authors have curated different FER datasets with various imbalance factors and the performance improvement on these imbalanced FER datasets is non-trivial compared with other SOTA methods, especially on the minor classes like fear and disgust. Ablation studies presented in Table 5 indicate that each of the proposed modules contributes to the overall performance. I also noticed that in Table 4, combined with Swin-T, the proposed method achieves remarkable results of 92.31% overall accuracy and 87.71% mean accuracy on the widely used RAF-DB dataset. 3. The proposed method is easy to implement and the paper is written in an easy-to-understand manner.

Weaknesses

1. Lack of experiments. As illustrated by the authors, they utilize the weight (i.e., a hyper-parameter of 0.9999) introduced by Ref. [5] as the re-balanced weight for both the re-balanced attention maps and re-balanced smooth label. However, I think there should be some studies regards of this hyperparameter, and what if we utilize different values for the two proposed modules? 2. Some claims were not supported by experiments. The authors claimed that "EAC employs attention maps from all expression classes, rather than just the labeled class, which is similar to label distribution learning". However, there is a lack of experiments to validate this claim. I believe the authors should include experiments to validate their point of label distribution learning, i.e., what is the performance if we only utilize the attention map of the labeled class instead of all classes? 3. Comparative techniques were not sufficiently performed. For example, what if we use GradCAM instead of CAM to implement attention consistency? What if we utilize the basic label smooth instead of the proposed re-balanced label smooth?

Questions

I have some questions regarding the weight design of the proposed method. Since the weight is utilized by both of the proposed modules simultaneously, I think it is very important for the performance of the proposed method. Therefore, I am curious what the performance would be if we simply set the weight as the inverse of the sample numbers of each class? There should be more discussions regarding the weight design of this method.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The limitations of this paper should be discussed. This method is designed specifically for imbalanced learning in FER as FER has the label distribution learning characteristic. However, things might be different in other imbalanced image classification tasks. For example, minor classes in imbalanced CIFAR-100 might contain little common feature with major classes in imbalanced CIFAR-100.

Summary

In the paper, the authors tackle the issue of imbalanced learning in the facial expression recognition task by introducing a fresh approach. Their proposed method revolves around the concept of attention consistency under spatial transforms, aiming to extract information about multiple classes from each sample. This approach effectively enhances the performance of under-represented classes. The authors evaluate their approach on two datasets and successfully achieve state-of-the-art performance with a transformer backbone.

Strengths

(+) They achieve state-of-the-art (SOTA) performance on the RAF-DB dataset with the Swin backbone. (+) Table 4 clearly demonstrates the performance improvement brought by the proposed approach. It highlights the significant increase in performance for both minor classes and the overall score. (+) Despite attention consistency being proposed in previous work, the authors cleverly and novelly employ a modified version of it incorporating rebalanced smooth labeling. They provide a reasonable hypothesis, stating that information for minor classes can be captured from all training samples. (+) The visualization results presented in both section 4.7 and the appendix provide further evidence that the proposed approach functions effectively.

Weaknesses

(-) To further support the idea, I suggest conducting evaluations on multi-label classification using other facial expression recognition (FER) datasets, such as BP4D. If the hypothesis is correct, the proposed approach should also work effectively in a different type of FER task. Testing it in this manner would provide a good opportunity for validation. (-) The two methods proposed in the paper, in terms of their structure, are not entirely novel. However, there is potential for further refinement and innovation in their design.

Questions

- What is the purpose of utilizing Global Average Pooling to extract features? Doesn't it result in a loss of information? - Based on the findings presented in Table 4, it appears that the performance improvement for minor classes surpasses that of other classes. Regarding the fairness aspect (although it may not be the primary focus of the paper), what can be said about the trained models? Do you believe that your method also facilitates the development of fairer models?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Given that it is a facial expression recognition task, privacy could potentially be a concern.

Summary

The paper presents an approach to tackle the imbalance problem in facial expression recognition. The approach makes use of information from the majority class to help improve performance of the minority classes. The proposed approach is evaluated on two public FER datasets. Different backbone networks are evaluated, along with comparisons to state of the art and ablation studies.

Strengths

The paper investigates a challenging problem. and the proposed approach using attention map consistency is interesting. Encouraging results are shown on two public datasets. Ablation study is generally well conducted.

Weaknesses

Experimental design is lacking. FER2013 is an old dataset; newer FER datasets should be evalauted (e.g., AffectNet - Mollahosseini, Ali, Behzad Hasani, and Mohammad H. Mahoor. "Affectnet: A database for facial expression, valence, and arousal computing in the wild." IEEE Transactions on Affective Computing 10.1 (2017): 18-31.). Along with being newer, AffectNet needs to be evaluated as the approach is on imbalanced FER datasets. The paper conducts experiments under different imbalance factors, however, AffectNet is one of the largest and most imbalanced datasets available for FER. For example, in AffectNet, Happy is ~46% of the data and contempt is ~1% of the data. How does the proposed approach handle these extreme imbalance issues? The comparison to CB [5] in Table 1 is not clear. This paper is not on FER and does not evaluate RAF-DB. What is this? Paper states the accuracy of each class is not commonly shown in FER papers. This is not accurate. If it is not in a table, this is often shown through confusion matrices (e.g., Farzaneh, Amir Hossein, and Xiaojun Qi. "Facial expression recognition in the wild via deep attentive center loss." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.)

Questions

How does the proposed approach handle extreme imbalance such as in AffectNet (e.g., Happy vs. Contempt)? What is reference [5] in Table 1? This is not a FER paper that is being compared to.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The limitations of the proposed work are not addressed. How does the approach work on extremely imbalanced data? What kind of societal impact can this have? There are always ethical concerns with FER systems.

Author follow-up

Dear Reviewer N9Q5, We sincerely appreciate the time and effort you dedicated to reviewing our paper. We hope that our responses have effectively addressed your concerns. If you have any additional points of concern, please do not hesitate to bring them to our attention—we would be more than willing to address them. We eagerly await your feedback. Thank you very much!

Thanks for your rebuttal comments to my review.

Thank you for carefully responding to my comments and feedback. I have also read the other reviews and your rebuttal comments to those reviews. Overall, I am convinced that this will be a valuable contribution to NeurIPS 2023 and I will stay with my original rating of Accept.

Thanks for your detailed response.

I've thoroughly reviewed the authors' rebuttal and am generally satisfied with their responses, both to my review and to the other reviews. They have effectively addressed my concerns regarding the re-balanced weight study. The authors provided comprehensive experiments and analyses on the re-balanced weight distribution and the corresponding performance with different hyper-parameters. The comparison with the one-class attention map reveals that there is indeed extra information from attention maps besides the target class. Overall, I recognize several merits of this paper: 1. The idea of extracting additional knowledge regarding minor classes from both major and minor class samples is novel. 2. The experimental results on RAF-DB using Swin-T as the backbone are highly promising. 3. In the rebuttal, the authors presented results on AffectNet and the extremely imbalanced RAF-DB. Considering the accuracy of minor classes, it's clear that the method is well-suited for the imbalanced FER task. To the best of my knowledge, there is no similar method that extracts minor-class information from both the major and minor-class samples to effectively address the imbalanced FER task. The motivation and idea are novel and interesting to me. Given the method's strong performance, achieving state-of-the-art results in the imbalanced FER task, I believe this paper deserves acceptance. Therefore, I recommend accepting this paper.

Response to the rebuttal

I appreciate your comprehensive feedback. As I comprehend your comments, it appears that you utilized the EmotioNet dataset. However, it seems that the outcomes are coming from a 'classification' problem, rather than a multi-label classification task where each instance could be linked to multiple labels concurrently. To illustrate, if the AU units are accessible within that dataset, could you kindly share the results involving a multi-label classification task? I am inclined to support the acceptance of this paper, as the authors have adeptly addressed both my concerns and those of fellow reviewers. Through their thorough experiments, as presented in both their response and the manuscript itself, they have convincingly demonstrated their model's advancement to state-of-the-art status.