Color Equivariant Convolutional Networks

Summary

This paper questions the importance of color variations for neural classifiers and proposes to use color-equivariant architectures in the case of unbalanced datasets. To demonstrate the validity of the presented approach, the authors conduct experiments both on synthetic controlled datasets and on common object recognition benchmarks. As the experiments show, the injection of color-equivariant layers leads to a slight improvement on almost all common benchmarks when the performance is measured on the original test set but the advantage of the presented method becomes more evident when the test data is corrupted with hue shifts.

Strengths

This paper studies an interesting and underinvestigated question of the importance of color representation for neural networks. The submission is easy to read, and the motivation is well explained in the example of the Flowers dataset. The authors have conducted a significant number of experiments to support their claims. Additional strength is that the demonstrated performance improvement is achieved without increasing the number of trainable parameters (line 238).

Weaknesses

1. While the idea of extending equivariance from geometric to photometric transformations is definitely interesting, the submitted manuscript, unfortunately, focuses on the only type of such transformations, i.e. hue shifts. Despite the case of the Flowers dataset is a perfect fit for this transformation, the authors do not discuss other use cases when this type of equivariance may be interesting in practice and just mention "accidental recording conditions" (line 3). For other datasets, hue shifts seem less meaningful, and the better robustness of the proposed CE-ResNets to such shifts at test time is explained by the fact the architecture was just intentionally designed for this scenario. Taking this into account, I find the scope of the paper a bit limited. 1. In addition to being limited in the number of considered photometric transformations, the paper also considers a single task of object recognition. I would encourage the authors to consider other tasks as well, e.g. unsupervised domain adaptation. 1. While the authors claim their approach makes networks more robust to test time corruptions (Tab. 1), they do not demonstrate other baselines aiming to provide robust outputs, e.g. adversarially robust models.

Questions

1. I ask the authors to discuss the weaknesses stated above. The main thing I am interested in is the usage of robust baseline models. 1. While preserving the number of trainable parameters is valuable, I wonder if the network throughput remains the same for color-equivariant architectures in comparison with the baselines. If this is not the case, how significant is the decrease?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are addressed adequately.

Summary

The authors introduce a color equivariant convolutional neural network. To achieve this the authors represent the image in HSV format, and achieve hue equivariance using methods for rotational equivariance. This is possible since hue can be represented by an angle. The authors show that the proposed approach out performs standard CNNs and color invariant CNNs when there is a hue shift between the train and test set.

Strengths

* Originality: The presented method of building a color equivariant CNN appears to be original. * Quality: The work appears to be of fairly good quality. * Clarity: The paper is well written. * Significance: The observation that color equivariance can be achieved by identifying hue with the rotation group is interesting. The results show the proposed approach leads to improved performance when there is a color based domain shift.

Weaknesses

* Quality: I have some questions about the mathematical presentation, and experiment design (see questions). * Clarity: Some aspects were unclear to me, due to presentation or motivation (see questions)

Questions

* regarding equation 3. I believe the correlation should be between the feature maps and $C^{l+1}$ filters [7]. I think it could be clarifying to write that $c$ is the input channel and $i$ is output channel. * regarding equation 6: I'm not sure I understand how does H_n(k) acts on \psi_c? In the case of group 2D rotation equivariance, the filters are transformed by in plane rotations. It is not clear to me how one can perform a 3D rotation of a filter with an arbitrary number of channels. * It seems to me that the network is designed for local hue equivariance, can the authors clarify the benefit of this over global hue equivariance (i.e, performing hue shift on the input image then processing all inputs with the same cnn, and combining representations at the final layer to get a hue-equivariant representation)? * Does computational expense improve if input images are converted to HSV before being passed into the network? In this case, I expect hue equivariance could be achieved by discretization of the 2D rotation group rather than a 3D transformation. * Have the authors experimented with finer/coarser discretizations of the hue/rotation group

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

* Limitations -- in particular the issue of computational cost -- are communicated

Summary

Paper proposes color-equivariant CNN layers by imposing equivariance to H_n (a discrete subgroup of SO(3)) in the RGB space which is imposes hue equivariance. Implementation follows the framework of Group-equivariant CNNs. Experiments show marginal improvements over standard CNNs for in-distribution test data but significant improvements when test data is hue-shifted.

Strengths

1. Color equivariance in CNNs is a relatively less-studied but an important topic for robustness. The proposed idea of incorporating equivariance to hue transformations via rotations in the RGB space is novel. 2. Experiments are setup well clearly showing when color equivariance is helpful vs color invariance vs no symmetry. Proposed approach shows improves over CNN even when in in-distribution test data, but major improvements come when test data is hue-shifted.

Weaknesses

1. Definition of color equivariance considered in the paper seems to be restricted as it only considers the hue dimension. One of the motivations for incorporating color equivariance is for robustness to illumination changes which I do not think is guaranteed here. A general definition of color-equivariance should consider other dimensions. Maybe the claims are better justified if Hue-equivariance is emphasized in the title/introduction/method name, etc. 2. The definition of hue-equivariance is not precise in the paper. Ideally, it should include all rotations in the RGB space (i.e., SO(3)), but also consider the fact that many of these rotations take the color values out of the RGB space (unit cube). In general, this issue occurs for the discrete subgroup $H_n$ as well. Simply projecting the color values back into the RGB space does not work as it breaks the invertibility property of these transformations. 3. Experiments compare with a standard CNN (+grayscale) as baseline. Other baselines can be included, for example [1], that considers invariance to illumination color/intensity. 4. Experiments in the main paper only consider the group $H_3$ (i.e., 3 rotations in the RGB space), which seems limited in robustness, as shown in Figure 1 without jitter augmentations. References: [1] Lengyel, Attila, et al. "Zero-shot day-night domain adaptation with a physics prior." _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2021.

Questions

1. How is the hyperparameter $s$ chosen for different datasets in Table 1, especially for hue-shifted test dataset (assumed not known during training/validation)? 2. Is the invariance captured by Grayscale-Z2CNN same as that considered in the paper, i.e., invariance to hue? Would a better preprocessing step be only considering a canonical representation in the hue dimension? 3. In the color-imbalance experiments, were the CNNs trained with loss weighted according to the class imbalance? 4. Except being computationally expensive, are there other issues with using $H_n$ for higher n? Ablation in the Appendix seem to show better robustness with higher n, and also good in-distribution performance (at 0 hue shift). So I am not sure why $H_3$ alone was considered in the main paper. 5. It would be helpful to add a few lines of summary on the ablation studies in Sec 4.2. 6. Can experiments in Section 4.1 include CNN+jitter as it seems to provide competitive performance in Section 4.2? **AFTER REBUTTAL** Authors have addressed most of my concerns, particularly, adjusted the claims regarding color vs hue equivariance and added relevant baselines. I am increasing my score as I think this paper can be an important addition to an understudied topic. However, I am still concerned about the mathematical definition of hue-equivariance (via H_n group) since these transformations can map a color outside the unit cube and projecting back loses invertibility requirement of a group. I accept that this may not reduce performance in practical tasks (except certain inconsistencies pointed out by the authors), but I believe it is not mathematically correct.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations should discuss lack of robustness to other color dimensions (e.g., illumination).

Summary

This paper proposes color equivariant convolutional networks (CE-CNNs), a novel convolutional neural network architecture that achieves equivariance to hue changes.  They introduce color equivariant convolutions that apply a discrete set of hue rotations to the convolution filters during the forward pass. This makes the network output invariant to corresponding hue changes in the input image. They propose a group coset pooling layer that pools feature maps over the group of hue transformations to achieve invariance. They evaluate CE-CNNs on several image classification datasets, showing improved robustness to hue changes at test time compared to regular CNNs. The method also improves absolute classification performance in some cases. Overall, the paper presents a novel and intuitive technique to build invariance to hue changes into CNNs. The evaluations demonstrate its advantages over standard networks, especially under shifted hue distributions between training and test.

Strengths

This paper introduces a clever yet intuitive technique to make convolutional neural networks invariant to hue changes in the input image. The core idea is to apply discrete hue rotations to the convolution filters during the forward pass, essentially "baking in" robustness to color changes. The paper is clearly written and easy to follow. The authors motivate the problem well, explain their proposed method succinctly, and provide thorough experimentation across image datasets. The visualizations offer useful insights, confirming that the networks learn consistent features across hues. Overall, I found this to be an original and significant contribution. Invariance to hue shifts is a practical problem, and this paper tackles it through an elegant approach that outperforms regular CNNs. The concept of encoding transformations into convolutions seems powerful. While not the flashiest technique, the method is thoughtful, principled, and achieves strong results. The paper is presented clearly and comprehensively, making the ideas accessible. In summary, this is a high quality paper with both theoretical and practical value.

Weaknesses

- The method is demonstrated on image classification, but it's unclear how well it would generalize to other tasks like detection or segmentation. Additional experiments on other applications could strengthen the claims. - The ablation study on number of hue rotations suggests performance varies across different shifts. It would be useful to dig deeper into why - is it an artifact of how shifts are applied? Better understanding this could improve results further. - The approach encodes discrete hue rotations. An interesting extension could be supporting continuous rotations for finer-grained equivariance. - The comparisons to "grayscale" networks should be interpreted carefully, as removing color information entirely handicaps models. Comparisons to networks pre-trained onImagenet may be more meaningful. - The Flowers-102 experiments indicate the method doesn't help much on datasets without color bias. Analyzing when color equivariance helps or hurts could guide adoption.

Questions

1. The amount of color jitter augmentation during training seems to significantly impact performance. Could you do an ablation or analysis to tease apart the direct benefits of the color equivariant convolutions versus the data augmentation? 2. Have you evaluated the method on larger-scale datasets like ImageNet? Results on more complex data could better demonstrate the scalability. 3. The choice of 3 discrete rotations seems arbitrary. Can you analyze the impact of the group size to help guide selection? Are there benefits beyond 3 rotations?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

1. The comparison to only a ResNet baseline limits the conclusions on the benefits of the proposed method. Comparisons to other approaches could provide useful context. 2. The long-term impacts of building color equivariance into models is unclear. Discussion of downstream effects on fairness and interpretability could be beneficial.

Reply

Authors, thank you for the detailed responses to my comments. Your rebuttal addresses most of my questions and provides valuable discussion. In particular, the additional experiments analyzing color jitter augmentation better explain the advantages of your method. I have some follow-up thoughts: 1. The ImageNet experiments show model capacity is also important. This reminds us that there is a balance between color equivariance and network width when designing models. Further analysis from the authors on how to strike this balance would be helpful. 2. Thanks for the comparisons to CIConv and AugMix. These do provide more thorough baseline comparisons. If space allows, I'd recommend including these results in the main paper rather than just the rebuttal. 3. You mention more hue rotations improves robustness but also increases parameters. I strongly suggest analyzing the impact of group size on performance with fixed parameters. This would better illustrate the trade-off for readers. 4. Very glad to see the authors discuss effects on algorithmic fairness. This is an important aspect. Expanding this discussion would provide value if space permits. Overall, your responses have addressed my questions and I'm more satisfied with the paper now. Thank you for the diligent work, I look forward to the final manuscript.