Determining the 3D orientations of an object in an image, known as single-image pose estimation, is a crucial task in 3D vision applications. Existing methods typically learn 3D rotations parametrized in the spatial domain using Euler angles or quaternions, but these representations often introduce discontinuities and singularities. SO(3)-equivariant networks enable the structured capture of pose patterns with data-efficient learning, but the parametrizations in spatial domain are incompatible with their architecture, particularly spherical CNNs, which operate in the frequency domain to enhance computational efficiency. To overcome these issues, we propose a frequency-domain approach that directly predicts Wigner-D coefficients for 3D rotation regression, aligning with the operations of spherical CNNs. Our SO(3)-equivariant pose harmonics predictor overcomes the limitations of spatial parameterizations, ensuring consistent pose estimation under arbitrary rotations. Trained with a frequency-domain regression loss, our method achieves state-of-the-art results on benchmarks such as ModelNet10-SO(3) and PASCAL3D+, with significant improvements in accuracy, robustness, and data efficiency.
Paper
Similar papers
Peer review
Summary
This paper proposes a method for object orientation estimation from images. The method predicts object orientation in the frequency domain of SO(3), by using SO(3) equivariant layers that operate on coefficients of Wigner-D matrices. The network is trained with a MSE loss between the predicted Wigner-D coefficients and the Wigner-D coefficients associated with a ground truth rotation label. The network is evaluated on two datasets, PASCAL3D+ and ModelNet10-SO(3), and outperforms all baselines in terms of rotation precision and accuracy. Additionally, an ablation study is performed which demonstrates that the proposed output formulation is better than traditional rotation formulations, like quaternions, euler angles and rotation matrices, and that the method is robust to the sampling strategy used at inference time.
Strengths
- The paper introduces a new way to train Wigner-D coefficients for accurate object pose prediction. They apply a weighted MSE loss between the predicted coefficients and ground truth coefficients. - The proposed approach achieves SOTA performance on the PASCAL3D+ and ModelNet10-SO(3) datasets. - The experimental section is thorough. Competitive baselines are included and statistical data is reported for the proposed method. The authors include a new variation on the ModelNet10-SO(3) dataset, which evaluates the sample efficiency of the method and baselines.
Weaknesses
- The contribution is interesting, but minimal compared to existing work. The neural network model and evaluation scheme used in this paper was introduced in [1]. However, this is not entirely clear the way the paper is written. For instance, Sections 4.2, 4.3, 4.4, and 4.6 are all describing specific techniques from [1]. It would be easier to understand the contributions specific to this paper's network if these sections were moved to Background. - The second contribution of the paper, as stated, is confusing. What is meant by the "intrinsic properties of SO(3) equivariant representations"? Also, the claim "structurally guarantee 3D rotational equivariance" is not correct. It is not possible to guarantee 3D rotational equivariance of the network when the input is an image. - The core contribution of the paper is the use of a regression loss applied to learned Wigner-D coefficients. However, this aspect of the method is not examined closely. There is no information of how the values of $w_l$ are defined in Eqn. 5, nor any experiment justifying why MSE is preferred. The contribution would be strengthened by a comparison of other sensible loss functions, an experiment that showed the sensitivity of the method to the values of $w_l$. Additionally, it is not clear how the proposed loss is affected by object symmetries. Could the authors explain why regression on Wigner-D coefficients does not suffer from inaccurate predictions when the orientation is ambiguous (e.g. tables have 180 degree symmetry in ModelNet10-SO(3))? - The ablation experiment in Section 5.5 needs to be described in more detail. How exactly was the method modified to produce euler, quaternion, etc. outputs? Were SO(3) equivariant layers still used? - Some of the writing could be improved for clarity. In Section 1, "Accurate determination of 3D orientation presents a more complex problem than translation due to the intricacies of rotation symmetries and the high dimensionality of pose space" is confusing: how is 3D orientation higher dimensional than 3D translation? Also, "SO(3)-equivariance is crucial for accurate pose estimation ..." is not entirely true, as evidenced by the competitiveness of non-equivariant baselines in Tables 1&2. - Figure 1, some of the images are taken from other sources without attribution (e.g. the healpix grid is from https://healpix.sourceforge.io [1] Klee, David M., et al. "Image to sphere: Learning equivariant features for efficient pose prediction." arXiv preprint arXiv:2302.13926 (2023).
Questions
- Did you explore evaluating the model with a higher resolution grid or with gradient ascent? Given that the regression loss can be precise, the model may be able to achieve even lower rotation error with more grid points. - The regression loss is very effective at achieving high precision, but seems to lose the ability to model object symmetries or uncertainty (according to qualitative results in Figure A2). Is there a way to combine the benefits of high precision and distribution modeling? Other pose works have used a classification-regression framework [1], and it would be interesting to see if this idea could be applied here. For instance, apply the classification loss over a HEALPix grid first, then apply the proposed regression loss if the prediction is close. - In Appendix C.6, it seems like the frequency level is set for both the S2 and SO(3) convolutions in the network. Did you try only modifying the frequency for the output of the network? The issue with higher frequencies reducing performance may be due to the non-linearity (ReLU in spatial domain), which could be a bit unstable with high frequency signals. Some other works have used gating functions for non-linearities on Wigner D coefficients [2]. - From the introduction, "in the context of spherical CNNs, ..., the 3D rotation parametrization in the spatial domain is inadequate because these SO(3) equivariant networks operate in the frequency domain." What do the authors mean by inadequate? Clearly, conversion from frequency to spatial domain is possible (the proposed method does so in its non-linearity layers). [1] Liao, Shuai, Efstratios Gavves, and Cees GM Snoek. "Spherical regression: Learning viewpoints, surface normals and 3d rotations on n-spheres." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. [2] Liao, Yi-Lun, and Tess Smidt. "Equiformer: Equivariant graph attention transformer for 3d atomistic graphs." arXiv preprint arXiv:2206.11990 (2022).
Rating
6
Confidence
5
Soundness
3
Presentation
2
Contribution
2
Limitations
The limitations are discussed in Appendix F. What is meant by, "our method can suffer from significant errors due to the loss of spatial information when projecting 3D data onto spherical harmonics"? What 3D data? Where are these significant errors described in the experimental results or discussion? The authors mention that, "computational cost associated with Wigner-D coefficients and SO(3) equivariant networks should be improved..."; this is a bit vague and contradicts Table A6, which shows the proposed method has the fastest frame rate during evaluation.
Thank you for the response. I appreciate all the new results presented in the PDF, and the decision to revise the background/related work sections. It is exciting to see the proposed MSE loss can be combined with the Cross-Entropy loss from I2S to achieve both high precision and accurate distributions. Regarding R3, what exactly is the takeaway from this plot? That the model puts more weight on higher-frequencies early on in training? Is this something that could be addressed with a weighted MSE loss to boost performance in the very low data regime? Regarding R4, it would be nice to understand the pitfalls of the MSE loss over Wigner-D coefficients. For instance, could the gradient of the MSE loss push the model away from the ground truth rotation in some cases? This doesn't need to be addressed here but it would be interesting to study in the future. Regarding R5 and R2, it is interesting that there is a sweet spot in terms of how precise the model can be. Do you think this is due to the limited precision of the truncated Fourier series? Given the proposed revisions and the new results, I updated my rating.
Response of the Official Comment by Reviewer f5v8
Thank you for updating your rating by reviewing our rebuttal and revision proposal of the background/related work sections, and thank you for your thoughtful feedback. * * * **Takeaway from the plot in Table R3**: The plot demonstrates that the model consistently assigns greater weight to higher frequencies ($y$-axis and value), which correspond to more complex rotational components in the Wigner-D matrix. This suggests that the model effectively captures and prioritizes these complex aspects, regardless of the number of training views ($x$-axis). Additionally, the plot further indicates that the weighted MSE loss plays a key role in enabling the model to focus on more complex rotations early stage in the training process, ensuring that these components are adequately emphasized even when data is limited (very low data regime). * * * **Understanding the pitfalls of the MSE Loss in Table R4**: Yes, exploring whether the MSE loss could potentially push the model away from the ground truth rotation is indeed a valuable direction for future research. However, at this point, we have not observed or verified instances where the model moves away from the ground truth rotation due to the MSE loss. * * * **The “sweet spot” in terms of the model precision in Tables R2 and R5**: The "sweet spot" observed in Tables R2 and R5 is indeed an interesting phenomenon. However, we do not attribute this is due to the limited precision of the truncated Fourier series. Instead, it seems to be more related to the trade-off between computational cost and performance. In Table R2, as the SO(3) grid precision increases, we observe that model accuracy improves and then saturates. This suggests that beyond a certain point, further increasing the precision (e.g. the number of points, $Q$) yields diminishing returns in accuracy relative to the additional computational cost. For example, we identified a good trade-off at $Q$=2.36M points (6th row), where further increasing to $Q$=18.87M points (7th row) improves accuracy at finer error thresholds (Acc@3° ) but also significantly increases space complexity. Similarly, in Table R5, employing gradient ascent does enhance model accuracy and reduces error, particularly for more precise pose estimations. However, this also results in a significant increase in inference time. Thus, the observed "sweet spot" in precision is more likely due to the inherent trade-offs in computational resources and the model's ability to effectively utilize them, rather than limitations imposed by the truncated Fourier series.
Summary
The paper tackles 3D pose estimation from a single image. It builds on Image2Sphere which maps CNN features to the sphere which are then processed by spherical CNNs to produce a distribution over SO(3). The submission proposes using an MSE loss function in the spectral domain instead of cross-entropy in the spatial domain. It outperforms the baselines on pose estimation on ModelNet10-SO(3) and Pascal3D+.
Strengths
S1) There could be value in studying losses and other operations in the spectral domain instead of spatial. S2) The method seems to outperform the baselines in the attempted benchmarks.
Weaknesses
W1) The paper seems to be a lightly modified version of Image2Sphere [1], but this is not properly acknowledged in the writing. Following Figure 2 and Section 4, the feature extraction, spherical mapper, Fourier transformers and pose predictor seems to be the same, the only difference being that in the submission, the last IFT is skipped and the loss is computed in the spectral domain during training (at test time it still needs the IFT so it is the same as Image2Sphere as far as I understand). Although Image2Sphere [1] is cited, the method section mostly describes the same things which implies they are new contributions. Please rewrite it focusing on the new contributions and being explicitly about what comes from prior work. W2) The main difference with respect to Image2Sphere is the loss being computed between Wigner-D coefficients instead of spatially. However how this is actually done is poorly explained. a) How does one L245 "convert the GT 3D rotations from Euler angles (...) to the Wigner-D matrices"? The text seems to imply that the Wigner-Ds are evaluated at the ground truth rotation, but then the loss in Eq (5) does not seem to make sense since the output of the model are the coefficients associated to each D matrix. I think what should be done is inverting a function on SO(3) that is an impulse on the ground truth pose but there is no mention of it in the paper. b) What is the "similarity between Wigner-D coefficients and an SO(3) grid" is L263? It seems that this somehow outputs a distribution on SO(3) that is queried spatially, so it seems like it should be an inverse SO(3) Fourier transform? W3) Experiments on SYMSOL are missing. The dataset is designed to assess performance on symmetric objects, so methods that predict distributions over SO(3) are necessary; the baseline Image2Sphere handles it well. I believe the proposed method might perform poorly on it because of the way the ground truth is computed or the MSE loss, which are the differences with respect to Image2Sphere. I believe a comparison and discussion of this possible limitation is warranted. W4) The paragraph L113 "Rotation representation in frequency domain." is confusing. It talks generally of 3D rotations but actually describes the specific case of rotations of spherical functions. I think the U in eq (1) should be the same as the D in eq (2) but they are described as different things. ## References [1] Klee et al, "Image to Sphere: Learning Equivariant Features for Efficient Pose Prediction", ICLR'23.
Questions
In L212, is looks like S is a function on the sphere defined in the spatial domain, but it is said to be C'xN where N is the number of spherical harmonics. Should N be the number of points on the sphere instead? Typos: L44: leverages -> leverage L70: faciltate -> facilitate L87: Foruier -> Fourier
Rating
3
Confidence
4
Soundness
2
Presentation
2
Contribution
1
Limitations
Possible limitation on symmetric objects should be discussed (see W3).
Response of the Official Comment by Reviewer HCET
Thank you for your thorough comment and insightful feedback. **[Reply to W1]** We will ensure that the main text is updated to reflect the clarifications provided in the rebuttal, as multiple reviewers have raised concerns about the clarity of the method section. * * * **[Reply to W2-a]** We apologize for any lack of clarity in our original submission regarding the "mathematical transformation”. Specifically, we map the Euler angles, which represent an element of $ SO(3) $ , to a set of coefficients corresponding to the Wigner-D matrix elements. These coefficients are critical in capturing the rotational properties in the frequency domain. **Mapping from SO(3) to Wigner-D Coefficients:** As you correctly pointed out, each Wigner-D matrix element provides a fixed map from $ SO(3) $ to $ \mathbb{C} $. Our method involves predicting these Wigner-D coefficients directly, bypassing the complexities and potential pitfalls of spatial domain parametrizations. To achieve this, we utilize an equivariant network to predict the Wigner-D matrix coefficients in the frequency domain directly from the input image features. The prediction process does not explicitly involve an inverse $ SO(3) $ Fourier transform of an impulse function centered on the ground truth rotation. Instead, our network learns to map the input features to the Wigner-D coefficients that represent the corresponding $ SO(3) $ rotation, leveraging the equivariant properties of the network to ensure rotational consistency. The output of our network is a vector of Wigner-D coefficients that directly encode the rotation in the frequency domain. These coefficients are then converted back to a rotation matrix or another suitable representation (e.g., Euler angles) as needed for evaluation or further processing. **Clarification of the GT Transformation:** We would like to gently remind you that Appendix A (and B.2) contains the detailed conversion equations between the Euler angles and the Wigner-D matrix, including the small Wigner-d matrix. We will further revise the transformation process in our final manuscript, elaborating on how the conversion between Euler angles and Wigner-D matrices is handled in our approach. * * * **[Reply to W2-b]** Thank you for your detailed feedback on the computational efficiency of our SO(3) grid generation and similarity computation for SO(3) distributions. We would like to clarify and justify our approach. **Clarification on the Inference Procedure and Similarity Calculation**: The similarity calculation described in L263 is designed to identify candidate Euler angles from the predicted Wigner-D coefficients by comparing them against a predefined SO(3) HEALPix grid at inference. You are correct that our approach involves precomputing the inverse SO(3) Fourier transforms to generate the set of predefined Wigner-D coefficients. This approach indeed requires significant space complexity, with a high time complexity of $ O(B^6) $ naively to initialize the predefined SO(3) grid. Consequently, our inference method is actually slower and uses more significantly more memory than just computing the IFT of the predicted coefficients. However, after the predefined SO(3) grid is initialized, both the training and testing phases benefit from this precomputation, leading to reduced computation times during these phases. Additionally, for repeated runs, the grid can be stored and reloaded as needed, further optimizing execution time. We apologize for the inaccurate expression: "The SO(3) iFT method produces a single value, making distribution modeling difficult”. What we intended to convey was that the inverse SO(3) Fourier transform results in a specific mapping, not a single value. We hope this explanation addresses your concerns and clarifies our design choices. Our approach aims to balance computational feasibility with the ability to accurately model complex pose distributions in SO(3) space. * * * **[Reply to W3]** Thank you for acknowledging the addition of the SYMSOL experiments. As you noted, the proposed MSE loss and the previously used likelihood loss work complementarily. However, our model does not surpass the performance of I-PDF [48] on SYMSOL, and we agree that future research should focus on further improving symmetry modeling to address these challenges.
## Mapping from SO(3) to Wigner-D Coefficients: Unfortunately it is still not clear to me how the map from Euler angles to Wigner-D coefficients happen. The appendices A and B mostly repeat textbook information that is not helpful: - definition of spherical harmonics (6) - relation of wigner D and d (7) - rotating spherical harmonics with Wigner-D (8) - rotation in 3D from Euler angles (A.2.1) - expression for Wigner-D (9) - different ways to represent rotation (B.1) - rotation of spherical harmonics again (10) -- same as (8). - relation of wigner D and d again (11) -- same as (7). - expansion into spherical harmonics (12). - rotation of spherical harmonics coefficients (13). - decomposition into spherical harmonics (14). Another suggestion is to clean up the repetitive writing. For example equations (1), (8) and (10) are the same and show the rotation of the spherical harmonics using Wigner-Ds which doesn't even seem relevant to the approach. Equations (2), (7) and (11) are also the same. Another guess at what might be happening is that the Wigner-Ds might be *evaluated* at the ground truth Euler angles to produce the supervision signal so the model is trained to predict a set of *values* of Wigner-D functions and not the Wigner-D coefficients that come out of a SO(3) FT. This would look a lot like "positional encoding" that maps coordinates to their evaluation in Fourier basis (sin/cos for the Euclidean spaces). But then I don't see how the output Wigner-D coefficients of the spherical CNN are mapped to the evaluation of the same coefficients at the ground truth. This would also contradict most of the text that refers to "Wigner-D coefficients" and not Wigner-D evaluations. ## Clarification on the Inference Procedure and Similarity Calculation: I think computing the similarity against $O(B^{3})$ vectors of $O(B^{3})$ coefficients each is also $O(B^{6})$ even if all vectors are precomputed so doing a fast SOFT would still be faster? ## Conclusion I think significant rewrite is needed, both to clarify technical details and also the contributions wrt Image2Sphere [see W1] so I do not recommend acceptance at this time.
Summary
The paper proposes a method for regressing the rotation of an object from an image, where several (~10s) of training views of the object with known rotations are available. In particular, the approach proposes a rotation-equivariant network that predicts continuous Wigner-D matrix coefficients in the frequency domain, which can be converted into a discretised heatmap in SO(3) to extract the predicted rotation(s). The contributions are (1) a method for predicting in the SO(3) frequency domain directly and (2) an approach for decoding this to a rotation at inference time. The approach is evaluated on two datasets with respect to rotation error and demonstrates good performance. Ablations and sensitivity analyses in the paper and appendix validate many of the design choices.
Strengths
S1. Originality. The idea of directly predicting Wigner-D matrix coefficients is original and is a sound design decision. It avoids some of the discretisation limitations of prior work and is intuitive. S2. Quality. On the whole, the paper is well-presented, with useful figures and a clear structure. The writing is also quite decent, and the results are clearly displayed. In particular, the related work is well-written, clear, concise and sufficiently complete to position the work in its research context. In addition, the experiments validated the overall efficacy of the approach well, except for a couple of points below, especially the graphs of performance w.r.t. training data cardinality. S3. Clarity. The paper is clear overall and does a good job of directing the reader's attention appropriately. One caveat is that it is not self-contained (with respect to the main paper and also the main+appendix), with some components left out of the main treatment that reduced the overall clarity of the treatment. Nonetheless, the introduction motivated the problem and approach very well, and each section was itself well-motivated, making the paper easy to follow. S4. Significance. The paper's contributions of predicting the frequency-domain coefficients and decoding them to a heatmap are likely to be of positive but mild significance to the NeurIPS community. The task itself is of high significance, since rotation estimation from an image (for known objects) is a challenging task that is important for many applications, including factory robotics, augmented reality, automotive, etc.
Weaknesses
W1. Contributions/claims. W1.1 The method in sections 4.1-4 is presented as part of the contribution, but appears to be the same as previous work, including [34]. If this is not the case, the design differences and the rationale should be better explained. If it is the case, this limits the contribution to sections 4.5-6, which is (a) directly predict and optimise the Wigner-D matrix coefficients, and (b) convert to a discretised heatmap and take the argmax. While (a) is likely key to the success of the method, it would appear to be a small variation on an existing approach; (b) on the other hand is very minor and indeed not claimed as a contribution. What this reviewer would like to see here is an expansion of these sections with some interrogation (and testing, in section 5) of the design choices. For example, is MSE a theoretically-grounded choice here, what are the weights w, why argmax instead of clustering for multi-hypothesis outputs? W1.2 Relatedly, the list of contributions has 3 items, but 2-3 do not seem to be contributions of this paper. That is, the contribution of leveraging SO(3)-equivariant representations to guarantee equivariance has clearly been done before (e.g., [27,34,40,33], among others); and achieving SOTA performance is not a contribution. W1.3 The claim that the proposed method is continuous and therefore better than the discretised methods (e.g., line 97) is dubious, when the predicted rotations are indeed discretised (section 4.6). The claim of loss of precision due to discretisation (for other methods) is also not validated in the experiment section; and one would assume that this method would also lose precision for the same reason (as Q decreases). W1.4 The claim that parametric models are insufficiently expressive for this task (L82) is not validated. It seems likely, but would depend on the model and would ideally be directly tested to back up the claim. W2. Design choices. (Partially overlaps with above) W2.1 Spherical mapper. While this is taken from existing work, it is not explained why it's a reasonable design decision to warp a 2D planar signal to the sphere, or given any motivation. It's clear that it's necessary for the method, to allow S2- and SO(3)-equivariant operations, but geometrically it seems quite dubious. Could the feature warping be made more geometrically meaningful by first running the image through DepthAnythingv2 and projecting out from a point on the crop's centroid ray? W2.2 The text mentions other approaches for non-linearities that avoid the extra (i)FFT steps (L233) but does not test or validate this design choice. W2.3 MSE (5). This does not seem to be a theoretically- or empirically-justified choice but is key to the contribution. Lines 254-258 loosely refer to this, but this section would seem to be a good opportunity to provide a justification for the design. W2.4 Discretised distribution on SO(3). The design choices around this part are not interrogated or justified - it reads as rushed or incomplete. For example, it states that the inference scheme "models objects with ambiguous orientations or symmetries by employing multiple hypotheses", but this is never done, as far as I can see. Instead, the argmax or gradient ascent is used, returning a single mode. Nowhere does the paper demonstrate how to use the approach to extract multiple hypotheses or test efficacy on extracting equivalent rotations for symmetric objects (see experiment section below). This is a missed opportunity and leads to the paper arguing against itself. W3. Clarity. W3.1 As alluded to earlier, the treatment is not self-contained. In particular, the Wigner-D matrix representation is central to the method and, for completeness and ease of understanding, should be included in the main paper. The reader would want to see the formulation that directly connects the task to the representation, instead of leaving that jump a bit vague (L121) and referring to the appendix. The function f in (4) and (12) are presumably not the same, but are not specifically defined for this task. Essentially, it reads as being incomplete, missing some of the material that connects the theory to the specific instantiation here. W3.2 L248. Please provide/derive the weights w for completeness. W4. Experiments. W4.1 The comparisons (especially in Table 1) would be more meaningfully grouped by the backbone extractor. For example, we should actually be comparing the similar method [34] with "ours (ResNet-50)", where a 0.6deg improvement is achieved. Similarly, it would be useful to compare the ResNet101 version on [34] in this table (like the version in Tab 2) since it is the closest existing method. W4.2 Missing dataset. SYMSOL, used in the closest related work [34,40], is not included in this paper, despite being a good opportunity to evaluate the performance with complex symmetries. Leaving it out gives the impression that the method cannot handle such cases, even though it seems that it ought to be able to. W4.3 Multi-hypotheses untested (despite L271). This claim ought to be experimentally tested, since it is such a relevant and interesting potential feature of the approach. W4.3 Ablations/sensitivity analyses. A sensitivity analysis on the influence of discretisation on precision (Q) is missing (despite claim in L96); the choice of MSE is not tested. W5. Minor. W5.1 The paper needs a proofread - there are many typos, syntactical and grammatical errors. W5.2 Confusing formatting error with footnote 2 appearing the page before it appears in the caption (L262.5).
Questions
1. Is 4.1-4 pre-existing work? If so, please make this clear in the paper. 2. Recommendation to rework the list of contributions to focus on the new aspects of the proposed approach and their significance. 3. Recommendation to validate the claim that the method does not lose precision due to discretisation. 4. Recommendation to expand on the design decisions W2.1-4. 5. Recommendation to test objects with complex symmetries via the SYMSOL dataset; and test the ability to predict multiple hypotheses as claimed.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
Yes, the authors adequately address the limitations and broader impact in the appendix, with an appropriate level of self-reflection and consideration.
Summary
In this work authors predicts SO(3) poses for objects by predicting Wigner-D coefficients in frequency space. Similar to other work [1], it first lifts 2D features to 2-sphere using pre-defined grid and orthographic projection to sphere using this grid, convert them to frequency domain, applies SO(3)-equivariant spherical convolution on top to predict a vector of Wigner-D coefficients. At inference time, these Wigner-D coefficients are mapped back to spatial rotation R. Authors show experiments on multiple datasets showing better accuracy compared to other work. Authors also show few-shot views training accuracy and ablation study for different design components of the network. References [1] Klee, David M., et al. "Image to sphere: Learning equivariant features for efficient pose prediction." arXiv preprint arXiv:2302.13926 (2023).
Strengths
Overall, the whole paper is well organized with sufficient figures to aid in understanding. Most of the math needed is included in the paper or cited the right resources. Experiments show that method is outperforming other pose-estimation methods that works in spatial / frequency domain. Ablation studies confirm authors claim of different network components. Authors also provide sufficient detail on the model architecture, spherical convolution design, and training hyper-paramters, suggesting reproducibility. Appendix has lots of nice extra experiments along with inference time.
Weaknesses
Some pose visualization on objects would be nicer and should be compared to other methods.
Questions
How would this method work for in-the-wild datasets? Would this method be able to handle occlusions? If we have to use "transformers" instead of convolution, what would change?
Rating
5
Confidence
3
Soundness
3
Presentation
2
Contribution
3
Limitations
Authors discuss limitations of their approach in the last section. Societal impact is discussed in appendix in detail.
Response of the Official Comment by Reviewer iMp8
First, thank you for taking the time to read and consider our rebuttal. * * * **Reply to R1 regarding pose visualization comparison:** We would like to clarify that the visualization of pose distribution using the surface of the 2-sphere with the Mollweide projection [A] conveys the same information as visualizations that use bounding boxes or the $x$, $y$, $z$ axes. For example, the pose distributions shown on the spheres in Figures R2, A2, and A3 can be converted into single pose values that can be directly plotted onto the object, as demonstrated in Figure 6 of [B]. [A] Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifold (Murphy et al., ICML 2021) [B] A Laplace-inspired Distribution on SO(3) for Probabilistic Rotation Estimation (Yin et al., ICLR 2023) However, we believe that the visualization method on the 2-sphere using the Mollweide projection [A], as used in Figures R2, A2, and A3, offers the additional benefit of enabling multi-hypothesis plotting in 3D pose space. This allows for deeper insights into symmetry and pose ambiguity modeling. * * * **Reply to Q1 regarding in-the-wild datasets (OOD scenarios):** Evaluating out-of-distribution (OOD) performance is generally not the primary focus in the context of this 3D orientation estimation task. However, we have conducted OOD generalization experiments using our proposed method by training the model on different datasets, between ModelNet10 and PASCAL3D+. The results are presented below: | Training Dataset | Evaluation Dataset | Acc@15 | Acc@30 | Rot. Err. | |------------------|--------------------|---------|---------|-----------| | ModelNet-SO(3) | ModelNet-SO(3) | 0.7590 | 0.7668 | 15.08° | | ModelNet-SO(3) | PASCAL3D+ | 0.0004 | 0.0019 | 112.98° | | PASCAL3D+ | ModelNet-SO(3) | 0.0015 | 0.0086 | 130.44° | | PASCAL3D+ | PASCAL3D+ | 0.7495 | 0.8965 | 8.92° | **Table:** Cross-dataset evaluation for validating out-of-distribution generalization on ModelNet10-SO(3) and PASCAL3D+ datasets. As the results indicate, the model does not perform well when evaluated on out-of-distribution dataset. Nevertheless, we recognize this as an important area for future research, and we appreciate your suggestion to explore this further.
I thank authors for the clarifications. I acknowledge the response.
Thanks for the rebuttal.
Reply W1: I was rather asking to fit bounding boxes onto the object to compare with Ground-truth. Errors on sphere doesn't show exactly, how well the predicted pose is fitting on the object. Reply Q1: I meant the performance on dataset outside the training distribution. Does the model generalize to different dataset that the one it is trained on?
W1: Thank you, I really hope this is updated in the main text since multiple reviewers pointed out that the method section might be seen as claiming contributions from prior work. W2-a: I am interested precisely in the "mathematical transformation" mentioned in the rebuttal which does not seem to be described anywhere. Specifically, the Euler angles represent an element of SO(3), while each Wigner-D matrix element is a fixed map from SO(3) to $\mathbb{C}$. So I think the map is from an element of SO(3) to a set of coefficients corresponding to the Wigner-D matrix elements; one way to obtain this map is by the inverse SO(3) Fourier transform of the impulse function on SO(3) centered on the ground truth rotation, but this procedure is not described anywhere so I am not sure if that's what is being done or I am missing something. W2-b: "The predefined SO(3) HEALPix grid includes a set of predefined Wigner-D coefficients corresponding 1:1 to Euler angles" -> sounds like this is exactly what the inverse SO(3) Fourier transform of function that is one at the cell corresponding to some Euler angles and zero elsewhere would give. So my understanding is that instead of computing the IFT of the predicted coefficients, the IFTs are precomputed for each discrete delta function of the grid, and the similarity between predicted coefficients and the precomputed is used as a distribution on SO(3). So I believe that for functions of bandwidth $B$ the approach stores $O(B^{6})$ coefficients (the grid has $O(B^{3})$ points and each delta function is represented by $O(B^{3})$ coefficients). The procedure to compute the distribution would also be $O(B^{6})$, which is the same as the naive algorithms for the SO(3) Fourier transform (SOFT). Since faster algorithms can reduce that to $O(B^{4})$ and even $O(B^{3}\log^{2}(B))$, I think the proposed method is actually slower and uses significantly more memory than just computing the IFT of the predicted coefficients. Please clarify. "The SO(3) iFT method produces a single value, making distribution modeling difficult." -> This seems incorrect; Fourier transforms and inverses are maps from function to function, not single values. W3: Thank you for adding SYMSOL experiments. They show that the proposed MSE and the previously used likelihood losses are complementary and combining both may help -- I think this is a much more convincing result than the current ones in the submission. It should be noted however, that it still doesn't surpass IPDF results on SYMSOL and I believe there are follow-ups that outperform it.
Thank you for your detailed response, and in particular the results included in the PDF. My primary concerns---regarding (W1) the contribution overlap with [34], (W2) the design choice of the MSE, and (W4) the lack of evaluation on SYMSOL and its symmetric objects---have largely been addressed in the rebuttal. Specifically, the authors have committed to moving the material from existing work into a background section, making it substantially clearer what contributions have been made. While the contribution is relatively small in my view, it is sufficient and the material in the rebuttal has strengthened it to an extent (Table R1 especially). The authors have also interrogated the choice of MSE quite convincingly, addressing its strengths and shortcomings, and ablating it in Table R4. Finally, and most importantly for my decision, the authors have evaluated on the SYMSOL dataset, showing that the method performs poorly for highly-symmetric objects, but interestingly can be combined with the loss from [34] to achieve greater performance than either, on this dataset. A small note in relation to the rebuttal: I maintain that the claim that the proposed method is continuous and therefore better than the discretised methods (e.g., line 97) should be removed. I understand that it is continuous in the training loop, but it most certainly suffers from discretisation at inference time. This claim should be more carefully worded, especially now that the authors have good evidence regarding this effect (in Table R2). Having carefully read the rebuttal and the responses from and to the other reviewers, I am inclined to increase my rating to a WA.
Response of the Official Comment by Reviewer HnJh
Thank you for increasing the score to WA and acknowledging that our rebuttal successfully addressed concerns W1, W2, and W4. We will remove the claim that the proposed method is continuous and therefore better than discretized methods, as in L97, by following your suggestion. We recognize that while the method is continuous during training, it does undergo discretization during inference. Consequently, we will carefully rephrase this point to emphasize the continuity of our method during training.
Decision
Accept (poster)