Summary
D2CSG presents a neural architecture for inferring CSG programs that reconstruct complex 3D shapes. The architecture is composed of two branches, a cover branch and a residual branch, which are differenced from one another to form the complete shape. The work is largely an extension of, and a strong improvement over, CAPRI-Net, wherein the major differences are: (i) the use primitive complements, (ii) separate primitive sets for the cover and residual branches, (iii) a third training stage, termed dropout, encouraging program sparsity, and (iv) switching from an encoder to an auto-decoder framework. Finding a CSG program to represent a target shape involves "overfitting" the network in a test-time optimization scheme, with an occupancy-based reconstruction loss (with regularizing terms, and various relaxations). Compared to previous neural relaxation approaches for CSG program inference, D2CSG net find programs that result in much better reconstructions (both quantitatively and qualitatively).
Strengths
The paper presents compelling, and well-supported, evidence that D2CSG provides a marked improvement for the task of 3D CSG program inference (and more generally shape abstraction). The insights that contributed to this improvement, while not a dramatic departure from past work, are sound and sensible.
Based on the qualitative results, and quantitative metrics, its clear that D2CSG outperforms a reasonable set of baseline methods, offering a new state-of-the-art bar for CSG program inference. The ablation experiments, in both the main paper and supplemental, are very helpful and generally well-structured. The fact that D2CSG is able to capture a provably larger set of possible shapes than CAPRINet, due to the use of complements in the primitive set, is also a nice property.
Weaknesses
Probably the biggest weakness of the paper is that its methodological improvements are largely incremental on top of the CARPINet system. In my opinion, this shouldn't disqualify the paper from being accepted. Despite the similarity to CARPINet, D2CSG offers real, supported improvement, and would be useful, and of interest, to the community. This type of paper requires strong experimental results, in the form of comparisons to related work and well-crafted ablation conditions, both of which are provided. To give a higher rating, beyond "accept", it would be useful to show that the insights found useful in improving CAPRINet to D2CSG could be extended to other architectures / systems (e.g. they would have more general applicability than this one system). While I think this could be possible, its by no means a given.
```Ablation issues```
There were a couple of small questions/concerns I had surrounding the ablation experiments:
- Is the top row of table 3 just CAPRINet (from the tables this looks like the case). But CAPRINet also has the encoder/decoder vs auto-encoder difference, so I would expect this to be a different architecture (e.g. the CARPINet without pretraining, provided in the supplemental). Please clarify what this row is supposed to report.
- I think it would be important to add rows to Table 3, where just DB, and just DO, are added to row 1. It would be fine to report these results in the supplemental material, but having access to their performance would help disentangle the effects of each change between systems.
```Clarity```
While I wouldn't say that the paper was unclear, I think the overall clarity of the method could be improved. The paper is not very self-contained, much of the technical details are omitted from the main paper, and pushed to either past-work or the supplemental material. Due to space constraints, a full explanation of the training scheme and architecture is likely impossible to have in the main paper, but I think the main paper could still be improved to give a better *motivation* for why the architecture and training scheme is designed in the way that it is.
- Most helpful would be to add more details and annotations to Figure 2 (e.g. the D, Q, T, Con, and a matrices). If these can't fit in the figure, then sub-figures showing parts of the pipeline should be introduced (e.g. to the supplemental).
- In the text, introductory paragraphs should be used to set the stage for the various matrices, and what information they are suppsoed to hold. For instance, there is no explanation given for what the "Con" matrix (first introduced on line 175) holds, beyond the formulas for how it is constructed. An additional paragraph at the start of 3.2, walking through all of the matrices that will be created (and what information they hold) would be very helpful.
- Similarly, while some technical details (though not all) are provided in 3.3 indicating how the loss functions change through the training stages, it would benefit the paper to have a high-level section at the beginning of 3.3 explaining *why* multiple stages are needed (while this reasoning might have been provided in past-work, it needs to be restated here, for self-containment). For instance, something like: in stage 0 no hard decisions are made for either (a) primitive to intersection members or (b) which convexes are added into union, then in stage 1 we make a hard decision for (b), and in stage 2 we make a hard decision for (a).
```Minor```
- It would be useful to know how the reconstruction metrics change when converting from quadrics to basic primitives for user-editing (especially compared with just directly optimizing basic primitives, and not quadrics).
- Some discussion of related work on shape program inference can be improved. [36] is not a supervised method, and should likely be grouped with [11], as RL-based methods for unsupervised shape program inference (see [1*]). Relatedly, [2*] should also be added to this section, as an alternative to per-shape optimization for 3D CSG inference.
- typos/phrasing:
- L:27, genera
- L:163, are float
[1*] Neurosymbolic Models for Computer Graphics, Eurographics 2023 STAR
[2*] PLAD: Learning to Infer Shape Programs with Pseudo-Labels and Approximate Distributions, CVPR 2022
Questions
- If I understood the dropout operation, its based on finding changes to the primitive / intersection contributions that make little difference (e.g. a small delta) to the output binarized shape from the network. I'm surprised that it isn't instead based in deltas to the networks reconstruction loss (e.g. difference versus the target shape). Did you explore any types of dropout operations that considered the target occupancy values?
- The conclusion has the following statement: ```We also have ample visual evidence that the CSG trees obtained by our method tend to be more natural than those produced by prior approaches```. Looking through the decompositions in the supplemental, I'm unconvinced that many people would posit the decompositions as `natural' -- I would consider either adding additional evidence to support this claim, or removing it.
- The conclusion implies that for datasets like ABC, that lack a consistent global part decomposition, 'overfitting' is more acceptable, with the subtext that its not helpful to learn over more than one shape at a time. I would encourage the addition of a little more nuance to this statement: while ABC-type shape are certainly less globally consistent compared with other domains (like ShapeNet chairs), mechanical parts share clear commonalities with one another, especially at the sub-component/local level, and other methods have found success in learning over distributions of these types of shapes (e.g. [46]). Thus, instead of tying the the merits of `overfitting' to this line of argument, it would be better to let it stand on its own.
- (minor) I'm about confused about Eq 7. From the explanation of the method, I would have thought that W is ignored after stage 0 is done. In table 1, it looks like it is not used at all in stage 1, but then re-introduced (and binarized) in stage 2. Line 109 in the supplemental indicates that stage 2 training is the same as stage 1, but how can this be the case if W is re-introduced (is it frozen in stage 1, then unfrozen/modified for stage 2)? I would request additional clarification on this mechanism.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
While the `overfitting' setup certainly seems to improve reconstruction performance (at least for D2CSG, though interestingly not for CAPRINet), there are certain downsides of moving away from a shared latent space: e.g., ill-conditioned inference tasks (e.g. program from partial point cloud, or program from image), and shape to shape interpolation, are no longer possible. It would be good for the paper to discuss the impact of this design decision a bit (beyond the change in reconstruction).