Summary
The paper presents a diffusion-based approach for reconstructing polygonal shapes from floorplan images and auto-drive sensor images. The idea is to refine an initial rough reconstruction through iterative denoising steps which revert a forward process that diffuses regular reconstructions into random noises through per-element noise injection (though the notions of elements are not clearly given for different tasks). The challenge with learning such a denoising diffusion model, as identified by the paper, is the robustness to permutations of orders of elements that are introduced by linearization (though it's not convincing why a set-based transformer model would introduce such ordering problem inevitably). The paper proposes to address this challenge by learning a guided forward diffusion process, such that the diffusion paths for different permutations are encouraged to be separate. In test time, the rough initial reconstruction is first converted to corresponding target noise distributions by the learned guided diffusion networks, and then reverted back to an accurate reconstruction through iterative denoising.
The approach is tested on two polygonal shape reconstruction tasks, including floorplans and autonomous driving maps, and shows improvements over the baselines that produce initial reconstructions. Ablation studies are conducted on diffusion guidance and choices of several hyper parameters.
Strengths
The paper proposes to improve polygonal reconstruction works that produce results in a single network pass by iterative denoising of the reconstruction. It is thus expected that the results can have better quality than the initial estimates, which is confirmed by tests on two tasks.
The paper formulates the diffusion model for structured elements like polygons and polylines, and identifies the problem of ordering permutation in representing the structures, which may cause severe ambiguity for a network to learn the reverse process.
The paper designs a learned guided forward diffusion process to distinguish the different permutations causing representational ambiguity, and derives the updated diffusion models biased by the learned diffusions. Such adaptive and learned diffusion can be inspiring to other situations where hand-crafting a diffusion process can be problematic.
Weaknesses
The computational cost of the iterative process can be better illuminated, to help readers appreciate the comparative advantage obtained over the initial estimates produced by other methods.
The paper identifies another challenge with diffusion-based approach for reconstruction: a reconstruction task has a single solution, so the initial noise needs to be chosen carefully. I do not fully understand this statement, and cannot find any validation of this challenge in the experiments and discussions. On the other hand, the proposed guided diffusion process does not produce a single noise for an input polygonal shape either, but samples from the predicted Gaussian distribution.
The guided diffusion process is trained through contrastive learning on the initial reconstructions of training data by a different method. It is not clear how well the guided diffusion generalizes for data samples outside the training set. In contrast, a fixed diffusion procedure as adopted by DDPM and others does not have generalization issue, as the final noise distributions are known a prior. Indeed, the paper acknowledges that when the initial reconstructions come from hand annotations that are largely different from training reconstructions for HD Maps, the guided diffusion and denoiser do not work robustly.
It might be difficult to regard this work as the first one using diffusion models for reconstruction. In fact, diffusion models have been widely used for conditioned generation that resemble reconstruction, e.g. ControlNet. Also, people have explored using diffusion models for segmentation [1], which is a kind of reconstruction.
[1] SegDiff: Image Segmentation with Diffusion Probabilistic Models.
Questions
1. Computational costs of the iterative denoisers should be reported, and compared with the base cost of running the initial reconstructions.
2. Please provide clarifications and experimental evidence that perturbations of the optimal noise for a particular reconstruction can cause failure of denoise-based reconstruction.
3. In the footnote of page 3, it's stated that even a permutation-invariant model like Transformer has to pick a permutation as the representation of a data sample. This statement seems self-contradictory and needs clarification. For example, in DETR like works (including RoomFormer and MapTR) there is no ordering of the elements, and they rely on matching to compute the loss functions between two sets. Why cannot this approach to extended to the diffusion setting?
4. When defining the permutation contrastive loss (12), why not compare the x_t and x'_t, but compare x_0 and x_t instead? Is it essential to constrain the similarity between x_0 the uncorrupted data and x_t the noised version? This is not common for standard diffusion models.
5. Sec.4, feature representation, it's not clear how polylines/polygons of different vertex numbers are encoded and represented. This is an important detail as its encoding directly impacts how likelihood-based refinement (Fig4) can be done.
6. The line matching metrics augmented with angle threshold is not defined clearly. There is no formal description of its computation in either text or supplemental.
7. Define the standard DM model of Sec.5.3 clearly. Otherwise it's hard to justify any proposed component.
8. What is training schedule? It looks like training epochs but why would one bother with the training iterations so much.
9. Since it is acknowledged that the guided diffusion network may not generalize to hand-annotated input, I hope the authors can thoroughly discuss the limitations of the learned diffusion network, including how to decide if it's not general enough for a particular test dataset or input pattern, and what are the practical ways to enhance its generalization scope.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.