VQ-Map: Bird's-Eye-View Map Layout Estimation in Tokenized Discrete Space via Vector Quantization

Summary

To address challenges facing generating BEV maps posed by occlusion, unfavorable imaging conditions, and low resolution, this paper introduces a generative model to facilitate BEV estimation. A codebook embedding is used to encoder prior knowledge for the high-level BEV semantics in the tokenized discrete space. Compared with other methods leveraging dense features to supervise BEV maps, this paper takes this discrete semantic feature as a signal to supervise BEV tokens learned from PV views. In a word, the article is logical and clear in writing. Moreover, it has excellent results and sufficient ablation experiments.

Strengths

This paper is well-written and easy to understand. Moreover, the experimental data is rich. Both surround-view and monocular map estimation tasks are assessed. The proposed model has superior accuracy results compared to other methods. The visualization result of the BEV codebook is quite interesting.

Weaknesses

In the experiment, the parameters of the ablation method and the table were not explained clearly. In Table 4, the difference between (a), (b) and (c) is not clearly described. The specific meaning of 'Supervision' in (d) - (f) is not described, which should be added.

Questions

1. The motivation of this paper is unclear. Why do sparse features work better than dense features, or do they work better together? 2. In 3.2, the description is not clear. Does this part need supervision? If so, how do you implement the supervision method? 3. The results of Table 3 show that 8-layer and 512-dimensional are a better choice. Doubtly, compared to other methods (6-layer and 256-dimensional), the proposed method has better results due to the increased number of layers and dimensions. The comparison is somewhat unfair. Would you consider discussing this? 4. Would you consider presenting more computational complexity results like FLOPs/MACs, the number of parameters, and running time when comparing your proposed method against existing methods in the experiment tables, e.g., Table 1 and Table 2? 5. Would you consider presenting a set of visual comparisons to directly compare your proposed method against MapPrior? This can better show the advantage of your proposed solution.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

This paper has well analyzed the limitations and broad implications of the study.

Summary

The authors propose to use a generative model similar to the Vector Quantized-Variational AutoEncoder (VQ-VAE) to obtain prior knowledge for high-level Bird’s Eye View (BEV) semantics in a tokenized discrete space. By leveraging BEV tokens and a codebook embedding that encapsulates the semantics for different BEV elements, the proposed method aligns image features with the BEV tokens obtained from discrete representation learning in a token decoder. The experiments are conducted on two benchmarks, nuScenes and Argoverse.

Strengths

+ The paper is well-written and very easy to follow. The figures are also very intuitive, significantly facilitating other researchers to follow and understand the gist. + The visualization of the proposed method helps readers understand how the proposed method works. + In fact, I quite like the tokenization idea since tokenization may be more robust against noise and geometric changes. However, tokenization may lose some information or detailed spatial information. Considering the output of this task, the output layout is much coarser than the input though. So tokenization may work well.

Weaknesses

At a glance, the proposed method achieves very good performance. In the experimental part, the proposed method outperforms the competing method. However, there are two main issues in the experiments: **The competing methods are not state-of-the-art** If we look at the reference [17] published in ICCV 2017, that is the state-of-the-art method (The same task and the same dataset). Moreover, the code of [17] is also publicly available. Comparison with reference [19] which is a generative model is not compelling. Based on the reported results of [17], the results of the proposed method are much worse than that of [17]. **The dataset split of nuScene is not right** In nuScene, there are 700 scenes for training, 150 scenes for validation and 150 scenes for testing. In this paper, (L197-198) there is no testing dataset. The authors mentioned to validate the performance of the proposed method. It is not clear whether the authors use the validation set or testing set. Moreover, how do the authors select the model weights if a validation set is not provided?

Questions

My biggest concern is the dataset split for this work. I expect the authors can explain this clearly.

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

This paper proposes to use a generative model to encode BEV semantic maps into tokenized sparse BEV representations with codebooks. Specifically, it consists a two stage training scheme. First, train a BEV generation VQ-VAE, then use the BEV Tokens as a ground truth to train the second stage network where it maps the surround view images into BEV map. The second stage reuse the first stage BEV generator. Experiments show the performance surpass the state-of-the-art by 0.2% - 3% in IoU, on average improving 2.8% IoU on nuScenes.

Strengths

+ This paper proposes a complicated pipeline to generate BEV maps, using a pre-trained VQ-VAE as a intermediate representation then trains a two stage network to firstly encode features from perspective view images into the BEV map. + The idea of using a pre-trained network as codebook seems quite interesting in the BEV map generation domain

Weaknesses

Q1. On average, is IoU a good metric for measuring the performance of BEV map? Considering the road can be a continuous structure, probably one of the most important aspect to measure the drivable area is its boundary, is this faithfully reflected in the IoU metric? Q2. I think the mean IoU increase is summing over the different settings, but not in terms of their raw pixel right? for example, in Table 1, there are 6 factors, f1 to f6, where each f is computed based on their raw pixels. Then the mean is computed by sum{f1, ..., f6} / 6. However, considering the Walkway, stopline, carpark and divider is much fewer than the drivable area, can you compute the average performance on based on normalization of their raw pixels as well? Q3 Complexity of the cost Can the authors list their methods complexity? Since it consists of two stage training and might introduce many overhead. Though it is not a big deal, but it is also interesting to the community.

Questions

as above

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

This paper cannot handle small regions but might be very important to realistic AD environment, where holes and some random obstacles can be dangerous to the vehicle. Compared to other method that can achieve multiple task at the same time, this is a specialized method only for BEV map generation.

Summary

The paper proposes a novel approach, VQ-Map, for BEV map layout estimation, addressing the challenges of occlusion and low-resolution images in perspective views. By leveraging a VQ-VAE-like generative model, the authors introduce BEV tokens to bridge the gap between sparse image features and dense BEV representations. The method demonstrates strong performance on both nuScenes and Argoverse benchmarks, setting new state-of-the-art results.

Strengths

The paper clearly identifies the challenges in BEV map layout estimation and provides a well-defined solution. The proposed method is based on the use of generative models like VQ-VAE for BEV map layout estimation and aims to address the limitations of existing methods. The proposed method achieves good results on nuScenes and Argoverse benchmarks.

Weaknesses

The proposed idea of using generative model is based on the previous methods like DDP or DiffBEV. The proposed method like toke-based decoder is not very novel for the feature alignment. For example, the deformable attention is used and explored by many other methods for feature alignment. The paper focuses on two specific datasets. The comparison with more recent diffusion methods is not included.

Questions

How about the proposed method using other attention instead of the deformable attention? What is the effect of the codebook size? In Table 1, what is the reason to make two different settings for the experiment results in nuScenes validation set?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The proposed methods are based on the previous methods like DDP or DiffBEV. The technical contributions are not very novel, like the use of the deformable attention.

Comment

The rebuttal helps to address many of the concerns. The added computation complexity analysis and qualitative visualization comparison results should be integrated into the final version. MapPrior also provides realism (MMD) and uncertainty awareness (ECE) results. Would it be possible to provide such results for analysis as well? This could better verify the superiority of your approach over the baseline method. Sincerely,

Questions

Thanks for the response. The rebuttal has addressed many of my concerns. An interesting point is that using deformable attention to replace standard attention can lead to such a large improvemence. Which of the other methods (in Tab. 4 ablation study) has or has not included deformable attention? How about if include? Apart from that, is it possible to list the respective improvements from different proposed components?