Thanks for your valuable comments. We provide more materials and details for understanding semantic tokens and the way to divide channels. Besides, there’re sufficient theories and experimental results to demonstrate the necessity of combination of blurring and additive noise in the diffusion process for SR, i.e., Blurring ResShift. We have also provided detailed ablation experiments and explanations to validate the significance of the proposed Blurring ResShift during the rebuttal period.
> Q3: The answer is still unclear. I still do not unstand the meaning of "semantic tokens" (O) and what are the relation with the visual features (F).
The same terms are employed in previous works [3, 5]. Encoder-decoder based transformer architecture for segmentation has been prevalently utilized in recent years, and the concept of semantic token is blatantly simple to understand and widely accepted. We firmly don't think it absolutely constitute a reason for rejection.
As clearly stated in the rebuttal, transformer-based segmentation network [1, 2, 3, 4] requires **two types of input, i.e., images and learnable tokens**, which is well-understood in the realm of visual tasks and has also been applied in numerous image segmentation tasks. The encoder of the transformer extracts multi-scale features from images via self-attention (Refer to Fig. 2 and its caption in [1]), yielding visual features (F). Meanwhile, the decoder of the transformer establishes relationships between learnable tokens and multi-scale features (Refer to “box embeddings”, “encoded image” and “Resnet features” of Fig. 8 in [1]), generating **semantic tokens (O) that can retrieve pixels related to instances and background from visual features** (Refer to “object queries” of Fig. 2 in [1]). They are combined to produce the final segmentation mask through MLP or element-wise multiplication.
[1] Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers[C]//**European Conference on Computer Vision (ECCV)**. Cham: Springer International Publishing, 2020: 213-229.
[2] Cheng B, Misra I, Schwing A G, et al. Masked-attention mask transformer for universal image segmentation[C]//Proceedings of the **IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)**. 2022: 1290-1299.
[3] Zou X, Dou Z Y, Yang J, et al. Generalized decoding for pixel, image, and language[C]//Proceedings of the **IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)**. 2023: 15116-15127.
[4] Li X, Ding H, Yuan H, et al. Transformer-based visual segmentation: A survey[J]. **IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI)**, 2024.
[5] Zhang H, Li F, Zou X, et al. A simple framework for open-vocabulary segmentation and detection[C]//Proceedings of the **IEEE/CVF International Conference on Computer Vision (CVPR)**. 2023: 1020-1031.
> Q4: I do not see any direct answer to the question of how to divide channels.
The direct answer has been elaborated in the second paragraph beginning with “we dig abundant”，but we’d like to explain it in a simple way. The cluster of channel relies on two zero-initialized parameters of network, i.e., channel-wise semantics ($C_j$) and textures ($T_j$). The semantic parameters initially model the relationship with channels of instance-centric tokens ($O_i$) via cross-attention along channel dimension instead of vanilla spatial cross-attention, which is formulated as:
$$
Q=C_j W^Q,K=O_i W^K,V=O_i W^V
$$
$$
A=\operatorname{softmax}((Q^T K)/\sqrt{d})
$$
$$
\widetilde{C}_j =AV+C_j
$$
After stacked cross-attention and residual connection, the semantics contained in channels of instance-centric semantic tokens are resembled into semantic parameters, thus grouping different channel-wise semantics. Then, the texture parameter is combined with semantic parameter via matrix multiplication. It appoints each channel semantic to a corresponding texture. The texture parameter assembles valuable textures from multi-scale features via cross-attention, similarly to the above process. The motivation and technical details are elaborated, we think that you should read our paper and rebuttal more attentively.
> Q5: Where is the "official comments"?
You may not be able to access the reformulation of Eq. (16)-(18) provided during the rebuttal period due to system. We’d like to exhibit the same contents here.
We reformulate Eq. (16)-(18) as follows:
$$
\left(C_j, T_j\right)=\mathcal{M}\left(\bar{C}_j, \bar{T}_j,\left\\{\bar{F} _{i, k}\right\\} _{k=1}^4\right)
$$
$$
\widehat{T}_j=\bar{C}_j \times \bar{T}_j
$$
$$
T_j=\operatorname{SA}\left(\operatorname{CA}\left(\widehat{T}_j,\left\\{\bar{F} _{i, k}\right\\} _{k=1}^4\right)\right)
$$
where $\bar{C}_j$ and $\bar{T}_j$ respectively denote zero-initialized channel-wise semantics and textures of j-th group. $\widehat{T}_j$ is the texture embedding that possesses one-to-one correspondence with semantic embedding via matrix multiplication.