We sincerely thank the reviewer for the constructive comments.
Weakness 1 (more params in training)
Although RefConv introduces extra params, these params are conducted on the basis weights instead of the feature maps, thus the extra Flops and memory cost introduced in training are negligible as stated in Sec.3.2 and quantitively shown in Table 1.
Moreover, these extra params can be equivalently transformed after training, thus no extra params are introduced in inference.
Weakness 2 (modern architectures)
We also test the effectiveness of RefConv on two modern architectures, ConvNeXt [1] and FasterNet [2], the improvements are 0.96% and 1.15%, as shown in Table 1.\
[1] A convnet for the 2020s. (CVPR 2022)\
[2] Run, don’t walk: Chasing higher flops for faster neural networks. (CVPR 2023)
Question 1
As requested, we conduct refocusing technique for the second time on the trained MobileNetv2 with RefConv, the accuracy is slightly enhanced from 72.35% to 72.43%. Then the accuracy remains close to 72.43% when more iterations of refocusing technique are conducted.\
Thus the performance improvement that RefConv brings may have an upper limit, which may be because conducting the refocusing technique once has been capable enough to effectively reduce the channel redundancy and learn new representations.
Question 2
DW conv was originally proposed to reduce the model size and FLOPs but at the cost of representational capacity. Consider a two-channel input feature map whose shape is (2, H, W), assume the conv layer is a two-channel DW conv whose kernel shape is (2, 1, K, K). The first channel of output only relates to the first channel of the input, and the second channel of output only relates to the second channel of input. But if the kernel is a regular (dense) conv, the kernel shape will be (2, 2, K, K), so that each of the output channel relates to all the input channels. From this perspective, we can see that a regular conv's representational is higher than a DW conv, i.e., it comes at a cost to let "each channel of the base weight kernel have its own role".
As evidence, in practice, if we only consider the performance, regular conv is always better than DW conv (namely, the accuracy will become higher when simply replacing DW conv with regular conv). Therefore, it's expected that implicitly connecting the channels of the DW conv can narrow the representational gap between the DW conv and the regular conv.
We explain how RefConv works. Taking the aforementioned two-channel DW conv for example, we denote its input, output, and kernel by X, Y, and W, respectively, so that using pytorch-style pseudo code, this layer can be represented by\
Y = conv2d(X, W)
RefConv can link the channels by re-parameterizing the kernel and such a re-parameterization is realized with a convolution on W. So that\
Y = conv2d(X, conv2d(W_b, W_r))\
Note that conv2d(W_b, W_r) can be seen as a re-parameterization of the original W. Since conv2d(W_b, W_r) is a dense conv, each channel of the output directly relates to each channel of W_b, so that each output channel of Y **indirectly** relates to each channel of W_b. Note that we omit the "identity mapping" (Eq.1 in the paper) here for simplicity.
Except for the improvements in performance, in this paper, we have also provided evidence that linking the channels brings benefits. In a DW conv, since there are no connections between the channels of DW conv, it is easy to result in redundancy. The evidence is shown in Section 4.5. In contrast, RefConv reduces the redundancy, as shown in Figure 4. This is because by linking the channels with W_r, the re-parameterized kernel W_t is encouraged to learn new representations through the learnable combination of the channels of W_b, which were originally independent.
Question 3
So far we have not encountered the performance degradation.\
As discussed above, in the procedure of refocus training, the basis weights W_b dose not conduct convolution on the feature maps, it actually act as the input"features" of the refocusing transformation with W_r. Then the output generated by the refocusing transformation is the transformed weights W_t, which replaces the W_b and conducts the convolution on the feature maps. To conclude, in refocus training, W_b does not serve as a kernel to convolve on anything, W_r conducts refocusing transformation on W_b and generates W_t, and W_t is the kernel of convolution on the input feature maps and generates the output feature maps.\
We reckon that RefConv is more than just simply fine-tuning the original conv layer. It uses learnable transformations to establish the connections between convolution kernels, learning new representations through combining the learned representations by the basis kernels, thus strengthening the model representational capacity.\
Moreover, due to the use of "identity mapping" (Eq.1 in the paper), the representational capacity of W_t should be no lower than the original basis kernel.