Summary
The authors propose a novel vision architecture, Parameter-Inverted Image Pyramid Networks (PIIP), which can be applied to different tasks, including classification, object detection, and instance or semantic segmentation. The authors aim to take advantage of the multi-scale information of image pyramids, without the usual excessive computational demands of processing an image in multiple different resolutions. The core idea is to use a family of networks of different computational requirements, e.g., ViT-S/B/L, and apply the lightest model (least parameters) to the image of the highest resolution in the pyramid, apply the second lighter model to the next bigger image, and so on, ending up to apply the heaviest model (most parameters) to the image of the smallest resolution. The intuition is that stronger models with more parameters can be used to extract semantically rich contextual features from images of coarser resolution, while lighter models can be efficiently applied to extract low-level features from high-resolution images. This way, the computational requirements of the different models are balanced, meaningful features from all scales are extracted, and images can be processed in higher resolution because the lighter and less expensive models are processing them.
The models are applied to the different images in the pyramid in parallel, and as features are extracted, the models communicate with each other through a proposed Interaction Unit. This way, features of different semantic meaning extracted from different models complement and inform each other. After the features from each individual model are extracted, they are merged to be used for the task at hand.
The authors perform extensive evaluation of PIIP on 4 different tasks, object detection, instance segmentation, semantic segmentation, and image classification. The authors initialize the PIIP backbones with pre-trained ViT models, and for each task they fine-tune appropriate heads. PIIP is compared to baselines of different scale, and in all experiments it demonstrates superior or comparable performance.
In addition, the authors perform multiple ablations, offering insights about the behavior of PIIP, and the importance of different design choices.
Strengths
Originality:
1. The core idea of using a parameter-inverted pyramid seemed to me counter-intuitive at first, since I would expect heavier models to be applied to the images of higher resolution, which contain more information. However, the authors make a good argument about why the parameter-inverted pyramid design is sensible, and I think it is a novel idea, that in its simplicity offers a valuable contribution to the community.
Quality:
1. The authors provide extensive experiments on multiple tasks, and in their experiments they control for the computational requirements of the models, offering meaningful comparisons.
2. The experimental evaluation includes multiple ablations, which shed light on different aspects of the proposed architecture.
Clarity:
1. The manuscript is well written, and easy to follow. The authors explain the intuition and the specifics of their method in detail, accompanying the text with clear visualizations.
Significance:
1. In addition to the novelty of the proposed idea of the parameter-inverted pyramid, I think a useful conclusion of the paper is the importance of using higher resolution inputs, even if they are processed with models of smaller size. To my understanding, this is the main reason PIIP outperforms the baselines, and is highlighted by the authors in Section 4.5 (ln 260 - 266). The importance of the input resolution is not something new, but the authors offer more evidence about its impact, and importantly, they offer a new paradigm to take advantage of it.
Weaknesses
Quality:
1. My main concern about this work is that the authors don’t provide actual timings and memory measurements. Computational requirements are quantified through FLOPs, however, theoretical gains in FLOPs don’t always translate to benefits in practice when the compared architectures have considerably different designs. For example, the PIIP models may be applied in parallel, but the interactions between the features may force the models to wait for each other, adding a sequential element that may add to the latency. I want to clarify that I am not claiming that this is the case, but that actual timings beyond theoretical FLOPs are needed to offer conclusive evidence, especially when efficiency is one of the main claims of this work.
2. About memory requirements, it seems to me that using a whole family of models may be prohibitive in many settings, so, memory should be explicitly measured. In addition, the authors control in their experiments for FLOPs, e.g., in Table 1 they show how PIIP models perform compared to baselines with similar FLOPs. I think it would be useful to control for memory too, e.g., in Table 1 the best PIIP models have more than 50% higher number of parameters compared to the baselines. Based on this, I think a question that naturally arises is what if the baseline was a larger model with equal parameters? If we have an extra memory budget, is PIIP the best way to use it? I would like to clarify that I don’t think the authors should necessarily have additional experiments controlling for memory, but if it is a disadvantage of PIIP, as it seems to be due to the parameter count, I think they should discuss it in the limitations.
Clarity:
1. In Table 2, the results for ViTDet-B and ViTDet-L are different compared to Table 1, why is that? Similarly, in Table 2, the results of PIIP-TSB and PIIP-SBL are different compared to Table 1. For the PIIP-SBL model, I see that the scores in Table 2 match the scores of the best model reported in Table 12, where models use higher resolutions, is this the reason for the discrepancy? However, the reported performance of the PIIP-TSB in Table 2 is higher than any performance reported for PIIP-TSB in Table 12, so, how the performance reported in Table 2 is achieved? I think the authors should clarify the configurations of the models they use in their experiments.
2. Some of the Table captions are not informative enough. For example, it is not clear why crop size is underlined in Tables 3, 6, 7. On a similar note, in the ablation about branch merging (ln 267 - 269, and Table 5), the authors don’t mention the dataset they use. I guess it is MS COCO, but I think it should be explicitly mentioned.
3. In the caption of Table 8, it is mentioned “‘PI’ and ‘IP’, ‘Inter.’ represent parameter-inverted, image pyramid and interactions”, I think it should be “‘PI’, ‘IP’, and ‘Inter.’”.
4. In Ln 262 the authors mention “green dashed circle in Fig. 5(a)”, but I think it should be “blue dashed circle”.
Significance:
1. PIIP requires a family of backbones for its processing, and the authors use pre-trained models in their experiments, while they also include some experiments on training from scratch in the Appendix. I think the need to train multiple backbones, or the need to find multiple pre-train models, may be an undesirable overhead for the adoption of the method.
Questions
1. In Section 4.1 the authors provide the pre-trained architectures they used for different tasks, how did they decide which models to use?
2. What is the formula the authors use to calculate the FLOPs of the models?
3. Why there are no comparisons with state-of-the-art models on image classification?
4. In Table 8, what is the range of resolutions that the model in the second row is trained on?
Limitations
The authors offer a short discussion of the limitations in Section 5, which I think should be expanded to address the memory requirements of PIIP.