Summary
This paper studies how to efficiently insert a mask output head into a pretrained DETR object detector: What layers can be left frozen, and how to insert extra adapters to make efficient use of the frozen features for the new task. It finds that with a few well-placed attention/mapping layers, almost all of the new model can be left frozen, so that instance masks can be added with little training time compared to retraining all of the det+mask model from scratch. The system is built incrementally, showing the effect of inserting adapters at different points, taking only those needed. The final system achieves very good performance, recovering SOTA.
Strengths
This is a clearly presented study of a kind of "transfer learning" with frozen pretrained model. It's a little different from the most common transfer learning settings in that the training task (instance masks) is meant to supplement the original output (boxes), rather than replace it. The ability of the system to do this with smaller amounts of compute is promising (see below).
Weaknesses
This is a nice study of this sort of frozen model adaptation, but currently of limited use. The setting uses COCO with all boxes and masks available, so there is little need to separate box and mask training: the data is fixed, and both box and mask labels come from the same source. At best, one might say that optimizing box det first followed by masks later could make efficient use of training resources by allowing a (human or machine) model developer to focus on optimizing one task at a time, reducing the overall space of hyperparam variables and interactions --- but this isn't a point that was argued or supported.
A promising direction to push this further could be to see how well this method performs with different-sized subsamples of the mask data --- since mask labels are more expensive to gather than box labels, it is useful to use fewer of them, and collect just as many as needed in an iterative fashion. This method may provide a simple means to do that while making efficient use of compute resources. I think that would be an interesting application (particularly since the faster mask training could be used on incrementally larger datasets, as more mask labeling is performed).
Overall, this is a simple method whose main components are well documented and developed. However, it hasn't yet been applied and evaluated in a context with clear benefit or advantages.
Questions
Additional questions:
- comparison to SOTA methods in sec 4: This appears to compare this method, which inits with an object detector already trained, to systems that jointly train object detection and masks from only a classification- or self-supervised pretrained backbone, without any obj det pretraining, unless I misunderstood. I don't think this an appropriate comparison, as staged detection-first then mask training could also be done with any of these systems, freezing ether all or subsections of them as well. IMO the apples-to-apples comparison would be to train the existing systems for obj det only, then add the mask outputs and train those with frozen features, possibly unfreezing some of the last layers or inserting trainable FFNs (but nothing more involved than that, as that would start to replicate this work)
- same for fig 1: what are being trained in the comparisons -- masks only or everything?
- tables 2, 3: would also be good to compare to FFN only or 2-hidden-layer FFN. baseline qF has no ffn, only comparing directly the frozen features. So it isn't clear how much of the improvement comes from just a FFN part, vs the attn/spatial combinations
- end of sec 3: "additional enhancements ...". while I appreciate these are in the appendix, I think more should be included in the main text, describing: (a) the impact / amount of improvement obtained from these enhancements, (b) whether they have been applied to all the models shown in the development (i.e. all frozen, + image feature encoder, + box encoder), only some, or none, (c) which of the other sota comparison systems use these or similar enhancements as well, and which do not but could still make use of them.
- I also didn't see a measurement (at least in the main text) of the inference-time cost of adding these extra layers, though that can be inferred approximately taking a fraction of the training-time flops.
- sec 3.3 channel mapper: is the channel mapping layer applied to all dims, (h x w x d) -> d' (so 32*32*d -> d'), or just on the d dimension, preserving the h x w as (h x w x d) -> (h x w x d') ?
- table 4: I think the number of hidden dims could go even lower --- what about going down to 8 (or even 1 or 2)? There isn't any large performance drop-off yet in this table, while flops is still going down substantially. same for table 10: this could go lower than 16x16.
Rating
5: marginally below the acceptance threshold
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.