Summary
The paper introduces a 3D- and -instance-aware matting framework with multiplane representations. The multiplane concept is designed from two different perspectives: scene geometry level and instance level. The approach builds feature-level multiplane representations which split the scene into multiple planes based on depth differences. This is followed by further splitting of the planes on an instance-level, at which point each instance is associated with an alpha value and a color value.
The detailed splitting into a complex multiplane representation permits other operations, such as instance-level image editing.
Strengths
The multiplane representation is novel, and the two-phase splitting by depth then by instance makes a lot of sense (in terms of different ways that a multi-plane representation could be built).
The paper extends the MPI technique of Tucker and Snavely (2020) by using deep features rather than raw pixel values.
The detailed splitting into a complex multiplane representation permits other operations, such as instance-level image editing. However, the representation presumably is rich enough to enable other image editing pipeline operations, such as creation of stereo pairs (for 3D movies), or doing video editing. This could be mentioned in the paper and the title of the paper could perhaps be changed to include the editing aspect, since this is a definite contribution of the approach, and somewhat compensates for the computation costs of building the multiplane representations.
Experiments show the method works well compared to other approaches, and the ablations effectively show the influence of the various parts of the method.
Weaknesses
An obvious cost of the proposed approach is that it requires depth information as input (e.g. an RGBD image). The paper states "The input depth map here can be easily estimated using off-the-shelf depth estimators, and the actual cost is comparable to pre-instance mask generation in existing mainstream mask-guided instance matting methods." No support is given for these claims in the paper, however. How do errors in the depth map affect performance? (i.e. how sensitive is the approach to depth map errors?)
No information is given in the experiment section how the value of N (number of depth planes) was set. There should be an ablation or hyperparameter search to determine an appropriate value.
The paper states "i ranging from 1 to S representing instance-level plane for each distinct human instances within the image", but does not say where these human instances come from. Are these detected by some external semantic (to identify humans) instance segmentation algorithms? If so, which ones were used? In the experiment section all that was mentioned is that certain datasets were used, and my assumption is that the ground truth instance labels were used as input. But some guidelines should be given on how the instances (and depth maps) would be generated when presented with unseen data which were not in the training datasets.
Also, no information or experiments are given as to the effect of errors in the instance segmentation on the performance of the approach.
In Table 1, both headings are labeled "instance agnostic". One of the these (the lower one) should be instance dependant.
A minor point, but the proposed method is not limited to human matting, so it is unclear why the authors chose to focus the paper on human matting. Nothing in the approach is specific to human instances. Any other type of instance (e.g. horses) could just as well be used, and would give the method more generality and applicability. There needs to be specific information which is unique to human instances for the approach to really be called human instance matting. Otherwise, it is just instance matting.
As an example of this point consider the paper by Tan et al (WACV 2018 - which should be referenced as an early work in this area) states "To simplify the task, we restrict our problem by focusing on human instance composition, because human segments exhibit strong correlations with their background and because of the availability of large annotated data. " This motivates the approach based on the focus on human instances, which the current paper under review does not do at all.
Tan et al. "Where and who? automatic semantic-aware person composition." 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2018.
The conclusion does shed some light on this, stating "our work mainly focuses on human instances. This is partially due to the lack of data for multi-instance matting for other categories.", but this does not mean that the approach is necessarily a human instance matting technique. It could be used for non-human matting if such training data was used.
In section 2.1 the paper writes "Previous image matting (...) is conducted under the single-instance assumption without multi-instance awareness...". This is not true. There are previous works that do multi-instance matting.
For example, the introduction of the paper writes: "InstMatte (Sun et al., 2022) first introduced the fomulation of multi-instance matting...". But even this statement is not true. Sun et al 2022 was not the first paper to formulate multi-instance matting. For example, the paper:
Hu and Clark. "Instance segmentation based semantic matting for compositing applications." 2019 16th Conference on Computer and Robot Vision (CRV). IEEE, 2019 uses instance segmentation to delineate foreground objects to be matted, and uses the simple iterative approach of matting these one at a time, with the other instances temporarily being considered as background.
There should be references to the early papers on semantic matting, as the more recent ones build on the ideas from these. Besides the two papers mentioned earlier, the following paper should be mentioned:
Sun, Tang, and Tai. "Semantic image matting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
There are two references for the same paper in the (Qinglin Liu, Shengping Zhang, Quanling Meng, Bineng Zhong, Peiqiang Liu, and Hongxun Yao. End-to-end human instance matting. IEEE Transactions on Circuits and Systems for Video Technology, 2023.) One should be deleted.