All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation

Image coding for multi-task applications, catering to both human perception and machine vision, has been extensively investigated. Existing methods often rely on multiple task-specific encoder-decoder pairs, leading to high overhead of parameter and bitrate usage, or face challenges in multi-objective optimization under a unified representation, failing to achieve both performance and efficiency. To this end, we propose Multi-Path Aggregation (MPA) integrated into existing coding models for joint human-machine vision, unifying the feature representation with an all-in-one architecture. MPA employs a predictor to allocate latent features among task-specific paths based on feature importance varied across tasks, maximizing the utility of shared features while preserving task-specific features for subsequent refinement. Leveraging feature correlations, we develop a two-stage optimization strategy to alleviate multi-task performance degradation. Upon the reuse of shared features, as low as 1.89% parameters are further augmented and fine-tuned for a specific task, which completely avoids extensive optimization of the entire model. Experimental results show that MPA achieves performance comparable to state-of-the-art methods in both task-specific and multi-objective optimization across human viewing and machine analysis tasks. Moreover, our all-in-one design supports seamless transitions between human- and machine-oriented reconstruction, enabling task-controllable interpretation without altering the unified model. Code is available at https://github.com/NJUVISION/MPA.

Paper

Similar papers

Peer review

Reviewer TGUq5/10 · confidence 4/52024-06-20

Summary

The paper proposed a unified image compression method with multi-path aggregation for joint human-machine vision tasks. The authors utilized a lightweight predictor to generate masks to allocate features into main and side paths. By leveraging the pre-trained main path module, shared features can be reused to support varied tasks by finetuning a relatively low amount of parameters.

Strengths

1. The authors provide an innovative unified image compression method with multiple path aggregation that can support multiple human-machine vision tasks with shared features. 2. Experimental results show that the proposed method achieves the SOTA performance. 3. The paper is easy to understand and the experimental results are well presented.

Weaknesses

1. For experimental results, the author compared several SOTA methods without the basic TinyLIC model. Besides, I think it is necessary to compare with SOTA baselines of unified models in terms of parameter amounts, time complexity, and computational complexity. 2. The effectiveness of the predictor remained to be verified. According to Table 3, the proposed Predictor provided a tiny improvement in performance with only 0.19% bitrate reduction.

Questions

Can the author provide more sufficient experiments to verify the effectiveness of the predictor module? As for the base model, is it possible to apply the proposed MPA to other more recently published learned image models?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

From my point of view, the proposed unified model needs to fine-tune as many sub-modules as the task amounts, and there are multiple training stages to adjust according to the specific tasks. The proposed framework still needs to accommodate different models adapted to the specific tasks, rather than an all-in-one image compression method.

Reviewer N4Rb7/10 · confidence 2/52024-07-17

Summary

This paper explores image coding for multi-task applications and introduces Multi-Path Aggregation (MPA), integrated into existing models to facilitate joint human-machine vision through a unified architecture. The MPA employs a predictor to distribute latent features among task-specific paths according to their importance, thus maximizing the utility of shared features and preserving task-specific features for further refinement. Additionally, a two-stage optimization strategy is proposed to mitigate multi-task performance degradation. Experimental results show that MPA achieves performance comparable to state-of-the-art methods in both task-specific and multi-objective optimization across human viewing and machine analysis tasks.

Strengths

1) This paper explores image coding for multi-task applications and introduces Multi-Path Aggregation, integrated into existing models to facilitate joint human-machine vision through a unified architecture. 2) Experimental results show that MPA achieves performance comparable to state-of-the-art methods in both task-specific and multi-objective optimization across human viewing and machine analysis tasks. 3) The paper is exceptionally well-written and has clearly articulated the problem, proposed method, and the significance of their contributions.

Weaknesses

I'm sorry that this paper is quite different from my research field, and I cannot make an accurate judgment on this paper. However, I think that this paper is exceptionally well written and has clearly articulated the problem, proposed method, and significance of their contributions. Therefore, I have no issues with it and am willing to defer to the opinions of other reviewers before making a final decision.

Questions

Please check the weakness.

Rating

7

Confidence

2

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have discussed the limitations of the work in Sec. 5.4 and Appx. A.

Reviewer sciG6/10 · confidence 3/52024-07-27

Summary

The paper introduces a Multi-Path Aggregation (MPA) architecture designed to unify image coding for both human perception and machine vision tasks. By integrating the side path, the authors aim to optimize performance across various tasks while maintaining efficiency in terms of parameter and bitrate usage. This approach promises seamless transitions between tasks and improved performance.

Strengths

1. This paper is presented very well, including but not limited to its readability and the rational presentation of experiments. 2. Each part is highly motivated. For example, in the MPA module, a learnable mask is first trained to decouple the features into task-shared features and task-specific features. By optimizing the side path, it adapts to various downstream tasks. Techniques such as Gumbel-Sigmoid are used to handle non-differentiable problems. By introducing the MPA, this paper unifies the encoder and decoder models. These techniques can be referenced and adopted by the corresponding fields. Additionally, the experiments are conducted very thoroughly.

Weaknesses

1. If we do not consider the issue of feature decoupling, introducing and fine-tuning the side path to adapt to downstream tasks is not a novel idea. For instance, in [1], a similar bypass fine-tuning method was introduced. 2. The proposed two-stage training strategy is somewhat complex, and it introduces a large number of training losses and hyperparameters, which may pose some inconvenience for its generalization and use. [1] Y. Guo, H. Shi, A. Kumar, K. Grauman, T. Rosing, R. Feris. SpotTune: Transfer Learning through Adaptive Fine-tuning. In CVPR 2019.

Questions

1. In the paper, MPA is used for classification and segmentation tasks, achieving good results. Can MPA be applied to other fundamental vision tasks, such as object detection? Alternatively, can MPA be used for knowledge transfer in cross-domain tasks, such as shifting from learning "task-specific" features to learning "domain-specific" features? 2. If possible, I would like to know whether this method is sensitive to changes in hyperparameters (such as those mentioned in Section 4) and how these hyperparameters affect the experimental results. 3. In terms of LPIPS and FID (from Bitrate 0.3 to 0.7), it seems that CRDR performs better. I completely accept that such a situation can occur, but I am curious about why this happens. 4. Can MPA be compared with parameter-efficient fine-tuning methods such as LoRA? Is it possible to conduct a simple performance comparison between them?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Please refer to the Weakness and Question sections. During the rebuttal process, I am willing to actively and promptly discuss with the authors and other reviewers. If the authors can adequately address my main concerns, I am willing to increase my score.

Reviewer N4Rb2024-08-14

Thank you for the author's response that addressed my concerns. Overall, I think the motivation is clear, and the writting is satifactory. However, considering the scores of other reviewers, I decide to maintain my score unchanged.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC