E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection

Summary

This paper introduces a novel end-to-end algorithm named E2E-MFD for multimodal image fusion and object detection. Unlike existing joint learning methods, its key innovation lies in the synchronous joint optimization approach, simplifying the fusion detection process into a single training step and enhancing efficiency compared to traditional multi-step methods. To harmonize the losses between the image fusion and object detection networks, a Gradient Matrix Task-Alignment method is proposed. This method balances the gradients of shared parameters between the image fusion and object detection tasks, addressing the challenges of task dominance and conflicting gradients in multi-task learning. Additionally, an image fusion network with an Object-Region-Pixel Phylogenetic Tree is designed to perceive information at different granularity levels. Experimental results demonstrate the performance of the proposed method in both image fusion and object detection.

Strengths

- The idea of learning image fusion and object detection tasks simultaneously to mutually benefit each other is intriguing and reasonable. - An end-to-end fusion detection algorithm is proposed, effectively avoiding the local optimum problem encountered in multi-stage training models. Specific modules such as Gradient Matrix Task-Alignment and Object-Region-Pixel Phylogenetic Tree are introduced to achieve this goal. - Sufficient Experiments demonstrate that these modules facilitate the learning process, and jointly optimizing these two tasks outperforms other existing pipelines. - The authors clearly describe their methods in the paper and enhance its comprehensibility through the judicious use of formulas and figures.

Weaknesses

The overall idea is pretty interesting and reasonable. I can see the insight in the proposed method. However, there are some typos in paper: 1.The L_SSIM in the line 179 seems not same with the Equation (7). 2.I see a dashed arrow in the figure 1, what’s this mean?

Questions

1.In "Study of branches in the Object-Region-Pixel Phylogenetic Tree", the authors analyzed the reasons for performance degradation under settings 0, 1, 2, 3, and 4, and provided visual evidence. Could you elaborate on why the fusion performance only starts to decline after adding the fourth setting? 2.For Figure 4, the targets are only circled. Enlarging the highlighted areas, similar to the other figures, would make them clearer.

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

Expanding new modal datasets or implementing modality conversion between multimodal data will become a solution to the single issue raised in the paper about publicly available multimodal object detection datasets.

Summary

This paper focuses on the task of multimodal image fusion detection, combining texture details and target semantic information. An end-to-end multimodal fusion detection algorithm named E2E-MFD is proposed, which employs synchronous joint optimization, differing from existing independent or cascaded joint methods. The authors introduce a Gradient Matrix Task-Alignment method to help resolve gradient conflict issues in the image fusion and object detection tasks. Experiments on horizontal and oriented object detection datasets demonstrate the effectiveness of this method.

Strengths

1. This paper presents the first attempt to achieve simultaneous single-stage training of image fusion and object detection, and the results appear very promising. 2. Inspired by multitask learning, the Gradient Matrix Task-Alignment method is introduced to reasonably balance the loss functions, thereby converging the fusion detection weights to optimal points. 3. The multi-granularity strategy in the Object-Region-Pixel Phylogenetic Tree demonstrates its effectiveness in learning shared parameters, thereby enhancing object detection performance. 4. The writing and figures in the paper are clear and easy to understand. Proper use of formulas enhances the comprehensibility of their method. 5. The experiments and ablation studies comprehensively demonstrate the results.

Weaknesses

The paper presents a thorough and well-executed series of experiments that significantly contribute to the strength and credibility of the research. However, I think some problems need to be addressed: 1. In the experiments, YOLOv5s is compared. But why not comparing with the latest yolo? 2. The three backbone networks involved in the Figure 1 of the paper are not specified in the text, which would limit other researchers to know the details. 3. In the line 22, "a MF network" should be "an MF network".

Questions

See weaknesses above.

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors note that current model validation relies on visible light and infrared modalities due to limited relevant datasets within the community. They express a need for new dataset guidelines and contributions to the open-source community to address multi-modal dataset validation challenges in the future.

Summary

This paper proposes a joint learning diagram for multimodal fusion and object detection with task alignment module. The suggested network achieves SOTA performance with affordable computational cost.

Strengths

This paper presents a novel approach to learning image fusion and objection detection in a synchronous and joint way The proposed network achieves the SOTA performance in both tasks.

Weaknesses

Globally, I am ok with the significance of the work with the SOTA performance, despite the fact that it comes with additional computational cost. I am more concerned by other issues: 1. After reviewing, this paper gives me the impression that the proposed method is more like a combination of existing works/modules/tricks to achieve the SOTA performance. In other words, I am concerned by the novelty. 2. Secondly, this paper is hard to read and follow. The motivation of the proposed work is not strong enough compared to SOTA works. The diagrams are a little bit confusing. The writing needs to be improved. 3. I am ok with all the proposed modules and blocks to be claimed as novel, such as the blocks in nodes 1 and 2. It seems that the main contribution that the author claimed is on task alignment. This seems to be a very generic learning strategy. The authors should further validate its effectiveness with other works such as MetaFusion or other applications.

Questions

n/a

Rating

9

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

See weakness

Summary

This paper proposed an end-to-end algorithm for multimodal fusion detection, experiments on fusion and detection tasks showed the better performance than some methods.

Strengths

This paper proposed an end-to-end algorithm with one-stage training process，for multimodal fusion detection, experiments on fusion and detection tasks showed the better performance than some methods.

Weaknesses

1. Why use the V channel in the HSV space of the fusion results to calculate the metrics? 2. The best result of car detection highlighted in table 2 is wrong. Additionally, add analysis of why the proposed algorithm couldn’t realize the best detection effect of car. 3. Provide a detailed justification for the chosen datasets. Explaining why these specific datasets are representative or challenging. 4. The details of GMTA should be added. 5. It is mentioned that the GMTA operation is executed every 1000 iterations, but more specific implementation details, such as the specific setting and selection basis of parameters, are lacking. 6. Ablation studies for CFDP should be added to verify how CFDP impacts the final results. 7. The advantages of ORPPT and GMTA compared with existing techniques are not fully demonstrated. It needs to be more explicit about how these innovations solve existing problems or lead to performance improvements. 8. More SOTA methods and metrics should be added for image fusion task.

Questions

See the weaknesses.

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

See the weaknesses.