Response to Reviewer uZLE (Part 1)
We sincerely appreciate your insightful reviews and constructive advice. We have provided detailed responses to your comment and updated the relevant content in the revised manuscript, hoping to address your concerns.
> **Q1: The paper lacks an overview figure, which may make it challenging for readers to understand the data collection pipeline.**
Thanks for your suggestion! A clear overview figure can aid readers in better understanding the methodology and contributions of our proposed data engine. Accordingly, we have included an overview figure in ***Figure 2 of the revised manuscript***.
> **Q2: This work does not propose a new model architecture or training approach, with only a mathematical dataset provided for fine-tuning. (Fortunately, the engineering effort is substantial)**
Thanks very much for recognizing our engineering effort. We would like to clarify the main purpose and contributions of this study, which are summarized as follows:
1. **The Main Goal of this work is to provide a general and robust paradigm** for training MLLMs in mathematical reasoning, including training pipelines and datasets. Our focus is not on proposing new fancy model architectures or training approaches, but on exploring a ***concise and effective*** method to target specific challenges in this field. This work provides valuable insights and a practical foundation for future research in mathematical MLLMs.
2. **Our Training Pipeline is specifically designed for key shortcomings in math.** Given our observation of MLLMs' three issues (*visual encoding of math diagrams, diagram-language alignment, and chain-of-thought (CoT) reasoning*), we respectively devise four training stages to alleviate them. ***The first*** pre-training stage of Math-CLIP bridges the semantic gap from general images to math diagrams. ***The second*** cross-modal alignment stage enhances the interpretation of LLMs on diagram embeddings from Math-CLIP. ***The third*** mathematical instruction-tuning stage endows MLLMs with fundamental problem-solving and reasoning skills. ***The fourth*** DPO training further elicit advanced reasoning capabilities and generate high-quality CoT rationales.
Therefore, while the techniques themselves are not new, our work represents ***the first effort*** to integrate and order them systematically for mathematical domains, collectively optimizing MLLM performance for mathematical reasoning, which provides reference for future work in this area.
3. **The Curation of Datasets is well tailored for the mathematical domain.** For each training stage we designed, we respectively support a corresponding dataset to fully unleash the training effectiveness, e.g., MAVIS-Caption for the first and second stage, MAVIS-Instruct for the third stage, and DPO ranking data for the fourth stage. Unlike conventional instruction-tuning datasets, ***our curation process incorporates the unique characteristics of mathematical tasks***. For example,
to mitigate the high costs of manual data collection and annotation, we develop ***an automatic data engine*** to efficiently generate math problems, including diagrams, captions, questions, and CoT rationales.
We refined these problems into ***text-lite versions*** by removing textual redundancy, ensuring the model focuses more effectively on diagram interpretation during training.
These designs leverage domain-specific insights to enhance the training process for mathematical reasoning tasks.
In conclusion, our work is far from a naive application of existing techniques. Instead, it integrates thoughtful designs in training methodologies and dataset curation, specialized for the multi-modal mathematical domain.