Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks

Rebuttal by Authors [Q5-Q8]

> Q5: *The results in Table 3 have shown the effectiveness of MDA when applied to GCN models. However, could the authors provide more details on the design of MDA for GCN models?* > We apologize for any confusion. We are pleased that Table 3 demonstrates the effectiveness of MDA on GCN models. In fact, the core computation of GCN models is given by $\mathbb{H}^{l+1} = \delta(\tilde{\mathbb{A}}\mathbb{H}^{l}\mathbb{W}^{l})$, where $\tilde{\mathbb{A}}$ is the degree-normalized adjacency matrix, $\mathbb{H}^{l}$ and $\mathbb{W}^{l}$ represent the features and weights at layer $l$, respectively, and $\delta$ denotes an activation function. This process can be simplified to linear operations involving $\mathbb{H}^{l}\mathbb{W}^{l}$. The MDA method proposed in our paper, which is designed for convolutional operations (discussed in Sections 3 and 4 of the paper), is also applicable to linear operations (as detailed in Appendix C). Therefore, no special modifications are required to apply MDA to GCN models; the same MDA method used for CNNs can be directly employed. This underscores both the versatility and effectiveness of the MDA method. > Q6: *Does model disassembling require additional training? If so, please present the details on how the numerical results in Tables 1 and 3 were obtained.* > Thank you for your question. Model disassembling does not require any additional training. The performance results presented in Tables 1 and 3 are derived directly from the disassembled models. As observed, the performance of these disassembled models generally exceeds that of the original models. The detailed reasons for this performance improvement can be found in our response to your first comment. > Q7: *The fine-tuning of the assembled model will inevitably cause a shift in the internal pattern of the model. Do the pathways of the fine-tuned model remain the same as the original ones?* > This is an interesting question. We have disassembled both the pre-fine-tuned and fine-tuned assembled models to examine the internal decision pathways. Our analysis reveals that fine-tuning does indeed result in changes to the internal decision pathways of the model. These changes allow the fine-tuned model to better adapt to the new data distribution, thereby mitigating the interference between parameters from different categories during inference. > Q8: *Have the authors considered the interference between categories when assembling the disassembled components? Are there any potential solutions to this issue?* > Thank you for your question. We have indeed considered the issue of category interference when assembling disassembled components. Details regarding this concern are addressed in our responses to your third and fourth comments. In the paper, we propose two approaches to mitigate this interference: the "parameter scaling strategy" and "minimal fine-tuning mechanism." These methods can alleviate inter-category interference to some extent without incurring significant computational overhead. Furthermore, in our future work, we plan to explore adaptive parameter adjustment strategies from the perspective of the parameter space of submodels. For example, we are considering leveraging diffusion models to adjust parameter distributions. This approach would involve adapting the parameters of the submodels based on the characteristics of each category, enabling the assembled model to rapidly adapt to new data distributions and potentially eliminating the need for fine-tuning. However, it is important to note that the current methods for model disassembly and assembly still hold significant value.

Rebuttal by Authors [Q2]

> Q2: *There is a lack of systematic comparisons with other works, such as deep model reassembly[1], and the study only includes self-constructed specific tasks, making it hard to evaluate the technical soundness.* > Thank you for your valuable feedback. Here is our response: **Systematic Comparisons with Other Methods**: We apologize for any confusion regarding this matter. In lines 44 to 47 of the main text and lines 503 to 504 of Appendix B, we highlight the key differences between our work and other related studies. To provide a systematic comparison, we have specifically compared our method, MDA, with the Deep Model Reassembly (DeRy) [1], as you pointed out. This comparison covers aspects such as problem definition, methodology, and effectiveness. It is evident that MDA differs significantly from DeRy and offers clear advantages. For instance, MDA disassembles models into task-aware components that handle different subtasks vertically. These disassembled submodels can be directly used for inference, are interpretable, and can be assembled to perform various tasks. Moreover, compared to other related work, MDA has several distinctive features. For example, there is no need for predefined subcomponents during the training of the original model, and no additional learning modules are required during disassembly and reassembly. In summary, this work represents the first approach to model disassembly and reassembly related to subtasks, allowing for the flexible creation of new models in a manner akin to assembling with building blocks. | | DeRy [1] | MDA [This Paper] | | --- | --- | --- | | Problem Definition | DeRy aims to partition different layers of models into equivalent sets and then reassemble layers from these sets. | MDA is the first to aim at disassembling a model into different task-aware components, with each component corresponding to one or more specific sub-tasks, which can be assembled to solve a larger task comprising these sub-tasks. | | Model Disassembly Method | DeRy uses covering set optimization to partition different layers of models into equivalent sets. To maintain these sets, each time a new model is partitioned, it requires calculating the "functional similarity" for relevant layers across all models, increasing unnecessary computational overhead. | MDA disassembles the model based on proposed contribution aggregation and allocation, independent of other models and solely related to the sub-tasks themselves. This direct approach significantly enhances disassembly efficiency and practicality. | | Effectiveness of Disassembled Models | The parameters generated by DeRy's method, which partitions models by layers, lack practical meaning, resembling a black box that cannot be understood or used directly for inference. | MDA disassembles the model according to the inference paths of different sub-tasks, making the model more transparent and interpretable. More importantly, the task-aware components disassembled can be used directly for inference without any training. | | Model Reassembly Method | DeRy reassembles models by solving integer programming to stack layers from different models horizontally. If the dimensions of two consecutive layers do not match, additional parameters for concatenation layers are introduced, necessitating retraining. | MDA reassembles models by vertically assembling different decision paths and uses a parameter scaling strategy that requires no training to mitigate sub-task interference. This approach usually yields excellent results and maintains interpretability during the assembly phase. | | Effectiveness of Reassembled Models | DeRy can only partition and reassemble models for the same task. For example, different models for ImageNet classification can be partitioned by layers and reassembled, but the new model can still only be used for ImageNet classification. | MDA can assemble models used for different sub-tasks. The reassembled model can solve new, previously unseen larger tasks comprising these sub-tasks. | | | | | **Self-constructed Specific Tasks**: Here is our response to your concern: 1. This work is the first to propose sub-task-based deep model disassembly and assembly (Section 2). Therefore, there are currently no standard datasets or tasks, nor comparable settings from previous works. 2. The datasets used to evaluate the final model performance are publicly recognized, and the evaluation process is standard. In addition to the commonly used model accuracy, we provide more detailed experimental results, such as parameter counts and FLOPs of disassembled models (Appendix G) and more granular baseline effects before model assembly (Appendix H). We understand your concerns and hope that the above response addresses your questions. Designing a comprehensive benchmark for model disassembly and reassembly will be one of our priorities in future work.

Rebuttal by Authors [Q3&Q4]

> Q3: *Given the widespread adoption of transformers in the field, the absence of experiments involving transformers is unfortunate.* > We understand your concern. In fact, the proposed MDA, such as contribution aggregation and allocation, can be applied to transformers, as discussed in our response to your first comment (Q1). We conducted a preliminary model disassembly experiment using ViT-Tiny on ImageNet-10, and the results are shown in Table 1. Table 1: Disassembly Performance of ViT-Tiny on ImageNet-10 | Disassembled Task | Base(%) | Disa(%) | | --- | --- | --- | | 0-1 | 87.70 | 89.25 (+1.55) | | 1-3 | 75.10 | 78.10 (+3.00) | | 3-9 | 77.51 | 79.29 (+1.78) | As shown in Table 1, the disassembled model can still be used for inference, and its accuracy is higher than that of the original model. Furthermore, we tested the model assembly performance of ViT-Tiny on CIFAR-10 and CIFAR-100. However, we must acknowledge that the results were not satisfactory. For example, when assembling the disassembled CIFAR-10 (0-3) and CIFAR-100 (11-17) models, the accuracy without fine-tuning was only 13.27%. As Reviewer gfoc mentioned in their final comment, task interference occurs when sub-models from different source models are assembled together, and this issue is particularly severe due to the self-attention mechanism unique to transformers. We believe that adjusting parameter distribution from different sub-model parameter spaces before assembly is a potential solution to this problem, which will be a focus of our future research. > Q4: *Could you provide an intuitive explanation for why your method sometimes outperforms the baseline, given that the baseline is end-to-end trained for the classification task?* > That's an excellent question. The superior performance of the proposed MDA method over the base model can be attributed to two main factors: 1. The source model, from which the disassembled components are derived, can be considered as trained on additional data relative to the target task. This means it has learned more features and possesses better feature extraction capabilities. 2. More critically, during model disassembly, our proposed method extracts only the parameters relevant to the subtasks, discarding those unrelated. This means that during inference, there is no interference from parameters associated with other subtasks, leading to higher inference accuracy. The performance improvement demonstrates that disassembling and identifying subtask-related model parameters is both feasible and effective. Besides the accuracy improvement, the experiments in our paper also show that our method accurately identifies subtask-related parameters and effectively removes irrelevant ones (as detailed in Table 5 of Appendix G).

Willing to Provide Further Clarifications

Dear Reviewer, Thank you for your valuable comments and feedback. We have thoroughly addressed the questions you raised in our detailed responses. If you have any further questions or concerns, we would be more than happy to provide additional clarifications. We believe this would be greatly beneficial in refining and improving our work. Sincerely, The Authors of Model Lego