The responses to the minor points are as follows:
> I would expect decision list trees to be much weaker than soft trees because they have less parameters.
You can check it in Tables 3 and 4. Performance differences between perfect binary trees and decision lists are small.
> Model merging is mentioned as one of the applications for linear mode connectivity (LMC), however, no results for model merging are provided
Figure 7 shows the results. It can be observed that performance has improved through merging.
> I could not find a related work section
We have not explicitly separated a section for related work, but we do discuss related work in the introduction and conclusion. If it is necessary to have a distinct section, we can address this in the camera-ready version.
> What is "Ours" in Table 2?
“Ours” in Table 2 refers to the results when considering not only tree permutation invariance but also subtree flip invariance and splitting order invariance for matching. In Section 3, our method is mentioned, and the term “Ours” is also used in Figures 1, 6, and 7.
> I did not find in the main text any explanation (even after looking into algorithms in appendix, which I found very confusing) for the operation of WM and AM...
As mentioned, detailed explanations are provided in the appendix. We also explained in line 202 in the main text. If the absence of algorithmic details (Algorithms 1 and 2) in the main text lowers the evaluation, we can move them to the main part. However, we have structured the content this way to maximize the information within the page limit.
> could not find any details for LAP
Are you requesting a detailed explanation of LAP? If needed, we can add it to the appendix in the camera-ready version. We assumed that the LAP is well understood within the community, which is why previous representative studies, such as [1], did not explicitly explain it. Please note that we mention the algorithms to solve LAP (Jonker-Volgendant) in the main text.
> Why does an MLP with depth 2 have lower accuracy than an MLP with depth 1 in Table 2?
This is because Table 2 presents the generalization error. Generalization performance does not necessarily improve as the model becomes more complex.
> Could you please explain or provide sources for how soft trees help in the development of LLMs?
A soft tree can be interpreted as a hierarchical mixture of experts [4]. The mixture of experts is a technique used in large language models like Mistral [5]. While tree ensembles might not be directly used in LLM development, considering a scenario where each expert module in a hierarchical mixture of experts is a language model, subtree flip invariance and splitting order invariance become important when performing model merging.
> Why is the accuracy of the interpolated model higher than the accuracy of the starting models?
As shown in Figure 1, the line segments connecting the models after matching can often result in better performance in terms of generalization error. This phenomenon frequently occurs in experiments involving split data training [1] and has been observed in models like MLPs, not just tree ensembles.
> Why do you say that if models are LMC then they are functionally equivalent? Doesn't it just mean that they are connected with a path in weights space along which loss value is lower than some threshold?
Yes, since the barrier is not strictly zero in practice, your expression is more precise. If we consider a threshold to be zero, it would be equivalent to reaching the same local solution, achieving functional equivalence in such a case.
> What is the sum of leaf parameters?
Each leaf has a vector whose length equals the number of classes (this information is explicitly mentioned only in the function arguments on line 119, so we will clarify it in camera-ready). The term "parameters" might have caused some misunderstanding by suggesting it was a scalar. The model output is a weighted sum of the vector with a length equal to the number of classes, as shown in Equation (3).
> why decision boundaries of oblivious trees are straight lines?
In an oblivious tree, the splitting rules at the same depth share the same decision criteria, which include the slope and the intercept of the decision boundary. This means that regardless of the root-to-leaf path, the data passes through the same splitting rules, resulting in straight decision boundaries.
> The paper does not propose any practical application for the linear mode connectivity between soft trees
Model merging can be considered a practical application. Model merging has potential applications in continual learning [6] and federated learning [7].
----
[4] Jordan and Jacobs, Hierarchical mixtures of experts and the EM algorithm, ICNN1993
[5] Jiang et al., Mistral 7B, 2023
[6] Mirzadeh et al., Linear Mode Connectivity in Multitask and Continual Learning, ICLR2021
[7] Adilova et al., Layer-wise linear mode connectivity, ICLR2024