More rebuttal
Dear reviewer,
Thanks for the quick and detailed response. We have further rebuttals as follows.
## Further clarification about the prior you proposed
- **One factual error**: the paper you cited, *(Papageorgiou, I., & Kontoyiannis, I. (2024). Posterior representations for Bayesian Context Trees: Sampling, estimation and convergence. Bayesian analysis, 19(2), 501-529.)*, considers $m$-ary trees, while in our paper we consider (full) binary trees only, as describe in Lines 104-105 and then emphasized in Line 177.
- **Further explanations about why the prior probabilities you proposed do not sum to 1**. For simplicity, consider the case when a decision tree has 3 nodes in total, for which apparently only one possible tree structure exists (i.e., one root node and two leaf nodes). However, if you put a 1/2 prior probability on whether to split each node, this "only" structure has a prior probability of $(1/2)^3 < 1$. Hence, in this case, the sum of all prior probability masses is $(1/2)^3 < 1$ as well.
## Further clarification about the hyperparameter tuning
- Again, as already mentioned in our previous rebuttal, we NEVER claimed that “no hyperparameter tuning is needed”; hence, I am not sure how we "tone it down".
- We indeed introduced $C$, yet we have shown that a global ad-hoc choice for $C$ can work reasonably well for all datasets in our experiments (and whether tuning $C$ can further increase the predictive performance of CDTree is not related to the main research question in this paper). By contrast, this is hardly possible for the regularization hyperparameter; i.e., it is hardly possible to pick a single value with an intuitive meaning for the $\alpha$ in the traditional decision tree learning optimization function, often in the form of "$impurity + \alpha |T|$", in which $|T|$ denotes the size of the tree, and $impurity$ can be MSE for regression.
## Regarding Condition (1) of page 426 of the MDL book.
- Notably, the text box on Page 426 does NOT give a rigorous definition of the "Refined MDL model selection", for the following reasons. First, see page 427, the 2nd paragraph, and here I quote *"The general idea is summarized in the box on page 426, which provides **something like a definition** of MDL model selection, **but only in a quite restricted context**. If we go beyond that context, these prescriptions cannot be used literally, but extensions in the same spirit suggest themselves."*. Second, the Condition (1.) of page 426 is also NOT a rigorous definition for the requirements of the priors, as the concept "quasi-uniform" is NOT defined throughout the book (see page 425, here I quote *"While we have **not formalized** what types of codes still count as quasi-uniform and what not ..."*). Hence, Condition (1) of page 426 is more like a general guideline summarized from the examples used in the book.
- The last two paragraphs of page 425 (starting from "we can choose the code ..."; and note that the last paragraph of page 425 ends on the top of page 427) are a more general guideline for choosing the priors for model selection than the condition (1) of page 426. Specifically, our proposed prior is close to the general description given by the last sentence of the first paragraph on page 427; here I quote, "In some cases we can apply conditional codes which do achieve a “conditional” minimax regret, ...". That is, in general, we encode the model by
number of nodes => tree structure => node splitting conditions => number of histogram bins;
each step is a conditional uniform/quasi-uniform code (as the integer code is listed as an example of quasi-uniform in the book), conditioned on the value encoded in the previous step.
- Notably, we quoted the content of page 427 (in our previous response) as it suffices to give the rebuttal to your previous statement that a mix of NML (one-part) and a model prior makes our MDL encoding "crude"; as shown on page 427, a mix of these two is by definition the form of the "refined" MDL model selection.