Deep Neural Networks (DNNs) have revolutionized artificial intelligence, achieving impressive results on diverse data types, including images, videos, and texts. However, DNNs still lag behind Gradient Boosting Decision Trees (GBDT) on tabular data, a format extensively utilized across various domains. In this paper, we propose DOFEN, short for \textbf{D}eep \textbf{O}blivious \textbf{F}orest \textbf{EN}semble, a novel DNN architecture inspired by oblivious decision trees. DOFEN constructs relaxed oblivious decision trees (rODTs) by randomly combining conditions for each column and further enhances performance with a two-level rODT forest ensembling process. By employing this approach, DOFEN achieves state-of-the-art results among DNNs and further narrows the gap between DNNs and tree-based models on the well-recognized benchmark: Tabular Benchmark \citep{grinsztajn2022tree}, which includes 73 total datasets spanning a wide array of domains. The code of DOFEN is available at: \url{https://github.com/Sinopac-Digital-Technology-Division/DOFEN}.
Paper
Similar papers
Peer review
Summary
The paper introduces DOFEN, a deep learning method for tabular data inspired by oblivious decision trees and random forests. DOFEN combines a novel differentiable oblivious tree model with a novel forest ensembling stage. On a well-established tabular benchmark, DOFEN performs close to boosted trees and mostly outperforms other deep learning baselines.
Strengths
- The model is evaluated on a well-established benchmark. - The model achieves strong results on the benchmark (not necessarily state-of-the-art in every respect, but still promising). - Many additional results are provided in the appendix. - The ensembling stage, which is shown to be useful in ablations, could be useful for other deep tree-based models as well. - Code is provided.
Weaknesses
1. I found the description of the method sometimes confusing, especially the mathematical notation (see below). Maybe writing the method as an algorithm could help? I was also missing some details the sub-networks $\Delta_i$ in the main paper (are they fully-connected?). 2. Training times are not reported (although inference times are reported) and from what I can infer from the text, training is rather slow.
Questions
**Questions:** - Section 3.2.2.: Is the shuffling process necessary to obtain good performance or would direct random sampling of coordinates be sufficient? Do you use more trees for datasets with more features? - In Appendix C you say that you do not use early stopping. Do you still select the best epoch based on the validation set? If no, why not? **Comments:** - l. 222: "Trompt is the current state-of-the-art tabular DNN." I would say this is at least unclear, since for example TabR (https://openreview.net/forum?id=rhgIgTSSxW) claimed to outperform XGBoost on the same benchmark. There are also many other deep learning methods that claim to be SOTA, e.g., GANDALF (https://arxiv.org/abs/2207.08548), GRANDE (https://arxiv.org/abs/2309.17130), BiSHop (new, so not required to cite, https://arxiv.org/abs/2404.03830). - For the related work I think GRANDE (https://arxiv.org/abs/2309.17130) is highly related, and they also propose differentiable trees that output weights together with the predictions. - I think oblivious decision trees are mentioned before they are explained, the authors might consider explaining them earlier. **Minor comments:** - Eq. 2: The notation is a bit strange, specifically the set notation for ODT(x), the long right arrow, the definition of $j$, and why j is an upper index and not a lower index. An alternative suggestion might be $ODT(x) = H(x_j - b), (j, b) = criterion(context), b \in \mathbb{R}, j \in \{1, \dots, N_{col}\}$. - l. 157: It is not clear how the shuffling of the matrix works. Is it equivalent to shuffling after flattening? If yes, then it might be better to write it in this way. But in Figure 2 there is still exactly one color per column, which might indicate that the shuffling works differently? - Eq. 6: From the text it seems that $\Delta_2$ is applied for each tree separately, producing a scalar output. From the equation, it seems that it is applied to the whole shuffled matrix, producing the vector $w$. - Eq. 9: The naming with one prime for the vector and two primes for the elements might be confusing. Also, it is not clear here if softmax is applied to each $w_i''$ separately or somehow together. - Eq. 8-11: It is unclear to me why you write everything as operations on big vectors, instead of just writing the computation for a single index $i$. - Eq. 12: use \mathrm or \operatorname to write names in formulas (then it doesn't look like products of single-letter variables) - An appendix table of contents at the beginning of the appendix would be nice to have. **Conclusion:** The paper provides a new method that achieves strong results on an established benchmark on a practically important problem. The results appear solid and comprehensive. Therefore I recommend acceptance.
Rating
7
Confidence
4
Soundness
4
Presentation
3
Contribution
3
Limitations
Limitations are discussed. Broader impacts are not discussed, but the method is general-purpose, so the impact depends on the downstream applications.
Summary
Tabular data is widely used in various fields. This paper studies the current situation that DNN still lags behind Gradient Boosting Decision Trees (GBDTs) in tabular data. This paper introduces Deep Oblivious Forest Ensemble (DOFEN), which is inspired by unbiased decision trees and implements on-off sparse selection of columns. In addition, DOFEN shows excellent performance on the recognized Tabular Benchmark, surpassing other neural network models and similar to GBDTs in multiple tasks. This achievement highlights the robustness and diversity of DOFEN in different tasks and proves its effectiveness on tabular data.
Strengths
First of all, this paper has sufficient technical contributions and corresponding theoretical analysis. I believe that this idea of introducing oblivious decision trees into deep neural network (DNN) architecture will promote the research progress of deep learning in tabular data. Secondly, this article is well written and easy to understand. The explanation of the formula is sufficient. Finally, the paper provides a comprehensive assessment of the DOFEN model, including model stability, efficiency, and hyperparameter effects, which provides valuable insights for further research and model optimization. 1. A novel deep neural network (DNN) architecture is proposed by DOFEN, which combines the concept of oblivious decision trees (ODT) and realizes the on-off sparse column selection of table data. This is a feature that is not common in existing DNNs. 2. DOFEN solves the non-differential problem of sparse selection in deep learning models through its innovative two-step process, which is a challenge in previous studies. 3. The paper not only proposes a new method in theory, but also verifies the effectiveness of the DOFEN model through experiments, showing its application potential on different types of data sets. 4. Although DOFEN is a complex model, the paper provides detailed model configuration and hyperparameter setting, which is helpful for other researchers or practitioners to implement and apply this model.
Weaknesses
1. Compared with other DNNs, the inference time of DOFEN may be relatively long, which may affect its practicality in application scenarios that require rapid response. 2. The randomization step of DOFEN leads to a slower convergence speed, which means that more training steps are needed to achieve optimal performance, which may increase training costs and time. 3. Although the paper mentions that increasing the number of forests can alleviate the instability caused by randomness, randomness itself may pose a challenge to the stability and reproducibility of the model. 4. Although the paper mentions that increasing the number of forests can alleviate the instability caused by randomness, randomness itself may pose a challenge to the stability and reproducibility of the model.
Questions
1. The DOFEN model mentioned in the paper has a long inference time. What are the specific performance bottlenecks in implementation? Does the author consider or implement specific optimization measures to improve the efficiency of the model? 2. The randomness introduced in the model is mentioned in the paper. How does this randomness affect the stability and reproducibility of the model? Is there a way to reduce this effect? 3. In deep learning models, model interpretability is an important consideration. Does the DOFEN model provide an explanation for the decision-making process? How are these explanatory features realized? 4. The paper mentions some limitations of the model. What are the key assumptions in the model design and experiment? How do these assumptions affect the performance of the model in practical applications? 5. Does the author test the generalization ability of the DOFEN model on different data distributions or domains ? If yes, please share the results and analysis of these experiments.
Rating
5
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
1. Compared with other DNN models, DOFEN has a relatively long inference time, which may limit its practicality in application scenarios that require fast response. 2. Due to the randomization steps introduced in the DOFEN model, the model requires more training steps to achieve optimal performance, which leads to a training process that may be more time-consuming than other models. 3. The performance of DOFEN may be sensitive to hyperparameter selection, and sufficient hyperparameter adjustment is required to obtain the best performance, which may increase the complexity of model tuning.
Summary
This paper proposes Deep Oblivious Forest ENsemble (DOFEN), a novel neural network-based architecture for tabular deep learning. DOFEN extends Neural Oblivious Decision Ensembles (NODE) by relaxing the interaction level between a sample and the conditions on nodes in oblivious decision trees. In particular, when constructing oblivious decision trees, DOFEN follows a two-stage procedure, where the first stage generates soft conditions and the second stage utilizes a shuffle-then-segment framework to generate multiple condition tables. The resultant combination of condition tables can be regarded as a differentiable counterpart to the vanilla oblivious decision trees, which the authors referred to as "relaxed oblivious decision trees". Experimental results on the widely adopted tabular benchmark demonstrate that DOFEN achieves excellent performance, outperforming a number of state-of-the-art tabular deep learning models.
Strengths
- The idea of using relaxed oblivious decision trees for tabular deep learning is, to the best of my knowledge, novel and technically sound. - The authors adopted the highly recognized tabular data learning benchmark, which ensures a rigorous evaluation of the proposed approach. Experimental results also suggest that DOFEN achieves very competitive performance.
Weaknesses
- The paper is not very well-written, and I found it quite hard to follow. - No ablation study is provided, and thus the readers cannot know how much each component in DOFEN contributes to the final performance. I would suggest that the authors dissect the difference between DOFEN and NODE, and conduct an ablation study to demonstrate how each newly introduced component improves the results.
Questions
- In the code provided in the supplementary material, is CCCModel the implementation of DOFEN? If so, could the authors please point to me which part of the code corresponds to $\Delta_2$ in DOFEN? - In the shuffle-then-segment procedure in DOFEN, is the shuffle random for each sample, or is it fixed during training and inference?
Rating
4
Confidence
3
Soundness
3
Presentation
2
Contribution
3
Limitations
The authors have adequately addressed the limitations and potential negative societal impact of their work.
I used the three regression datasets used in the NODE paper, and searched for hyperparameters for 500 trials. The following is the search space I have defined: args.num_cond_per_column_scale = 2 ** trial.suggest_int(name='num_cond_per_column_scale', low=4, high=6) args.num_cond_per_subtree = 2 ** trial.suggest_int(name='num_cond_per_subtree', low=1, high=3) args.num_subtree_per_condset = 2 ** trial.suggest_int(name='num_subtree_per_condset', low=0, high=1) args.train_num_estimator = 2 ** trial.suggest_int(name='train_num_estimator', low=5, high=10) args.test_num_estimator = args.train_num_estimator args.hidden_dim = 2 ** trial.suggest_int(name='hidden_dim', low=6, high=8) args.dropout = round(0.1 * trial.suggest_int(name='dropout', low=0, high=8)) args.shuffle_type = trial.suggest_categorical(name='shuffle_type', choices=['row', 'random']) args.lr = trial.suggest_float(name='lr', low=1e-5, high=5e-3, log=True) args.weight_decay = float('1e-{}'.format(trial.suggest_int(name='weight_decay', low=2, high=7))) Can you please let me know what's the best hyperparameters on these three datasets? Once I verify the results, I would potentially raise my score.
To obtain the performance of DOFEN on these three datasets (Year, Microsoft, Yahoo) in a timely manner, we did not conduct a hyperparameter search. Instead, we chose a reasonable hyperparameter setting, which is shared among these three datasets, based on dataset size and resource limitation. Below is the detailed hyperparameter setting we used for these three datasets: * `num_cond_per_column_scale` (refers to $m$) = 64 * `num_cond_per_subtree` (refers to $d$) = 8 * `hidden_dim` = 16 (for Year and Microsoft), 4 (for Yahoo) * `batch size` = 2048 * `number of epochs` = 50 We decided on these hyperparameters for a couple of reasons: 1. **Dataset Size**: The sample sizes of these three datasets are about 10 times larger than the largest ones used in the Tabular Benchmark. Therefore, we increased the model capacity by setting `num_cond_per_column_scale` and `num_cond_per_subtree` to the largest possible values defined in Table 39. 2. **Memory Constraints**: The number of features in these datasets is relatively large and can cause out-of-memory (OOM) issues. As a result, we downscaled the `hidden_dim` from 128 to the largest available value that allows DOFEN to fit in GPU memory, which is 16 for Year and Microsoft, and 4 for Yahoo. 3. **Experiment Speed**: To expedite the experiments, we increased the `batch size` from 256 to 2048 (a value also used by NODE for the Year dataset) and decreased the `number of epochs` from 500 to 50. Other hyperparameters not mentioned here are set to their default values, as defined in Appendix C of our paper or in the code we provided. The results of DOFEN with these hyperparameters are provided in Table A6, with NODE performances taken from their paper. Please note that we only report results for one random seed (seed=0) to timely provide the outcomes. The random seed for both training and inference is crucial for reproducing the results of DOFEN. In the provided Python script `ccc_train_and_eval.py`, the training seed is set in lines 201 to 203 before model initialization, while the inference seed is set in line 145 for each batch. Table A6 demonstrates that DOFEN can achieve performance comparable to NODE, and we believe that DOFEN could achieve even better results with a thorough hyperparameter search. To achieve better hyperparameter tuning result, please follow the defined search space in Table 39 of our Appendix, as the model capacity of DOFEN is primarily influenced by $d$ and $m$. We highly recommend focusing on these two hyperparameters when searching for the optimal capacity for each dataset. Other hyperparameters, such as `hidden_dim` and `batch size`, can be adjusted manually to fit different situations (e.g., decreasing `hidden_dim` to avoid OOM and increasing `batch size` to speed up training). Regardless of whether DOFEN can outperform NODE on these three datasets after hyperparameter search, we want to emphasize that each model can have its advantages in handling different tasks. To evaluate different methods fairly and comprehensively, we argue that conducting experiments on a more diverse set of datasets, such as those in the Tabular Benchmark, can provide a more complete view of a model’s capabilities under different scenarios. We welcome any further discussion after you verify the experimental results. Thanks! --- Table A6: DOFEN Performance on Three Regression Datasets Used by NODE. Numbers are in Mean Squared Error (MSE), with lower values indicating better performance. | | Year | Microsoft | Yahoo | | ------------------------------------------- | ---------- | ----------- | ----------- | | DOFEN (a reasonable hyperparameter setting) | 77.4012 | 0.5584 | 0.5699 | | NODE (default) | 77.43±0.09 | 0.5584±3e−4 | 0.5570±2e−4 | | NODE (search) | 76.21±0.12 | 0.5666±5e−4 | 0.5692±2e−4 |
I tried the hyperparameters that you have suggested, but was not able to get the performance that you have reported. For example, on Year, the best result that I got is ```test_score = {'mse': 120.05630365986323, 'r2_score': -0.019386768341064453}```, which is a lot worse than NODE. Below is the modification that I have made to the code. Is there anything that is wrong? I do not know if feature normalization would help DOFEN, so I tried both with and without feature normalization. ``` import pandas as pd df = pd.read_csv('datasets/YearPredictionMSD.txt', header=None) train_df = df[df.index.isin(pd.read_csv('datasets/stratified_train_idx.txt').values.reshape(-1))] valid_df = df[df.index.isin(pd.read_csv('datasets/stratified_valid_idx.txt').values.reshape(-1))] test_df = df[~df.index.isin(pd.concat([pd.read_csv('datasets/stratified_train_idx.txt'), pd.read_csv('datasets/stratified_valid_idx.txt')], axis=0).values.reshape(-1))] train_features, train_targets = train_df[list(range(1, 91))].values, train_df[0].values valid_features, valid_targets = valid_df[list(range(1, 91))].values, valid_df[0].values test_features, test_targets = test_df[list(range(1, 91))].values, test_df[0].values from sklearn.preprocessing import QuantileTransformer quantile_transformer = QuantileTransformer(output_distribution='normal') train_features = quantile_transformer.fit_transform(train_features) valid_features = quantile_transformer.transform(valid_features) test_features = quantile_transformer.transform(test_features) tr_dataloader = DataLoader(TensorDataset(torch.tensor(train_features), torch.tensor(train_targets)), batch_size=opt.batch_size, shuffle=True, drop_last=False) valid_dataloader = DataLoader(TensorDataset(torch.tensor(valid_features), torch.tensor(valid_targets)), batch_size=opt.batch_size, shuffle=False, drop_last=False) test_dataloader = DataLoader(TensorDataset(torch.tensor(test_features), torch.tensor(test_targets)), batch_size=opt.batch_size, shuffle=False, drop_last=False) ### build model ### model = build_model( [-1] * 90, -1, num_cond_per_column_scale=opt.num_cond_per_column_scale, num_cond_per_subtree=opt.num_cond_per_subtree, num_subtree_per_condset=opt.num_subtree_per_condset, num_subtree_per_estimator=opt.num_subtree_per_estimator, dropout=opt.dropout, shuffle_condition=not opt.no_shuffle_condition, condition_shuffle_type=opt.condition_shuffle_type, device=device ) ``` Can you please provide me with the working code of DOFEN that can get 77.4012 on Year? Thank you.
To ensure that all data preprocessing settings applied in NODE are also implemented in DOFEN, and to guarantee that we use the exact same train/validation/test split as NODE, we decided to utilize the data preprocessing code provided by NODE on their official GitHub. Instructions for using this preprocessing code can be found in their notebook examples. After comparing NODE's preprocessing code with the code you provided, we identified a few differences that might influence the model's final performance: 1. It appears that a standard scaler normalization on the target is missing from your code. If standard scaler normalization is applied before training, an inverse transformation is needed for the normalized predictions and targets before calculating the mean squared error. This can be seen in NODE's notebook example at `/notebooks/year_node_8layers.ipynb`. 2. The Quantile Normalization used by NODE has quantile noise set to 1e-3. This is also configured in their notebook example at `/notebooks/year_node_8layers.ipynb`. 3. It seems NODE only tested the last 51,630 samples when using the YEAR dataset. This can be found in line 246 of the file `lib/data.py` in NODE's repository. We are currently packaging our code and will provide it to you through the AC after an anonymized check. In the meantime, you may want to consider modifying the differences mentioned above or using NODE's preprocessing code directly to ensure there are no implementation discrepancies, and see if this improves your results. Thank you for your attention to these details.
Update: To better understand the impact of standard scaler normalization on DOFEN's performance, we reran the experiment on the Year dataset without applying normalization to the target. The resulting metrics are as follows: * Loss: 129.10622376662033 * RMSE: 10.95666790008545 * MSE: 120.0485610961914 * R²: -0.019392279646307298 These results are significantly worse than those obtained when applying normalization and closely resemble the results reported by the reviewers. This suggests that standard scaler normalization is crucial on regression tasks and can greatly influence the model’s performance. Thank you for your attention to this matter.
Dear Reviewer, The code to reproduce DOFEN’s results on the Year dataset is accessible through the following Google Drive link: https://drive.google.com/file/d/1QW2vEdTLDDkIMVcapS587gpgPuk9078H/view?usp=sharing This link includes a zip file containing the NODE repository, the DOFEN model, and a notebook for DOFEN training and evaluation. Both the link and the code have been anonymized to prevent any identity leaks. We hope the provided code will help in reproducing our results. Thank you for your attention and understanding.
Dear authors, Thanks for providing the code. I can confirm that I am able to reproduce your results now. Do you have any idea why target transformation plays an important role in training models such as DOFEN and NODE? Is it due to the nature of tree-inspired architectures? For traditional NN-based models, normally standardizing targets would not result in too much performance difference.
Dear reviewer, Target transformation (e.g. standardization, min-max scaling, log-transformation, etc.) in the label space is a widely used preprocessing technique that has shown effectiveness in training deep neural networks [1-5]. This approach is not limited to tree-inspired architectures. Its importance in regression tasks lies in its ability to reduce the influence of outliers and smooth gradients during training, which can enhance model performance. We hope this clarifies the reasoning and appreciate your consideration of this aspect. --- [1] Bishop, Christopher M., and Nasser M. Nasrabadi. Pattern recognition and machine learning. Vol. 4. No. 4. New York: springer, 2006. [2] LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." nature 521.7553 (2015): 436-444. [3] James, Gareth, et al. An introduction to statistical learning. Vol. 112. New York: springer, 2013. [4] Kuhn, Max, and Kjell Johnson. Applied predictive modeling. Vol. 26. New York: Springer, 2013. [5] Kuhn, Max, and Kjell Johnson. Feature engineering and selection: A practical approach for predictive models. Chapman and Hall/CRC, 2019.
Summary
This paper primarily investigates DNN models for tabular data. Current DNN models have limited performance and do not outperform GBDTs. The authors believe the key lies in the need for sparse feature extraction from tabular data. Existing attention-based models lead to dense selection of columns, while tree-based DNNs can only achieve near-sparsity. The paper introduces a novel deep neural network architecture called DOFEN (Deep Oblivious Forest ENsemble), aiming at enhancing performance on tabular data. Inspired by Oblivious Decision Trees (ODT), DOFEN implements a new two-step process to achieve on-off sparse selection of columns, thereby gaining advantages in feature diversity and mitigating overfitting. The paper evaluates DOFEN on the recognized Tabular Benchmark, demonstrating its excellent performance across various tasks, especially in competitiveness when compared with Gradient Boosting Decision Trees (GBDTs).
Strengths
1. Innovation: DOFEN integrates the concept of ODT with deep learning techniques, proposing a new neural network architecture, which is an innovation in the processing of tabular data. 2. Performance: Evaluation results on the Tabular Benchmark show that DOFEN has achieved state-of-the-art performance levels in multiple tasks, demonstrating competitiveness compared to GBDTs. 3. Robustness and Universality: DOFEN has shown promising performance across various types of datasets, whether they feature numerical attributes or a combination of numerical and categorical features.
Weaknesses
1. Implementation Complexity: The architecture of DOFEN may be more complex than traditional tree models and some DNN models, which could increase the difficulty of implementation and understanding. 2. Long Inference Time: Although DOFEN performs well in some aspects, its inference time is relatively long, which may affect its use in applications that require real-time response. 3. Slow Convergence Speed: The randomized steps in DOFEN may lead to a slower training convergence speed, requiring more training steps to achieve optimal performance.
Questions
1. The paper mentions that DOFEN employs randomized steps to enhance the model's generalization ability, but what is the specific mechanism by which these steps affect the training dynamics and ultimate performance of the model? 2. The performance of DOFEN largely depends on the selection of hyperparameters. Is there an automated strategy for hyperparameter adjustment? 3. While DOFEN shows excellent performance in experiments, how does it perform in real-world application scenarios, especially when dealing with large-scale datasets or in resource-constrained environments? 4. Compared to traditional tree models, how interpretable is the decision-making process of DOFEN as a deep learning model, and are there ways to improve the model's interpretability?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
see the section of weakness.
I thank the authors for their answers. Some comments: - Weaknesses 1: Maybe adding a sentence like this for $\Delta_2$ and $\Delta_3$ to the main paper would help. - Weaknesses 2: I understand that providing training times averaged over the whole benchmark is difficult. However, I think that even some "cheap" way to get some idea of training times could be instructive, for example, running one random HPO configuration of all methods on a single train-test split of all or a few datasets (make sure to cover different #features), perhaps subsampled to a manageable size (e.g., the small 15k version of the original benchmark). Of course, this could still be some effort if you don't have all methods in the same codebase. - Questions 1-1: If it's equivalent then it might simplify the exposition to explain it as sampling with replacement, but I leave this to the authors to decide. - Questions 2: Interesting, it seems like you're giving your own method a disadvantage, but if it performs so well anyway then I can't complain about that. Thank you for the responses to the comments. I can't see a global rebuttal right now. I think you could use the comment function even though I'm not officially obligated to read it, but it might not be necessary since I already gave you a score of 7.
Update: I can see the global rebuttal now.
Thanks for the feedback from the reviewer. Regarding the suggestions mentioned in `Weaknesses 1` and `Questions 1-1`, we will definitely add additional explanations and simplify the exposition in future revisions to make the content easier for readers to follow. For `Weaknesses 2`, we conducted an experiment to compare the training time of DOFEN with other deep learning methods included in our paper (Trompt, FT Transformer, and NODE). We measured the training time on medium-sized datasets using both default and optimal hyperparameter settings, where the optimal hyperparameters refers to the settings that deliver the best performance for each model on each dataset. This experiment was conducted using a single NVIDIA Tesla V100 GPU. During model training, we carefully ensured that no other computational processes were running concurrently to enable a fair comparison. Additionally, we excluded datasets that would cause OOM (Out of Memory) issues during training, resulting in the selection of 50 out of 59 medium-sized datasets. The average training time across datasets for each model is provided in Table A7. The results show that the training time for DOFEN is relatively long, approximately twice as long as Trompt when using optimal hyperparameters. This extended training time may be due to the inefficient group operations involved in DOFEN, which consume about 80% of the computation time during the forward pass. For more details, please refer to the “Long Inference Time of DOFEN” section in the global rebuttal. Therefore, improving the efficiency of group operations could reduce both the training and inference time of DOFEN. --- Table A7: Average training time of different methods using default and optimal hyperparameter settings on 50 medium-sized datasets. Numbers are in Seconds, with lower values indicating faster training speed. | | Training time (Default) | Training time (Optimal) | | -------------- | ----------------------- | ----------------------- | | DOFEN | 332.6998 +/- 125.1965 | 1143.7674 +/- 804.3809 | | Trompt | 552.3495 +/- 213.3278 | 535.1781 +/- 291.9933 | | FT Transformer | 80.3425 +/- 57.2647 | 99.1068 +/- 79.2272 | | NODE | 95.0274 +/- 54.7463 | 427.8625 +/- 394.1191 |
Thank you for the training time results! While the times for DOFEN are relatively long, I think these results are an interesting addition to the paper.
I would like to thank the authors for the rebuttal. I agree that it is difficult to ablate individual components of DOFEN, since most components seem to rely on each other to work properly. I ran the code in the supplementary material, and compared DOFEN against NODE on a number of regression tasks. I found DOFEN is consistently outperformed by NODE, and I wonder if it is due to an inappropriate setting of hyperparameters. Can the authors specify what is the hyperparameter search space you have used for DOFEN? For train_num_estimator and test_num_estimator, do you set them to the same number?
The train_num_estimator and test_num_estimator both refer to $N_{forest}$ in our paper, and we set both of them to 100, as defined in the default hyperparameter space in Table 2. For hyperparameter search, we adjust $d$, $m$, and $drop\\_rate$ for DOFEN, and the corresponding search space can be found in Table 39. Regarding the variable naming in the code, the num_cond_per_subtree refers to $d$, the num_cond_per_column_scale refers to $m$, and the dropout refers to $drop\\_rate$. Could you provide the regression tasks you ran? We will experiment on them and get back to you with the results.
Thanks for your reply. I will keep the positive score.
Dear Reviewer, Please reply to the rebuttal. AC.
Please check to see and discuss with the author if your issues are addressed.
Dear Reviewer PzWm, Since the deadline of discussion period is approaching and now we have mixed ratings for this paper, please discuss with the authors at your earliest convenience to see if your concerns are addressed. Your timely review is important to make the whole process smooth. Thank you very much. Thanks, AC.
The deadline of discussion period is approaching
Dear Reviewers, As the deadline of the discussion period is approaching and we have mixed ratings currently, please check the review of other reviewers if you have not done it yet and discuss the rebuttal with the authors to see if your concerns are well-addressed. Thank you very much. Thanks, AC
Decision
Accept (poster)