TabMT: Generating tabular data with masked transformers

Summary

This paper introduces TabMT, a new Masked Transformer architecture designed for generating synthetic tabular data. While Transformers are predominantly used in natural language processing (NLP), TabMT demonstrates their effectiveness in dealing with heterogeneous data fields, such as images and tables, and efficiently manages missing data. The authors propose to sample the masking probability from a uniform distribution and predict masked values in random order during generation. By employing advanced masking techniques, TabMT can generate synthetic data with high performance across a wide range of dataset sizes. Moreover, the model proves valuable in privacy-sensitive applications, as it is capable of producing high-quality data while adhering to privacy restrictions.

Strengths

The motivation is easy to understand and the problem is important but less explored than common domains. It is not straightforward to apply existing techniques on images and texts to the tabular data. The experimental designs are comprehensive, including several sections assessing data quality, Privacy and Sample Novelty, Missing Data, and Scaling, which are solid.

Weaknesses

The motivations for designs are not very clearly written. Please consider reorganizing the structure, adding highlighted paragraphs, and moving line 129-138 to the beginning of the method or merging them with the methodology introduction texts. The introduction of temperature scaling is not well-motivated, is it for privacy purposes and why will it work? The introduction of privacy tabular data generation is not detailed. The conclusion claims that "the model is able to function under arbitrary privacy budgets." but I even didn't see how the budgets are computed and its defitions, motivations etc. Same for the definition of data "novelty". And the questions of "why many designs in TabMT can help those aspects are not well explained".

Questions

The work can benefit from discussing and comparing the work of causal inference using transformers, which is a particular kind of missing data generation problem where privacy protection is also pressing, e.g. [1], [2]. [1] Exploring Transformer Backbones for Heterogeneous Treatment Effect Estimation [2] Differentially Private Synthetic Control

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

See the weaknesses part

Summary

The paper proposes a masked transformer model that can be used to generate tabular data. The authors propose modifications in the masking strategy in the transformer, in order to make it more effective for generating data. Empirically, the proposed generator improves performance in various datasets compared to other baselines. Moreover, the paper presents cases of privacy-focused applications, presence of missing values, and large datasets to further depict the usability of the proposed method in real-world situations.

Strengths

- The paper is well-written and very clear to its points. The authors define the problem with the masked transformer for generating tabular data, and presents an effective method to overcome the difficulty. The empirical results are well presented, and seems promising.

Weaknesses

As noted in the paper, the major weakness would be the resource it takes to train the model. For general users, a pre-trained model with learning across multiple tables might be beneficial.

Questions

- Is it possible for the model to generate new categories (or numbers) not present in the train set?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The presented work might not be accessible to regular users, given the amount of resource required to train the model.

Summary

This paper proposes a new generative model of table-type data based on the transformer. And it can address the unique challenges posed by heterologous data fields and natively handle missing data.

Strengths

1. TabMT is a simple but effective Masked Transformer design for generating tabular data. 2. We highlight the applicability of our model in privacy-focused applications, illustrating TabMTs ability to arbitrarily trade-off privacy and quality through temperature scaling. 3. Experiments show the effectiveness of the proposed method.

Weaknesses

1. The TabMT structure has only been modified in terms of the input layer, without improving the multi-layer transformer structure. In other words, this paper only made adaptive improvements to the data format and is not on par with NeurIPS. 2. The application method of the model structure proposed by TabMT is not clear. Categorical and Numerical use independent inputs, which makes it necessary to determine the type of data input into the model. How to accurately and automatically identify this type is challenging, which also limits the complexity of generating table data for the model, making it difficult to cope with complex scenarios. The effectiveness of the model should be evaluated for inaccurate category recognition in the experiment.

Questions

See weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This paper explores the effectiveness of masked transformers as generative models for synthetic tabular data generation. The proposed TabMT architecture effectively handles challenges related to heterogeneous data fields and missing data. The model shows promising experimental performance and demonstrates good performance even under privacy constraints.

Strengths

- The paper proposes a promising masked transformer approach TabMT for generating tabular data. This is an elegant application of masked transformers. - The paper provides comprehensive experimental evaluation and compares to a diverse set of baselines on 15 datasets - The paper explores privacy-preserving capabilities of TabMT and shows promising results

Weaknesses

- While the paper provides adequate experimental evaluation, it would be helpful to include recent state-of-the-art generative methods in evaluation such as STaSy [1] - In the privacy experiments, only TabDDPM was used as the baseline, it would be interesting to see more comparisons with other baselines [1] Kim, J., Lee, C. and Park, N., 2022. Stasy: Score-based tabular data synthesis. arXiv preprint arXiv:2210.04018.

Questions

1. Could you please clarify the training procedure? When training the masked transformer, does every feature have its own classification head for predicting the mask? 2. Since you explore hyperparameter tuning for the downstream Catboost model, how was hyperparameter tuning performed for each of the generative models? 3. In the privacy experiments, only TabDDPM was used as the baseline, would it be possible to add more comparisons with other baselines? 4. It would be great to add recent state-of-the-art generative models into evaluation, such as STaSy [1] [1] Kim, J., Lee, C. and Park, N., 2022. Stasy: Score-based tabular data synthesis. arXiv preprint arXiv:2210.04018.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Addressed

Mixed reviews

Dear reviewers, Thanks for the hard work so far! This paper received mixed reviews and the authors have responded each of you. We need to ideally reach a consensus in the rebuttal period, and at least should have active discussions, updated recommendations, and acknowledgment that you have read the response. Can you please check other reviews and the author rebuttal and see if your opinion has changed? Please give your reasoning in as much detail as possible. AC

Response to the author rebuttal

I thank the authors for their response and provided clarifications. Regarding hyperparameter tuning, my question was about the hyperpameter tuning of the generative models rather than catboost. How was hyperparameter selection performed for both TabDDPM and the baseline generative models it was compared to? Regarding implementing StaSy, in fact it’s official GitHub provides a user-friendly implementation: https://github.com/JayoungKim408/STaSy making experimentation on at least a few datasets feasible within the rebuttal period. As StaSy is a recent and strong generative tabular model, it is important to include it in comparison with TabDDPM.