HYTREL: Hypergraph-enhanced Tabular Data Representation Learning

Summary

This paper proposes to transform a source table into a hypergraph. Each cell is a node in the graph, where nodes in the same row, column, and table are connected using three types of hyperedges. The authors claimed that by modeling the hypergraph of the table, the proposed method, named *HyTrel*, is able to generate representations of tables robust to permutations. Two self-supervised objectives: (1) cell or header corruption indication prediction and (2) in-batch negative sampling with InfoNCE contrastive loss for paired nodes. The experiments were performed on (1) pre-training on 27M tables and (2) fine-tuning for four tabular understanding tasks: column type annotation, column property annotation, table type detection, and table similarity prediction. The empirical findings include - Pre-training on HyTrel leads to marginal improvement over the variant without pre-training. - No clear evidence of which pre-training objectives are better, considering the four tasks evaluated.

Strengths

- It is reasonable to seek permutation invariance in tabular data representation as the existing table serialization approach fails to achieve this property. - The proposed method surpasses previous methods by utilizing directly supervised training on the target data without pre-training.

Weaknesses

**Complexity** The main concern lies in the efficiency of the proposed method. It needs to build a huge hypergraph on the tables, where the number of nodes is equal to the number of cells in the table, and the number of edges explodes when we combine nodes, columns, and rows. As we know, it is very common that a table may contain millions of rows and thousands of columns. However, there seems to be lacking complexity analysis of the overall running time and memory cost. Although HyTrel was observed converging faster than previous BERT-based methods in terms of the number of epochs, there is no direct comparison of the inference time needed for each sample/batch. Also, the time required for building the graph in the training and the prediction phase should be taken into account.

Questions

Please refer to the weaknesses section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to the weaknesses section.

Summary

The paper proposes a framework for tabular representation learning by modeling the structure of the tables as a hyper-graph working on different granularities, namely, rows, columns, and the entire table.

Strengths

- The paper addresses a substantial question of how to best incorporate the structure of a table while pre-training models for tabular data. - The proposed framework outperforms the baselines on tasks purely dependent on the learned table representations. - The paper is well-written and easy to follow. - In addition, the authors also include quantitative analysis to demonstrate that their framework can assimilate the table structures to generate robust representations.

Weaknesses

There have been recent works proposing techniques to incorporate the table structure while pre-training or fine-tuning (by biasing the attention layer or tasks other than the ones considered in the paper). It will strengthen the paper if the authors discuss how their work differs from those. - MATE: Multi-view Attention for Table Transformer Efficiency Julian Martin Eisenschlos, Maharshi Gor, Thomas Müller, William W. Cohen EMNLP 2021 - Learning Enhanced Representations for Tabular Data via Neighborhood Propagation Kounianhua Du, Weinan Zhang, Ruiwen Zhou, Yangkun Wang, Xilong Zhao, Jiarui Jin, Quan Gan, Zheng Zhang, David Wipf NeurIPS 2022 - Learning Representations without Compositional Assumptions Tennison Liu, Jeroen Berrevoets, Zhaozhi Qian, Mihaela van der Schaar ICML 2023

Questions

- Does the model also include a classification head like other TaLMs? If not, why did the authors not consider including it? This simple change would have opened up evaluation for some more benchmarks. - What's the rationale behind the modified ELECTRA and contrastive objective for pre-training (as opposed to other objectives)? - How would the approach perform if tables have a lot of numerical values? Will numeric tables lead to a degradation in performance?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors include a section in the paper that goes through the relevant limitations of the proposed approach.

Summary

This work presents a novel tabular language model that represents tables as hypergraphs called HyTrel. HyTrel is designed to capture the structural properties of tabular data, including 1) invariance to row/column permutations, 2) structural similarity within columns, 3) high-order multilateral relations, and 4) hierarchical organization. However, many of the Language models pre-trained on tabular data do not consider these structural properties that exist in tabular data. Experimental results are provided to demonstrate the superiority of the HyTrel on four downstream tasks, and the robustness of the representations from HyTrel is further validated through qualitative analysis.

Strengths

1. Unlike previous methods, the HyTrel model can handle changes in the order of rows and columns without significantly affecting its performance. 2. Also, HyTrel can handle input tables of arbitrary size, making it versatile and adaptable to a wide range of datasets. 3. The model is highly efficient regarding the number of epochs for pretraining compared to prior works. 4. HyTrel can incorporate the hierarchical table structure into its representations. 5. By incorporating these structural properties, HyTrel consistently achieves superior performance over the competing baselines.

Weaknesses

No detailed comparison of computational efficiency or runtime on the number of epochs needed for pretraining compared to prior works.

Questions

Compared to the previous method, how would HyTrel representation of Tabular Data benefit Multilingual Tabular data alignment?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

HyTrel is designed for tables with simple column structures and struggles with tables that have complex hierarchical column structures.

Summary

This paper proposes a tabular language model (HyTrel) that utilizes the hypergraphs of the data table. Specifically, the hypergraph is constructed with cell values representing each node and the row, column, table representing the hyeredges. In the proposed framwork, the nodes and edges are first fed into the embedding layer, followed by 12 layers of HyperTrans structure, which is consisted of two HyperAtt blocks connected by a hyperedge fusion block. The model is pretrained through corruption of certain parts of the table/hypergraph. To use this model, one can first compute the table encoder of a table, which can then be used to finetune downstream tasks. This paper reports competitive performance of HyTrel in 4 downstream tasks (CTA, CPA, TTD, TSP).

Strengths

The HyTrel architecture produces a table encoder that is permutation invariant with both theoretical supports and empirical experiments. It analyzes different functions in the HyperTrans layer and shows the output encoding will not be affected by row/column permutation of the table. Tasks finetuned with such encoder outperform other baseline models. There are also some qualitative analysis of the hierarchical structure of the table being learned through the proposed framework.

Weaknesses

The method only applies to a single table scenario. In real-world applications, this could be a rare case where tabular learning is involved. Another dimension that can be added to the hypergraph can be key/foreign key relationships, which may extend the model to work in multi-table settings.

Questions

What are some examples of realistic permutation of the table other than simple row/column permutation? If there are other non-trivial permutation types that can benefit from the invariance brought by HyTrel, I think it adds additional value and persuasion to the adopt this method from a practical point of view. Or, in the case where row/column permutation is the main target, it would be helpful to show how it negative affects downstream tasks, motivating the development of an permutation invariant encoding representation.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See the weakness and questions section above.

Summary

In the paper, the authors aim to capture the structural properties of tabular data using hyper-graphs with four different types of hyperedges based on the co-occurrences in the table. The experimental results on four downstream tasks show the advantages of the proposed method on other competitive baselines.

Strengths

The manuscript is well-structured and easy to follow. The overall idea of modeling high-order cell/rows/columns interactions using hyper-graph is novel. 1. The proposed hyper-graph encoder captures tabular structures via attention learning. 2. The experimental results, which extensive ablation studies, convincingly demonstrate the advantages of the proposed methods. 3. The visualization in Sec. 4.2 shows that hierarchical representation can be learned by hyper-graph.

Weaknesses

Minors 1. Since hyper-graph excels majorly in handling hyper-edges of varied orders. It would be sub-optimal for hyper-graph to handle the hype-edges of fixed order in the tabular data? 2. It’s expected that some baseline hyper-graph models are included in the compassion results such as [1,2]. 3. It’s suggested to elaborate on the hierarchical structures of tabular data in the Introduction otherwise a little bit confusing. Further, it's still unclear the relationship between hierarchical structures and hypergraphs. [1] Hypergraph Neural Networks, 2018 [2] Hypergraph convolution and hypergraph attention, 2020

Questions

NON

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

NON