Towards Hybrid-grained Feature Interaction Selection for Deep Sparse Network

Summary

The authors propose a novel feature selection algorithm, aiming to detect interactions between features at instance-level, contrary to the usual feature selection algorithms that selects the same features for every sample. Initially, the authors propose a highly memory-demanding approach, requiring an $m \times m$ matrix, being m the number of different values the features can take. Later, a less memory-demanding approach is presented, using matrix decomposition. The experimental results are slightly better to the state-of-the-art.

Strengths

- **Quality:** Several DSN methods were included in the state-of-the-art section. The proposed algorithm is able to obtain very similar results. - **Clarity:** The paper is easy to follow and to understand. The decisions made are clearly motivated. - **Significance:** The idea of selecting, per each sample, the most important interactions between features is very interesting and it can provide a good explanation about the decision making.

Weaknesses

- **Originality:** The algorithm is a combination of well-known techniques. The innovative part is focused on how to merge all of them. - **Quality:** There exists other field of methods that also address feature interaction: the so-called *dynamic feature selection'. Techniques like L2X [1] are focused on the same goal, without the need of using DSNs, which highly reduces the memory consumption. Some information regarding these techniques should be included in the paper. - **Clarity:** Fig. 3 is clearly misleading. Although it constantly suggest the proposed method outperforms the state-of-the-art, the granularity of the y-axis is almost non-existent. There are very little differences amongst all algorithms. - **Significance:** I have concerns regarding two critical aspects of the experimental results: 1. The experimental results show very little improvements against the baseline methods. An statistical analysis is mandatory, in order to establish whether the obtained results provide a real improvement over the state-of-the-art or not. 2. Although I agree with the authors that feature interaction selection can provide insightful information regarding the decisions provided by the network, the authors do not mention anything related to this in the experimental section. [1] Chen, J., Song, L., Wainwright, M., & Jordan, M. (2018, July). Learning to explain: An information-theoretic perspective on model interpretation. In International Conference on Machine Learning (pp. 883-892). PMLR.

Questions

See the weaknesses sec

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Not applied.

Summary

This work proposes a hybrid-grained feature interaction selection approach for deep sparse networks, which targets both feature field and feature value. The proposed approach uses a decomposed space that is calculated on the fly to explore the expansive space of feature interactions. The work also introduces a selection algorithm called OptFeature, which efficiently selects the feature interaction from both the feature field and the feature value simultaneously. The proposed approach is evaluated on three large real-world benchmark datasets, and the results demonstrate that the proposed approach performs well in terms of accuracy and efficiency. The work concludes that the proposed approach can effectively select feature interactions in deep sparse networks, and it has the potential to improve the performance of prediction tasks with high-dimensional sparse features.

Strengths

1. The hybrid-grained feature interaction selection approach goes beyond traditional field-level selection, and the decomposed space and sparsification-based selection algorithm make the work appear to be a cutting-edge method to some extent. 2. This work ran the repetitive experiments with different random seeds five times and reported the average value for each result, and provides information about the parameter setup, metrics, datasets, baseline and parameter setup, so the experimental results appear to be reliable.

Weaknesses

1. Novelty: The proposed approach that targets both feature field and feature value levels and introduced a decomposed space and a sparsification-based selection algorithm to explore the selection space, which appears to be a novel contribution to the field. But it does not provide a comprehensive review of related Feature Interaction Selection work in Section 2, and the novelty is not so obvious. 2. Experiments: What GPU was used in this work? How many were used? Were all the experiments conducted on the same GPU? Why formulate the hybrid-grained feature interaction selection as a binary selection according to Equation 6? Taking either 0 or 1 doesn’t seem to reflect ‘hybrid’. 3. Writing: the introduction does not summarize the main contributions of this work, so readers cannot intuitively get the advantages of this work. In addition, the content of Section 2.1 introducing Neural Architecture Search seems to be not very relevant to this paper. Furthermore, Section 3.3.2 does not explain how to determine the parameter α in Equation 6, which makes one wonder how to choose between value-grained and field-grained.

Questions

1. Experiments: What GPU was used in this work? How many were used? Were all the experiments conducted on the same GPU? Why formulate the hybrid-grained feature interaction selection as a binary selection according to Equation 6? Taking either 0 or 1 doesn’t seem to reflect ‘hybrid’. 2. Writing: the introduction does not summarize the main contributions of this work, so readers cannot intuitively get the advantages of this work. In addition, the content of Section 2.1 introducing Neural Architecture Search seems to be not very relevant to this paper. Furthermore, Section 3.3.2 does not explain how to determine the parameter α in Equation 6, which makes one wonder how to choose between value-grained and field-grained.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

See the weaknesses.

Summary

This paper tackles the problem of modeling fine-grained feature interactions in high-dimensional sparse features. A hybrid-grained feature interaction selection method is proposed, which operates on both field and value for deep sparse networks. To handle the increase in computation, a decomposed form of the selection space is done, which greatly reduces the computational requirements of modeling. Results on deep sparse networks benchmarks show that the proposed method achieves SOTA results while being more computationally efficient.

Strengths

- Strong results in terms of performance on established benchmarks and computational efficiency, demonstrating the effectiveness of the proposed method. - The proposed method seems generalizable and can be applied to other methods. - All experimental parameters are provided, making reproduction straightforward. - The writing is fairly clear and easy to understand.

Weaknesses

- Experimental results: - The proposed method is a simple tensor decomposition for improved efficiency and the additional consideration of more features. Such choices are generalizable to other architectures (as mentioned in lines 109-112) but this is not demonstrated in the paper. I would like to see the application of the proposed components to other existing approaches. - Significance of results: - The AUC and Logloss scores differ by less than 0.001 between the proposed method and the previous SOTA. Is this significant? I suggest the authors add confidence intervals to Table 1 and 2 for easier comparison.

Questions

See weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The limitations are discussed at the end of the paper.

Summary

This paper introduces a hybrid-grained feature interaction selection approach that targets both feature field and feature value for deep sparse networks and decomposes the selection space using tensor factorization and calculating the corresponding parameters on the fly.

Strengths

Extending the selection granularity of feature interactions from the field to the value level. Introduce a hybrid-grained feature interaction selection space, which explicitly considers the relation between field-level and value-level. The tensor decomposition and the sparsification are combined to perform selection on the shrinking space.

Weaknesses

1.The evaluation datasets are pretty small (the feature number is around 11-26). For recommendation systems optimization work, it is usually better to show the results in large-scale datasets like industrial datasets to demonstrate the scalability and performance. 2.Missing several references: AutoFAS， NAS-CTR， AutoIAS, GAIN: A Gated Adaptive Feature Interaction Network for Click-Through Rate Prediction Maybe adding some discussions or comparisons to them is better.

Questions

See weakness

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes, the authors discussed some limitations.

Thanks for providing the results on large-scale industrial dataset.

Thanks for providing the results on the large-scale industrial dataset. I think the new comparisons addressed my concerns. I increase my score from borderline accept to weak accept.

Follow-up by Reviewer

Thank you for providing detailed responses to my concerns. I have read through all the other reviews and responses. I will maintain my rating of "Weak Accept".

Answer to authors

I would like to thank the authors for their responses. Regarding to the Clarity in Fig. 3 issue, the problem is that the differences between the models seems higher than reality. For instance, in the first figure, the y-axis moves between 0.8090 and 0.8115. Visually, a value in the lower bound of the axis should mean a bad performance, and not just a minor decrease in the performance. The margin between the top and the bottom should be higher than the one used in the figures.