Summary
This paper investigates the one-epoch overfitting (OEO) phenomenon in CTR prediction models and empirically identifies the overfitting of embedding parameters as the primary cause. Based on this insight, the authors propose a method of randomly reinitializing embedding parameters at the beginning of each epoch and continually learning the parameters of MLPs. This proposed method is simple yet effective, enabling multi-epoch learning.
Strengths
The one-epoch overfitting (OEO) phenomenon is a common issue often encountered in industry and has been reported in previous work. This paper specifically investigates this issue and proposes a simple workaround to enable multi-epoch learning, thereby improving model performance. In summary, the paper has the following strengths:
+ The paper reproduces the OEO phenomenon in open benchmark datasets, validating its existence.
+ Extensive experiments and analyses are conducted to identify the main cause of OEO, which is the overfitting of embedding parameters.
+ The paper presents a simple solution to avoid OEO by randomly reinitializing embedding parameters at the beginning of each epoch, a method tailored for both incremental and non-incremental learning scenarios.
+ Both offline experiments and online A/B tests have been conducted to validate the effectiveness of the proposed method.
Weaknesses
I think the paper has the following major limitations that need further improvement.
1. The paper claims, 'We provide theoretical analyses of the reason,' but I did not find any theoretical analysis throughout the paper. The authors are suggested to give a theoretical analysis before the approach.
2. The authors primarily experiment with the Amazon dataset and presume the existence of one-epoch overfitting (OEO). However, according to the results reported in the benchmark paper 'Open Benchmarking for Click-Through Rate Prediction,' OEO does not always occur (some datasets exhibit OEO, while others do not; some models experience OEO, while others do not). Given these variations, the paper lacks a rigorous analysis of the conditions under which OEO occurs. Comparing datasets where OEO occurs with those where it does not could better help identify the causes, for example, by highlighting differences between the datasets.
3. While the proposed method to avoid OEO is simple, it introduces multiple groups of embedding parameters in the incremental MEDA setting. Given that the size of embedding parameters is substantial, this approach will require multiple times the memory resources. I am unsure whether the return on investment (ROI) is sufficiently high to justify this increased resource usage. The authors are suggested to include some metrics on memory usage to indicate the practicality of the method.
4. The method is simple and effective. But the authors lack a discussion whether some alternatives are considered, for example whether using a smaller learning rate for embeddings and a larger learning rate for MLPs could mitigate the OEO issue.
Questions
1. The term SEL is not defined upon its first occurrence.
2. In the related work, the authors mention graph learning and state that "graph learning research has delved into fine-tuning pre-trained models for new graphs." However, there is no clear connection drawn between graph learning and CTR prediction research, particularly for this paper. The authors are suggested to explicitly state the relevance of graph learning to the work on CTR prediction, or just remove this section if it's not directly related.
3. The paper states, "Figure 3 shows that the embedding overfits on each trained data sample," but the plot does not track individual data samples. It is unclear how the authors determine that overfitting occurs on each sample. For example, the training loss may drop and then slightly increase without MEDA, and the testing loss may also fluctuate. The authors can clarify how they drew this conclusion from the data presented in Figure 3.
4. The paper claims, "the initial embedding of the second epoch precisely memorizes the information of any data sample in the first epoch." However, there is no empirical evidence provided to support this statement. Given that models typically experience information loss during training, this claim seems unlikely. The authors are suggested to provide empirical evidence supporting this claim, or to clarify if this is a hypothesis rather than a proven fact.
5. The sentence "Because, first, Figure 2(a)..." contains a syntax error.
6. On line 269, the figure reference may be incorrect. The text states, "Furthermore, our results in Figure 4(a)..." but the figure shows the cosine similarity between MLPs, not the convergence of MLPs.
7. On line 311, the details about incremental MEDA are unclear. Specifically, the algorithm "selected based on requirements such as computation/storage costs" is not described. How are these selections made according to computation and storage costs? Additionally, Figure 1(b) mentions two operations, "copy" and "select," but the "copy" operation is not described in Algorithm 2.
8. In the A/B testing section, the authors indicate that they use incremental MEDA. However, the reported numbers (test AUC increase, duration of A/B test, training time) are identical to those reported in the first Arxiv version, which only discussed non-incremental MEDA in that experiment. It is unclear whether two separate A/B tests were conducted or if the same results are being reused.