Decomposable Transformer Point Processes

Summary

The work designs a novel transformer-based approach for modelling time series (e.g. predicting next event). The main novelty is the decomposition of the log-likelihood into a conditional probability mass and density functions. The former, implemented with a transformer, models the distribution over the event types; the latter models the event occurance with a log-normal mixture. The experimental results are compelling: the approach achieves convincing performance on next-even prediction (in terms of log-likelihood) and long-horizon prediction.

Strengths

* The writing and presentation are clear and precise. The technical introduction and the contribution have solid foundations. * The experiments are convincing. I appreciate reporting the variance for transparency. * I also appreciate the details provided here and there without distracting from the main story (e.g. the hyperparameter values in the supp. material).

Weaknesses

I do not see any major problems, but would encourage minor revisions towards reaching broader audience. Specifically: * The thinning algorithm plays an important role in motivating the approach and interpreting the results. The readers will appreciate a high-level recap of the algorithm, intead of having to look it up in the reference. * The justification for the decomposition (ll. 90-93) comes across a bit weak. Perhaps it could be improved by providing an analytical argument why it should offer the same benefits as the intensity function, and why depending on the thinning algorithm at inference time is a problem.

Questions

* How does the size of the mixture model (M) affect the prediction and what was the methodology for choosing the optimal M? * The scale in Fig. 1 is wildely different. How does one explain the differences and what does it say about the data quality and/or task complexity? Minor remarks: l. 80 The integral’s upper bound coincides with the integrand variable. l. 150-155: Perhaps one could provide a comparison with previous work, what does this simplification of the problem translates to in practice (how much faster it is to train/optimize).

Rating

6

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

The limitations are discussed sufficiently in Sec. 6.

Summary

This paper presents a novel framework for modeling marked temporal point processes (MTPPs) using Transformer-based architectures. The authors address the limitations of traditional methods that rely on computationally intensive thinning algorithms by proposing a decomposable approach that partly uses a Transformer. This approach models the conditional intensity function (CIF) using a Transformer architecture while separating the modeling of inter-event times and event types into two distinct components: the conditional probability density function (CPDF) for inter-event times and the conditional probability mass function (CPMF) for event types.

Strengths

- Novel framework that uses a Transformer architecture for the first time to tackle this problem - Decomposing the likelihood into conditional probability density function (CPDF) and conditional probability mass function (CPMF) components - The proposed DTPP model is technically solid, with a well-structured approach to decomposing the MTPP likelihood and modeling the components using Transformers. The mathematical formulations are clearly presented. - The authors provide detailed implementation details and make their code available. This ensures that the results can be reproduced and verified by other researchers, which is important for the credibility and quality of the work. - The demonstrated improvements in predictive accuracy and computational efficiency over state-of-the-art methods highlight the significance of the proposed framework. The speedup achieved in long-horizon prediction tasks is particularly noteworthy.

Weaknesses

- The primary contribution of the paper is the application of Transformer architecture to the problem of modeling marked temporal point processes. While this is a useful application, it does not introduce significant theoretical advancements or novel methodologies beyond leveraging existing models in a new context. To enhance the novelty, the authors could consider integrating more innovative elements or demonstrating new theoretical insights specifically tailored for MTPPs. - The paper could benefit from more detailed ablation studies to isolate the contributions of different components of the proposed framework. For example, assessing the impact of various hyperparameters, the influence of the Transformer architecture's depth and width, or the role of specific design choices in the decomposable framework would provide deeper insights into the model's functioning and robustness. - While the paper includes several figures and tables, additional visualizations could enhance clarity. For example, visualizing the learned intensity functions, the attention mechanisms within the Transformer, or case studies showing specific sequences predicted by the model could provide more tangible insights into the model's behavior.

Questions

1. Have you considered including more visualizations of the learned intensity functions, attention mechanisms, or specific case studies showing predicted sequences? Adding more visual representations of your model’s predictions and learned parameters would improve clarity and provide tangible insights into the model’s behavior. 2. Discussion about potential theoretical extensions that could lead to the modifications to the Transformer architecture specifically suited for MTPPs. Probably some interesting inductive biases could be added here. 3. Still, I do not see too much scientific novelty because literally Transformer was partly applied to a problem where it was not applied before.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Authors already discussed the limitations in the main paper.

Summary

The paper introduces a Decomposable Transformer Point Process (DTPP), a novel framework for modeling marked point processes. It maintains the advantages of attention-based architectures while avoiding the computational intensity of the thinning algorithm. The model uses a mixture of log-normals for inter-event times and a Transformer architecture for the conditional probability mass function of event marks, achieving state-of-the-art performance in next-event prediction tasks and outperforming thinning-based methods in long-horizon prediction.

Strengths

1. **Innovative Approach**: The paper proposes a new way to model marked point processes by decomposing the problem into manageable sub-problems, which is a creative advancement in the field. 2. **Empirical Performance**: The DTPP model demonstrates improved performance over existing methods, particularly in next-event prediction and long-horizon forecasting, which is a significant contribution.

Weaknesses

1. **Unclear motivation.** I am not an expert in this field, so I do not have a deep understanding of the field of neural point processes. I was pretty confused when I tried to understand the necessity of decomposing the log-likelihood of a marked point. 2. **The writing is a bit difficult to read.** The writing lacks clarity for readers outside the field. The overall quality needs improvement, as certain sections are challenging to comprehend due to complex sentence structures and unclear presentation of ideas. Specific examples include lines 60-63 and 128-136, where the convoluted language may hinder understanding for those unfamiliar with the subject matter.

Questions

See above.

Rating

5

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have adequately addressed the limitations

Discussion

I thank the authors for their response. What is the value of $M$ in the main experiments (e.g. in Fig. 1)? Since $M=2$ seems optimal in the provided ablation study, what is the value of log-likelihood for $M=1$ on the same datasets?