Integration-free Training for Spatio-temporal Multimodal Covariate Deep Kernel Point Processes

Summary

This work proposes to fuse Deep Kernel Learning (DKL) into the Deep Mixture Point Processes (DMPP), resulting in Deep Kernel Mixture Point Processes (DKMPP) which can handle complex relationships between events and covariates in a more flexible and expressive manner. The authors also leverage the denoising score matching technique and come up with a training procedure that does not require integration and computation of second-order derivative. The proposed method is evaluated on a variety of datasets and compared with other baseline point process models to demonstrate its advantages.

Strengths

• The authors extend the score matching to point processes and make the training more scalable by leveraging denoising technique, which, I believe, is a fair contribution to the field of point process learning and can be applied to a variety of scenarios. • The authors conduct a wide range of experiments using both synthetic and real-world datasets, demonstrating the efficiency and effectiveness of the proposed method. The choice of model hyperparameters is appropriately studied. • The organization and the presentation of the proposed method are clear and easy to understand.

Weaknesses

On lines 158-159, the authors argue that Euclidean distance in the kernel may not be a suitable measure of similarity, especially for **high-dimensional inputs**. I think this statement needs more clarification. If I understand correctly, the input $\textbf{s}$ to the kernel function $k_{\phi}$ in the original DMPP lies in the space $\mathcal{R} \times \mathcal{R}^2$, but in DKMPP, based on (7), the input to the kernel function is first mapped by a deep neural network $g_{w_2}: \mathcal{R} \times \mathcal{R}^2 \rightarrow \mathcal{R}^D$ where $D$ can be much larger than the original dimension of $\textbf{s}$. In other words, the **high-dimension** of the input is induced by our design choice of $g_{w_2}$ instead of the data, is that right? Can $\textbf{s}$ itself be high-dimensional in spatio-temporal point processes? Minor comments: • Since $\lambda(\textbf{s}|\mathcal{D})$ in (6) and (7) is an approximation of the true intensity of DMPP, I recommend using a different notation such as $\hat{\lambda}(\textbf{s}|\mathcal{D})$. • Figures 1(c)-(d) are displayed but never referred to. I suggest the authors either remove the figures or compare 1(c)-(d) to the true intensity function.

Questions

• If $f_w$ is a network that outputs nonnegative mixture weights and $k_{\phi}$ is positive semi-definite, why do we need the link function $\eta(\cdot)$ to ensure non-negativity in (7) but not in (6)? • Why do all point process models yield relatively low accuracy on the NYC Complaint data? Does this mean point process models are not good at processing textual information?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The limitations are adequately addressed. I’m not aware of any direct negative societal impacts of this work.

Summary

This paper argues that there are two common approaches for modeling intensity functions: traditional and covariate based methods, and this paper focuses on the latter one. In detail, the intensity function is designed in a kernel convolution form: $\lambda(s|\mathcal{D})=\int f_w(\mathbf{u},\mathbf{Z(u)})k_{\phi}(\mathbf{s,\mathbf{u}})d\mathbf{u}$, in which contextual information is embedded in $\mathbf{Z(u)}$. In practice, its integration is replaced by summation. This method makes integration $\int \lambda(s|\mathcal{D})ds$ intractable. However, the kernels lack expressiveness both because of the unknown relationship between covariates and event occurrence and improper usage of Euclidean distance. To solve this issue, authors propose to use deep kernels which are modeled by neural networks. To address parameter estimation, the authors further propose to use of a score matching-based estimator to estimate parameters.

Strengths

1. The theoretical part is reasonable and complete, and the proposed model effectively takes into account both solving the difficulty of integrating the intensity function and maintaining the strong expressiveness of the intensity function. 3. The score matching-based modeling method is novel and interesting and plays a positive role in promoting research in the field of point processes. 3. The advantages of the model are adequately and effectively demonstrated by experiments.

Weaknesses

1. The motivation for using a score-based approach is not clear. In fact, the score-based approach is a special generation model. In that case, why not use another generation model, such as GAN or VAE? The authors point out that the score-based approach can effectively solve the parameter estimation problem, but it seems that other generative models can also solve the problem. 2. The proposed model seems to be a combination of existing frameworks, which actually hinders its nolvity to some extend.

Questions

1. I'm confused about the reason why the authors use score-based methods. 2. Actually, there are some efforts concerning the embedding of generative models into point processes. Is there any comparison between the proposed model and some existing methods, see, for example [1-2]. [1] Xiao S, Farajtabar M, Ye X, et al. Wasserstein learning of deep generative point process models[J]. Advances in neural information processing systems, 2017, 30. [2] Mehrasa N, Jyothi A A, Durand T, et al. A variational auto-encoder model for stochastic point processes[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 3165-3174.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.

Summary

The paper proposes an enhanced version of Deep Mixture Point Processes with a flexible neural network-based kernel. The intractable training process of the point process with deep kernel is handled by a score-matching technique with the denoising method.

Strengths

1. The proposed deep kernel goes beyond the parametric kernel and substantially improves the model flexibility and expressiveness. 2. Address the learning challenges of MLE (a long-lasting problem in neural pp training) by proposing a score-matching technique which achieves better modeling performance and computational efficiency. This is also supported by the experimental results.

Weaknesses

1. Related works are not comprehensive enough. There is plenty of work on point processes equipped with deep kernel, such as Okawa [1] and Zhu. [2]. These works can be reviewed to make the paper more comprehensive. 2. More experiments can be included. For example, one baseline of Hawkes process equipped with a deep kernel can be included. Also model performance on synthetic point process data (such as self-exciting point process or self-correcting process) would make the numerical results more convincing. --- [1] Maya Okawa, Tomoharu Iwata, Yusuke Tanaka, Hiroyuki Toda, Takeshi Kurashima, and Hisashi Kashima. Dynamic hawkes processes for discovering time-evolving communities’ states behind diffusion processes. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1276–1286, 2021. [2] Shixiang Zhu, Haoyun Wang, Zheng Dong, Xiuyuan Cheng, and Yao Xie. Neural spectral marked point processes. In International Conference on Learning Representations, 2022.

Questions

1. How do we calculate the model density $p_\theta(\tilde{S}_m)$ in equation 12? 2. Could the authors include additional synthetic experiments, such as modeling self-exciting point process data, because such data is becoming ubiquitous and gaining popularity in recent research and real-world application.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The current structure still imposes parametric constraints on the kernel. An alternative form of the kernel can be considered [1]. --- [1] Shixiang Zhu, Haoyun Wang, Zheng Dong, Xiuyuan Cheng, and Yao Xie. Neural spectral marked point processes. In International Conference on Learning Representations, 2022.

Reply to the author rebuttal

The authors' rebuttal appropriately addresses my concerns and questions. I believe the neural network-based kernel method and the score-matching technique would contribute to the point process community in the future. In light of this, I would raise my score.

Response to author rebuttal

Thank you for answering my questions. The responses mostly make sense to me and I'm inclined to maintain the score as is.