Add and Thin: Diffusion for Temporal Point Processes

Autoregressive neural networks within the temporal point process (TPP) framework have become the standard for modeling continuous-time event data. Even though these models can expressively capture event sequences in a one-step-ahead fashion, they are inherently limited for long-term forecasting applications due to the accumulation of errors caused by their sequential nature. To overcome these limitations, we derive ADD-THIN, a principled probabilistic denoising diffusion model for TPPs that operates on entire event sequences. Unlike existing diffusion approaches, ADD-THIN naturally handles data with discrete and continuous components. In experiments on synthetic and real-world datasets, our model matches the state-of-the-art TPP models in density estimation and strongly outperforms them in forecasting.

Paper

References (50)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer 6sim7/10 · confidence 4/52023-06-30

Summary

The submission derive a probabilistic diffusion model for TPPs. By proposing the Add-Thin framework, the proposed method can naturally handles the continuous and discrete nature of point processes and directly models the whole event sequences. Compared with the traditional autoregressive approaches for the temporal point process (TPP), the proposed framework is immune to the accumulation of errors caused by sequential sampling.

Strengths

The submission proposed a framework that learns the probabilistic mapping from complete noise, i.e., a homogeneous Poisson process (HPP), to data. More specifically, a model is learned to reverse the noising process of adding (superposition) and removing (thinning) points from the event sequence by matching the conditional inhomogeneous denoising intensity. This idea is very interesting.

Weaknesses

I do not find any obvious weaknesses.

Questions

None.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discussed the limitations and potential negative societal impact of their work.

Reviewer RwFj6/10 · confidence 4/52023-07-05

Summary

The forward process adds points from homogeneous poisson process (HPP) into the sequence and removes points from original sequence ($\mathbf{t}^{(0)}$). The goal is such that $\mathbf{t}^{(n)}$ is HPP. The neural network is trained to approximate missing information about $\mathbf{t}^{(0)}$, e.g. figure out which points from HPP came from $\mathbf{t}^{(0)}$ and which points were removed at intermediate steps. The thinning and superposition theorems were used multiple times to derive intensities of various sets, where the sets categorize the points.

Strengths

1. The adding and thinning approach, training via classification and matching, to modeling and generating temporal point processes is original and interesting.

Weaknesses

1. The writing and notation is extremely confusing. I read the paper multiple times. 1a. e.g. please clearly specify the observation, is it a time sequence $x(t), t\in[0,T]$ where $x(t) = 0$ for no event, $x(t) = 1$ for event, discretized in time. or is it a varying length sequence $\mathbf{t}=(t_1,...,t_K), 0< t_1<...<t_K\leq T$ with $K$ arrival times. also, section 3.3 writes about sequence embedding to get an input in $R^d$ but I am still confused. 1b. e.g. what are the sets A and F? what are the differences between B and C, D and E? B seems to be the points preserved from $\mathbf{t}^{(0)}$ all the way to HPP, C seems to be the points preserved from $\mathbf{t}^{(0)}$ but removed in the next step. is there a clearer way of writing the material in section 3.2? 1c. perhaps the background section can be shortened if we need more space to explain the method clearly. 1d. would potentially increase the score if the model and method is clearly explained.

Questions

1. Would like to introduce this recent other work that connects point processes and diffusions. in this case, the interarrival times of a point process is related to the length of a diffusion excursion. https://openreview.net/forum?id=KIlwyX7nCi 2. What is the benefit of applying the proposed model compared to other methods of generating sequences?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

1. Please address the potential limitations of using a neural network to approximate missing information about $\mathbf{t}^{(0)}$.

Reviewer 1xQ25/10 · confidence 4/52023-07-06

Summary

The paper proposes a probabilistic diffusion model for TPPs, ADD-THIN, that naturally handles the continuous and discrete nature of point processes and directly models whole event sequences. While autoregressive methods are expressive in modeling event sequences, ADD-THIN does not suffer from the accumulation of errors caused by sequential sampling for long-term forecasting applications. The proposed method matches the performance of state-of-the-art models in density estimation and outperform them for forecasting on both synthetic and real-world datasets.

Strengths

- To the best of my knowledge, this work represents the first attempt to combine diffusion models and Temporal Point Processes. The potential of generating complete sequences without relying on auto-regressive mechanisms presents a promising avenue for future research. It is worth noting that this paper shares similarities with the unpublished work available at: https://slideslive.com/embed/presentation/38922857. - The proposed method is compared to multiple baselines for both unconditional and conditional sampling (forecasting).

Weaknesses

- The paper is poorly written and hard to read. - There are multiple statements that are either inaccurate or without references - "The intensity of a TPP describes the expected number of points" - Line 65-70, you either need to prove it or provide references that do so. - Line 72, "the expected number of points on [0, T] follows a Poisson distribution with rate ..." - "Even though Poisson processes can model seasonality" - How does a Poisson model seasonality? - "We know that conditioned on ... is a random sequence of events with a density that can be completely described by its (unconditional) intensity function" - "Since the sets are disjoint, ..." - Is there a difference between a counting process and measure? an intensity function and measure? You use these terms interchangeably. - Multiple TPP datasets that are used in the experiments have been shown to be inappropriate benchmarks for neural TPPs. See https://openreview.net/forum?id=3OSISBQPrM - Overall, the presentation of the paper could be greatly improved, especially Section 3. Specifically, it feels like adding more formal notations (e.g. for Section 3.2.) would allow to shorten the text, while helping the reader more easily grasp the idea behind the method. For instance, why defer to the Appendix the pseudo-code for the sampling procedure? Additionally, several design choices are not clearly motivated, while the experiments and evaluation of the proposed model falls a bit short of expectations. These concerns are summarized in the points below: 1) Why include both $\boldsymbol{e}i$ and $\boldsymbol{c}_i$ in the parametrization of $g\theta$? Are the $\boldsymbol{c}_i$ constructed from neighboring events? Otherwise, it looks like there is redundancy between the information contained in the two embeddings. 2) There is no real motivation behind the use of a positional encoding for $n$. Why do you need it? 3) What is the motivation for using an unormalized mixture of Gaussians to parametrize $\lambda^{(A \cup C)}$? Have you considered alternative parametrizations that can be found in the neural TPP literature, e.g. an unmarked version of RMTPP [1]? Additionally, more details regarding the architectures employed would be appreciated (e.g. what is $\sigma$, and what is the model behind the MLP on page 5?). 4) In the NLL objective of equation 5, is the objective solely computed on times from $A \cup C$? If yes, this should be indicated explicitly in the objective. 5) On page 6, it is not clear how you find that $\Lambda^{(A \cup C)}(T) = K \sum_{i}^H w_i$. Given your definition of $\lambda^{(A \cup C)}(t)$, I would have expected this result to hold only if the integral was taken over $-\infty, \infty$. 6) The paragraph on 'conditional sampling' requires more detailed explanations, as it is impossible to see from the text how you handle this scenario. Specifically, how is $\boldsymbol{h}$ integrated into $g_{\theta}$ and $\lambda^{(A \cup C)}(t)$? 6) As $\hat{t}^{(0)}$ is successively approximated at each time step $n$ during the reverse process, how can you ensure that the propagation of errors is mitigated and won't affect significantly the generated sequences after $N$ steps? 7) Why only reporting the MAPE between the generated and truth sequences lengths? Isn't there a way to evaluate the quality of the generated samples within a sequence itself (e.g. using a measure of distance)? [1] Du, Nan and Dai, Hanjun and Trivedi, Rakshit and Upadhyay, Utkarsh and Gomez-Rodriguez, Manuel and Song, Le. (2016). Recurrent Marked Temporal Point Processes: Embedding Event History to Vector. SIGKDD.

Questions

See Weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

1 poor

Contribution

3 good

Limitations

The authors have a section on some limitations of their work, although a more in-depth discussion would have been appreciated. Specifically, they mention that sampling from their diffusion model can be quite expensive, but they do not provide computional trade-offs in terms of time/resources across baselines.

Reviewer 1d9q7/10 · confidence 3/52023-07-17

Summary

The paper introduces a novel probabilistic diffusion framework for temporal point process. Its significance lies modeling a whole event sequence directly, overcoming common limitations of autoregressive models.

Strengths

Originality: This paper is very novel. It connects diffusion models with TPPs and model a whole event sequence , overcoming issues from autoregressive models. Quality: The formulation of the approach and method itself are sound by leveraging two properties of point process: thinning and superpositions. The discussion of the reverse process are reasonable by discussing 4 cases of transition from nth step to n-1 step. The empirical results demonstrate the good performance of the proposed approach, esp. in forecasting tasks. Clarity: The paper is overall well-presented. Improvement can be made by better explaining A,B,C,D,E,F sets. Significance: The paper introduces a new way to model event sequences probabilistically which is appealing and significant to the TPP community.

Weaknesses

A. Experiments. The experimental evaluation on density estimation. The results of Add and Thin is only comparable to Intensity-free model by Shchur (RNN in the paper), if not worse. B. Application and Limitation. One minor limitation is it only deals with TPP without marks. It would be interested in see something for Marked TPP. C. Overall presentation. It took me a while to understanding the notation of sets A-F. It would be nice to include such information in the caption of figure 2. D. No codes are given, I am not sure how reproducible the results are.

Questions

A. Posterior approximation. How good is the posterior approximation in Add and Thin? Are there any theoretical insights. B. Neural network. Could the authors provide some movitation of using 3-layered-d CNN? C. Loss. 1). Is the mixture of Gaussians eqn. (4) used to model the conditional intensity good for modeling an event time(or interevent time) t which is positive? Add and Thin is a latent variable model with latent variables t^{n-1}’s. Usually int the diffusion models reference 40 in the paper, latent variable models leverage variational inference to maximize the evidence lower bounds. But this model is not. Could the author shed some insights on this? D. Cox process models. Do the authors have any experiment results with these baselines form reference [5,8,17]? E. Experiments. How many diffusion steps (n) are used?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The limit of the paper according to the authors is it is limited to TPP without marks. In addition, current sampling routines may not be efficient. TPP models can be applied applied to many fields, where common application domains include traffic, social networks, and electronic health records.

Reviewer 1d9q2023-08-14

Thank you for your response.

Thank you for your clarification. I found it helpful and clear. Therefore, I keep my recommendation and will stay tuned about the discussion from other reviewers on your interesting work.

Reviewer 1xQ22023-08-15

Rebuttal

Thank you for your clarifications on my concerns regarding the design choices, and for taking my recommendations into account. There was indeed a misunderstanding regarding the metrics employed. Upon reading your updated description of the reverse process and your replies to other reveiwers, I have increased my score. Nevertheless, in light of the fact that we won't have the opportunity to review the revised paper, I remain hesitant to recommend its acceptance.

Reviewer RwFj2023-08-15

The unpublished work is ``Deep Point Process Destructors'' David I. Inouye, NeurIPS Workshop on Learning with Temporal Point Processes, 2019.

Reviewer RwFj2023-08-15

W.1a. Obervations - Please further clarify ``temporal embedding followed by a sequence embedding''. In addition, is it e.g. $\mathbf{t}^{1}=(1,5,7,9)$, $\mathbf{t}^{2}=(2,5,6)$, where $\mathbf{t}^{1}$ and $\mathbf{t}^{2}$ are two data samples, and the lengths may be different? What is the input to the network at each stage? Or is it possible to make code available, thanks! W.1b. Unclear distinction between sets A,...,F. - Reviewer 1d9q had the same comment too. In the initial submission, I was guessing what the model and method is. Read the response to all reviewers. Is it also possible to have a figure, thanks! Q1. Connection to work on diffusion excursion for TPPs. - Thank you for pointing me at those references! Please address the potential limitations of using a neural network to approximate missing information about $\mathbf{t}^{(0)}$ - Read the response to reviewer 1d9q. Approximating Dirac deltas with Gaussians may not work as well... Read all the reviews and the responses, I am positive and interested in the work, increasing score to 5 for now, will wait for the authors to respond.

Authorsrebuttal2023-08-17

Thank you for your response and active engagement in the rebuttal discussion. We are delighted that you are positive and interested in our work. In the following, we would like to clarify your comments: ### Observations > In addition, is it, e.g., $\mathbf{t}^{1} = (1,5,7,9)$, $\mathbf{t}^{2} = (2,5,6)$, where $\mathbf{t}^{1}$ and $\mathbf{t}^{2}$ are two data samples, and the lengths may be different? Indeed, instances or data samples of a TPP can have different lengths and could, for example, look like $(1,5,7,9)$ and $(2,5,6)$. In other words, both the arrival times and number of points are random. This defining property of a TPP is introduced in the background section 2.1 lines [59-61]. Further, we would like to highlight the notational difference in your response. With the superscript $\mathbf{t}^{(n)}$, *we* refer to an event sequence at the $n$-th diffusion step, where $\mathbf{t}^{(n)} \sim \lambda_n$ and not to different samples from the data distribution. > Clarify ``temporal embedding followed by a sequence embedding'' and the input to the network at each stage (can you maybe add code). The temporal and sequence embedding were explained in Sec. 3.3 lines [178-183] with the additional sentence in our last response and further depicted in Fig. 3. Here, we will additionally present some background and slightly informal explanation of the embedding types for TPPs: **Temporal (Positional) embedding:** Applying temporal embeddings, e.g., log-transform and trigonometric functions (similar to the positional embeddings of Transformer [3]), to event times is a common practice for TPPs (see Sec. 3.1 of [1]). With a temporal embedding, we refer to a function $\mathbb{R}_{>0} \to \mathbb{R}^d$, that maps a time, e.g., $t_i \in \mathbf{t}$ independently to a $d$-dimensional embedding space. Then for each ${t}_i$, we have a feature vector $\mathbf{c}_i$ encoding its temporal information. **Sequence embedding:** To attain an event embedding (also sometimes called context embedding) $\mathbf{e}_i$ for each event $i$ that also incorporates information about all other events, we apply a three-layered CNN with dilation and circular padding on the sequence of temporally embedded event times $(\mathbf{c}_1, \cdots, \mathbf{c}_K)$. Ultimately, a global sequence embedding $\bar{\mathbf{e}}$ is attained by computing the average over $(\mathbf{e}_1, \cdots, \mathbf{e}_K)$. The code will be published upon acceptance, but we present a simplified pseudo-code of our implementation of the sequence embedding here: ``` # temporal embedding time_embedding = self.time_encoder([x_n.time, x_n.tau]) # Kxd # event embedding event_embedding = self.cnn(time_embedding) # Kxd # sequence embedding seq_embedding = event_embedding.mean(dim=0) # d ``` As described in Sec. 3.3 line [185; 196] and Fig. 3, ```time_embedding``` $\mathbf{c}_i$ and ```event_embedding``` $\mathbf{e}_i$ are the inputs to the classifier and ```seq_embedding``` $\bar{\mathbf{e}}$ is the input to the MLPs parameterizing the intensity function. ### Set distinction > Is it also possible to have a figure (Distinction between sets A,...,F)? We might be wrong but believe that Fig. 1 (right) and 2 (left) are what you are looking for. Fig. 2 presents the case distinction of the sets **A**,...,**F** with a Venn diagram as the logical relations between $\mathbf{t}^{(0)}$, $\mathbf{t}^{(n-1)}$ and $\mathbf{t}^{(n)}$. Further, in Fig. 1, the denoising process illustrates the case distinction for one specific example of $\mathbf{t}^{(0)}$ and $\mathbf{t}^{(n)}$. Please note that the illustrated posterior intensity can be separated into the sets **B-E**: The base intensity relates to $\lambda^{(E)}(t)$, while the highest peaks (Diracs) refer to $\lambda^{(B)}(t)$, i.e., the points that are in $\mathbf{t}^{(0)}$ and $\mathbf{t}^{(n)}$. The medium peaks and lower peaks (thinned Diracs) represent $\lambda^{(D)}(t)$ and $\lambda^{(C)}(t)$, respectively. Please let us know if this addresses your point, and we will, for the camera-ready version, add an additional reference to the denoising process in Fig. 1 in the discussion of the reverse process in Sec. 3.2. ### Approximation > Approximating Dirac deltas with Gaussians may not work as well... A Gaussian is the standard approximation of the Dirac delta function and can, in the limit $\sigma \to 0$, perfectly approximate it [2, 4]; a property that is not true for many approximations of other TPP models (e.g., parametric distribution for the intensity function of *RNN*, *Trans* and *TriTPP*). Furthermore, the KL divergence between Dirac and our Gaussian is directly minimized to ensure the best possible approximation. ### References 1. Citation 38 in the paper 2. Ghatak, Ajoy, et al. "The dirac delta function." Quantum Mechanics: Theory and Applications (2004) 3. Citation 42 in the paper 4. Saichev et al. "Chapter 1: Basic definitions and operations", Distributions in the Physical and Engineering Sciences (1997)

Reviewer RwFj2023-08-17

Thanks for the clarification! Increased score to 6. Q. additional reference to the denoising process in Fig. 1 in the discussion of the reverse process in Sec. 3.2. Yes please, for the camera-ready, it would be very helpful to label each peak/ addition/ thinning with the set.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC