Dynamic Tensor Decomposition via Neural Diffusion-Reaction Processes

Summary

The paper presents the framework to generate dynamic embeddings for multi-entity (e.g., traffic type, severity, latitude, and longitude) interactions with timestamps. The temporal high-order interactions are modeled via neural diffusion-reaction processes, and to tack the temporal drift of underlying node, authors propose the dynamic embeddings that follows a diffusion-based ODE model. The diffusion process uses a multi-partite graph to coevolve the trajectories, and the reaction process models the individual evolution of each object. The combination of these two processes can capture both the commonalities and personalities in learning the embedding trajectories for different objects. Authors demonstrate that the proposed model can capture the temporal dynamic well through both synthetic and real-world datasets with comparison to a large number of existing methods.

Strengths

1. The proposed ODE-based model looks like a reasonable way of representing the dynamics of embeddings. 2. The dynamic embeddings for entities are generated in continuous, rather than discrete time, unlike most existing approaches. 3. Numerical experiments demonstrate that the proposed method outperforms baselines on 3 real-world datasets.

Weaknesses

1. It seems that the text can be further improved to present better. Some phrases are repeated several times unnecessarily, and some points, on the contrary, are covered too briefly. I would recommend the authors to carry out additional proofreading of the text. 2. The source code is not provided, and this complicates the reproducibility of the results.

Questions

1. [Line 6] It seems that "I" in "EMbedIngs" should be lowercase. 2. [Lines 92-100] It seems that there are not enough conclusions to this paragraph. What are the advantages and disadvantages of the indicated "continuous CP decomposition" approach? 3. [Line 106] "typical" -> "typically". 4. [Line 270] "We compared <our approach>..."

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The proposed method is limited to a small number of entities, however, further development of the approach using the graph cut algorithms could potentially improve its scalability.

Summary

In this work, the authors introduce DEMOTE which can both consider both structure information and temporal information via dynamic tensor decomposition. Structure information is captured by multi-partite graph diffusion process which in substance is a special ODE, whereas temporal information is analyzed using another ODE. Experimental results demonstrate DEMOTE in general outperforms one baseline for synthetic data and many baselines for real-world data.

Strengths

1. It is interesting to model the spatial interactions for values at the same time point using multi-partite graphs through graph diffusion, for the small-scale datasets. 2. Addressing the sparsity in tensorial data is another strength of this work. It has been an important problem in tensorial data and appeal to the tensor learning community.

Weaknesses

1. There have been some existing work on dynamic tensor decomposition using neural ODEs and relational information in the recent years, whereas not discussed and compared in this work, for example, TANGO [1]. 2. The presentation should be improved. The notations should be defined in a clear manner, for example, the adjacency tensor W needs more clarification on its order and how it is computed, from its indices. Reference [1] Han, Z., Ding, Z., Ma, Y., Gu, Y., & Tresp, V. (2021, November). Learning neural ordinary equations for forecasting future links on temporal knowledge graphs. In *Proceedings of the 2021 conference on empirical methods in natural language processing* (pp. 8352-8364).

Questions

1. For the statement in Page 3 Section 3 Lines 113-115, why can learning dynamics produce more reliable and long-term trajectory predictions? The time derivative can provide the general dynamics of the whole time series and mainly focus on the global sequential information. Please check this paper "Neural Rough Differential Equations for Long Time Series" [1]. 2. Could the authors provide any explanation and examples on why K should be involved and represent structure information using multi-partite graphs is necessary? It seems that including the multi-partite graphs is very complicated, and in what circumstances the multi-partite graphs should be applied to describe what types of intercorrelation among multi-dimensional data, without incurring overfitting? 3. In Figure 2(c), why do the red lines decline? In the long-term, the performance of DEMOTE is worse than the baseline model NONFAT. Could the authors articulate it? 4. For the results of DEMOTE in Figure 5(a) and (b), why are the entry value predictions with high and sharp spikes across time? It seems that DEMOTE can capture the periodic fluctuations, but DEMOTE by nature is based on two ODEs. In specific, ODE and neural ODE assume that the modeled trajectory should be smooth, or at least Lipschitz continuous [2,3]. Could the authors also provide their code? References [1] Morrill, J., Salvi, C., Kidger, P., & Foster, J. (2021, July). Neural rough differential equations for long time series. In International Conference on Machine Learning (pp. 7829-7838). PMLR. [2] Chen, R. T., Rubanova, Y., Bettencourt, J., & Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems, 31. [3] Süli, E., & Mayers, D. F. (2003). An introduction to numerical analysis. Cambridge university press.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitations are sufficiently addressed by the authors.

Summary

In this paper, to capture higher-order interaction among different modes of tensor data and their evolution process over time, the authors proposed an ODE-based method called DEMOTE to capture the dynamics of latent embeddings. In detail, the authors propose to use a diffusion-reaction process to model the evolution of these embeddings. What's more, structure knowledge within tensor data is considered in this diffusion-reaction process, which enhances the model's representation ability. Experiments show the effectiveness of the proposed method.

Strengths

1. The experimental results show an advantage compared with related work. 2. The way of using the structure knowledge is novel and effective in modeling tensor data. It is particularly interesting to embed this structured knowledge with the framework of ODE.

Weaknesses

1. The main concern lies in its novelty since there are some efforts studying the interaction results of higher order continuous time data, as clarified in the paper (that is [1]). Actually, in [1], they use ODE to model the entry values' trajectories and depend on $m_{i}$. In this work, the authors use ODE to model $m_{i}$ directly. Although these two papers correspond to different ways of handling problems and the authors have stated the advantage of the proposed method, they do share many similar intuitions towards the same task, which hinders their novelty. Typo: Page 3, last paragraph: $(v_{j}^{k}, v_{j'}^{k'})\in \mathbb{E}$ $\rightarrow$ $(v_{j}^{k}, v_{j'}^{k'})\in E$. [1] Li S, Kirby R, Zhe S. Decomposing Temporal High-Order Interactions via Latent ODEs[C]//International Conference on Machine Learning. PMLR, 2022: 12797-12812.

Questions

1. I think the contribution should be further clarified.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

This paper proposes a dynamic tensor decomposition (TD) model, in which the latent factors are functions of time stamps described by neural ODEs. Moreover, they encode an adjacent matrix into the diffusion process to better model the correlations among different tensor modes. Finally, they use an additional neural network to map latent factors to tensor entries. Using differentiable ODE solvers, all the parameters can be learned by optimizing the joint likelihood with SGD. For empirical evaluations, the authors first conduct a simulation study to show that the proposed model can learn latent factor trajectories. Then, they demonstrate the completion performance on several dynamic tensor datasets.

Strengths

1. The authors propose to use a diffusion process to learn tensor factor trajectories, which is new in the field. 2. The experiments show that the proposed model can learn factor and entry trajectories. Besides, it achieves a significant improvement on the completion performance.

Weaknesses

1. In the paper, the authors construct a complex TD with ODEs and NN to parameterize the latent relation. Besides, the model parameters are learnt from MAP estimates (Eq. 3). I am wondering if such a model suffers from overfitting for sparse tensor data? 2. In Equation 1, the authors define the ODE of the factor trajectories. Since $\mathcal{F}(\mathcal{U}, t)$ is already parameterized by flexible neural networks, I am wondering if it is necessary to add the multi-partite graph term, i.e., $(\mathcal{W} - \mathcal{A})\mathcal{U(t)}$? Can we construct a specific network structure that is equivalent to Eq. 1?

Questions

1. In the equation between Line 178 and 179, does $\mathcal{L}$ equal to $\log (Prior)$ **plus** the likelihood?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

None