Temporally Disentangled Representation Learning under Unknown Nonstationarity

Summary

This paper addresses the problem of identifying latents representations from sequential data which is stationary within contexts, and non-stationary between contexts. Existing work has either conditioned on observed auxiliary variables that indicate which context one is in (e.g. time contrastive learning and the followups), or they relied on mutually (conditionally) independent latents and did not allow time-delayed latent causal influences. The paper presents an approach called Nonstationary Temporal Disentangled Causal Representation Learning (NTDC)and shows conditions under which they can identify time-delayed latent causal variables and their relations without the need for observed auxiliary variables. They formulate the problem as a discrete Markov process and establish identifiability of the latent independent components. They give impressive experimental results on both synthetic and real-world datasets to demonstrate the proposed method's success in recovering latent variables.

Strengths

* Really nice experimental results section. The results on simulated data and I appreciated the section on the MoSeq dataset analyzing mouse behaviour. * I think the graphical model that the paper analyses is very practically useful for abstracting time series data. By breaking a time series into conditionally stationary domains, $c_t$, that change more slowly, you gain a more interpretable model of the underlying behaviour of the system (the mouse experiments are nice illustration of this). Empirically, we've seen results similar to this before in the original MoSeq paper, but it is nice to have identifiability results to go with these approaches.

Weaknesses

* Assumptions (6) and (7) in Theorem 2 are unintuitive to the point that I'm not sure the theorem is useful at all. The main value of identifiability theorems are to give a set of preconditions on the data under which a practitioner can expect a disentanglement routine would work. But I'm not sure that there are any practitioners who would be able to judge whether those assumptions hold in their data, and there was no attempt to give any intuition for when they might hold and when they would fail.

Questions

1. Can you give any more intuitive assumptions that are sufficient for Theorem 2 to hold? Or give examples of distributions for which they hold and examples for which they fail? If done well, this would make the theory section stronger.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The paper already does a reasonable job of outlining the limitations of their work.

Summary

This paper gives identifiability results in a new setting, along with an estimation method and experiments validating the estimation method. There are existing identifiability results in the nonstationary setting if you assume that the auxiliary variables are observed. There are other existing identifiability results in the nonstationary setting if you assume that the auxiliary variables evolve as a Markov chain but the latents only affect the current timestep. This paper gives an identifiability result in the nonstationary setting where the auxiliary variables are *not* observed and the auxiliary variables evolve as a Markov chain but the latent variables can affect the observed variables across time steps. The estimation method is an extension of Sequential VAEs, where there is an autoregressive Hidden Markov Model model that models the nonstationarity, a prior network that estimates the prior using a conditional normalizing flow, and a VAE where the encoder fits the demixing function and the decoder fits the mixing function. They perform experiments on two synthetic datasets that satisfy their identifiability conditions, a video dataset for a modified Cartpole environment, and a video dataset of mouse behavior. For the synthetic datasets, they compare MCC with seven baselines (BetaVAE, iVAE, TVL, SlowVAE, PCL, TDRL, and HMNLICA). For the video cartpole dataset, they compare MCC against one baseline (TDRL). For the mouse video dataset they don’t have ground-truth independent components so instead of numerical results they provide a visualization of the recovered components.

Strengths

Strengths: - I am not especially up-to-date on identifiability literature, but as far as I am aware this is a novel problem setting and novel result. - I agree that the “observed auxiliary variable” condition in many nonlinear ICA identifiability results is not realistic, and it is a useful direction to prove results that do not rely on this assumption. - The choice of baselines is very thorough for the synthetic datasets. - They use two video datasets to demonstrate their estimation method in a more realistic setting than synthetic data, which I appreciate since this is mostly a theory paper.

Weaknesses

Weaknesses: - The writing is okay but not great. It gets the point across but it could use proofreading and better word choices in some places. Ex: “It is obvious that the causal relations among those latent variables are more meaningful and the identifiability is urgently needed.” <- this sentence could be improved. - See questions below.

Questions

- In the Cartpole experiments, why do you only provide Accuracy and MSE A metrics for your method and not for baseline methods? And can you provide evaluations for the other baselines for this experiment?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

- As they mention in the conclusion section, they assume that the nonstationary variables follow a Markov chain, which is a somewhat restrictive assumption. - No concerns about negative societal impacts.

Summary

This paper presents identifiability results that handle nostationary time-delayed causally-related processes without auxiliary variables. In addition, the authors propose NTDC, which is a neural network that is based on their identifiability results. The method is evaluated on a few tasks in comparison to a few baselines, showing significance of toy tasks generated by the authors.

Strengths

This paper introduces identifiability results on sequential information which is causally-related without additional variables. Thus, this paper extends prior work from a theoretic viewpoint. Another strength of the paper is the new deep framework that incorporates the theoretical results.

Weaknesses

One of the main weaknesses of the paper is its presentation. In particular, a lot of jargon is used without properly introducing the terms. For instance, domain indices are used throughout the paper without ever defining what it means. Similarly, Sec. 4.1 is missing a lot of details (what is ARHMM? what is the prior network?). Sec. 4.2 seems to be unfinished in terms of writing. Missing details and descriptions make it hard to properly evaluate the proposed NTDC and position it with respect to prior work. Another main shortcoming of the work is the lack of discussion on related work for sequential disentanglement. Only paper [20] is mentioned (extremely) briefly. However, there are already at least ten papers or more since [20] which significantly improved the state-of-the-art results, the neural network models, the evaluation and theory. Another shortcoming of the paper is the dicsussion in the introduction. In particular, the authors argue their mouse example might be challenging for existing work. In addition, they argue to be first to consider scenario Fig. 1c. However, in a recent ICLR'23 paper with the title "Multifactor Sequential Disentanglement via Structured Koopman Autoencoders" (SKD) by Berman et al. the authors show a framework that considers a similar model to Fig. 1c, and it is able to handle data such as the mouse example. The authors should compare their work with SKD both qualitatively and quantitatively. Similarly, the work "Contrastively Disentangled Sequential Variational Audoencoder" (C-DSVAE) by Bai et al. can also deal with a similar scenario and should be compared with. Another shortcoming of the paper is its evaluation. In particular, the proposed method is compared with several baselines only on toy datasets that the authors create. No sequential disentanglement baselines are considered. On real world data, the authors only compare with a single method.

Questions

See above.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

See above.

Summary

The study focuses on unsupervised representation learning for sequential data with time-delayed causal influences. Identifiability results for disentangling causally-related latent variables have been established in stationary settings using temporal structure. However, existing work only partially addresses this in nonstationary settings by using observed auxiliary variables or introducing the Markov assumption, but not both simultaneously. The authors introduce NTDC, to reconstruct time-delayed latent causal variables and identify their relations without the need for auxiliary variables, by utilizing deep generative models. The experimental results demonstrate that the proposed methodology outperforms existing baselines by effectively exploiting nonstationarity and distinguishing distribution shifts.

Strengths

- The paper is clearly written. - The authors tackle the setting of previous works and propose a non-existing work, which is described in Figure 1. - The theoretical grounds are sufficiently provided with proper theorems.

Weaknesses

- There are some grammatical errors in the main paper. - There is no explanation of the "disentangled" which is included both in the proposed model name and the title of the manuscript. - The authors provide their code with anonymity, but there is limited explanations of the codes that which is directly linked to the proposed NTDC and how one can run the code.

Questions

- Could you provide more motivational examples of the model structure in Figure 1(d) other than the mouse movement example? - Are there any derivation with probabilistic graphical model settings of Figure 1(d)? I wonder whether the problem could be solved in a statistical manner rather than just utilizing DGMs. - I guess the model structure can be viewed as a doubly-hidden Markov model, or it could be more generalized as a multi-layer hidden Markov model. Could proposed theorems, as well as the proposed NTDC, be generalized in such settings? - What's the specific meaning of "disentangled" representation in the title as well as the model name? And how the disentangled representation is achieved?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

- There are no potential negative societal impacts of their work. - Even though the authors provided a new work, my concern is that it lacks significance considering the NeurIPS standard.

Thank you and we are looking forward to your continued feedback!

Dear Area Chair and all Reviewers: Thanks again for all the insightful comments and advice, which helped us improve the paper's quality and clarity. The discussion phase has been on for several days and we have not heard post-rebuttal discussion yet. We would love to convince you of the merits of the paper. Please do not hesitate to let us know if there are any additional information or clarification that we can offer to make the paper better. We appreciate your comments and advice. Best, Authors

Sufficient variability

Thanks for the clarification. This discussion is useful and should be in the paper: I'm someone who knows this literature fairly well, and while it looked like a sufficient variability-style assumption, I missed that it was exactly sufficient variability. Despite the fact that sufficient variability is now a commonly used assumption, I think a description of what it requires and when it fails should still be in every paper that uses it so that the paper has a self-contained explanation of when we can expect the method to work / fail. On the assumption that you'll add this discussion to the paper, and I've increased my score to a 7.

Reviewer response to rebuttal

Thanks to the authors for your responses to my questions and comments about the experiments and baseline comparisons and about the writing. I don't have any further questions and I'll keep my score as is.

increase score from 4 to 4.5

Thank you for your clarification, especially for Q3, Q4, and Q5. I've read the authors' feedback, as well as other reviewers' reviews and the corresponding rebuttal. I've increased my score to 4.5, which is a definite borderline. In the meantime, I've left the rating to 4 since there is no 4.5 option.

Thanks again and we are looking forward to your continued feedback

Thank you very much again for the insightful comments. We have been dedicated to absorbing them and providing our responses accordingly. As Aug 21 is the last day for us to respond, we hope for the chance to see and respond to your feedback. It seems that Reviewer 5DXD hasn't got time to discuss whether our response properly addresses the concerns. We are eager to see the feedback. We summarized our response into following categories: **Updated presentation** 1. We have provided additional motivational examples with explanations to demonstrate the use case for our proposed model structure (**t3u7**) 2. We clarified domain specific terminology and provided detailed explanation on unclear descriptions in the manuscript, we also updated the related work part (**5DXD**) 3. We also revise and improve the presentation (**eHJB**) **Newly conducted experiments** 1. We conducted additional experiments on the baseline settings that reviewers suggested (**5DXD**) and the result is consistent with our original claim. 2. Some of the metric doesn't fit the experiment setting, either the algorithm doesn't have such output or the algorithm doesn't converge in that setting. We added detailed explanation in the discussion (**eHJB**) 3. Some reviewers suggest comparison with methods with orthogonal contribution directions (**5DXD**), we thank the reviewer and conducted the experiments, the result is also consistent with our original conclusion, i.e. it is necessary to gain identifiability in representation learning and only an identifiable model can recover the ground truth information. **Clarification on the Assumptions** 1. Thanks the reviewer for pointing out, we further discuss when the assumptions hold and when the assumptions fail. We clarified the sufficient variability assumption with examples which is well received by reviewer (**PKrt**)