Switching Autoregressive Low-rank Tensor Models

Summary

This paper introduces a new unsupervised probabilistic model for time series analysis: the Switching Autoregressive Low-rank Tensor (SALT). It combines a Low Rank Tensor parametrization of autoregressive (AR) models with switching Dynamics. The two main contributions are: 1) The SALT model itself. Although the tensor parametrization is not new, they succesfully augment it with switching dynamics and derive an EM-based learning and inference algorithm with closed form updates. 2) A theoretical connection between Linear Dynamical Systems (LDS) and Low-Rank Tensor Autoregression. The model is evaluated on simulated, behavioral and neural datasets.

Strengths

The paper is very well written despite the need for numerous notations. The author fluently expose their model and relate it to existing approaches. The link between LDS and Low-Rank Tensor AR models is important and novel (to my understanding). Not only it helps grasping the relationship between those two models, but it also provides a bound linking the dynamics spectrum, the tensor rank and the approximation error one makes when using the tensor AR model. Among other things, I believe it can guide the specification of the tensor rank, which is a non trivial operation. SALT builds upon existing approaches, and although it is not more expressive than sLDS, inference and learning use closed form updates which is very convincing from a practical perspective. Importantly, it retains the possibility to analyze a low dimensional representation of the observed time series. Both simulated and real world experiments are convincing and the code is provided.

Weaknesses

1) Title 3.3 is confusing. I agree that the theoretical connection discovered by the author is significant, but it does not concern SALT and sLDS. The link is between LDS and Low Rank Tensor AR models. 2) I think the graphical of Figure 1 is wrong. Why isn't there any arrow between $z_t$ and $z_{t+1}$ despite $z_{t+1} \sim \text{Cat}(\pi^{(z_t)})$ ?

Questions

1) Would it be possible to concatenate the 3-way autoregressive tensors $\mathcal{A}^{h}$ in a 4-way low rank tensor $\mathcal{A}$ of size $N \times N \times L \times h$ ? This might yield an even more compact description of the data. 2) I understood Proposition 1 was a novel contribution from the authors. If it's indeed the case, it's worth mentioning it (more) explicitly in the introduction. 3) How stable are the CP-SALT discovered factors from one initialization to the next ? It could be interesting to look at the factors directly to characterize dynamics, but I think the discovered dynamics could be roughly preserved despite different $\mathcal{A}^{h}$ ?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

1) The proposed method is based on Gaussian noise models, which is acknowledged by the author. In its current form, it cannot, for example, efficiently handle image data without pre-processing steps. 2) The link between SALT and SLDS is "only" demonstrated for non switching dynamics ($h=1$ ).

Summary

The paper introduces a new model for time-series called SALT (switching autoreg. low-rank tensor). The goal with SALT is to offer a "best of both worlds" alternative to AR-HMMs and switching linear dynamical systems (SLDS). SALT's relative advantages are: * enjoys closed-form parameter estimation (unlike SLDS) * enjoys lower parameter count, meaning less likely to overfit for long-range models on small datasets (unlike AR-HMM) | | AR-HMM | SALT | SLDS | |------------------|-------------|-------------|-----------------------| | param estimation | closed-form EM | closed-form EM | need MCMC/variational | | parameter count | O(N^2) | O(N) | O(N) | | hyperparameters | H, L | D, H, L | D, L | where D = rank / number of latent state dims, H = num hidden states, L = autoregressive lag The key idea behind SALT is to essentially force the autoregressive coefficient matrix of an AR-HMM to take a specific factorization structure (See Eq 6-7). In Sec 3.2, a closed-form EM estimation algorithm is derived. In Sec 3.3, Prop 1 presents a formal error analysis of how well a SALT model can approximate a *stable* LDS Experiments in Sec 4 cover * 5.1: analysis of Prop 1 ideas on a simulated dataset * 5.2: evidence that SALT might be as expressive as SLDS on synthetic data * 5.3/5.4: real experiments on mouse behavior and worm neurons

Strengths

+ elegant proposal for factorizing coefficients of AR-HMM + parameter estimation via EM allows all steps to be closed-form + thoughtful experiments on synthetic and real data

Weaknesses

### Why prefer SALT over SLDS? What real computational advantage? In Sec 5.4, on a big real dataset it is argued that SALT "can perform as well as" SLDS. While this is all fine, is there a strong advantage that makes a practioner studying this data prefer SALT? Seems like ultimately, parameter count SALT is larger than SLDS (Table 1). Closed-form parameter estimation is nice of course, but if the approximate methods for SLDS worked well enough on this dataset, when should a practitioner think that SALT is preferrable, if ever? ### Missing comparison to AR-HMM with L2-regularized coefs The paper's story suggests that AR-HMM overfitting is the major reason a new model is needed. However, a natural way to prevent overfitting (that preserves closed-form EM parameter estimation) is to provide L2-regularization of (some) AR coefficients. As best as I can tell, the results shown emphasize maximum likelihood estimation, without regularization. While SALT is elegant, why should a practitioner invest in SALT and its more complex parameterization over a well-studied way to prevent overfitting? ### Hyperparameter settings need to be clarified **Update after response**: The author response resolved this concern. Text below from original maintained for posterity. In my understanding, SALT needs the user to select D, H, and L as hyperparameters. In the main paper, I found it difficult to understand which settings were used in several cases and how those were determined. My concern is that their SALT model could be using substantially higher D/H/L values than alternatives. There needs to be a transparent way these hyperparameters are selected on each dataset. I'd suggest in Fig 3 caption, reporting N/D/H/L for both racetrack and Lorenz data. Similarly with Fig 4 for mouse data and Fig 5 for worm neural activity. ### Missing some related work **Update after response**: The author response resolved this concern. Text below from original maintained for posterity. I'd suggest the authors consider a conceptual comparison to deep switching auto-regressive factorization (DSARF), published by Farnoosh et al at AAAI '21. Like SALT, DSARF produces discrete segmentations and continuous latent trajectories for a time-series. SALT's factorization is conceptually distinct and has closed-form estimation, while DSARF requires non-conjugate inference via VI with Monte Carlo approximations of gradients. However, DSARF could be more flexible.

Questions

1. How can we use Prop. 1 to decide which D value is optimal? Caption of Fig 2 implies we can use Prop. 1 to pick/rank D for SALT given a known LDS model. But doesn't Prop 1 as stated just provide a statement on error when SALT's D is set to one specific value D = n + 2m? 2. Seems like SALT has more hyperparameters (H, L, and D) to select than either AR-HMM or SLDS (each requires 2 of the 3). Are there good rules of thumb to avoid the expense of 3-dimensional grid search? 3. Can you report typical runtimes on the large datasets you study? Not just for training, but also hyperparameter selection as well.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Sec. 6 is missing entirely a proper paragraph discussing limitations of the work at the end of the paper.

Summary

This paper proposes a new time-series model called Switching Autoregressive Low-rank Tensor Model (SALT) that combines the advantages of autoregressive hidden Markov models and switching linear dynamical systems while addressing their weaknesses. SALT allows for longer range dependencies without overfitting and has been proven to be effective in various prediction tasks. The paper also explains the low-rank factorization used in SALT parameterization and provides experimental results demonstrating the effectiveness of SALT in various real-world applications.

Strengths

1. Novelty: The paper proposes a new time-series model, SALT, that combines the advantages of two existing models while addressing their weaknesses. This is a novel approach that has not been explored before. 2. Clarity: The paper is well-written and easy to understand, even for readers who are not experts in the field. The authors provide clear explanations of the model and its components, as well as the experimental setup and results. 3. Empirical evaluation: The paper provides empirical evidence of the effectiveness of SALT in various real-world applications, including neural and behavioral time series. The authors compare SALT to other commonly used time-series models and demonstrate its superior performance. 4. Reproducibility: The authors provide code and data to facilitate reproducibility of their experiments. This is important for other researchers who want to build on this work or apply SALT to their own datasets. 5. Generalizability: The authors demonstrate the effectiveness of SALT in various real-world applications, suggesting that it is a generalizable model that can be applied to a wide range of time-series data.

Weaknesses

1. Lack of theoretical analysis: While the paper provides empirical evidence of the effectiveness of SALT, it does not provide a detailed theoretical analysis of the model. This may limit the understanding of the model's properties and limitations. 2. Limited comparison to state-of-the-art models: While the paper compares SALT to other commonly used time-series models, it does not compare it to the most recent state-of-the-art models. This may limit the understanding of how SALT compares to the best-performing models in the field. 3. Limited discussion of limitations: The paper does not provide a detailed discussion of the limitations of SALT. This may limit the understanding of the situations in which SALT may not be the best choice for modeling time-series data.

Questions

1. How generalizable are the results of this paper to other types of time-series data and applications? 2. How robust are the results of this paper to variations in the experimental setup, such as the choice of evaluation metrics or hyperparameters? 3. How does the complexity of SALT compare to other time-series models, and what are the implications of this for its practical use? 4. How does the choice of low-rank tensor regression in SALT affect its performance, and how might other regression methods be more appropriate for certain types of data? 5. What are the potential ethical implications of using SALT or other time-series models for analyzing sensitive or personal data, and how can these be addressed?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1. Limited scope: The paper focuses on a specific type of time-series data and does not explore the applicability of SALT to other types of time-series data. This may limit the generalizability of the model to other domains. 2. Limited sample size: While the paper provides empirical evidence of the effectiveness of SALT, the sample size of the experiments is relatively small. This may limit the generalizability of the results to larger datasets. 3. Limited evaluation metrics: The paper primarily evaluates the effectiveness of SALT using prediction accuracy metrics. While these are important metrics, they may not capture all aspects of the model's performance, such as its ability to capture complex temporal dependencies. 4. Limited discussion of hyperparameters: The paper does not provide a detailed discussion of the hyperparameters used in the experiments. This may limit the understanding of how sensitive the model's performance is to the choice of hyperparameters. 5. Limited discussion of computational complexity: While the paper briefly mentions the computational complexity of SALT, it does not provide a detailed analysis of the model's computational requirements. This may limit the understanding of the practical implications of using SALT in real-world scenarios with large datasets.

Summary

The paper proposes Switching Autoregressive Low-Rank Tensor (SALT) models, a variant of an Autoregressive Hidden Markov Model (ARHMM) in which the model’s temporal dynamics are captured by a low-rank tensor approximation, thus combining the parameter efficiency of Switching Linear Dynamical Systems (SLDS) with the simple learning and inference techniques available for ARHMMs. The paper further shows that a SALT model can approximate a stable LDS model to a degree that depends only on the eigenspectrum of the LDS model and the order of the SALT model.

Strengths

- The paper makes two main contributions: (1) a parametrization of ARHMM dynamics with a low-rank Tucker decomposition; and (2) a theoretical analysis of the resulting model’s approximation error relative to a stable LDS. The latter is an interesting theoretical insight and includes a rigorous proof in Appendix B. - The background section (Section 2) provides a useful recap of (switching) autoregressive models and (switching) linear dynamical systems that motivates the need for parameter-efficient architectures with tractable learning and inference algorithms. - The experiments (Section 5) confirm that SALT models can learn (S)LDS dynamics and require less training data than their ARHMM parent due their parameter-efficient representation. The experiments on real-world data demonstrate that SALT models can learn semantically meaningful filters / state representations and outperform ARHMMs in terms of test log-likelihood.

Weaknesses

- As discussed in Section 4, structural decompositions of temporal dynamics, including low-rank approximations, have been explored in a variety of different contexts. This includes the representation of the autoregressive tensor as a Tucker decomposition and, while I appreciate the differences in the graphical structure compared to these models and the theoretical insights provided in Proposition 1, the technical contribution of this paper is not particularly strong. - Although the core ideas of this paper are relatively simple, the confusing tensor notation makes the paper unnecessarily hard to follow. Uncommon operations like the $n$-mode tensor-matrix product or the $n$-mode tensor matricization must be properly defined, ideally including visual illustrations. Having to figure out which high-dimensional slices are being multiplied is a burden on the reader and diverts the focus from the underlying ideas. Where possible, I would recommend to express the model dynamics in summation notation instead of tensor operations; it would greatly improve readability. - Ultimately, the representation of the autoregressive tensor with a Tucker decomposition is a structural assumption that modulates the spectrum between flexible models prone to overfitting and robust models prone to large bias. Since SALT models cannot be more expressive than a generic ARHMM, I view them primarily as a form of implicit regularization and, as such, would have liked to see a comparison with other regularization techniques (e.g., a Bayesian treatment with strong priors). - The experimental validation compares the proposed method to two traditional time-series models (ARHMM and SLDS) but does not include any state-of-the-art baselines (e.g., based on deep variants of autoregressive or state-space models, Gaussian processes, Transformer architectures, normalizing flows, etc.). Even if the proposed method is more related to ARHMM or SLDS, it is expected that the evaluation takes other time-series architectures into account, including more competitive and more recent developments. The data is relatively simple as well, with two toy datasets consisting of (S)LDS simulations and a small number of real-world sequences (3 mice, 1 worm).

Questions

Minor comments: - It would improve the accessibility of the paper if it included graphical models of (S)LDS and (S)VAR, similar to Figure 1, so that the advantages and disadvantages of the different model classes can be analyzed in terms of their (marginalized) conditional independence assumptions. - I wish the presentation had pointed out the strong connections of the Tucker decomposition to (higher-order) PCA/SVD. In contrast to the Tucker decomposition, this is something most readers are familiar with and would have helped with the intuition of an orthogonal/unitary factor approximation. Questions: - Section 5.1: how were the random matrices sampled? - Figure 2(A): why does a rank *higher* than 7 (Tucker-SALT) and 10 (CP-SALT) lead to *worse* approximations. Should the MSE not be strictly decreasing?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

- The paper briefly mentions possible extensions but does not include an explicit discussion of limitations. - The paper does not discuss any ethical concerns related to the proposed method.

Upgrading score to an accept

Thanks to the authors for the insightful response. I appreciate the artifacts in the PDF. Fig 1: The new comparison to DSARF, which suggests at least that --- based on RMSE of reconstructions on one dataset, SALT is competitive with DSARF --- the binary segmentations indeed do look better for SALT to my eye from a quick review Tab 1: The complete listing of hyperparameter settings for all experiments --- This is helpful for reproducibility and ensuring fair comparisons across models. It relieves my original concerns about fairness. Tab 2: Runtimes --- This helps clarify that ARHMM runtime at training is surprisingly high and that SLDS is definitely more costly than SALT (esp at inference) If the paper is revised suitably as promised (and includes these new artifacts), I am now persuaded to accept. ### Remaining questions Q0: Regarding L2-regularization of coefs for the AR-HMMs, can you confirm that the experiments in the paper did use this regularization? Your response seems to indicate this, but I can't find any clear description of your regularization strategy in the paper or the supplement. Please confirm. If you did use regularization already, I suggest revision to make clear how you pick the regularization strength, etc. Q1: Can you clarify what you mean by the following? Just that selecting between MCMC vs variational vs other methods is tough? Or that even within one choice, there are other "hidden" costs like learning rates? > the inference pipeline used in SLDS is a hidden hyperparameter which dramatically impacts performance. I think a clear description of the "hidden" costs of existing SLDS methods would help improve the paper