TimewarpVAE: Simultaneous Time-Warping and Representation Learning of Trajectories

## Introduction The authors would like to thank reviewer joWW for this review. We have added the reviewer's suggested related works "TrajVAE: A Variational AutoEncoder model for trajectory generation" and "Trajectory VAE for Multi-Modal Imitation" to our Related Work section and explain how our work is different from those works. Those works incorporate timing and spatial information together into a combined latent space, while our approach is able to learn a separate embedding of spatial and timing vectors. Additionally, we have improved the manuscript now to clarify the questions mentioned by the reviewer, that the trajectories are all sampled to length T=200, and to provide some intuition for the regularization term $\lambda$. ## Is it possible to pass in the input trajectories as sequences (with some temporal information appended alongside them) like you would for a Transformer or RNN? The TrajectoryVAE formulation of an encoder, decoder, and timewarper could easily accommodate a variety of different neural network architectures, including RNNs or Transformers. We would still be able to separate the latent space into timing and spatial components in the same way. The key is to have two separate encoders (the spatial encoder and the timing encoder), and then the decoder would generate the sequence using the spatial latent vector and using timestamps that have been warped by the time warper. We have tried a Transformer decoder architecture, for example, with similar results, but the model training time was too long to do an exhaustive analysis. ## If good reconstruction would include both the spatial reconstruction and temporal reconstruction, how does the model perform? This is an interesting question. One downside to our model is that in the current implementation, we use a very large latent space for the time warping latent variables. This is because our focus was on compressing the spatial latent space to lead to good spatial reconstructions. We are able to fairly compare to other models with small latent spaces because we do not use the temporal information at all in our TimewarpVAE evaluations to generate the reconstructed trajectories and to compute our spatial reconstruction error. However, to do a fair comparison to measure both spatial reconstruction and temporal reconstruction between our model and other models which combine together the timing and spatial latent variables into a single latent space (like the TrajVAE and Trajectory VAE for Multi-Modal Imitation), we'd need to count the size of the timing latent space as well in order to be fair. In future work we plan to reduce the size of our timing latent space into a more compressed space (like we already do for the spatial information). Our intuition is that this will lead to an interpretable small latent space with separate variables for timing and separate variables for spatial information. However, our current approach is not designed to address this particular problem of "How small can the sum of the sizes of the timing latent space and the spatial latent space be?" Given the nuances of this question, we leave it to future work. ## Do all the training trajectories have the same length T? Yes, we have now added this clarification to the paper. ## What is the intuition of lambda, e.g., what does it mean when having small lambda vs big lambda? We have now added more insight to lambda in the paper. Since $\lambda$ penalizes the timewarping in the model, a large $\lambda \to \infty$ would drive time-warping to be the identity function (meaning the spatial latent space would have to dedicate some of its modeling capacity to also account for timing differences between trajectories), and $\lambda=0$ could allow the model to learn severe time warps (which is a problem we see in some models, where the time warping module learns to completely skip over parts of the template trajectories). ## These papers may be related "TrajVAE: A Variational AutoEncoder model for trajectory generation" and "Trajectory VAE for Multi-Modal Imitation" We have now added these papers to our related work section. The fundamental distinction between those papers and ours lies in their utilization of a combined latent space encompassing both timing and spatial variations, whereas our TimewarpVAE approach separates information about timing and spatial information into different latent spaces. ## Conclusion We again thank this reviewer for their questions, which have allowed us to modify the paper to clarify some details we previously missed and to strengthen our related work section.

## Introduction We thank reviewer qQMc for drawing our attention to these related works, especially the interesting paper "Rate-invariant autoencoding of time-series". We agree that this paper tackles the same problem addressed in our work, so we have updated our manuscript to clarify the differences in our approaches, we have added that method as a quantitative and qualitative baseline, and we have additionally provided some insight into why our approach is better at generating trajectories for execution on a robot. Additionally, we have now been able to provide an additional theoretical contribution by showing that the "Regularization-free Diffeomorphic Temporal Alignment Nets" did not properly consider the time warping degeneracy we address with our time warping regularization term. ## Rate-invariant autoencoding of time-series For "Rate-invariant autoencoding of time-series" (RIA), the primary differences are: 1) RIA is restricted to generating a canonical trajectory that is piecewise linear. RIA's canonical trajectory (the output of the convolutional neural network) gives the position at pre-specified time steps. Linear interpolation is used during warping to look up positions from the canonical trajectory. By contrast, our method can parameterize the template trajectory as an arbitrary neural network function of the canonical timestamp. This is especially important when combined with the second difference below. 2) The second difference is that RIA does not contain any regularization on the time warping function. This can allow the canonical trajectory to be arbitrarily warped compared to the data trajectories. 3) For completeness: additionally, we use a VAE instead of a simple autoencoder, and, related to point 1 above, our time warping mechanism happens before trajectory decoding, while in their approach it happens after trajectory decoding. Together, as we have updated our paper to show, these first two differences mean that RIA can easily learn to skip parts of the canonical trajectory during learning. That is, without the time warping regularization, it's easier for RIA to have severe warping, skipping large pieces of the canonical trajectory. Likewise, once RIA has learned to skip a portion of the canonical trajectory, the use of linear interpolation means that no gradient information will flow to the skipped parts of the canonical trajectory, so RIA will not learn to fix that part of the trajectory. Our approach does not have that issue. This comparison can now be seen qualitatively in Figure 1 of our updated paper and quantitatively in our updated results section. ## "Diffeomorphic Temporal Alignment Nets" and "Regularization-free Diffeomorphic Temporal Alignment Nets" Additionally, we would like to thank the reviewer for drawing our attention to the interesting papers "Diffeomorphic Temporal Alignment Nets" and "Regularization-free Diffeomorphic Temporal Alignment Nets". We found it especially interesting to consider the claim in the second paper that any regularization of time warping is not needed, since that argument presented there would also translate to our work. We find, however, that "Regularization-free Diffeomorphic Temporal Alignment Nets" does not properly consider the degeneracy of the overall choice of timing for the canonical trajectory (which we address in Section 4.5). To better align our contributions with the literature, we have additionally added Appendix A.3, in which we prove that this timing degeneracy also applies to the Inverse Consistency Averaging Error proposed in that work, a degeneracy which could be solved by adding a time warping regularization term like ours. ## "SrvfRegNet: Elastic Function Registration Using Deep Neural Networks" "SrvfRegNet: Elastic Function Registration Using Deep Neural Networks" gives another approach to properly handling the reconstruction loss when comparing warped trajectories (performing the type of change of variable analysis we perform in our Appendix A.2), we note that "adding a constant to a function does not change its SRVF". Though we mean-center our overall training set of trajectories, individual trajectories will have different mean positions, and we want our model to be able to differentiate the spatial position of trajectories, as that is important when executing a trajectory on a robot. ## Conclusion The authors would again like to thank reviewer qQMc for this review, as it has significantly improved this manuscript, both in terms of better baseline comparisons of our method, and also allowing a clearer theoretical contribution on the importance of the time warper regularization term.

## Introduction The authors would like to thank reviewer ui6D for their comments. We have now added Figure 10 to explicitly show the effect of the timewarper parameters on the generated trajectory, we have clarified that we preprocessed the quaternion representations to ensure they are near each other in $R^4$, and we have now differentiated our work from "Dynamic movement primitives in latent space of time-dependent variational autoencoders" in our Related Work section. ## Can TimewarpVAE show advantageous performance on data involving many human limbs? We do not think our model would necessarily suffer from high dimensional input spaces. The encoder already takes in a very high-dimensional input (on the size of 200 timesteps times 7 DOF = 1400 dimensions). There might need to be some optimization of the decoder network architecture to generate high dimensional outputs. To the point about the TimewarpVAE model working better for lower-dimensional _latent_ spaces (different from the dimensionality of the _input_ and _output_ spaces), we believe this reflects the fact that for larger latent spaces many competing approaches, like PCA, already work well, so there is less room for improvement for our approach. A key question would be in the type of motion. For trajectories of walking, running, dancing, or acrobatics, for example, where multiple limbs are all moving in a coordinated fashion (but possibly with different timings) we believe that the variations in the spatial styles of movements could be encoded into a low-dimensional spatial latent space. However, arbitrary uncoordinated movements would likely be very hard to model into a low-dimensional space, as uncorrelated movements of different body parts would require a latent space that has size increasing exponentially with the number of limbs. ## Can we have more direct evidence showing the spatial latent variable and canonical time index can independently control the variations of trajectories along the expected dimensions. We have now added Figure 10 to show explicitly that the timing latent only affects the timing of the trajectory. Although intuitively, based on the architecture diagram in Figure 2, we know that the temporal latent variables $\Theta$ can only affect the time warping and can therefore only affect the timing of the generated trajectory, we agree that it is helpful to show this explicitly in a figure in the paper. ## The fork dataset has part as the scaled quaternion while the error appears to be evaluated in the sense of Euclidean. Will this impact training and evaluation? We have now clarified in the paper the important point that we did preprocess the quaternion representations before feeding them to the model to ensure that they are near each other in $\mathbb R^4$. This could have been a problem since (q,x,y,z) and (-q,-x,-y,-z) represent the same orientation. Because the quaternions are near each other, we did not notice issues with using the Euclidean metric. In general, we believe this is because our reconstructions were close enough to the correct rotations that the Euclidean metric was a good proxy for the difference in orientation between the quaternions. ## Can TimewarpVAE work with multiple data modalities, e.g. handwriting trajectories for all alphabetical letters? Will it need prior other than isotropic Gaussian? Will the identification of time variation help us to have a more structured latent space to group each data modal? This is a very interesting question. For the purpose of this paper, we were primarily interested in building a generative model that could be used by a robot to create new trajectories. We agree with the reviewer that if we were to build a classification model, it may be beneficial to have a more structured prior other than an isotropic Gaussian in order to beat current state-of-the-art. ## How TimewarpVAE is related to other generative models with dynamical latent space, such as VAE-DMP [a] ? We have now added this work to our related work section. The main difference between our work and that work is in the meaning encoded by the values in the latent space. In VAE-DMP, an individual latent vector represents a single state, and a trajectory is formed by a sequence of different states in the latent space. So, that work is able to compress information about individual states into the latent space. Our work, by contrast, compresses information about the whole trajectory into the latent vector. So one point in our latent space can generate a full canonical trajectory (consisting of 200 different positions in the trajectory). ## Conclusion The authors would again like to thank this reviewer for their questions and especially for the revisions we were inspired to make to our paper because of them, including adding further details on our data processing, and adding an important figure to demonstrate the effect of the time warping latent variables.

## Introduction The authors would like to thank reviewer sdX1 for their helpful review. Based on their feedback we have now added an empirical comparison to Dynamic Movement Primitives and have shown that our method outperforms Parametric DMPs on this task. We note that the purpose of the fork experiment is to contextualize this work by learning a generative model that robots could use to execute useful trajectories, for example picking up food off of a plate to assist someone to eat. ## A bit too much background given on the beta-VAE, which is not novel We have condensed the introductory section on beta-VAE and moved the figure of the beta-VAE architecture along with its optimization objective to Appendix A.1. ## DMP style dimensionality reduction Dynamic Movement Primitives (DMPs) are able to write trajectories in terms of control parameters. This is very useful for changing trajectories during execution and for motion feedback in changing environments. However, we find that building a latent variable model on those control parameters does not build a positional model of a set of trajectories as accurately as when the model is built directly on the positional data (instead of on DMP control parameters). We have added this quantitative comparison to our paper in Figure 5. ## Classification Classification is an interesting challenge. In this work, the primary focus and consideration was on building an accurate generative model to create trajectories that could be executed on a real robot. The general architecture of VAE provides us this ability to generate new trajectories, and since this is our primary goal, we compare the reconstruction accuracy of the generated trajectories. For classification accuracy, it would likely be beneficial to use a model that is tailored specifically to handle discrete classes of data. As mentioned by reviewer ui6D, training on multiple classes would likely benefit from a generative model with a more structured prior in order to outperform state of the art. ## Conclusion We would like to thank the reviewer for this helpful review, as it has inspired us to add a quantitative comparison to Parametric DMP, showing that our approach outperforms this baseline method.