Summary
This paper uses stochastic optimization to estimate the covariance of Kalman filters in supervised tracking problems. They find that online stochastic optimization in this manner tends to generalize better than either neural Kalman filters with LSTM base models or the older plug-in estimators for Kalman filter problems. Their argument for the outperformance of stochastic optimization in these problems relies heavily on the fact that the assumptions for the plug-in estimators ends up being violated in some manner, producing at least some amount of model mis-specification. The experiments cover a bunch of simulated data where the assumptions of the non-optimized Kalman filter can be tested rigorously, as well as less toy video tracking and lidar state estimation experiments.
After the authors rebuttal I'm a bit happier about this paper. Thanks for showing the value of the Cholesky parameterization and answering the rest of my questions.
Strengths
Originality
- I don’t have a great background in the Kalman filtering literature so I find it difficult to fairly judge originality. However, based on a cursory reading of the other literature, it seems to be a quite original work.
Quality
- Overall the paper seems to be of pretty high quality experiment-wise and technically. See typos at bottom.
Clarity
- The assumptions are spelled out pretty well, especially as each experiment tends to have a different set of assumptions.
Significance
- This paper ends up being yet another demonstration of “stochastic optimization tends to work better in practice than in theory” which seems certainly like a good addition to the ML community, even if it comes from a place that is a bit outside.
Indeed, appendix b.2 shows that stochastic optimization is a bit more robust to model mis-specification in what you describe, as you show an example where adding more data definitely makes the error worse. Honestly, you could move this into the main text with some tigher writing.
- Overall, between the authors’ software and the simplicity of the approach, it also seems like a fairly reasonable contribution to the system identification literature.
Weaknesses
Originality
- From my limited knowledge here, I do think that the Bayesians have been estimating parameters in a similar manner for GPs in state identification for a while. See for example [1], [2]
Quality
- A lack of comparison to non-Kalman filter benchmarks on some of these tasks limits the relevance to the ML community – perhaps physics NNs or even a direct LSTM would be a good baseline.
Clarity
- I think that I am a little confused as to the broader application of the Kalman filter in this type of supervised learning setting however. The motivation for this specific subset (supervised like training estimation) could be explained a bit better.
- The implications of Assumption 1 should probably be explained a bit better right after the assumption and theorem is sketched out. Theorem 1 is also not really well written or formally stated – please either write it more formally or just drop the theorem heading.
Significance
- Touched on a bit above, but I am a little puzzled by the when is the supervised estimation Kalman filtering problem actually used? From what I can tell all of the experiments are very distinguished from reality and simplified.
- At the end of the day, the approach is really just covariance matrix estimation for trajectories, which seems like it has been done before in things like [3], [4]
References:
[1] Reece and Roberts, 2010. An Introduction to Gaussian Processes for the Kalman Filter Expert
[2] Deisenroth, Huber, & Hanebeck, ICML, 2009. Analytic moment-based Gaussian process filtering.
[3] Xiao and Wu, Ann. Stat, 2012. Covariance matrix estimation for stationary time series.
[4] Wang et al, Neurocomputing, 2017. An adaptive Kalman filter estimating noise covariance.
Questions
When Assumption 1 is not violated, does the stochastic optimization approach converge to the same solution (just at a slower rate)?
You may wish to break Assumption 1 into four separate assumptions – stationarity, i.id-ness, known covariances, and good initial states.
This would probably cleanup some of the writing in the experiments by being able to which specific piece of assumption 1.
Could the neural Kalman filter be combined with the optimized Kalman filter with both being optimized jointly?
Could the authors run an ablation where the L Cholesky factor is parameterized diagonally? What is the impact of the full parameterization?
I believe that this should become much less well-specified and harder to optimize.
Is the formulated stochastic optimization problem still convex or at least reachable to a global optimum?
Why can we not map observations from cartesian to polar coordinates in the lidar and doppler experiments, thereby producing observations that are much less violating of assumption 1.
Section 4:
In the doppler radar problem, could another benchmark (either a physics neural net or a neural ode) be used for the trajectory problem as well?
typos / writing:
L357: studied -> studies
Section 5: use a real name for section heading – e.g. “Comparing Optimized KF to KF”
Personally, I would probably combine sections 4 and 5 into a single section with the same sub-headings: experiments. Currently, the headings seem a bit informal.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
I think motion tracking in crowds may not be a great application socially, but otherwise no other societal impacts.