Summary
In this paper the authors propose a graph based network that combines feature extraction with test time optimization to perform two-frame particle tracing. The core problem is that given two sets of points, e.g., point cloud data, to find the correspondences between the point clouds and estimating their velocity from this correspondence. While this problem can be well analyzed in synthetic data, real world problems are often sparse in data and, accordingly, using methods that do not require labeled data (and not a lot of it) would be beneficial. The proposed methodology works by proposing an unsupervised loss formulation based on assuming smoothness and divergence-freedom of the solution space and uses parts of this loss function as a test-time target for optimization. The method is evaluated and trained on synthetic data and then evaluated on dissimilar data to show the generalization capabilities of the approach.
Weaknesses
Overall the paper has an interesting core idea, however, several issues exist, especially regarding evaluation and the lack of statistical evaluations and ablation studies in general. There are a few core weaknesses listed below in a short list. Following these a list of all potential issues and weaknesses is included for completeness at this point. Note that many of these issues are also addressed separately and are not questions in themselves.
* The authors perform no evaluation regarding the statistical significance of their proposed method. While the computations are deterministic (as they note in the checklist), this does not mean that network initialization does not play a role, especially when many of the evaluation metrics are relatively close and some results seem counter intuitive, e.g., the EPE improving for smaller datasets. Furthermore, there is no clear evaluation on the choice of the 1% of data, i.e., did the authors try different subsets? Were the 1% chosen consistently across class? Across different trained methods?, etc.
* It is not clear why the proposed DVE test-time optimization cannot be performed on top of data trained on supervised data, especially as the authors change the loss functions anyways from training to testing.
* The authors make a fairly broad claim in the introduction as to synthetic data being always limited due to hand-crafted priors but show that their method does not suffer from this problem, even though it uses the same synthetic data. This needs to be clarified.
* The divergence-free loss term is not very well evaluated. On the one hand the assumption seems to be relatively central to the approach and the authors go out of their way to state that assuming divergence-freedom can even be assumed in compressible cases (where it does not hold), but then also state that they disable the divergence-free term during test-time optimization especially for cases where it does not hold. * Either the authors need to more clearly evaluate the impact of the divergence-free loss term during test-time optimization in _all_ cases, or consider the importance of the argument.
* The formatting of the paper at times is very odd with subsubsubsubsections and randomly highlighting of statements with inconsistent (and incomplete) highlighting in tables. This should be improved.
General:
* In the paper checklist under 7 the authors note that their ‘method is deterministic’ and thus does not require error bars. However, choosing random subsets of data, training different architectures and more usually involves sources of randomness and seed variance can be quite significant. This needs to be evaluated properly and adjusted accordingly.
* Table 3 seems odd as the EPE _improves_ for the proposed method when training on smaller datasets.
* Table 9: Why is the Epoch count increased to 300 for the 1% train size and is this done consistently across all methods and what would the results be with only 100 Epochs or a normalized training with the same number of weight updates across dataset sizes?
* It is wholly unclear as to why the proposed scheme is 6 times slower during testing just because it is trained on a larger dataset.
* Why can other methods not utilize DVE as a test-time optimization process? This would make for much fairer comparisons
* The authors should make it more clear in the related work as to which graph-based feature extractor they use as their architecture is heavily informed by prior work.
Dataset:
* The authors claim that synthetic data is always limited as it leads to ‘hand-crafted priors’ but they utilize a synthetic dataset to train their method and show that it is not limited by the biases induced from the prior, i.e., their method can generalize even from synthetic data.
* The authors state that in cases of boundary conditions, or other situations, smoothness and divergence-freedom is not holding up. This should be more clearly shown, i.e., a demonstration that using these regularization terms leads to issues in such cases.
Reduced Data Training:
* The 1% setup is interesting, however, the authors do not clearly show how sensitive their method (or methods in general) are to selection, i.e., there is no clear evaluation on how the choice of the 1% affects the outcome
* There are no statistical evaluations, which would be especially important for the 1% evaluations and with the difference between methods being so narrow at times, initial seed choice might be important too
* There is no evaluation as to what the method would do if the data is incomplete, i.e., if particles are missing from one of the frames. This would be important for real-world applications but due to their reliance on hand-crafted synthetic priors and other narrow datasets, this is not evaluated clearly.
Divergence-Freedom/Loss Terms:
* The Divergence-free loss term is interesting but not using it during testing seems odd as it should be helpful in all cases. However, even the authors note that this would be a limitation in non divergence free settings (which they evaluate one of). Considering the prominence of the divergence-free claim, this needs to be further elucidated (especially considering the statement in line 674)
* The authors claim (178) that even for compressible fluids divergence-freedom can be assumed in practice. While sometimes done, this should be more clearly evaluated as the authors clearly do not use the divergence-free loss in datasets where the assumption would not hold.
* The distinctions when smoothness and divergence-freedom are not to be used for optimization are not clear as there might be reason to not use the divergence-free loss, i.e., if one expects a dataset with divergence, but not requiring smoothness is odd. This should be done in an ablation study.
* The neighborhood sizes are odd and should be more clearly written as the information in 715 is not easily understood, i.e., is the divergence-loss computed with only 2 points?
* The formulation of the divergence as a central finite difference scheme is not necessary, other information would be more prudent.
Formatting/Writing:
* The formatting is very inconsistent at times, e.g., table layouts, subsubsubsubsection headings are sometimes italicized, sometimes underlined and sometimes bold.
* The authors do not need to state that they are excluding information due to page limits, twice.
* The highlighting in Figure (mostly table) 4 is inconsistent. In the top part the second best performing method is not highlighted with underlining but also boldfaced and the T_test column has no highlighting
* Table 6 should also highlight cases where the method without divergence-freedom performs better
Missing Details:
* How were the lambda parameters chosen?
* KNN should be properly introduced (208) and evaluated as to why it is not efficient
Evaluation:
* The method performs significantly worse in some cases, e.g., the EPE is worse by a factor of 2 for the MHD case compared to the turbulent channel flow. It would be nice if this was more clearly evaluated and highlighted, especially for the Beltrami case (Figure/Table 5) where Gotflow performs worse by a factor of 2 relative to the proposed method, which is fairly close considering this case is about 8 times worse than the best case for the proposed method and only 3 times worse for Gotflow.
* The improvement of 22882 matches over 22001 seems fairly minor (315) and it would be prudent to add the tracking timing in this section in the main paper.
* How does the proposed method perform on its own (323), i.e., when it is not used as an initializer?
* The difference in EPE from 100% to 10% with and without DVE is relatively close even though the text (800) notes that the method is highly sensitive.
Questions
Note that these questions are not in order of importance.
1. What is the variance and statistical certainty of the training across different initializations for the proposed methods?
2. Why is the EPE error lower for the 1% case in Table 3?
3. What would the training results be with equal weight updates across methods, and if they do not change from the proposed hyperparameters, why were these hyperparameters chosen?
4. Is it possible to apply DVE to other state of the art methods and, if so, what would the results be?
5. How was the 1% subset of the data chosen? Does this influence the result, i.e., do different subsamplings (and strategies) result in consistent (within a small deviation) results?
6. Is there any data to highlight the limitations of including the regularization terms during test time optimization?
7. How were the lambda parameters chosen?
8. Why is the computational time of the method at test time different based on the dataset size used during training?
9. What was the total time to train a network from start to finish for the proposed method with and without the various terms?
Limitations
The primary limitations of the paper relate to the evaluation of the methodology and the divergence-free loss term. In summary there are three key issues:
1. The evaluation should include a broader evaluation that at least considers seed initialization and variance due to the 1% sampling. As the paper currently stands, it is impossible to tell, with certainty, that the results are consistent and not just due to a lucky seed. While performing such evaluations for all setups is computationally prohibitive, at least the core contributions should be evaluated
2. There is a lot of inconsistency regarding the divergence-free loss. On one hand it is argued as very important during training and how it can work even in compressible cases. Then the term is wholly disabled during test time optimization because it isn’t needed and then towards the end in a case with divergence the term is left off anyways. There is no clear evaluation as to the influence of including the divergence-free condition during test optimization, especially for cases which are compressible (and/or not divergence-free). This makes it difficult to judge how the method truly generalizes
3. While not a direct limitation, the authors themselves state that the use of synthetic data leads to hand-crafted biases (in the introduction), but they also demonstrate how their method is immune to this. As this does not seem to be a limitation of the synthetic data nature, this should be either clarified in the introduction or an actual limitation where the method does not generalize due to its limitation from hand-crafted data.
Rating: