A Functional Extension of Semi-Structured Networks

Summary

This paper extends the application of semi-structured networks to functional data. The orthogonalization technique makes this method more scalable than existing methods.

Strengths

This paper is well-structured and well-written. The idea is easy to follow despite the experiment being conducted in an abstract functional space. The authors have conducted extensive experiments to show the new method's applicability and generalizability.

Weaknesses

The authors emphasize the interpretability of the proposed method, but they have not demonstrated this property in their experiments. Besides, the FFR $\lambda^+$ can only explain the linear part of the target function $y$, but most of the time, we hope to explain the nonlinear interaction in the neural network.

Questions

Q1: The authors claim that FFR can serve as "a functional version of a residual connection "(lines 141-142) and show that the FFR component can empower FNN for better model performance, as shown in Figure 5. Why do we not add the residual design to the FNN directly? Comparing the model performance between FNN, FNN with residual network design, and FNN+FFR in Figure 5 could answer this question. Q2: Under the discretization setting, what are the differences between time series modeling and functional networks?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The limitations are well discussed, but the authors have not stated the GPU specification in Appendix D.2

Summary

In this paper, the authors develop a semi-structured model for functional data, summing an interpretable linear model with a more general nonlinear functional neural network. The authors validate the improved performance of the combination relative to the individual components on a variety of biomechanics datasets. The authors also discuss how to preserve the interpretability of the semi-structured model with a linear weight surface by projecting the neural network predictions into the subspace of the linear model.

Strengths

The approach provides a natural adaptation of semi-structured models to functional data and is worthy of detailed investigation. Experiments performed on synthetic and real data are reasonably thorough and help validate the claims on efficacy and interpretability. The ability to retain the linear model interpretability using the projection is interesting and specific to the setting.

Weaknesses

The paper accomplishes what it sets out to do fairly well, though the contributions of the paper through providing evidence for new phenomena, novel insights, or new algorithmic ideas seem much more limited. In terms of the structured part of the model, the linear regression with the functional encoder and decoder bases for function -> function regression, I assume that this approach has been used before but I did not see it in any of the explicit references. Can the authors comment on whether this component is original to their work or whether this component is building on what already exists in the literature?

Questions

One aspect I did not see mentioned in the paper was the actual choice of basis functions for encoder and decoder. These should certainly be specified along with any justification or tuning of hyperparameters involved (e.g. bandwidth parameters?). Fig2a and 2b present two different potential options for constructing this semi-structured model, might it be worth evaluating the performance of structure a)? Would we expect the performance to be similar, what are the drawbacks? When fitting the parameters of the linear model, it is mentioned the large computational cost. I did not find the explanation of how the scalable implementation circumvents this to be particularly clear. Also, even for solving the linear system could one not use more sophisticated matrix free methods such conjugate gradients with the kronecker product matrix, or for example SVRG which would converge quickly even when the number of data points and features is large?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

I think the limitations are sufficiently discussed.

Summary

An extension to semi structured networks is introduced that combines a traditional linear and interpretable model component $\lambda^+$ with a flexible $\lambda^-$ neural network component to better approximate relations in biomechanical application. The model performs equally to traditional methods but requires less memory and remains interpretable thanks to a post-hoc orthoganalization.

Strengths

_Originality:_ The introduced method nicely combines established approaches and compares it to some, but not to all relevant related articles (see details under weaknesses). _Quality:_ The submission is technically sound in that most claims are well supported by experimental results; some exceptions are listed under weaknesses. The paper is nicely structured but, for my taste, lacks an intuitive examples and explanations to nudge and inform readers that are unfamiliar with the topic. An exhaustive number of experiments is conducted on several problems, both on synthetic and real-world data, allowing a quantitative assessment of the introduced method. _Clarity:_ Although authors could add more intuition about the effect and function of certain (mathematical) concepts, the manuscript is well organized and clearly written. In particular, the authors very nicely outline their hypothesis and refer to them in the results section. Apart from some aspects detailed in the weaknesses, the manuscript informs the reader adequately. The submission is complemented with code, yet the data to reproduce experiments is lacking (due to space constraints). _Further comments_: - Figure 2 nicely illustrates the two proposed implementation options for hybrid models. - Figure 3 is helpful, but I find it hard to understand entirely. What exactly does the x-axis represent and are the values normalized to [0, 100] or what is the data range and what does it represent? The last sentence in the caption is not clear to me, i.e., when stating "... early time points $s$". This is probably formulated suboptimally, as $s$ is reported to relate to a sensor signal and $t$ to time.

Weaknesses

_Significance_: Even though the authors emphasize their method's superiority in lines 144-146, the performance improvement over existing approaches is rather mediocre and does not seem significant, in particular when considering the comparably large standard deviations. The error distributions of the different methods seems to overlap to large extend. In this vein, it would be appreciated if error bars were reported in Figure 5. In that figure, you may consider to flip the y-axis to make it more intuitive and report the relative error reduction over the baseline. _Further comments_: - It took me very long to get a grasp of the concept of functional data/analysis/regression. In fact, until the very end of the manuscript, I did not fully understand the actual task at hand and permanently felt somewhat lost. Authors may give a quick practical example of functional analysis early to catch readers sooner. Even the example in lines 109--123 did not resolve my uncertainty about where such processes find application. I thus suggest to add one clear and illustrative practical example and spend more effort to introduce the problem at the beginning of the introduction rather than starting with related work. - While novelty for functional data analysis seems reasonable, I cannot find a substantial contribution to the machine and deep learning community (given the paper is submitted to a top tier ML conference). Even though I like the paper's argumentation and substantiation and perceive it as a generally sound work, the manuscript might better be suited to a journal that deals with functional analysis or with biomechanics. - Claim in l265 not supported with experimental results. Please provide error and dispersion metrics (such as in Table 1 and 2) to allow a quantitative comparison of the methods. - Typo in line 331: Either "reveal that" or "yields"?

Questions

1. What are the additive structures you are referring to in l15, can you provide some examples? 2. What is the proportional contribution of $\lambda^+$ and $\lambda^-$ to the overall solution? How is guaranteed that $\lambda^-$ is not solving the entire problem solely, thus loosing all interpretability? The orthogonal formulation seems to take a step into that direction, but I am not sure what the orthogonal formulation actually does, and if the patterns revealed after its application are helpful and realistic. 3. What are the integration weights $\Xi$ concretely in line 182 and what are they required for (can they be dropped for simplicity yielding a conventional loss function)? 4. The orthogonalization approach in Section 3.3 appears very beneficial, yet, frankly, I could not follow it as the math is quite abstract. Can you share an intuition what the orthogonalization implements and how it effects $\lambda^-$ to be less dominant, if so? Is it some kind of regularization to minimize its contribution (or how would, e.g., L2 regularization compare)? 5. Relating to Figure 4, how does runtime compare for the three implementations for $\lambda^-$? Given the memory is capped for the batched neural network, it should take multiple evaluations to process the same amount of data. How does the memory compare if the other methods are implemented in a batched fashion too? 6. Can you share an intuition of why the neural network performs worse in high SNR regimes? I'd expect a neural network be superior in all conditions, given it is sufficiently large and regularized appropriately. 7. Could you include the results from reference [24] into your results in Section 4.2.1 to allow a comparison between your and an established method? It is crucial to assess how your approach performs in comparison to existing methods.

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

1

Limitations

Limitations are addressed by the authors.

Summary

The paper proposes a hybrid approach that combines the benefits of neural networks with those of more structured models (generalised additive models). They show benefits on real and simulated data.

Strengths

- The proposed idea makes sense and is novel AFAIK - The empirical results show good performance compared to either purely deep models or purely structured models - The description of the method in sections 3.1 to 3.4 is clear.

Weaknesses

- How much tuning has been done for the pure deep-only network baseline from Section 4.2.2? I would be a bit worried that the findings are very dependent on the choice of architecture of this model? E.g. does it have any increased capacity over the deep part of the semi-structured network? And does this architecture have suitable inductive biases for fitting linear functions (maybe residual connections which add a linear transformation of the input to the output)? - The presentation could be made clearer in a couple of areas. In particular, (1) Equation 3 is the first time that the “double function call” notation of e.g. $\lambda^-(X)(t)$ is used. Having a double function call makes sense given that there is a function which takes X as input and outputs another function. However it is not clear to me why this is then not used in Equation 1, where you could presumably replace $\mu(t)$ with $\mu(X)(t)$? And e.g. in the last part of Section 2.2, why write $h^{(L)}(t)$ instead of $h^{(L)}(X)(t)$? Would it be possible to make this clearer? And also (2), when functional data analysis is introduced in Section 1 and 2, it would be helpful if an example application was described

Questions

See weaknesses. I am willing to increase my score if these are addressed.

Rating

5

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

Adequately addressed

Major concerns resolved

Thanks for your efforts in providing additional results, in particular verifying study [24] in your experimental setup, and for justifying your study's suitability to NeurIPS. I must admit that I am not an expert of this field and still have difficulties in assessing the relevance, but your arguments appear sound to me. The additional results provided in your rebuttal are convincing. They help me to further understand where your model is superior; A thorough interpretation and discussion of these results would be helpful for readers. That is, for what reason does you semi-structured approach boost performance in some cases but not in others? Assuming the explanations and results from the rebuttal are carefully woven into the manuscript, I am raising my score from 4 to 6 but lower my confidence further from 3 to 2, as I do not feel qualified to provide a strong and well justified assessment of this work. Good luck!