Summary
This paper proposes Differentiable Modal Synthesis for Physical Modeling (DMSP), which is a neural-model-based method to predict the vibration of nonlinear strings. DMSP takes the physical parameters for the differentiable equation as the input, predicts the mode frequencies and AM/FM effects, and finally outputs the waveform of vibration.
Strengths
S1: This paper is among the first attempts to study neural audio synthesis using physical properties, which makes a meaningful contribution in this sub-area.
S2: Section 2 provides some background which is helpful for understanding the context of the proposed method.
Weaknesses
W1: Some method details are not introduced with sufficient clarity. In the loss section, the detailed mathematical forms for pitch loss and mode frequency loss are not explained. Is the pitch loss L2 loss, L1 loss, or in other forms? What type of regularization is used for the mode frequency loss? Also, since there are typically many mode frequencies for a linear/nonlinear system, it is unclear how many mode frequencies the mode estimator predicts. Is it a fixed number or a variable number? If it is a variable number, how did the neural model output a variable number of mode frequencies?
W2: As the authors have also discussed, the mode predictor turns out to be not functioning appropriately based on the results in Table 2 (the big gap between DMSP and DMSP-N). Since the mode predictor is a major component of the proposed system, the design of the proposed pipeline is not well-justified.
W3: It is generally unclear what advantages the proposed physics-driven sound synthesis paradigm has over neural-based generative models, such as AudioLM or diffusion models. The latter has been shown to synthesize high-quality sounds, music, and speech. Does the proposed paradigm have higher quality, lower computational overhead, greater generalizability, or better in other aspects than the latter paradigm?
W4: (Minor) There are many empty references, e.g., line 135, line 229, and wrong references, e.g. Table 1 is line 243 which should probably be Table 2. Also, in line 243, ‘the ablation of the mode information is studied..’ Shouldn’t this table be the main result? Calling it an ablation result rather than the main result may cause some confusion, especially when Table 3 shows another ablation study result.
W5: (Minor) The problem statement in 3.1 is a bit confusing to me. For example, it is unclear to me why the initial condition $u_0$, is an element of $\mathcal{U}$. $u_0$ is a function on $\Omega \times \{0\}$, whereas $\mathcal{U}$ is a set of $(x, t)$ pairs. Also, in line 138, it seems that the mapping $\mathcal{S}$ is not only a function of $\mathcal{P}$, but also a function of initial conditions. Why is the latter omitted?
Limitations
Limitations are minimally discussed in Section 5. It would be nice to extend the discussion to include the failure of the mode predictor, the computational complexity of DMSP-N, etc.