Summary
The authors introduce a new model for generating multi-track accompaniment given a lead sheet. While being strictly a two-stage method (first generate a piano arrangement from the lead sheet and *then* generate the accompaniment), the authors' contribution is focused on the 2nd stage, and they rely on existing modules for the 1st stage. Their proposed method features two key components: **first**, a VQ-VAE submodule whose aim is to learn (quantized) representations of "orchestral functions" (orchestral functions being 1-D time-series representations of a track, with each element representing the sum of notes active in each segment, more on that in weaknesses), and, **second**, a transformer-based "mixing" VAE which combines the representations of the VQ-VAE (i.e., the representations of the orchestral functions) with learnt representations of "piano reductions" of each song (derived by averaging all individual tracks) to generate latent representations which are used to synthesize the full song. A key novelty of their model is the "mixing" part of the VAE, with the decoder utilizing interleaved time-wise and track-wise cross-attention layers -- meaning, that each output "sees" information both from past frames in its own track and from other tracks. Their system compares favorably to 3 standard baselines taken as-is from prior work or reproduced.
Strengths
Overall, barring some presentation issues discussed below, this is a well-written paper that features state-of-the-art results and enough novelty to make it relevant for both the broader generative AI community (given the difficulty of generating long-form, complex music which the authors tackle with hierarchical modelling) and the more niche music-generation community. This is why I am leaning positive towards accepting the paper. I will mention some more of its strengths in detail: + The authors target a challenging task which is to generate a multi-track arrangement given a lead sheet. Even though they "only" focus on the 2nd stage of that pipeline, namely the generation of the multi-track arrangement given a piano arrangement, it is still a valid contribution. + The idea to disentangle the piano reduction and orchestral function and combine them in the way they do is interesting and novel. This also provides a nice element of controllability. + The fact that the model can be trained in "self-supervised" fashion (i.e. without manual annotations) is commendable. + The authors have done a thorough job of documenting experimental parameters (data, GPU runtime, etc.) and submit code to reproduce their experiments. + The authors have accounted for different sources of variability and provided multiple "knobs" to twist (and counteract domain mismatches) through the clever use of multiple "positional" embeddings (Fig. 2).
Weaknesses
The main weaknesses of the paper are primarily in terms of presentation, rather than methodology or evaluation. My comments are mainly meant to help the authors improve their presentation.
* Clarity of music concepts: Sometimes, the authors introduce or reference concepts which are not straightforward without domain knowledge. Examples of that are: * The orchestral function of x is introduced as a column sum of activation indicators over the different columns of X. Given that the columns of X represent MIDI pitches, this would mean that at each frame, the authors are summing the notes that are active. Yet, in p.3, l.115, they write that this indicator function "counts each note onset position as 1". Why onset only? This would mean that each note is counted once. But the equation states $x_t^k>0$, which appears to be evaluated for each frame (thus, not only at the onset of each note, but for its entire duration). Further, they state that "it [orchestral function] describes the rhythm and groove of each track". It is not at all clear to me why this orchestral function should describe the rhythm? It only appears to describe the amount of notes active at a given time. It's even less clear why it should describe the groove, given the subjective nature of "groove". It is okay that the authors attempt to give a layman's explanation to their method in this part, but it is also recommended that they better describe those concepts (and do so in a way that would be acceptable even to those that are not familiar with concepts from music theory).
* There are other terms, such as "counterpoint", "phrase", etc. which will not be evident in an audience not familiar with them. While it is understandable that in a music generation paper, there will be a lot of domain language, it would be useful if the authors commented on their importance in their discussion of results. For example, the ILS metric measures the similarity within a music phrase vs the rest of the song. What does it mean that the authors' method does better than the baselines in this metric? That it produces more coherent phrases that are more easily distinguishable from the rest of the song? Adding a sentence to clarify the importance of each result (in, as far as this is possible, layman's terms) would improve the readability and outreach of the paper
* Comparability to baselines: The authors compare to three established baselines. However, with the exception of PopMAG, which the authors reproduce on part of the data they used to train their own model, the other two baselines are not strictly comparable. AMT requires ground-truth accompaniment (arguably a limitation of that method compared to the authors'), and GETMusic was trained on a different data (and additionally handles different instrumental tracks). While this is not a no-go, and is common practice for generative methods in general, it should be mentioned in 5.2.
* The description of *interleaving* layers in the autoencoder -- according to the authors themselves, a major novelty of the work -- does not feature as prominently in their text as it should (given its stated importance), and could be improved in terms of clarity. Specifically, it is not entirely clear what the "Track Encoder" does in Fig. 2. Is it a standard residual "transformer" (it is not clear from A.2 if a residual connection is there) layer that essentially combines information across tokens? It would be beneficial to describe this in 3.3 using simple equations like in 3.2. One important open question is why this track encoder, if it is indeed based on self-attention, is different than the decoder? Specifically, why does one integrate information in the time-axis and the other on the track-axis? It would be important to show the dimensions here. I guess you have: $(B, K, N, P)$, where B is the batch size, K the number of tracks, N the duration of each note, and P the number of notes (I guess N&P are eventually downsampled from their original dimensions). So the inter-track encoder would operate on tensors $(B\cdot N, K, P)$ and then the intra-track decoder would operate on tensors $(B\cdot K, N, P)$? All this of course would be done in autoregressive fashion, so it would go from $\{1...N\}$. It would be nice to show this using the notation of the authors. * The VQ-VAE is described (A.1) and portrayed (Fig. 1) as having a convolutional encoder with a stride of 4 (thus a downsampling of 4) and a fully-connected decoder. It would be useful to show how this fully-connected decoder mitigates the downsampling of the encoder to have an output size equal to the input. Minor comments: + p.7, l.240: Best substitute "denoising process" with denoising diffusion probabilistic model (if this is what this meant), as the term denoising can be ambiguous