Summary
Recently, diffusion models have been released that can generate high quality video clips, indicating that they strongly capture natural appearance and motion in video. This paper seeks to adapt one recent model, Stable Video Diffusion (SVD), to explore if what it has learned can be leveraged to improve video recognition performance, and investigate whether co-training on generation and recognition using this model can improve on both tasks.
The two tasks, video diffusion and recognition, have quite different setups: the first takes uncorrupted video as input, while the second starts with a single conditioning frame as well as noise-corrupted video. This paper proposes to unify the tasks, adapting the SVD model to take masked-uncorrupted video plus noised video, concatenated, as input. This unifies the tasks, allowing joint training on recognition and generation. It also permits new tasks: generation and recognition using a variety of partially-masked video configurations as input.
The paper shows competitive results on classic video clip classification tasks, and very strong results on generation FVD. Further they demonstrate strong classification performance on masked input scenarios including early action prediction, and recognition with limited frame-rates.
Strengths
This paper addresses what is to me, a researcher in video recognition and generation, a really burning question: generative video models have clearly learned an incredible amount about the visual world, so how can we harness that to advance recognition performance?
But the setups of the problems, recognition and generation via diffusion, are really quite different. The authors show how to unify them, and co-train on the two tasks, in an original way that actually produces synergies for both. This is an exciting result to me.
The paper is very clearly written and easy to follow (although the math dump in section 3.3 could be a little clear).
I appreciate the inclusion of the two strong baselines (I and II, described on line 213), that compare the unified model to models trained only on generative or only on discriminative tasks.
Weaknesses
In the recognition results, Table 1 doesn't represent the current SOTA. E.g. InternVideo (https://arxiv.org/abs/2212.03191) from 2022 claims 77.2% on SSv2 and 91.1% on K400. It's fine if GenRec doesn't beat SOTA, but SOTA should be accurately represented in the table.
The generative experimental results show a surprising improvement on FVD on SSv2 and EK-100, which I don't feel the authors do enough to explain or analyze. (See an associated "Question" below.)
There is no exploration of how different values of the generative/discriminative loss balance ratio, gamma, affects performance on the benchmarks.
Similarly, there is no exploration of different latent masking schemes or ratios during training.
Minor weaknesses:
* The method is described as "analogous to a temporal extension of MAE" in line 53. VideoMAE [38] does already exist.
* On line 61, please define the accuracy retention ratio of include a citation.
Questions
The generative results in Table 1 demonstrate a really surprising decrease in FVD compared to the other methods. Can you explain why this is the case in more detail? Is the model really that good? Is it a typo? Is it because this benchmark is not particularly competitive? (I think FVD is more often presented on UCF and Kinetics-400 in the papers I've read, e.g. https://arxiv.org/pdf/2212.05199). If you double-check the FVD result, and convince me, then I'll gladly list this surprising decrease as one of the paper's strengths.
Would it be useful to clarify to readers why the focus is "particularly the unconditioned or image-conditioned models" in line 29?