VideoClusterNet: Self-Supervised and Adaptive Face Clustering For Videos

Response to Reviewer B82Y

We thank the reviewer for providing a thorough review of our paper. This would help us greatly in improving the quality and legibility of our manuscript. Below we address specific questions/concerns raised in the weakness section: 1. **Clarification on method novelty:** We kindly refer the reviewer to the “Joint Representation and Clustering” paragraph in Section 2 regarding the comparison of our proposed method with closely related methods. We believe that our method’s performance increase is a result of both the self-supervised (SSL) model finetuning and the final clustering modules. Empirical evidence for this fact can be observed in Section 5 (Ablation Analysis). Specifically, Table 3 and Table 4 prove the individual effectiveness of our model finetuning mechanism and clustering algorithm, respectively. Unlike closely related prior work, the model adaptation stage is more effective majorly due to the self-distillation procedure and its sole dependence on positive match pairs. Our final clustering algorithm directly benefits from the model finetuning stage by incorporating the SSL training objective as a distance metric. Such a learned metric helps boost the algorithm’s performance as it efficiently evaluates inter-track distances directly in an embedding space, which is optimized for reduction in the same identity distances. Also, any sub-optimal performance that may be induced due to any user-defined matching thresholds is avoided through the adaptive custom threshold computed for each track. Such a thresholding mechanism helps the clustering algorithm automatically adjust to how effectively the model can match a given track’s identity across a true positive match pair. We will update our paper with this detailed explanation for the reader’s enhanced understanding of our method’s salient novelty. 2. **Need for new benchmark dataset:** We kindly refer the reviewer to the second last paragraph in Section 1 (Introduction) where we detail a few primary reasons for the need for a comprehensive movie benchmark dataset for video face clustering. To justify it further, the problem we address in the paper, and its application for TV series/movies has seen comparatively lesser interest in the last few decades as opposed to other Computer Vision related tasks. Also, complicated usage and copyrights for TV series/movies meant that providing open source datasets based on them wasn’t feasible or otherwise costly and time-consuming w.r.t. providing annotations. As a result, the majority of related approaches in this domain end up evaluating their method on a very specific TV series/movie dataset while not being able to provide means to access it publicly. A direct consequence is thus not being able to have fair comparisons with prior works, forcing authors to evaluate across a large set of niche datasets. We present our dataset with this view in mind and sincerely hope that it will largely benefit the research community in this niche domain. Finally, we refer the reviewer to our response to the first reviewer (**RNxJ**)’s concern regarding dataset analysis and comparison. We also refer the reviewer to Appendix J, which summarizes the uniqueness and advantages of our proposed dataset over existing ones. We would appreciate it if you could let us know of any further questions/concerns that you may have and if any finer clarification is required for any of our responses listed above. If not, we would greatly appreciate it if the reviewer could consider increasing his/her overall rating.

Response to Reviewer wXAX [Part 1/2]

We thank the reviewer for taking the time to provide a thorough and detailed review of our paper. This would help greatly improve the quality and legibility of our manuscript. Below we address some specific questions/concerns posed by the reviewer in the weakness section: 1. **Missing details on model finetuning module:** We would first like to state that the proposed model finetuning module (details in Section 3.4) is fully self-supervised, i.e., completely independent of any manual ground truth labels. This facilitates the selection of all available tracks in a given video for the finetuning process. Also, the model finetuning hyperparameters such as training epochs, batch size, and learning rate (see Appendix A) are kept constant and are agnostic to the training dataset statistics. This removes dependence on a traditional train and validation set based mechanism for performance evaluation. These hyperparameters have been carefully tuned across a wide range of video datasets to ensure convergence of the fine-tuned model. This also helps the model learn all observed natural variations across a given set of dataset face crops. An alternate way to describe this finetuning mechanism is that in our self-supervised setup the training and validation/test set are indeed exactly the same, but where performance validation isn’t essential. Also, certain image augmentations (detailed in Appendix A.2) are applied during the finetuning process. This ensures that the model observes enhanced facial variations during its training and helps reduce its primary dependence on natural variations present in the dataset. In addition, since the model is finetuned on all face tracks that need to be clustered, it gives the model more flexibility to slightly overfit the training dataset if the track count is low. Both of these factors make the finetuning process fairly agnostic to the number of tracks and the available natural variations within a given video dataset. Table 2 shows empirical evidence of this fact wherein the track count ranges between 119 and 917 across our movie dataset. Our method manages to achieve significant performance improvements over benchmark methods across all the movies despite these track count variations. We hope that these insights help the reviewer better understand the salient novelty and merits of our proposed method. We will update future manuscript versions with this explanation accordingly. 2. **Positive track pair contribution clarification:** We thank the reviewer for mentioning the ambiguity in one of our listed paper contributions and apologize for the confusion. We would like to provide clarification about the second contribution in the introduction. Our method only relies on positive pairs in the model finetuning/adaptation stage, a unique feature from prior works. Previous methods leverage negative track pairs and incorporate either manual ground truth labels or temporal track constraints to obtain them. We have updated our second contribution accordingly to reflect this clarification. 3. **Clarification on baseline comparison with advanced pre-processing modules:** For the comparisons with baseline methods listed in Table 1, we used the exact same set of face bounding boxes and motion track information provided by the benchmarked methods for a fair comparison. We skipped the usage of our track pre-processing modules (detailed in Section 3.2) and directly processed the available data through the model finetuning and final clustering modules. Note that this data was made publicly available by (Tapaswi et al. 2019)[1] at this link: https://github.com/makarandtapaswi/BallClustering_ICCV2019. We believe this facilitates a fair evaluation of our self-supervised clustering algorithm w.r.t. baseline methods and eliminates any influence of advanced pre-processing modules, specifically shot and face detection. We kindly refer the reviewer to this fact mentioned in Section 4.1 of our paper. Similarly, for a fair comparison on our dataset in Table 2, we use our track pre-processing module to evaluate both the baselines and our method on the same set of face boxes. **Continued in Part 2/2**

Response to Reviewer wXAX [Part 2/2]

**Continued from Part 1/2** 4. **Dataset and code availability:** We would like to assure the reviewer that the proposed MovieFaceCluster dataset will be open sourced to the benefit of the wider research community and to encourage further research into this relatively niche domain. As for the concern about movie copyright issues with our dataset release, the reviewer should rest assured that we possess valid internal usage licenses for all the movies listed in our dataset. We will release the dataset in a format such that it doesn’t breach any copyright or license restrictions. In its current state, along with ground truth identity information, we will release the bounding box and motion track annotations referring to each frame in every movie. This would enable the dataset user to obtain the face crops given source media. We hope this effectively addresses the reviewer’s concern about the dataset's legal compliance. We will also work on making face crop data itself available in the next dataset version. We also confidently believe that we have provided thorough implementation details (in Appendix A) to accurately reproduce our proposed method. In addition, we will make available the re-implementation of the baseline code (accurate up to the best of our knowledge) upon the paper's publication. We would appreciate it if you could let us know of any further questions/concerns that you may have and if any finer clarification is required for any of our points listed above. If not, we would greatly appreciate it if the reviewer could consider increasing his/her overall rating. References: [1]: Makarand Tapaswi, Marc T. Law, and Sanja Fidler. Video face clustering with unknown number of clusters. In International Conference on Computer Vision (ICCV), pp. 5027–5036. IEEE, 2019.

Response to Reviewer RNxJ [Part 1/2]

We would first like to thank the reviewer for taking the time to provide a thorough and detailed review of our paper. This would greatly help us improve the quality and legibility of our manuscript. Below we provide some specific comments on questions/concerns pointed out in the weakness section: 1. **Dataset analysis and comparison:** In addition to the details on our proposed dataset presented in Section 4.3 and Appendix E, we have updated our paper to provide a comprehensive analysis and comparison of our dataset with existing literature in Appendix F. This includes a comparison of some key dataset attributes like unique character and track count, cast racial diversity, and average track face quality score (computed using our face quality estimation module presented in Section 3.6). We also present a detailed histogram comparison of the dataset face quality distribution to give more insights into each dataset’s average quality score. In addition, we provide dataset histogram comparisons across critical attributes, specifically scene lighting and facial blur. We are confident that this revision further strengthens our argument of providing a challenging, more diverse movie face clustering dataset. We sincerely thank the reviewer for his comments on this specific aspect of our paper. Also, in Figure 1 the term “varying parameter” denotes the dominant image attributes that are significant outliers for a given face crop. It isn’t part of the available dataset annotations and is used simply for enhanced reader understanding. We have added this clarification in the updated paper version in Appendix J. 2. **Comparison with MovieNet dataset:** Even though MovieNet dataset [1] comprises a wide variety of annotations on a large number of mainstream movies, we would like to kindly point out that its annotations aren’t directly useful for our task of video face clustering. It unfortunately consists of person bounding boxes (covering the entire character's body) and not face bounding boxes, which are essential for our task. Furthermore, complex post processing could be used to detect face bounding boxes in each dataset frame and match them against the ground truth person boxes, in order to associate the identity of each face. For this, the movie frame data would be essential to predict face boxes. To the best of our knowledge, neither the source movie dataset nor the links to the respective movies are available for download from https://movienet.github.io/. Due to these reasons, we strongly believe that a dataset comparison with MovieNet in its current state wouldn’t be fair for our intended task as it doesn’t contain relevant annotations for the same. We would however like to thank the reviewer for mentioning this specific dataset as it could potentially be relevant in a future extension of our work, wherein the video clustering is applied for person bounding boxes rather than just face boxes. We have updated our paper to acknowledge the same in the “Limitations and Future Works” paragraph in Section 5. 3. **Clarification for certain empirical system parameter values:** We would like to highlight here that the track quality estimation module (see Section 3.6) can be viewed as a pre-processing module for our final clustering algorithm (see Section 3.7). It is highly effective in filtering out non-identifiable face tracks that are often present in mainstream media content. Also, the coarse track matching module (see Section 3.5) is used in conjunction with the model finetuning module (see Section 3.4). We do acknowledge that both of the referenced modules contain minor empirically evaluated parameter values. However, we claim in our listed contributions (fourth contribution in the Introduction) that we specifically have a parameter-free video face clustering algorithm. Both of the aforementioned modules are essentially external to our core clustering algorithm. We would like to confidently reiterate that the core clustering algorithm is fully automated in the true sense. We would also like to provide clarification on specific values set for certain parameters in both referenced modules. For Equation 4 in the main paper, backed by empirical evidence, the value of 2.7 was first loosely set by fitting a Gaussian distribution onto the given set of track face quality scores. To select/filter the lower ~1.5% outliers, which are often less than the threshold of (mean - 2.8*std) in a Gaussian distribution, we sampled values in the range of 2.6 to 3.0 and empirically found that 2.7 worked optimally for our test set consisting of a wide range of movies. **Continued in Part 2/2**

Response to Reviewer RNxJ [Part 2/2]

**Continued from Part 1/2**: Regarding the coarse matching module, the motivation to select crop pdf values in the lower 25% range was again based on empirical evidence. To effectively select the true outliers, which would result in a more accurate computation of a given track’s custom match threshold, we experimented with values ranging from 5% to 40%. We found out that setting it to 25% provides effective true positive track matches while avoiding any false positive matches (which is a critical performance parameter) over a wide range of movie track sets. We will update this in the future revised version accordingly. 4. **Anomalous clusters in t-sne visualization:** We believe that the majority of the anomalous clusters visible in Figure 5 for the ground truth visualization are a combination of two main factors. First, there exist certain sub-optimalities in the t-sne embedding method [2] used to find a two-dimensional visualization sub-space. That partly results in the appearance of those outlier clusters due to failure in finding the optimal subspace to have ground truth clusters in a more compact representation. Second, a few of the outlier clusters seen as part of the cluster sets: cluster_1, cluster_17, cluster_5 consist of tracks possessing higher than average difficulty for face clustering. As a result, these tracks appear distant from their respective cluster centers. We would like to highlight here that our method still accurately manages to assign such outlier clusters to their correct ground truth cluster even though they provide a harder face clustering challenge. We kindly refer the reviewer to the “Limitations and Future Works” paragraph in Section 5 of our paper, where we highlight in detail the core limitations of our work and specify future directions to improve on them. 5. **Literature method comparison:** There are two major selection criteria we used to choose the methods to benchmark our proposed method’s performance in Table 1. First, methods having results reported on The Big Bang Theory (BBT) and Buffy the Vampire Slayer (BVS) datasets were selected as we noticed a major overlap in the use of these two datasets for evaluation in the literature. Certain methods were omitted based on the fact that they only reported results on certain rare TV series datasets that weren’t publicly available for evaluation and/or didn’t have necessary usage rights. Secondly, methods that reported results on publicly available versions of BBT and BVS datasets were selected. For a fair comparison, all methods need to have a constant face track pre-processing module (details in Section 3.2) to ensure that clustering performance is evaluated on the same set of face tracks. Methods that did not satisfy these criterias also, unfortunately, didn’t provide publicly available implementations of their work, which prevented us from re-evaluating these methods on the available input data. For the same reason, we decided to re-implement certain closely related works ourselves for a fair comparison in Table 2. We remark that to the best of our knowledge, Aggarwal et al. (2022)[3] is the only paper published within the last 3 years that we haven’t reported in Table 1. It unfortunately failed to meet the second aforementioned criterion. We sincerely hope that this clarification helps the reviewer provide more insights into our evaluation methods selection criteria shown in Table 1. We would be open to any paper suggestions from the reviewer that may be closely related to our work that we might have missed. 6. **Inconsistency in citations:** We would like to thank the reviewer for mentioning this minor issue. We have resolved it in the updated paper. We would appreciate it if you could let us know of any further questions/concerns that you may have and if any finer clarification is required for any of the points listed above. If not, we would greatly appreciate it if the reviewer could consider increasing his/her overall rating. References: [1]: Huang, Qingqiu, et al. "MovieNet: A Holistic Dataset for Movie Understanding." European Conference on Computer Vision. 2020. [2]: Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008. [3]: Abhinav Aggarwal, Yash Pandya, Lokesh A. Ravindranathan, Laxmi S. Ahire, Manivel Sethu, and Kaustav Nandy. Robust actor recognition in entertainment multimedia at scale. ACM International Conference on Multimedia (ACM MM), pp. 2079–2087, 2022