PLIP: Language-Image Pre-training for Person Representation Learning

Summary

This paper presents a language-image pre-training framework and a large-scale synthetic image-text dataset for person representation learning. The proposed framework contains three elaborately designed pretext tasks: 1) Text-guided Image Colorization (TIC); 2) Image-guided Attributes Prediction (IAP); and 3) Identity-based Vision-Language Contrast (IVLC). Meanwhile, a large-scale person dataset with image-text pairs (SYNTH-PEDES) is constructed by automatically generating textual descriptions using the proposed Stylish Pedestrian Attributes-union Captioning (SPAC) method. The dataset contains over 4.7M images from 300K identities with 12M descriptions. Finally, the paper utilizes the pre-training framework to pre-train models on SYNTH-PEDES. Extensive experiments demonstrate the significant improvements brought by the pre-trained models to a range of downstream person-centric tasks on various settings. Also, the paper conducts extensive experiments to verify the superiority of the proposed pre-training framework compared with other SOTA pre-training methods.

Strengths

1. This paper is well-written and easy to follow. 2. The proposed pre-training framework considers the key characteristics of persons and effectively utilizes the correlations between images and texts. The learned representations through this framework show good generalization ability on extensive person-centric tasks. 3. The SYNTH-PEDES dataset contains a large number of person images along with their corresponding text descriptions. The quality and diversity of the texts are quite good. This dataset can facilitate research on more cross-modal person-related domains. 4. This paper conducts comprehensive experiments on many aspects, validates the powerful transfer and domain generalization capabilities of the pre-trained models, and evaluates the quality of the dataset.

Weaknesses

1. The paper does not conduct pre-training on commonly used ViT. 2. The entire content of the paper, including the appendix, seems somewhat verbose, totaling about 31 pages. The authors could be more concise in their presentation.

Questions

1. As shown in Table 4, for the proposed pre-trained model PLIP, CMPM/C achieves the best results, significantly surpassing the other two downstream algorithms. However, for all other pre-trained models, the LGUR achieves the best results. Can you explain the reason for this? 2. In my opinion, the TIC task utilizes the textual descriptions with color words to restore the original color information of the gray-scale images, which could lead to learning comparably fine-grained information. Nevertheless, in some specific scenarios, like encountering a white T-shirt with a blue logo and vaguely described as “a white T-shirt,” could TIC result in learning suboptimal representations?

Rating

8

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

Since the research in this paper involves persons, it inevitably contains some privacy and security issues. However, the authors have discussed these concerns in detail in the “Broad Impact” and “Limitations” sections of the appendix and have taken measures to minimize the risk of privacy breaches. Therefore, I think the authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

A novel vision-language pre-training framework is proposed in this paper, termed PLIP, for person-centric downstream tasks: image-based Re-ID, text-based Re-ID, attributes recognition, human parsing and person search. To form PLIP, three pre-training tasks are designed: text-guided image colorization, image-guided attributes prediction and identity-based vision-language contrast. Also, the authors design a new caption model named SPAC to generate stylish textual descriptions for each person image. Utilizing SPAC, they construct a large-scale synthetic person dataset with image-text pairs to validate PLIP's effectiveness. A range of experiments have shown that the pre-trained models achieve state-of-the-art results on various downstream tasks, without the bells and whistles.

Strengths

1. The proposed PLIP is well-motivated and the combination of three pretext tasks explicitly helps to learn fine-grained cross-modal associations. That is to say, the design of the pretext tasks takes into good consideration the distinguish features of this research field, and it is simple and effective. 2. The paper presents a new dataset called SYNTH-PEDES, which contains whopping 4.7M person images and 12M textual descriptions, making it by far the largest person dataset with image-text pairs. This dataset will undoubtedly become an important resource in this research field. 3. The authors commit to open-sourcing the code, dataset and weight. This practice of open-source will have a positive impact on this research field. 4. Extensive experiments provide strong support for the paper. Notably, the ablation studies and analyses are very comprehensive, investigating various settings and dataset quality evaluations. 5. The improvements of PLIP models to downstream tasks are obvious. Compared with existing pretrained models, the PLIP models achieve consistent leading performance across a range of tasks and settings. Specially, PLIP brings a significant improvement to unsupervised ReID methods. For example, on the MSMT17 dataset, it improves the mAP of PPLR and ISE by 14.7% and 11.4%, respectively.

Weaknesses

1. The proposed PLIP comprises three tasks, which seems somewhat complex. The popular CLIP, for example, only includes one contrastive learning task, whereas the method in this paper has three tasks, which should introduce greater computational overhead. I think it’s worth trying to make the pre-training method simpler. 2. Since the identities in SYNTH-PEDES are obtained based on tracklets, and the textual descriptions are also synthetic, this dataset will contain some noisy data. 3. There are some typo errors. For instance, the period in the ninth line should be changed to a semicolon. The authors should carefully review the full text to avoid these typo errors.

Questions

1. For the IAP task, why don’t you use multiple binary classification heads instead of just one multi-classification head? Please explain. 2. How does the TIC task help with learning person representations? I am looking forward a more detailed explanation.

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have discussed the limitations and broader impact in detail. I hold the view that they have adequately addressed the limitations and potential negative societal impact of their work.

Summary

This paper introduces a Language-Image Pre-training framework, termed PLIP, designed for enhancing person representation learning. In order to better adapt to downstream person-centric tasks, three pretext tasks are designed to pay more attention to critical person-related characteristics, including Text-guided Image Colorization, Image-guided Attributes Prediction, and Identity-based Vision-language Contrast. In addition, a large-scale person dataset named SYNTH-PEDES generated by an image captioner named SPAC is employed during pre-training. And PLIP performs remarkable ability in various downstream person-centric tasks.

Strengths

The authors highlight a significant domain gap that exists between general pre-training technologies and person-centric tasks. To address this gap, the authors suggest a language-image pre-training framework, which represents a meaningful direction.

Weaknesses

1. What is the core difference between the proposed pretext tasks and Lapscore [93]? The authors assert that the proposed IAP masks attribute phrases instead of color words, as done in Lapscore. While I agree with this distinction, it seems to be a minor difference based on certain operations. 2. Some of the representations are unclear and lack specificity. For instance, the abbreviations in Figure 2 are not explained either in the caption or in the main text; the method/technology represented as Baseline in Table 1 remains unclear. 3. The writing should be improved. For example, the last paragraph in Section 2.1 appears to be somewhat absurd; Fig 4 and Fig 12 are same. 4. The experimental evaluation is incomplete, and certain comparisons lack logicality and fairness. The proposed framework comprises a methodology and novel datasets, which require separate meticulous verification. Otherwise, some comparisons may be unfair due to the proposed framework being trained on a larger dataset, SYNTH-PEDES. Additionally, there are existing works [1-2] that aim to construct large-scale text-image person datasets, and their performance should be compared with the proposed SYNTH-PEDES dataset. 5. The authors propose training an image captioner specifically for annotating person images. However, it begs the question of why not directly utilize the existing state-of-the-art MLLM technologies for image annotation? Contemporary MLLMs possess exceptional modeling capabilities. It is worth mentioning and discussing the existing person image captioning methods [1-3] that have been previously proposed. [1] 2024-CVPR-Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReID [2] 2023-ICCV-Unified pre-training with pseudo texts for text-to-image person re-identification [3] 2023-ACMMM-Text-based Person Search without Parallel Image-Text Data

Questions

The description in line 206 seems to violate the blind review rules. [such as the Cross-Modal Projection Matching (CMPM) loss [110] we adopt].

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

This paper introduces a person pre-training framework PLIP, which consists of three pre-text tasks: text-guided image colorization (TIC), image-guided attributes prediction (IAP) and identity-based vision-language contrast (IVLC). Furthermore, a large-scale person dataset SYNTH-PEDES is constructed to facilitate the pretraining. Extensive experiments demonstrate the effectiveness of the proposed method.

Strengths

1. **Good Writing**: This paper exhibits an overall clear storyline. I can easily catch the motivation and the contribution of this work. 2. **Adequate Experiments**: The experiments are comprehensive and validate the importance of pedestrian pre-training data.

Weaknesses

1. **Unfair Experimental Comparison**: My primary concern lies in the fairness issue of this paper. As mentioned in line 1046, SPAC uses CUHK-PEDES and ICFG-PEDES as training datasets, while the subsequent PLIP model leverages SYNTH-PEDES, constructed using SPAC, as the training corpus. **This results in information leakage from CUHK-PEDES and ICFG-PEDES into the PLIP model**. Therefore, the claimed zero-shot or domain generalization experimental comparisons are not fair, and such comparisons can not effectively demonstrate the superiority of the proposed PLIP method. 2. **Undefined Abbreviations**: The abbreviations SIC, VLC, and VAP are not explained in Figure 2, making their significance unclear. 3. **Lack of Comparison with Latest Works**: The compared methods are outdated. For example, the most recent methods referenced in Tables 6 and 7 were proposed in 2022. 4. **Lack of Novelty**: The three pretext tasks proposed in PLIP are not novel. For instance, TIC from LapsCore, IAP from APTM, and IVLC from TBPS-CLIP have already been introduced. Therefore, these tasks are unlikely to provide valuable new insights for subsequent research in this field.

Questions

In the IAP task, the paper employs attribute phrases masking, which differs from random masking. However, attribute phrases masking may overlook the fact that certain verbs can reflect relationships between entities. Have the authors conducted the experiment to make an experimental comparison between attribute phrases masking and random masking?

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Please refer to Weaknesses and Questions.