Latent Representation Matters: Human-like Sketches in One-shot Drawing Tasks

Summary

This paper investigates how different regularization techniques, applied to the latent space of Latent Diffusion Models (LDMs), impact their performance on one-shot drawing tasks. The Authors explore many regularization methods: KL divergence, vector quantization, classification, prototype-based, SimCLR, and Barlow Twins. They evaluate these methods against human performance using quantitative metrics (originality vs. recognizability) and qualitative analysis of feature importance maps. The results show that LDMs with prototype-based and Barlow Twins regularizations produce sketches that are most similar to human drawings in terms of both recognizability and originality.

Strengths

- The paper provides a comprehensive comparison thorough examination of six different regularization techniques, offering insights into their effectiveness for one-shot drawing tasks. - It introduces a novel method for generating feature importance maps in LDMs, allowing for direct comparison with human perceptual strategies. - From the practical side, the findings have potential applications in improving generative models for tasks requiring human-like generalization abilities. - The study has an highly interdisciplinary approach since it integrates computer science, cognitive science, and neuroscience, potentially offering insights into human visual cognition. The paper is very clearly written, and the Authors provide detailed information about their experimental setup, hyperparameters, and code availability, enhancing reproducibility.

Weaknesses

Overall, I think that the paper makes a nice contribution to our understanding of how different regularization techniques affect the latent representations in generative models and their ability to produce human-like sketches. The weakness I see regards the possibility to generalize from the "simple" datasets analyzed to more complex creative processes. This study primarily focuses on the QuickDraw-FS dataset, with limited exploration of the Omniglot dataset. It not very clear to me how sound can be the extrapolation from these very simple (although relevant) contexts to more complex ones. I know that this does not provide a concrete and actionable insight, but I would appreciate a comment on this.

Questions

How generalizable are these findings to more complex drawings?

Rating

6

Confidence

2

Soundness

3

Presentation

4

Contribution

2

Limitations

Limitations are adequately discussed, but they are not part of the main text (they are discussed in the Appendix, pag. 34).

Summary

The authors propose to explore how different regularizers impact the performance of LDM on one-shot sketch generation, with a specific focus on evaluating the similarity between the generated sketches and real ones. It reveals that prototype- and Barlow-based regularizers are among the best, and claims the gap between humans and machines in the one-shot drawing task is almost closed.

Strengths

- This seems the first time that the latent diffusion model (LDM) is applied to one-shot sketch generation. - The authors discussed how different regularizers impact the generation results regarding the originality (diversity) and recognizablity, which is valuable. Interestingly, prototype- and Barlow-based approaches are the best. - The paper is well-written and easy to follow.

Weaknesses

- This paper is more like an incremental work based on [30], the key idea of using the diversity vs recognizability framework, and importance maps to measure the generation quality of diffusion models is the same. It differs in extending the diffusion model into latent feature space and applying different regularizers, which seems minor. - It is a bit over-claimed that the gap between humans and machines is almost closed on the one-shot drawing task by using LDM plus proper regularizers. The qualitative results shown are not as good as the actual sketches, suffering from clear blur and distortion. The experiments are conducted on relatively simple sketch cases, which makes it hard to justify its scalability. - DDPM used in [30] for this task is not compared (with or without the same regularizers if possible), it would be helpful to understand how effectiveness of pushing the denoising into latent space.

Questions

- It is unclear how to construct the sketch codebook used for VQ-VAE.

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Please refer to the weaknesses.

Summary

i)This paper uncovers the impact of representational inductive biases on Latent Diffusion Models through one-shot tasks, particularly in the realm of human-like sketching. It compares three distinct groups of regularizers: a standard baseline, supervised methods, and a third group consisting of self-supervised techniques. ii)This paper aims to uncover the strategies employed by each regularization method to generalize to novel categories. iii)This paper conducts a comprehensive comparative analysis of the effectiveness of different regularization methods in one-shot drawing tasks. The study explores various dimensions, examining the performance and differences of various regularization strategies in such tasks.

Strengths

This paper is dedicated to analyzing the specific manifestations of various inductive biases in one-shot drawing tasks, contributing new findings and demonstrating a high level of originality. The paper is well written, with a clear and logical structure, reflecting a rigorous academic attitude. The experimental section is well-designed, with substantial research effort, and the execution process strictly adheres to scientific methods. The comparative analysis strategy used is both comprehensive and detailed, effectively ensuring the precision and credibility of the research findings. Additionally, the conclusions of this paper provide valuable insights for the field of one-shot drawing tasks and have significant guiding significance for subsequent research.

Weaknesses

i)This paper conducted an extensive analysis of the impact of different inductive biases in one-shot drawing tasks. However, the analysis remains superficial, merely briefly revealing the experimental results of various methods without delving into the fundamental reasons behind the differences in outcomes among the methods. Furthermore, the paper does not propose specific solutions to the research questions addressed. ii)The experimental method adopted in this study is not limited to specific modalities and is not restricted by data scale. This method may be applicable to the generation tasks of more modalities(such as photos or text2img). However, the experiment specifically chose handwriting and sketches as the research subjects for one-shot generation tasks. iii) The core argument proposed by this paper is the generation of samples that resemble human-drawn sketches or handwriting. However, the paper does not provide sufficient elaboration on how to quantify the “human-like” characteristics of the samples, especially with an in-depth analysis from the perspective of stroke features.

Questions

Major: i)In one-shot drawing tasks, originality constitutes a key evaluation metric. However, when measuring the “human-like” characteristics of samples, the impact of originality is relatively minor, and its role in the evaluation process appears to be more limited. ii) Does this paper quantitatively analyze the stroke correlation between the generated sketches and those drawn by humans? Although the generalization curve has been considered in the evaluation of originality and recognizability, the paper does not explicitly reveal the interrelationship between the strokes. iii)Has this study explored the reasons for or proposed hypotheses about the performance differences exhibited by various inductive bias methods in one-shot drawing tasks? Minor: i)Given that classification models may be influenced by their own biases or uneven distributions in the training data, the recognizability of the model does not necessarily equate to the “human-like” level of the samples. Does the paper take this potential issue into consideration? ii)In the exploration of effective methods to improve the performance of one-shot drawing tasks, did this paper consider approaches other than simply adding the prototype-based and Barlow regularizers with weights? iii) In Figure 2, does the “pixel baseline” refer to the use of rasterized sketches during training or testing? On this point, this paper could analyze the differential impact of inductive biases when dealing with vector versus raster sketches.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

No. How the evaluation criteria effectively align with human judgment or aesthetic standards is a question of considerable research value, especially in generative tasks. Particularly when dealing with sketches or handwriting, the consideration of strokes is an indispensable key dimension that cannot be overlooked.

Summary

This paper investigates how different representational inductive biases in Latent Diffusion Models affect their performance on one-shot drawing tasks, aiming to close the gap with human abilities. The authors explore six regularization techniques: KL divergence, vector quantization, classification, prototype-based, SimCLR, and Barlow twins. They evaluate these models using the originality vs. recognizability framework and a novel method for generating feature importance maps. The results show that prototype-based and Barlow regularizations significantly narrow the gap between LDMs and human performance in one-shot drawing tasks. These regularizers outperform standard LDM regularizers (KL and vector quantization) as well as classification and SimCLR regularizers. The authors also demonstrate that the feature importance maps of LDMs with prototype-based and Barlow regularizations align more closely with human attentional strategies. Additionally, they find that combining these two regularizers yields even better results. The study highlights the potential of incorporating specific representational inductive biases in generative models to achieve more human-like generalization capabilities, with implications for both AI advancement and understanding human cognitive processes.

Strengths

- The paper presents a study by systematically exploring various representational inductive biases in Latent Diffusion Models for one-shot drawing tasks. The application of regularization techniques from one-shot classification to generative models provides new insights into improving model performance. - The authors employ a wide range of regularization techniques and evaluate them using multiple metrics, including the originality vs. recognizability framework and a newly developed method for generating feature importance maps. The statistical analysis of the results adds credibility to their findings. - This paper demonstrates how specific inductive biases can substantially improve the performance of generative models in one-shot tasks, potentially leading to more versatile and human-like AI systems. It also shows the alignment between the most effective regularizers (prototype-based and Barlow) and prominent neuroscience theories provides interesting insights into human cognitive processes. The paper's findings could have practical applications in areas requiring rapid generalization from limited examples, such as design prototyping or creative tasks.

Weaknesses

- The paper doesn't provide a detailed analysis of how different components of the regularizers contribute to the overall performance. This makes it challenging to understand which specific aspects of each regularizer are most crucial. - While the authors use human-derived feature importance maps, they don't include a human evaluation of the generated samples. Such an evaluation could provide additional insights into the perceived quality and human-likeness of the generated drawings. - Although the authors briefly mention combining prototype-based and Barlow regularizers, this aspect is not thoroughly explored. A more systematic investigation of different regularizer combinations could potentially yield even better results.

Questions

- How well do your findings generalize to more complex datasets beyond simple sketches? Have you considered testing your approach on datasets with more detailed or realistic images?

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

N/A

Official Comment

The author's comprehensive response in the rebuttal section effectively addresses most of my initial concerns, particularly by delving into the underlying mechanism of the regularizers' excellent performance. Based on this, I am inclined to give this paper a positive evaluation and change my votes to Borderline Accept.