DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models

Summary

This paper proposes DiffSketcher, a stroke-based rendering model using a pre-trained text-to-image diffusion model. It performs the task by directly optimizing a set of Bézier curves with an extended version of the score distillation sampling (SDS) loss, which allows using a raster-level diffusion model as a prior for optimizing a parametric vectorized sketch generator.

Strengths

1. This paper proposes a stroke-based rendering method for synthesising vector graphics. 2. This method does not need to retrain the very cost Diffusion model. 3. This method could use text to control the painting content.

Weaknesses

1. Lack of novelty. The model's generation ability essentially origins from the pre-trained diffusion model and the differentiable rasterizer module process is just a fitting process. I will discuss this in the below section "Limitations" further. 2. Lack of comparison. Figure 5 shows the raster images generated by the diffusion model and the corresponding sketching results. However, the compared methods Canny and CLIPasso are too weak. The authors should compare their results with more robust image-to-sketching rendering methods such as "Combining sketch and tone for pencil drawing production" and image style transfer methods such as StyleGan. In the supplementary material, I have seen comparisons with CLIPDraw. But this kind of comparison could not support anything since their artistic styles are totally different. 3. Lack of a very important citation. This paper is a sketching version of the published CVPR 2023 paper "Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models". Both papers follow the same idea that using diffusion models to generate raster images and then using diffvg to optimize stroke parameters. Although the task and technique details are different, no citation of such similar work is unacceptable. 4. Poor writing. The authors should explain their proposed abbreviation the first time they use it. For example, the "ASDS" in line 183. When I got to line 185, I saw the "Augmentation SDS" invented by the author. And I did not see what the abbreviation "JVSP" mean.

Questions

1. What is the total stroke number of a sketch? Is it user-specified or automatically determined by some means? 2. Control points (anchors). In line 225, how do you sample the control points from the distribution map? Using rejection sampling or adaptive sampling or other methods? And the stroke width. In the supplementary material, the authors handly set this parameter. Why not make it learnable as the opacity? 3. Running Efficiency. It seems the optimizing process is very costly. What is the running time to generate a sketch? Please give the used image resolution, stroke number, and iteration steps. 4. In figure 3, is the $\varepsilon$ and $\epsilon$ are two different things?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

I doubt the meaning of this kind of technique route that uses a pre-trained diffusion model to generate a raster image and then translate the raster image into a vector graphic. This technique route is just an engineering combination of a generative model and a raster-to-vector algorithm. I prefer to see the authors proposing improvements to the diffusion models or the raster-to-vector algorithms, but not just combining them.

Summary

The authors introduce DiffSketcher. The model makes use of a frozen pre-trained latent diffusion model (Imagen) to guide the parametrisation of bezier curves and reproduce the generated image in vectorised sketch form. The method works in a similar way to DreamFusion, but instead of using a differentiable 3D renderer (DreamFusion is a text-to-3D model) the authors use a differentiable bezier curves renderer.

Strengths

- The paper shows very "believable" results: "human-like" sketches. - The first text-to-sketch model to work at the scene and object level. - They make good use of the methods available in the literature, the combination of the Li et al [15] differentiable rasteriser with the score distillation sampling loss from DreamFusion being particularly effective - The authors additionally improve the random startup of the bezier curves and show how the inclusion of a perceptual (LPIPS) and "semantic" (CLIP feature comparison) loss improves fidelity of the sketch to the prompt.

Weaknesses

- I was not able to identify what is the benefit of using the Augumentation SDS loss instead of the original SDS loss or why it "encourages plausible images to have low loss and implausible ones to have high loss"? - The authors claim it is possible to control the abstraction level of the sketches but that seems to be through the number of strokes. There are multiple ways of defining abstraction and because these sketches are always based on image inputs it is improbable that the method would be able to reproduce higher abstraction levels such as the ones found on the QuickDraw dataset. - Some of the notation is confusing. Equations 1, 2 and 3 are probably hard to parse if the reader is not familiar with the original references. The use of S_a and S_phi for sketches and images is also particularly confusing on first read.

Questions

- The current work is constrained by its direct connection to a model that produces realistic images. Do you see how a future method might be able to achieve true abstraction (e.g. cats with whiskers and a house as a triangle+rectangles)?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

The authors do not discuss current limitations of the model.

Summary

The authors introduce DiffSketcher, an algorithm for generating vectorized free-hand sketches from text input. DiffSketcher leverages pretrained text-to-image diffusion models to optimize parametric Bézier curves and generate sketches that align with the text prompt. The method incorporates Score Distillation Sampling (SDS) loss and a differentiable rasterizer to guide the optimization process. Additionally, a joint attention-based stroke initialization strategy improves convergence towards semantic depictions. The paper also demonstrates the effectiveness of DiffSketcher in generating sketches with various levels of abstraction. Overall, this is a promising approach to bridge the gap between natural language and free-hand sketches.

Strengths

- The paper is written really well, and the concepts are explain clearly and intuitively for the most part. - addressing a relatively underexplored area of text-to-sketch synthesis - The overall idea is simple (combining differentiable rasterizer with text-to-image diffusion) and it is well-designed specifically for generating vector sketch from text. - the stroke initialization based on fused attention map is also neat - interesting idea to incorporate opacity in vector sketch to mimic the way human draw - the results are good

Weaknesses

- the discussion of the results and comparison is somewhat lacking. For example, the ablation study is really short and I'd love to see more insight on the effect of each components - The evaluation is also a bit unsatisfying. While the results are impressive on their own, the comparison with existing work seems unfair since the style are quite difference (and if we are using diffusion model to generate image anyway, any image to sketch techniques could have been used?). And I feel like the quantitative evaluation which relies on CLIP embeddings trained on images, may favor sketches with more detailed styles, or more human-like style anyway. So when the baselines are weak, the comparison doesn't really give us much insight into the other strength of the method. - I also cannot find much information in the paper about the VAE decoder in the proposed method (that convert image to sketch?)

Questions

- Would appreciate some clarification regarding the VAE decoder - How long does it take per sketch? - How do you set the number of strokes? Shouldn't complicated prompt need more strokes than simple prompt in order to get the same level of abstraction? paper that might be related: Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I might have missed it but I don't see any mention of the limitations of the proposed method in the paper. I think this is quite important to see since the evaluation for this sort of task can be quite subjective (qualitative results could have been cherry pick). So I'd appreciate some discussion about the limitation.

Summary

This paper proposed an algorithm that creates vectorised free-hand sketches using text as input. Particularly, the authors use a pre-trained text-to-image diffusion model and transfer its prior knowledge into a differentiable rasteriser. It performs the task by directly optimising a set of B\'ezier curves with an extended version of the score distillation sampling (SDS). The loss is backpropagated through the differentiable rasteriser to update the control points and opacity of b\'ezier strokes at each step until convergence of the loss function.

Strengths

The paper is relatively easy to follow and the qualitative results are quite convincing. This line of work -- synthetic sketch generation -- is recently gaining quite some traction and it is nice to see another addition to the community. The authors did a commendable job in writing the paper clearly with nice diagrams that are intuitive.

Weaknesses

The biggest weakness of this paper is its novelty. While there might not be a paper that ONLY focuses on text-to-sketch using SDS loss, the authors missed an important paper that does the same job -- VectorFusion (https://arxiv.org/pdf/2211.11319.pdf). Particularly, VectorFusion uses the same differentiable rasteriser (including optimising the opacity parameter) to generate 2D line drawings. See Fig. 2 in VectorFusion and section 5.5 Like in this paper, VectorFusion can also control the level of detail using B\'ezier controls. I would love to think that DiffSketcher and VectorFusion are parallel works (and the authors did not know about VectorFusion paper). Nonetheless, it should be noted that VectorFusion was uploaded on arXiv on Nov 21st, 2022 and accepted to CVPR'23 long before NeurIPS submission deadline.

Questions

Apart from my aforementioned weakness, there are a few additional questions/confusions: First, the authors mention in L3 that the "algorithm that creates vectorized free-hand sketches" -- I think we should carefully consider the definition of what is a "free-hand sketch" -- a sketch that has stroke deformations typically seen in a human drawn sketch. As DiffSketcher was never trained on a human-drawn sketch, so how can the authors assert that it is generating free-hand sketch and not something that looks realistic but follows edge-maps? One way of evaluating if DiffSketcher truly generates free-hand sketch and not some edge-map is to train a Fine-Grained Sketch-based Image Retrieval (FG-SBIR) model on sketches from DiffSketcher and evaluate on real human drawn free-hand sketches. The authors can use some standard FG-SBIR dataset like Sketchy or QMUL-Shoe-V2. On a minor note, the authors should provide a clear description of why SDS is better than something like CLIP. This description could be accompanied with some insightful analysis or experiment. I would leave this to the authors to decide and think what experiments or analysis to show that helps to clarify SDS vs CLIP for DiffSketcher. Also the authors should better motivate "LDM for Vector Sketch Synthesis". Although I know it is not true but the current introduction reads like -- LDM is a hot topic, hence we extended CLIPasso with LDM. Finally, to make the paper more interesting, the authors can add an interesting discussion as to why something super simple like "StableDiffusion+CLIPasso" is not a good alternative for text-to-sketch generation.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Since this work is quite literally a sub-section of another paper "VectorFusion", it shares the same as mentioned in the discussion section 6 of the paper. I am simply quoting those limitations (application to this paper as well) for completeness: "VectorFusion faces certain limitations. For instance, forward passes through the generative model are more computationally expensive than contrastive approaches due to its increased capacity. VectorFusion is also inherently limited by Stable Diffusion in terms of dataset biases and quality, though we expect that as text-to-image models advance, VectorFusion will likewise continue to improve"

Summary

This paper proposes an algorithm to leverage pretrained image diffusion models for text-guided sketch synthesis. The sketch to be optimized is represented as a set of Bezier curves (and their opacities) and optimized with the (augmentation) score-distillation sampling (ASDS) loss. Attention maps from the diffusion model's UNet are fused and used to initialize the Bezier curves to accelerate convergence. A variety of results ranging from abstract to concrete sketch generation are demonstrated.

Strengths

The paper is easy to understand and makes a first-of-its-kind contribution: using pretrained image diffusion models to optimize vector graphics sketches. While it is thematically similar to DreamFusion and related works, the problem and solution are novel enough and interesting. The results are convincing are significantly better than prior work, and evaluation is comprehensive on a wide range of concepts. The ASDS loss is interesting although a similar augmentation strategy was also used in CLIPDraw. The stroke initialization strategy by analyzing the attention maps is an excellent idea and could potentially be helpful in other future works that have to deal with domain gaps and difficult optimization landscapes. Ablation studies are quite comprehensive. Overall, the paper is technically strong and demonstrates excellent results.

Weaknesses

This is a strong paper, I do not see any major weaknesses. It would be good to report some failure cases and discuss limitations.

Questions

Minor suggestions: Line 177: this is good place to define the JVSP abbreviation Line 183: the abbreviation ASDS is not defined

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are not discussed, it would be good to include those in the revision along with some failure cases.

Any unclear explanations?

Dear Reviewers, Thanks for your efforts in reviewing this paper. We have tried our best to address the concerns and improve our work. Are there unclear explanations here? We can further clarify them. Best wishes, Authors

Thanks!

Thank you for the detailed response; it clarifies many of my confusions. While I believe that VectorFusion somewhat diminishes the impact of the work, I still think it remains within the acceptance threshold. Especially given that VectorFusion is a somewhat recent publication. So, although I'm not as optimistic as I was initially, but I'd still like to keep my original rating for now.