Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalization

Summary

The paper introduces Tina, a text-conditioned neural network diffusion model designed for train-once-for-all personalization. Tina utilizes a diffusion transformer model conditioned on task descriptions embedded using a CLIP model. This innovative approach aims to generate personalized models for various end-users and tasks based on text prompts, demonstrating significant generalization capabilities even when trained on relatively small datasets (~1000 samples). The model is evaluated under zero-shot/few-shot image prompts, varying numbers of personalized classes, natural language descriptions, and predicting unseen entities to assess its understanding of world knowledge.

Strengths

- The paper provides a comprehensive explanation of the design and framework of Tina. - It conducts a detailed ablation study and experiments across different datasets. - The topic is interesting, and the presentation is clear and easy to understand. - very detailed and robust comparison with previous works.

Weaknesses

- The model parameter size in the experiments is too small; larger models are needed to evaluate effectiveness. - In Table 1, the results of direct fine-tuning should be included. - We might need an ablation study on the impact of text prompts. - We might need an ablation study to determine if the model merely memorizes and reproduces parameters. - Figure 2 requires polishing for better clarity.

Questions

Can you provide your prompts for Appendix A.2? If the authors solve my questions, I will raise my scores.

Rating

4

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

The model size is too small in exp.

Summary

To generate personalized models for a variety of end-users and tasks via text prompts, this paper introduces Tina, a text-conditioned neural network diffusion model. Tina employs a diffusion transformer model, complemented by a CLIP model to embed task descriptions. Remarkably, Tina demonstrates superior generalization capabilities even on small-scale datasets, performing well both within and outside the distribution of the training data. Furthermore, Tina exhibits robust performance under zero-shot/few-shot image prompts, natural language instructions, and unseen categories.

Strengths

1.The method demonstrates excellent generalization, showcasing significant in-distribution and out-of-distribution performance even when trained on small datasets. It also exhibits robust behavior in predicting entities that have not been seen before. 2.Compared to existing text-to-image models (such as stable diffusion), text-to-video models (like Sora), and large language models (such as GPT-4), the concept of Tina, which generates personalized models suitable for specific tasks directly from text descriptions, is quite novel. 3.The experimental process is comprehensive and reliable. The paper conducts comparisons against baselines across multiple datasets, and it also undertakes experiments to validate generalization performance as well as performs ablation studies. 4.The experiments involves multiple datasets to verify the effectiveness of the proposed methods.

Weaknesses

1.It is better to includes more comprehensive and competitive baselines to show the model’s effectiveness and advance. The two baselines come from one paper published in 2023. As for the experimental setting involves three widely-used datasets, I am wondering whether the experimental results excels or perform similarly to the SOTA performance on some of the three datasets. In other word, is it possible to apply the proposed strategy to some more advanced framework to make the performance similar to the SOTA, which ensure the proposed method have real applications in the real use. 2.The base model is CNN or ResNet in the experiments. Is the proposed method generalized to more advanced framework? Applying the proposed method on more advanced framework and obtain more advance performance indicates that the method has potential to be used in the real life. 3.We suggest providing necessary explanations in the captions of the model framework overview.

Questions

None

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The limitations are fine with me.

Summary

This work introduces Tina, a text-conditioned neural network diffusion model designed for generating personalized models from text prompts. Tina aims to enable efficient personalization by training a generic model once and then customizing it for various end-user tasks using task descriptions. Leveraging a diffusion transformer model and CLIP-based text embeddings, Tina demonstrates the ability to generate models for a wide range of personalized tasks. The approach shows promising results in generalizing to both seen and unseen tasks, achieving state-of-the-art performance in several benchmarks.

Strengths

1. Tina's train-once-for-all approach effectively addresses the need for personalized models without requiring extensive retraining, making it a practical solution for diverse end-user scenarios. 2. The model achieves competitive performance across multiple datasets, demonstrating its robustness and effectiveness in both in-distribution and out-of-distribution tasks. 3. Tina can handle various types of input prompts (text, images) and generalize to unseen classes and tasks, highlighting its versatility and potential for broader applications.

Weaknesses

1. Some methodological details are sparse, such as the specific configurations and hyperparameters used for training Tina. Providing more granular details could help readers replicate the experiments. 2. The reason for adopting DiT as the weight generation model is not well justified. It would be good to see some results of adopting different kinds of diffusion models.

Questions

1. It would be good to clarify how exactly the parameters are inherited from G.pt for initialization. 2. Tina encodes and decodes the parameter tokens with linear layers, which are trained along with the DiT model in an end-to-end manner. I'm curious about what are the benefits of this one-stage manner compared to the original neural network diffusion, which trains an autoencoder and diffusion model separately. And what would the performance be if you adopted the same pipeline of neural network diffusion for weight generation?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

While limitations are discussed, the manuscript could benefit from a discussion of the scalability of Tina to larger datasets and more complex tasks.

Thanks for responding my comments

I have checked the authors' rebuttal and reviewers' comments. The authors' rebuttal sounds fine with me, which addressed most of my concerns. The authors add some additional experiements (i.e. on ViT) for my concerns as well as other reviewers' questions. I still would like to vote for this paper. Considering other reviewers' comments and the overall quality of this paper, I tend to keep my score.