We appreciate your response and are happy to clarify further.
**“Biggest concern is the collecting a large number of weights…”**
We agree that collecting a large number of model weights is more complex than a large number of images. However, the bigger picture of this work is the “idea of using weights as the dataset” (215A) in order to demonstrate "the existence of a LoRA model manifold” (rzPz). As mentioned in the global response, we introduce this idea of a **meta**-latent space **creating new models**, with “broad applications in generative modeling” (wMSF), which we believe is beyond just the personalization application. We would like to note that although creating this space requires a notable amount of compute, using the space for the variety of applications is lightweight. We discuss these practical details below.
**“...in terms of the number of parameters, Celeb-Basis only has 1024 learning parameters.”**
While Celeb-Basis uses 1024 coefficients to define an identity in token space, it requires optimizing an MLP at test-time with 525,312 parameters and backpropagation through the text-encoder, leading to \~3 minutes and \~21GB VRAM on a single NVIDIA A100. In contrast, we directly optimize 10,000 coefficients (\~2% of Celeb-Basis's parameters) without text-encoder backpropagation, reducing optimization time to \~1 minute and VRAM usage to \~8GB. Furthermore, our studies (Table 1 and 2 in the rebuttal PDF) show *w2w* offers better identity preservation without overfitting.
We would also like to note that Celeb-Basis works in token embedding space, which is already fairly low dimensional. In their case, they use 1024 dimensions to span the original 1536 dimensional space (768 dimensions for first name + 768 for last name), resulting in ~66% of the original dimensionality. In contrast, we compress a much higher dimensional LoRA space (100,000 dimensions) to 10,000 dimensions (10% original dimensionality). The fact that even LoRAs, which are by design low rank, have significantly lower rank semantic subspaces is quite intriguing, potentially inspiring further exploration of interpretable and controllable subspaces in model weights.
**“...what guarantees that information like cartoon drawing is not encoded in the principal components?”**
When you say, “the weight information represents a Buzz Lightyear,” the weights of the base Stable Diffusion model encodes prior concepts such as “Buzz Lightyear,” “Pixar,” etc, but not the LoRA weights added on top. During Dreambooth fine-tuning, we train LoRAs on realistic human identities, so the only new inserted concept is the identity. We apply PCA to the LoRAs, not the base model, and since there’s no style variation among the encoded identities, no principal components encode style. However, if “style-editing” means generating images of the identity in different styles with prompts, that’s possible because the base model retains prior concepts. For example, in Fig. 7 and Fig. 8, we generate a person as a statue or painting using prompts. Furthermore, in rebuttal PDF Fig. 8, we show that a separately trained style LoRA can be merged with an identity model from *w2w* space.
**“... whether the identity information of each person can be maintained when generating multiple characters at the same time…”**
Thank you for the clarification. Identities from *w2w* space can be generated alongside existing characters or concepts in the base diffusion model (e.g., the celebrity examples in Figures 6 and 7 of the paper). However, two *w2w* models can't be naively merged by adding weights, as they share the same subspace and will interfere, leading to identity interpolation. Separate LoRAs can be trained to encode identities and then be merged with *w2w*, but they cannot live in the same subspace. This can be enforced using something like [1]. On the other hand, a LoRA encoding another concept such as style can be seamlessly merged as seen in rebuttal PDF Fig. 8. We will discuss this as a limitation in the paper.
**Summary**
We hope we addressed your main concerns. Overall, we have shown from an efficiency and performance standpoint, that *w2w* performs stronger than Celeb-Basis as well as other baselines. More importantly, as you mention in your original review, our work provides “new ideas and insights into personalized generation models.” We believe that the merits of our work should not be subsumed by Celeb-Basis just because of the personalization application. If that were the case, then works like Dreambooth can be thought of as just Textual Inversion but optimizing model weights instead of token embeddings. But works like Dreambooth provide further insights beyond the application.
[1] “Orthogonal Adaptation for Modular Customization of Diffusion Models.” Po et al. 2024