CAR: Controllable Autoregressive Modeling for Visual Generation

We would be more grateful if the feedback you provided was about the novelty and contributions of our work.

Thank you for your detailed feedback, but what do you think about the novelty and contribution of our work? We appreciate the thorough review and understand that addressing minor issues is part of the peer review process. However, we feel that the numerical and technical details you raised, while important, do not fully capture the core idea of our work. We believe that evaluating whether our work provides valuable insights and advancements to the community is equally important. Despite our disagreement that these detail issues are sufficient reason for the low rating score, we still find your feedback valuable for improving the quality of our paper. Here are our specific responses to your questions: **More comparison with ControlVAR**: We will provide a detailed comparison with ControlVAR in the revised version. However, it is important to note that we do not consider ControlVAR to be superior to our method, as was suggested. ControlVAR requires doubling the number of output tokens, which significantly increases the computational cost for both training and inference. Additionally, it necessitates full training on ImageNet. In contrast, our method only requires training 50% of the original parameters, 10% of the original training data, and 50% of the original training epochs, resulting in a total training computational cost of just 2.5%. Actually, if we do not consider generalization, even 1% of the training data is sufficient. This makes our method far more efficient compared to ControlVAR. **Details of Eq3 and Model Architecture**: We will provide more detailed information about Eq3, the proposed fusion function, transformer, and injection function, including their parameter counts. However, we want to emphasize that these components add minimal additional parameters and are minor issues. Our focus is on the overall effectiveness and efficiency of the control mechanism. **Number of Categories**: The choice of 100 categories was somewhat arbitrary due to our limited computational resources (using 8 V100 GPUs). After testing with 100 categories, we observed satisfactory control effects and quantitative metrics, so we did not explore more categories. However, our method is flexible and can handle more or fewer categories. For instance, we have tested with 10 categories and achieved good visualization results. We hope these clarifications help to better convey the significance and value of our work. We are committed to improving our manuscript and ensuring that our contributions are clearly communicated to the community.

We believe that we are the first to achieve controllability while maintaining visual AR models‘s inherent capabilities.

Thank you for raising many detailed questions. We will address them one by one. However, we still believe that these detailed issues do not overshadow our core contribution, which merits a more favorable initial score. Our work is the first to achieve controllability while maintaining the inherent capabilities of AR models, a goal that existing solutions have not been able to accomplish. Below are our specific responses to your concerns: **Weakness 1 & 2:** We will correct the writing issues regarding the mixing of terms like "control token," "control map," and "control information," as well as the placement of the model architecture description, in future versions. Thank you for pointing this out. **Weakness 3:** While our method does indeed add approximately 0.5 times the parameters of the original model, this is also the case with the popular ControlNet method used in diffusion models. This trade-off brings stability and rapid convergence during training. We believe there is currently no method that can maintain the model's original capabilities and achieve good controllability with a negligible increase in parameters. **Weakness 4:** We did not notice the work Lumina-MGPT, which was published in August. However, ICLR submissions's deadline is in September, and according to the reviewer guidelines, papers published on arXiv within the past four months can be considered concurrent work. Therefore, we do not consider this an oversight. Nevertheless, we reviewed this work and found that it is fundamentally different from ours. Lumina-MGPT is a multimodal model fine-tuned on a powerful Chameleon base model, using over 50 million text-image pairs and finally completely altering the pretrained model's parameters. In contrast, our method focuses solely on pure AR visual generation and only trains the control capabilities on top of a frozen pretrained model using merely 0.1 million data points. Comparing the two is neither fair nor meaningful. However, we would like cite this work and explain the differences in our next submission. **Weakness 5:** We believe the concern about efficiency is misplaced. Our method uses only 50% of the trainable parameters, 10% of the training data, and up to 50% of the training epochs of the original model. When combined, the total computational cost is only 2.5% of the pretraining process. Even with this reduced computational load, we achieved comparable or better control performance without losing generalization. Compared to methods like ControlVAR, which require full training, our method offers significant efficiency advantages and is currently the most efficient approach for controlling AR models. **Weakness 6:** We will add more comprehensive comparisons with ControlVAR and other relevant methods. **Question 1:** The control map is obtained by resizing the raw input image and then passing it through several convolutional layers. **Question 2:** We will supplement the comparison with the original model, noting that the model size increases by 50%, and inference speed is also affected. However, this is not the primary focus of control-oriented work. **Question 3:** To our knowledge, LoRA (Low-Rank Adaptation) methods are not applicable in the context of introducing control conditions in both diffusion and AR models. LoRA is designed for parameter tuning and cannot introduce external control signals. **Question 4:** Thank you for the suggestion. We will strive to include more quantitative experimental results. We hope these clarifications address your concerns.

CAR is the First Control Method to Retain the Original Capabilities of Visual AR Models

We appreciate your objective and detailed feedback and are committed to addressing your concerns. Below are our responses to the specific points you raised: **Weaknesses**: **1**.Training Details for Baselines: We acknowledge that in Section 4.2 , we did not provide sufficient details about the training settings for T2I-Adapter and ControlNet for fair comparision. We would like to supplement this information to ensure clarity. **2**.Supplementary Material: While we agree that the code is crucial, we also recognize the importance of visualizations. We will enhance the supplementary material by adding more extensive visual results, Thank you for your suggestion. **Questions**: Canny Conditioning in Figure 2: Yes, c_1 and other images are obtained by resizing. Since the resizing involves scaling down from larger images, there might be some loss of detail, but the overall structure and integrity of the Canny information are retained. This should not significantly affect the prediction of tokens at smaller scales, and our experiments have confirmed the effectiveness of this approach.

Despite your mentioned specific limitations, our method is still a pioneering and significant advancement in the field of controlled generation for visual AR models.

Thank you for your insightful comments and suggestions, they have helped us to better understand potential areas for improvement in our work. Below, we address the concerns and questions raised in your review: **Weakness 1:** we acknowledge that our current experiments are limited to VAR models operating under the next-scale paradigm. However, we believe that our methodology could be adapted to other AR models, such as those using the next-token approach in LLamaGen or the masked autoregressive technique in MAR, by incorporating conditions during the decoding process based on the spatial positioning of visual tokens. While this represents a current limitation, we view it more as an opportunity for future exploration and expansion of our work. It is also worth noting that CAR stands out as the first method to achieve controllable generation while preserving the original capabilities of visual AR models, something that, to the best of our knowledge, has not been achieved by other methods like ControlVAR or ControlAR, which typically modify the original model parameters and capabilities. CAR's modular design, allowing for inference with the same base model through simple parameter replacement, underscores its flexibility and adaptability. We will strive to further highlight these aspects in our subsequent revisions. **Weakness 2:** Concerning the second limitation regarding computational overhead, we appreciate your feedback. It is true that the Transformer module in CAR utilizes approximately half of the parameters of the original VAR model, which suggest higher costs inference. However, it is important to note that similar techniques, such as ControlNet in diffusion models, employ an equivalent strategy without facing significant criticism regarding computational efficiency. In fact, ControlNet is celebrated for its capacity to deliver effective control outcomes with minimal training data and epochs. Our method achieves comparable results with just half the training parameters of the original model, alongside a mere 10% of the training data and fewer than 50% of the training epochs, all while requiring aprroximately 2.5% of the computational resources needed for the initial pretraining phase. We hope this clarifies any concerns regarding the computational demands of our approach. To address your questions: **1**. Regarding the implementation details of the baseline models ControlNet and T2I-Adapter, we agree that providing more context could be beneficial. However, given the established recognition of these models within the industry, we opted to focus our paper on the innovative aspects of CAR. We will consider adding a brief overview of these models in our next revision to offer readers a more comprehensive understanding. **2**. Comparing the training cost and convergence speed of CAR with baseline models is a valid point. While we agree that such comparisons are valuable, they may not fully capture the unique characteristics of AR models versus diffusion models. The differences observed could largely be attributed to the intrinsic properties of these model types rather than the control mechanisms employed. Nonetheless, we will reflect on how to better contextualize these aspects in our discussion. **3**. Your question about applying CAR to other AR generative models that do not utilize multi-scale tokens is insightful. Although our current experiments are centered around models with multi-scale latents, we are optimistic that the core principles of CAR can be extended to other AR frameworks. By focusing on the spatial positioning of tokens, we envision that similar conditioning strategies can be implemented. It is important to recognize that CAR represents a significant step forward in the field of AR model control, particularly in maintaining the original model's capabilities. We will explore these possibilities in future work. **4**. As the performance of smaller CAR modules with larger pretrained VAR models, it is a interesting question. That may work, we consider it needs either initialize control modules from smaller VAR for larger VAR, or just using less copy parameters from original models. We would like to have such a try in the future, again thanks for your advice. We sincerely thank you once again for your thorough review and constructive feedback. We are eager to incorporate your suggestions to improve the quality and impact of our work.