Large language models (LLMs) can be seen as atomic units of computation mapping sequences to a distribution over sequences. Thus, they can be seen as stochastic language layers in a language network, where the learnable parameters are the natural language prompts at each layer. By stacking two such layers and feeding the output of one layer to the next, we obtain a Deep Language Network (DLN). We first show how to effectively perform prompt optimization for a 1-Layer language network (DLN-1). Then, we present an extension that applies to 2-layer DLNs (DLN-2), where two prompts must be learned. The key idea is to consider the output of the first layer as a latent variable, which requires inference, and prompts to be learned as the parameters of the generative distribution. We first test the effectiveness of DLN-1 in multiple reasoning and natural language understanding tasks. Then, we show that DLN-2 can reach higher performance than a single layer, showing promise that we might reach comparable performance to GPT-4, even when each LLM in the network is smaller and less powerful.
Paper
References (56)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The paper introduces a framework called Deep Language Network (DLN) that involves stacking multiple large language models (LLMs) as stochastic layers in a deep network. The prompts at each layer serve as tunable parameters, and the output of one layer is fed as input to the next layer. The LLMs are trained jointly using variational inference. The DLN architecture achieves higher performance than a single layer and can sometimes match the performance of larger and more powerful models like GPT-4. The authors discuss the analogy between LLMs and traditional parametric and nonparametric models, explore the limitations of LLMs, propose the use of variational inference for training DLN, and demonstrate the performance of DLN on various datasets. They suggest that DLN can serve as a framework for characterizing and optimizing prompt learning techniques. The paper also discusses future directions such as testing with different language models, fine-tuning stackable LLMs, and expanding DLN to accommodate arbitrarily directed acyclic graphs. The authors hope that DLN and modular approaches like theirs will address the challenges associated with LLMs and make them more adaptable to different use cases. However, they acknowledge the limitations of technical solutions and the need to consider deployment and ethical considerations when using such models in real-world applications.
Strengths
The paper introduces the concept of Deep Language Networks (DLNs), which stack multiple Large Language Models (LLMs) and train them jointly using variational inference. This approach offers a new perspective on leveraging the power of LLMs and demonstrates improved performance compared to a single-layer model. It presents a layered architecture that decomposes the task into a series of smaller sub-tasks, each of which is more easily solvable by an LLM. This decomposition allows for more efficient training and better performance. It details prompt engineering and in-context learning techniques for optimizing the prompts associated with each layer of the DLN. These techniques enable fine-tuning of the DLN's performance and allow for better adaptation to different tasks and datasets. The paper utilizes variational inference to learn the prompts in the DLN. This approach enables the joint search over the prompts, addressing the challenge of optimizing multiple prompts in a deep architecture. The paper provides experimental results on various datasets, demonstrating the effectiveness of DLNs compared to single-layer models. The results show that DLNs can achieve performance comparable to higher-capacity models like GPT-4, even when each LLM in the network is smaller and less powerful. It discusses practical aspects of DLN implementation, such as proposal diversity, learning in-context learning, and backtracking and memory optimization. These considerations make the DLN approach more feasible and effective in real-world scenarios. The strengths of the paper lie in its innovative approach, theoretical framework, practical insights, and experimental validation, showcasing the potential of Deep Language Networks for improving language modeling tasks.
Weaknesses
The proposed Deep Language Networks (DLNs) introduce a more complex architecture compared to single-layer models. This complexity may make the implementation and training of DLNs more challenging, requiring significant computational resources and expertise. Although the paper compares DLNs to single-layer models, it does not provide a comprehensive comparison to other state-of-the-art language models or architectures. Without such comparisons, it is difficult to assess the relative performance and advantages of DLNs against other advanced models. Additional evaluation metrics would provide a more comprehensive assessment of DLNs' strengths and weaknesses. The experiments and results presented in the paper may not generalize to all types of language modeling tasks and datasets. The performance of DLNs could vary depending on the specific domain, language, or task requirements. The paper should provide a detailed analysis of the limitations and potential failure cases of DLNs in different scenarios. DLNs consist of multiple layers of Large Language Models (LLMs), which can increase the computational overhead during training and inference. It does not extensively discuss the computational efficiency of DLNs, including the training time, memory requirements, and inference latency. These factors could limit the scalability and applicability of DLNs in resource-constrained environments.
Questions
Can you provide more details on the computational and memory requirements of the proposed Deep Language Networks (DLNs) compared to single-layer models? Are there any specific challenges or limitations associated with implementing and training DLNs that were not mentioned in the paper? Could you include a more comprehensive comparison of DLNs against other state-of-the-art language models or architectures? This would help understand the relative performance and advantages of DLNs in a broader context. Are there any plans to explore alternative metrics in future work?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
Yes, the content you provided appears to adequately address the requirement of addressing the limitations and potential negative social impact of the authors' work. The authors acknowledge the limitations of addressing societal issues through technical work and emphasize that their modular approach aims to alleviate some of the issues associated with large language models (LLMs), such as concentration of power and difficulty in training. They also express the hope that their approach will make LLMs more adaptable and suitable for a wider range of use cases. However, the authors explicitly state that they do not address the deployment of such models, when and how they should be used, or provide additional guarantees against their misuse. They highlight the importance of considering the performance of these models in uncontrolled environments and the need for justification before deploying them in high-stakes situations. By acknowledging these limitations and potential negative societal impacts, the authors demonstrate transparency and responsibility. They recognize the need for further consideration of ethical and societal implications beyond the technical performance of their work. The content provided covers the requirement and fulfills the guidelines for addressing limitations and broader societal impacts as specified in the checklist.
Summary
This paper suggests stacking multiple large language models (LLMs) together, with tunable parameters represented as prompts at each layer. Given that these prompts are discrete natural language elements, their direct optimization using a gradient-based method is challenging. To address this, the authors propose a prompt optimization framework: (1) generating N local candidates using a proposal distribution, and (2) scoring each candidate to select the one that maximizes the scoring function. To apply this framework to a 2-layer setting, they propose a variational inference objective introducing an additional hidden proposal. The effectiveness of this method, tested on multiple datasets, is demonstrated in both 1-network and 2-network settings.
Strengths
The proposed prompt optimization framework is both general and plausible, with its extension to the 2-layer setting using variational inference technically sound. The exploration of stacking multiple LLMs together is an intriguing and under-researched area. The paper's comprehensive experimentation on various datasets effectively showcases the method's applicability in both 1-layer and 2-layer settings.
Weaknesses
The paper's title, "Deep Language Networks", is misleading, as the main content only employs 2-layer LLMs. The term 'deep' should be avoided to prevent any misconceptions and exaggeration. While the introduction claims that existing modular approaches are heavily dependent on prompt engineering to break a task into smaller tasks, the proposed method also relies on an initial given prompt. This initial prompt prescribes the subtask for each layer. For instance, in the 2-layer setting experiments, the first layer always serves as the COT step, and the second layer provides the final answer. The roles of the two layers are determined by the initial human-provided prompt. Though the paper suggests that it's easy to generalize from a 2-layer network to multiple layers, it's not straightforward to establish the initial prompts for each layer. While it's manageable in a 2-layer setting, mirroring the COT steps where the first layer handles reasoning and the second layer provides the final answer, it becomes substantially more challenging as the layer size increases. Therefore, the main motivation of this paper may not be as practically applicable as suggested.
Questions
In the context of variational inference, is the LLM that parameterizes q(h) separate? A straightforward baseline would be COT+APE. How does the performance of this simple baseline compare? Stacking multiple LLMs increases computational complexity. An alternative is to directly use an ensemble of multiple LLMs. How does the proposed method compare with a simple ensemble of multiple LLMs? This could offer insights into whether LLMs should be combined horizontally or vertically.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
Please refer to the Weaknesses section for a detailed discussion of the paper's limitations.
Summary
The authors provide an interpretation of LLMs as shallow language networks. They explained how One-layer language networks can be used for joint prompt training, and then moved to stacked (two) language networks. The authors propose to use variational inference for the training, and figured out a few practical instantiation that makes the optimization work well.
Strengths
Viewing LLMs as stochastic layers and prompts as tunable parameters, the authors provide a framework for joint prompt training of stacked LLMs using variational inference. The overall idea looks sound, and the framework and variational inference solution look elegant. The authors also experimentally verify their Deep Language Network on a few artificial datasets, and achieve comparable performance. The ablation study aligns with the authors' claim.
Weaknesses
-- As also claimed by the authors, it is difficult to make Eq1 (the bound) tight, and the optimization is also not straight-forward. In order to make the whole system work, a lot of practical instantiation is needed, making the whole system more cumbersome. In order to get diverse prompts and identify the near-optimal prompts, I expect the value of K and N in algorithm 2 will not be too small. An ablation study looking into that, to study the computational cost and performance effect given different K and N could be interesting. -- Also, this work only evaluates on small scale datasets, and works on some artificial tasks. This may prevent the audience from understanding the benefit of the proposed methods. Is it because the training/evaluation is not so efficient, so the authors decided to work on a small scale? -- I wonder if the proposed methods can be easily extended to three layers or beyond. Technically, it seems possible, but the optimization could be very challenging. -- In line 186, the authors mentioned that they selected lambda for each task. This could be cumbersome when working on a new task/dataset.
Questions
Can the authors explain the results in Table 2 with richer information. Why does the approach seem to work very well on Nav and Subj, but worked clearly worse than other strong baselines? The 2-layer DLN end-to-end worked badly presumably due to the difficulty of optimization mentioned in section 4?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors present some bullet-points on potential limitations in the draft, I did not identify significant negative societal impacts.
Summary
This paper introduces a novel discrete prompt tuning method. The paper is discussed under a scenario that regarding discrete prompt as the only tunable parameters while freezing all other model parameters. In this sense, the author focuses on getting optimized prompt from one LLM and feeding which as the input prompt to next LLM. Such kind of stacking LLMs workflow is regarded as the main contribution as the authors claimed. Moreover, the authors introduce a new latent variable to further improve this pipeline. In the experiment, the authors verify the effectiveness of the proposed method via comparing with a bunch of baselines including several in-context learning method and one instruction-tuning method APE. The authors highlight the outperformance of their method over 0-shot GPT-4 is achieved via using text-davinci-003 backbone on Hyper, Tree, Disaster dataset.
Strengths
originality: This paper shows a novel discrete engineering method from a new perspective that interprets the process of getting optimal prompts as the stack of two LLMs. This angle is novel and interesting. From the methodology perspective, authors develop their method based on improved APE and generalize which to 2-layer case. quality: overall quality is mediocre, though the idea is novel, the authors do not provide enough evidence to support their claim. clarity: the authors attach the codes for 1-layer & 2-layer case, which makes it easier to interpret their ideas. However, the intuition for some settings need to be improved. significance: authors evaluate the proposed 1-layer method on 9 tasks and 2-layer method on 4. They re-run the experiment 3 times for verify the significance.
Weaknesses
1. The paper is not comparing with enough correct baselines. The authors are mainly comparing with 0-shot GPT3&4 and APE method in two main tables. However, if you regard the number of training example that proposed method can touch with as the x-shot, then the comparison objective should be x-shot GPT3&4 instead of 0-shot GPT3&4 (especially in Table 2). It's widely acknowledged that increasing the number of few-shot examples (in a range) would enhance model's performance. Is it still fair to compare with 0-shot model with your proposed method? Then it means, the only baseline authors are correctly comparing with is APE. I'd encourage authors to compare more baselines in both discrete prompt engineering and continual-space prompt tuning area. 2. The generalization of the proposed method is questionable. The author claims their proposed method can generalize to more number of layers. But the comparison between 2-layer and 1-layer model is insufficient. Authors only compare them on 4 datasets and on some of them the 1-layer results are even better than 2-layer's. It's not convincing. 3. Even the authors have mentioned in conclusion&future work that testing with other LM is needed. I want to highlight this as a major weakness of the paper. The value of this paper should be reevaluated is it only works on GPT-3 model. Why not stacking two GPT-4 models which should not be hard to do (just change slightly in API calling)? Or maybe on BERT/RoBERTa models. Without corresponding evidence, it's hard to justify the effectiveness of the proposed method. 4. The intuition and some setting of the proposed method is unclear. For example, in Line 137, the authors suddenly define the a weight without illustrating why the sum of these two terms is a good fit for their purpose. I am little confused to interpret. 5. The authors also mentioned that they hope to match the performance of the largest LLMs without incurring the large computational and data cost. However, one should also be careful that GPT-3 API calling is also a considerable computational cost. Without disclosing the number of times they call the GPT-3 API, it's hard to support their claim as incurring low computational cost.
Questions
1. I kind of curious why you are adopting a subset of test set for evaluation. Any reason for that? 2. I kind of doubt the performance of 0-shot GPT-4. For example, on Subj dataset, FT RoBERTa and prompt tuning RoBERTa can achieve ~97% score, while 0-shot GPT-4 can only get 65.8. I am little shocked by GPT-4's bad performance on this task. It would be better for the authors to release the prompt of 0-shot gpt-4 they use for reproducing purpose. 3. Why not reporting the 95% confidence interval on APR & 1-Layer LN & 2-Layer LN in Table 2? 4. Why you split the main results into two tables?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
1 poor
Presentation
2 fair
Contribution
1 poor
Limitations
typo: Algorithm 2 Line 13 \beta_k -> \beta_1 Algorithm 2 Line 18 \pi_0^n -> \pi_0^I Algorithm 2 Line 19 \pi_1^n -> \pi_1^I
further improvements are necessary for this paper
Thank you for your reply. Here are a few thoughts I'd like to share regarding your responses. 1. **"Insufficient comparison with correct baselines"** Table A indeed reveals the substantial benefit of incorporating more examples, resulting in a notable improvement of approximately +10 for both the ICL and KATE models in navigation and subjectivity tasks. As previously pointed out, it's worth contemplating whether this comparison is equitable, given that you extract information from 400 examples while the baselines have access to only 5/10/32 examples. To promote thorough analysis, I recommend conducting an ablation study where your models are trained with the same number of examples as the baselines. Additionally, I reiterate my belief that the current version lacks an adequate comparison with a comprehensive range of baselines. Expanding the scope of baselines for comparison could enhance the robustness of your findings. 2. **"Scale beyond two layers; two layers is not always better than one layer"** You've highlighted a significant discrepancy in your response. While you explained that "scaling beyond two layers might be a challenging issue," your submission predominantly emphasizes the ability of your model to incorporate numerous LLMs and attains superior performance compared to a single layer (as evidenced between Line 3-5). This might be perceived as an overstatement. Notably, your original submission omits any mention of the unfavorable generalization to additional layers. 3. **"Testing with other LM is needed. I want to highlight this as a major weakness of the paper."** You've made a valid point regarding the inclusion of GPT-4 and the open-source model WizardLM, which is a positive addition. However, I find it difficult to concur with the methodology of computing solely based on the number of tokens during inference. Indeed, your model necessitates training, which involves multiple API calls, unlike ICL which doesn't require training. To ensure a fair comparison, wouldn't it be more appropriate to report the number of tokens involved in both stages? Furthermore, I'd like to emphasize the importance of presenting comprehensive results across a broader range of datasets, rather than solely focusing on the four datasets mentioned earlier. **"Why you are adopting a subset of test set"** Utilizing a subset of the test set for evaluation becomes more reasonable when evaluating the entire test set is resource-intensive or when baseline methods themselves utilize such an approach. However, in your case, neither of these circumstances seem to apply. Without aiming to be overly critical, I'd like to point out that the paper you referred to has two distinctions from your submission: firstly, it lacks a direct relationship to your work, and secondly, it employs a subset of the development set rather than the test set. I still not get the response for my questions **"Why not reporting the 95% confidence interval on APR & 1-Layer LN & 2-Layer LN in Table 2?"** and **"Why you split the main results into two tables?"**. Considering the current quality of the submission and the need to incorporate more comprehensive experiments, I maintain my stance that further improvements are necessary for this paper.
clarifications on your responses
Thank you for the answer. Before replying, we want to emphasize that these models are expensive and we need to be mindful when selecting which experiments to run. We would love it if you could provide a bit more information about what experiments you would like us to prioritize, which conclusion you would get from the additional results, and how that would affect your assessment of our work. While waiting for an answer, we still decided to run as many of the requested experiments as we could, hoping that this can address your concerns. --- ***As previously pointed out, it's worth contemplating whether this comparison is equitable, given that you extract information from 400 examples while the baselines have access to only 5/10/32 examples.*** KATE consists of a retrieval step and a prediction step. The retrieval step is executed using *400* examples. Due to the limit of the context length in davinci-003, the prediction step can take at most 32 examples. Therefore, the KATE baseline we report uses all of 400 examples. We would be happy to implement other baselines you think are more equitable. --- ***the current version lacks an adequate comparison with a comprehensive range of baselines. Expanding the scope of baselines for comparison could enhance the robustness of your findings.*** We would be happy to implement other baselines you suggest that can enhance the robustness of our work. --- ***To promote thorough analysis, I recommend conducting an ablation study where your models are trained with the same number of examples as the baselines.*** We currently ran DLN-1 on the 4 tasks from the rebuttal and updated our results: | | nav / #tok | date / #tok | logic 7 / #tok | subj / #tok | | - | - | - | - | - | | ICL - 10-shot | 61.3 / 151k | 62.9 / 263k | 38.9 / 529k | 72.0 / 119k | | ICL - 32-shot | 66.0 / 449k | 63.5 / 786k | exceeds ctx len | 83.2 / 343k | | DLN-1 - 32-shot | 69.1 / 26k | 55.0 / 136k | 42.1 / 77k | 83.1 / 55k | --- ***While you explained that "scaling beyond two layers might be a challenging issue," ... your original submission omits any mention of the unfavorable generalization to additional layers.*** We do not think that “scaling beyond two layers might be challenging” means “unfavorable generalization to additional layers”. Instead, while it might be hard to train multiple layers, very much like it was hard for neural networks at the beginning, the potential gains could also be very real if one manages to do so. Our experiments with DLN-2 show that this could be the case. As mentioned in the general response, we will tone down the abstract and intro in the camera-ready. --- ***However, I find it difficult to concur with the methodology of computing solely based on the number of tokens during inference.*** Please refer to the Appendix where we report the cost of training DLN-2. It costs 60 dollars to train a dln-1 and 273 dollars to train a dln-2. We therefore believe such amounts during training are thus negligible when serving the model for a large number of users: tokens spent per user request at test time is more important in this context, as this is a cost you can't amortize. --- ***Furthermore, I'd like to emphasize the importance of presenting comprehensive results across a broader range of datasets, rather than solely focusing on the four datasets mentioned earlier.*** We restricted ourselves to the datasets reported because every run is fairly expensive. Please suggest datasets you find particularly useful to strengthen our work. --- ***“it employs a subset of the development set rather than the test set.*** The reported results in Lu et al. are computed on a 256 subsample of the dev set. Please refer to Table 2 caption – first line – and their github repo: https://github.com/yaolu/Ordered-Prompt/blob/e93f70d0a5f6a8cfcadf6f2917c26eed265cd0be/config/cb.yaml#L2 The reason the authors give is for computational efficiency. --- ***"Why you split the main results into two tables?".*** Because running DLN-2 is more expensive and we thought it would be sufficient to do it on 4 datasets, and we feel these 4 datasets are the most representative of the range of behaviours we want to analyze. --- ***"Why not reporting the 95% confidence interval on APR & 1-Layer LN & 2-Layer LN in Table 2?"*** | | **Nav** | **Date | **Logic 7** | **Subj** | |:-----------------------:|:--------------:|:---------------:|:--------------:|:--------------:| | **0-shot** | 64.1 | 56.4 | 45.9 | 61.7 | | **APE-15** | 67.3 $\pm$ 7.7 | 32.1 $\pm$ 28.6 | 45.5 $\pm$ 4.7 | 61.3 $\pm$ 7.2 | | **APE-400** | 56.9 $\pm$ 32.9| 23.5 $\pm$ 14.1 | 45.6 $\pm$ 12.4| 63.7 $\pm$ 9.2 | | **DLN-1** | 67.1 $\pm$ 7.6 | 55.7 $\pm$ 4.5 | 47.5 $\pm$ 2.1 | 83.2 $\pm$ 6.5 | | **DLN-2 (end 2 end)** | 83.1 $\pm$ 24.7| 65.9 $\pm$ 4.0 | 45.7 $\pm$ 3.5 | 85.9 $\pm$ 8.7 |
I know that conducting experiments is expensive for sure. However, this shouldn't be a reason for preventing you do some meaningful experiments if they are necessary for your paper. **32-shot ablation study** That's precisely in line with my understanding. When considering an equal number of accessible data points, the proposed approach manages to outperform ICL baselines on just one dataset (also other baselines are missing here, the performance of other methods' 32-shot versions compared to your model remains uncertain). If this holds true, can I deduce that your method excels over ICL only for one dataset while employing fewer tokens (thus, incurring fewer costs)? However, in a broader sense, can we confidently recommend your method to users seeking optimal performance whenever they encounter a new dataset? I'm concerned that the answer might be negative. **The claim about generalizing to more than 2-layer networks** Indeed, it's hard to overlook the fact that your initial submission's assertion could be seen as somewhat misleading, a sentiment shared by Reviewer SaT7. Moving forward, a more cautious approach to highlighting the capabilities of the proposed method might be advisable, particularly since it hasn't been subjected to testing involving more than two layers. This could contribute to a more accurate representation of the method's potential in your upcoming version. **Costs in training/testing** Despite your assertion that training costs are a relatively minor factor, it's noteworthy that the training cost for the proposed 2-layer and 1-layer methods is 2.9 million and 1.4 million respectively, on a single dataset (in contrast, ICL-5-shot GPT-4 costs 83k, 291k, 66k on three test sets). When these training costs are combined with practical testing expenses, the overall expenditure does indeed largely exceed that of the baseline methods. I do see your point that testing takes majority part when the number of the test is larger. However, in your current situation, wherein the number of test set is small and the total cost of your model surpasses the baselines. Moreover, the separation of reporting training costs and emphasizing testing costs might be perceived as inequitable, especially for methods that do not entail training costs, even if the focus is primarily on testing expenditures. **Why you are adopting a subset of test set** I read the paper and I know what they did. But you were still not explaining why you test under a subset instead of full set. As training on 2.9 million tokens takes roughly $273, according to table C the ICL-5-shot GPT-4 takes 66k and DLN-1 takes 57k on subj dataset. Given this context, it raises the question of whether the expense of testing on the entire subj test set would truly be prohibitively expensive. **Why splitting two tables** I just feel the two tables are very duplicated as they are comparing under the same datasets for the same purpose. ---- **I am summarizing my review comments for this submission as a conclusive evaluation. There is no need for a response.** I find it challenging to discern the advantages of employing the proposed method, especially when considering the scenario of controlling the same number of training data accessible. Firstly, traditional trainable methods such as fine-tuned Roberta showcased notably superior performance in some of the tasks tackled by the authors. Secondly, in instances where GPT API calls are exploited, the outcomes do not exhibit a distinct pattern compared to baselines that also employ blackbox models. Certain aspects of the submission appear misleading (as discussed in my previous response) — notably the claim about stacking l-layer LM, and the presentation choice of not combining training and testing costs. The experimentation aspects require greater rigor. While the authors included numerous additional experiments during the rebuttal, some were conducted on partial datasets rather than the complete ones specified in the paper. I hope the authors will incorporate and conduct experiments on all nine datasets mentioned in their next version, rather than solely for the purpose of persuasion. Furthermore, some of the baselines' implementations could benefit from enhanced rigor. For instance, the authors mentioned that the ICL method randomly samples five examples from the training set in Line 226. Given our understanding that the order and quality of ICL examples influence results, conducting a significant test considering the randomness involved would be prudent. **In conclusion, I believe this submission is not yet prepared for acceptance.**
Questions/Concerns?
Dear Reviewers, Thank you for your time. We know it’s a busy moment, but we would like to know if there’s anything else we can do to clarify your concerns about our paper. --- ***Reviewers Y4LZ, Ttvx:*** Thank you for your positive feedback, we hope the rebuttal helped in answering your questions. --- ***Reviewer SaT7:*** - Your first concern was the paper contained “deep” in the title while only dealing with 2-layers DLN. We are open to changing the title, and we already toned down the abstract and intro by making the limits of this work to only 2-layers more evident. Would you have suggestions for the title? - We also implemented your proposed APE+CoT baseline as you can find in the common response. Was that helpful in addressing your concerns? --- ***Reviewer 2Jdr:*** Thank you for engaging with us. If you could be more specific about which baselines and which other datasets you think could robustly improve our work, we would be happy to work on that, always in the limit of our computational resources. Please note that we also added a APE+CoT baseline following SaT7’s suggestion. --- Is there anything else that we can do to address your concerns? Thanks!
Any further questions ?
Dear SaT7, Do you have any further clarifications you need from the authors ? Thanks.
Thanks for the Rebuttal
I appreciate the detailed rebuttal you've provided. Considering the points raised, I would like to suggest that the paper title include the phrase "Towards Deep Language Networks..." to better encapsulate its focus. However, it's important to acknowledge that several of my concerns are contingent upon a substantial rewriting of the paper. Given the current state, I am inclined to maintain my current score and continue to view the paper as a borderline case.
Looking forward to
Thanks for the clarifications, looking forwarding the seeing your updates.
Common response: abstract update and clarifying test set sizes
Dear reviewers, we wish to provide both an update and a clarification. ## Abstract update We updated the abstract to align it better with the current state of the framework. We are confident it will address the issues several of you raised. The new abstract is: We view large language models (LLMs) as stochastic language layers in a network, where the learnable parameters are the natural language prompts at each layer. We stack two such layers, feeding the output of one layer to the next. We call the stacked architecture a Deep Language Network (DLN). We first show how to effectively perform prompt optimization for a 1-Layer language network (DLN-1). We then show how to train 2-layer DLNs (DLN-2), where two prompts must be learnt. We consider the output of the first layer as a latent variable to marginalize, and devise a variational inference algorithm for joint prompt training. A DLN-2 reaches higher performance than a single layer, sometimes comparable to few-shot GPT-4 even when each LLM in the network is smaller and less powerful. ## Dev / Test set confusion After discussion, we felt maybe some points regarding test set size were still unclear and thus we would like to clarify. ***Why don’t we use the full test set?*** The test sets of the original data are below: | Task | Test Examples | |:------:|:------:| | Hyperbaton | 250 | | Navigate | 250 | | Date Understanding | 250 | | Logic. 7 | 250 | | Subj | 2000 (8x) | | Mpqa | 2000 (8x) | | Trec | 500 (2x) | | Disaster | 5430 (~22x) | | Airline | 7319 (~29x) | Given that we have a large set of baselines, ablation study settings and 3 seeds, we would like to be mindful of the number of queries to GPT API. We chose to subsample every test set to use 250 examples. We would like to stress that we are not the first in doing so. This closely follows the setting in Lu et al. (ACL 2022 outstanding paper). They report: > For evaluation, we sub-sample 256 samples of the validation sets for all datasets to control for the GPT-3 inference costs as it requires the usage of a monetary paid-for API. This means they sample 256 data points from the dev set and ***report scores on those 256 data points***.. We can confirm this from [their official code](https://github.com/yaolu/Ordered-Prompt/blob/main/config/subj.yaml#L2). We believe this could have caused the confusion because Lu et al. do not conduct a hyperparameter search and directly use the 256 dev data points for testing.
Thanks for the information, and I agree that the planned changes would make the paper stronger, and I’m looking forwarding a later version. I would keep the score as it is for now.
Decision
Accept (poster)