Summary
The paper presents OWL model and OWL-Instruct dataset aimed at IT operations where the seed dataset is crafted by subject matter (IT) experts across 9 prevalent operations and maintenance domains. Inspired by NBCE, the authors propose HMCE to extend the context length. In the supervised fine-tuning of the model they suggest a Mixture-of-Adapter method to improve the instruction-tuning performance. The dataset is rooted in some manually labeled data from domain experts which are enriched in a Self-Instruct fashion with supplementary samples. The Owl-Bench benchmark is another contribution of this work derived from real-world questions.
Strengths
The paper comes with multiple orthogonal contributions each of which could be significant in their own sense contingent on proper evaluation.
The MoA adapter strategy is a significant contribution introducing task-specific adaption with a mixture of experts.
Expansion of token set by domain-specific tokens, while not novel, enhances the value of the proposed work.
The HMCE, inspired by NBCE, is yet another novel (to the extent of reviewer's knowledge) context extension approach.
Weaknesses
Regardless of the above strenghts, the paper is proposing a diverse set of ideas and tries to evaluate them within the context of the paper. However, given the orthogonality of the ideas and the relatively high number of them, the authors haven't been able to thoroughly assess each idea, particuarly the theoretical ones (e.g. HMCE, MoA) independent of the empirical setting of OWL and IT operations.
The HMCE contributions does not go well with the rest of the story. Based on the Evalutions in section 5.3, HMCE was evaluated with an independent set of tests where the concatenated question to create long input squences. While the results show that HMCE outperforms NBCE as a training-free context extension approach, it does not seem to have anything to do with the OWL model and its dataset.
From the understanding of the reviewer, in Figure 3, the baseline models have not been benefitng from MoA adapting (or LoRA). If that is the case, the comparison is not entirely fair.
Typos:
Probably a typo: "...where a group of LoRA adapters is lightweight compared to the pre-trained model."
Typo page 8: "We have propose HMCE..." --> proposed
Typo page 8: "Without NMCE" --> HMCE
Questions
In Appendix A, when embedding matrix is appended with new rows to expand D to D' and include the newly trained IT-operations token, do you retrain any of the LLaMa layers? Please elaborate on how the heads encoding the embedding space were retrained to incorporate the new tokens.
The results in Figure 3, has OWL leveraged MoA? If so, have the other baseline models benefitted from the same MoA training so that the comparison remains fair?
Rating
6: marginally above the acceptance threshold
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.