Building a Large Japanese Web Corpus for Large Language Models

Summary

The paper constructed a high-quality Japanese web corpus which was extracted from Common Crawl dataset for the training of Japanese large language models. Compared with existing Japanese text corpus which was developed overseas as a part of multilingual corpora, the corpus in this paper was specially developed for Japanese and enjoys better quality. The paper also demonstrates a Japanese text collection and cleaning pipeline and empirically analyze the improvement that the newly collected Japanese web corpus can bring to Japanese large language models.

Rating

5

Confidence

5

Ethics flag

1

Reasons to accept

1. The paper developed a high-quality Japanese text corpus which will be beneficial to Japanese language model training. 2. The paper also provides a text data collecting and cleaning pipeline which is not limited to Japanese but can also be applied to any other non-English language. 3. The improvement that the developed Japanese corpus brought on language models looks very impressive.

Reasons to reject

1. The paper has limited innovations and technical depth. The paper proposed very little novel techniques or innovations and its data collecting and cleaning pipeline looks straight forward and only involves data engineering components such as rule-based quality filtering (in which the rules are not hard to design). 2. The motivation of this paper is not convincing. Although the paper claims that all existing Japanese text corpora are just part of some multilingual corpora and suffer from unsatisfactory quality, there are no concrete evidence to support this argument. In addition, the paper does not provide any concrete problems of the existing Japanese corpus and the reason why they have such problems. Even if the existing Japanese corpora have some problems, I am afraid that they can be easily resolved given that the data collecting and cleaning pipeline is so straight forward. 3. Although the paper claims that it is dedicated to develop a Japanese-only text corpus, I can hardly identify any techniques proposedin this paper which is specifically designed for Japanese. 4. The presentation should be further improved. For example, the term "Llama 2 7B" looks wired and looks like the name of some model at the first glance. However, the model's name is "Llama 2" and the number of parameters is "7B".

Questions to authors

1. Could you please demonstrate some cases in the existing Japanese corpus whose quality is not satisfactory? It seems that from Table 1, each existing Japanese corpus also employed deduplication and data cleaning and they are supposed to be good enough. 2. Please also demonstrate the exact problems of the existing Japanese corpus and the reason why they have such problems and the reason why the problems cannot be easily tackled.

Summary

The paper presents a corpus building project to build a large Japanese web corpus for LLMs. Using Common Crawl WARC as a base is a good idea but a lot of other Japanese related corpora are driven by Japanese academics in their silo networks that has yet been tapped in modern LLM training, e.g. NICT LLM https://www.nii.ac.jp/en/news/release/2023/1020.html and https://www.aozora.gr.jp/#main and more on https://github.com/llm-jp/awesome-japanese-llm?tab=readme-ov-file. Also, references to those work is lacking in this paper (but it's understandable since a lot of the work done in those research labs are disseminated in Japanese only). The steps taken in the data processing are clearly described, esp. the language ID and ngram filter process. As for the "quality filtering" process of using character sets (Section 3.2), it might be that the filtering by character sets have implicitly also restrict the domain that the model will be learning from, e.g. there's a lot of Japanese BBS (bulletin board sites) / forums where substrings like "kaomoji" might be used e.g. ٩(ˊᗜˋ*)و https://detail.chiebukuro.yahoo.co.jp/qa/question_detail/q11297327146 and https://komachi.yomiuri.co.jp/ Lastly, there are also many nuances in cleaning Japanese texts e.g. half-width / full-width characters and furigana mixed into the texts. It would be nice to have some breakdown of how much each of the filtering process taken out, esp. how much of the final text is made up of the Wikipedia dump.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

Details on data cleaning and data creation is precious and these paper should be out in the public more to make LLM training more transparent and open.

Reasons to reject

Personal preference to accept papers on data creation and processing (thus the 6 score) but I'll say it's a weak accept, so reasons to reject includes: - No reference to other Japanese compilation work not using common crawl and web crawled data (see summary comments above) - Lacking some breakdown on how each much data was filtered out in each step - In data compilation study, depth of ablation (how much data affects a handful model) is more preferable then breath (how much of the performance the data affects different many LLMs). Priorities of computation usage would be better if it's covering depth.

Questions to authors

Suggestion ===== - Please add the breakdown of how much data is filtered out in each of the process and if possible sub-processes in the data creation. - Increase the spacing between lines in Table 2, esp when there are dotted or solid lines. - Cherry pick some examples where the model performs better after training with the data presented in this paper - If possible, please reference these other LLMs too https://huggingface.co/tokyotech-llm/Swallow-70b-instruct-hf and https://huggingface.co/Rakuten/RakutenAI-7B Questions ===== - Is there reason or particular examples for XL-Sum where the 7B models performs worse with the data in this paper? Why did it happen? Is it because the data after the cleaning process inherently (not on purpose) filtered out specific domain or specific length of the texts?

Summary

The authors created a novel Japanese corpus for training large language models from Common Crawl archives. It consists of 312B characters, which is substantially larger than previous training corpora for Japanese LLMs, such as CC-100 (25.8B characters), mC4 (239.7 B characters), and OSCAR (74B characters). Using their Japanese web corpus, the authors performed continual pre-training on various LLMs, such as Llama 2 and Mistral, and gained consistent improvements on Japanese benchmark datasets.

Rating

8

Confidence

5

Ethics flag

1

Reasons to accept

The authors have created a novel open Japanese web corpus comparable in size and quality to publicly available English web corpora. It will significantly contribute to the advancement of Japanese LLMs compared to training LLMs using the Japanese portion of a multilingual corpus. The authors proposed a lightweight language detection method to balance the speed and quality of Japanese text extraction. Using the Japanese web corpus, the authors performed continual pre-training on more than ten publicly available LLMs, and they extensively compared their performance using representative Japanese evaluation benchmarks, including llm-jp-eval and lm-evaluation-harness.

Reasons to reject

Many aspects of the proposed method, such as filtering based on character type and length, are only applicable to Japanese. The evaluation was only when the authors performed continual pre-training on existing LLMs using the Japanese web corpus. They do not demonstrate whether their corpus would be effective if they train an LLM from scratch.

Questions to authors

I want your current estimates of whether the Japanese web corpus has the quantity and quality necessary to train an LLM from scratch. This is not a question, but a comment. Using shorthands such as BL (billion letters) and BW (billion words) in Table 1 is okay. However, the authors should avoid their use in the body of the paper.

Summary

This paper presents an effort to create a very large corpus of trustworthy Japanese text that could be used to train LLMs. They describe the method used for obtaining candidate data, cleaning, and deduplicating it. In order to show the quality of the resource, the authors train different LLMs on this data and show that it actually improved the metrics on a benchmark of Japanese NLP tests.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

Although Japanese is not a low resource language itself, it is not one of the highest resources languages in NLP either, so the effort to create a very large corpus is very interesting. I particularly liked the thorough explanation of the data collection pipeline (including downloading, filtering, deduplicating, cleansing, etc.), which could be used as an inspiration to create corpora in other languages as well. The resource built seems very relevant, and the results shown over standard benchmarks seem to show it has high quality as well.

Reasons to reject

Some parts of the document are in need of more clarifications, for example the quality and hostname filtering, and the explanation of section 4.1 and associated Table 2.

Questions to authors

- By continual pretraining (or is it continued pretraining?) I understand you take the pretrained model weights and keep on training on your new data. Is this the case? Please make this clear, perhaps indicate some references where the technique is introduced or used. - Please explain what NG expressions are. Does is refer to js or other programming languages content? - When you refer to "blocklists" I think the correct term is "blacklists", or is it another concept? - I am intrigued by the Japanese Copyright Act mentioned in the ethics statement that says the purpose of the person is not to "enjoy" or cause another person to "enjoy" the work (quotes are from the text). Is this some translation issue, or what is being referred to as enjoy in this case? Does this mean the use for monetary gains of the data?

Initiating the discussion

We sincerely thank the reviewer for the valuable suggestions. We hope our response has thoroughly addressed all concerns and that the reviewer can consider improving their score based on our response. There is nothing we want to add to the response, but we are also happy to discuss with the reviewer if they still have a concern.

Initiating the discussion

We sincerely thank the reviewer for the valuable suggestions. We hope our response has thoroughly addressed all concerns and that the reviewer can consider improving their score based on our response. There is nothing we want to add to the response, but we are also happy to discuss with the reviewer if they still have a concern.