General Detection-based Text Line Recognition

Summary

This manuscript provides a new approach to line recognition of offline handwriting and printed documents with regard to multilingualism. This method can recognize a line based on its simultaneously recognized features based on transformers. This approach is interesting for document analysis. Promising results have been obtained for different datasets.

Strengths

This paper is well written and organized. Since the method recognizes the characters simultaneously, this approach is efficient. This paper explores the impact of using a synthetic dataset to improve accuracy in practice.

Weaknesses

This is not a whole system for the field of document analysis. This task is a part of document analysis that depends on line segmentation. This means that if the previous part (line segmentation) of the process has errors, these are transferred to the next steps. Therefore, this is an essential task for the work. What happens if you have a whole document for the task? What happens to your results if you use existing methods that segment lines? This method is not generally suitable for multilingual contexts. If there are several characters in different languages, this paper has no clear solution. You should therefore have several models for each language. It is better to explain the method for your synthetic data in more detail in the Appendix. It is better to clarify this with an illustration. Please check your text again for some errors, e.g. line 263 (annotations annotations). Is line 230 correct? "the English Volume 0002"

Questions

When I compared your transformer architecture to the original [57], Figure 2 is different from the original. They use a query selection and matching. Why do not you have these parts? I can see them in your description, but not in your illustration. You cite Figure 1 in subsection 4.2, why do you give the figure as Figure 1 without citing it earlier? What happens if you have a document with multiple languages? Do you test your method for the type of input? Or should you train your network for this type of data? Why don't you create the synthetic data for other languages and use a fine-tuning model based on English? Have you considered the scenario?

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

They showed some limitations in Figure 4. But it needs to add more samples in an Appendix.

Summary

The paper introduces a novel transformer-based character detection for text line recognition. The authors use a diverse set of synthetic data to enable localization part of the detection network to generalize to unseen characters during training. The transformer-based detectors can identify all characters in a text line in parallel and a masking strategy has been adopted to encourage detection interactions. The methods proposed by the authors also includes a process to fine-tune the detection network using only line-level annotations.

Strengths

S1: The approach is novel for character detection for text line recognition using transformer-based models. S2: The authors demonstrate strong performance by the model across several datasets and outperformance for cipher recognition. S3: The model can generalize well to unseen characters and variations in text as the authors utilize synthetic data.

Weaknesses

W1: Several typos throughout the paper. Ex: typo in the Fig 1 caption: “Our model is general can be”, W2: The authors can provide additional information on the fine-tuning process and the adaptability of the masking strategy. Did they explore other strategies? W3: The performance on real-world datasets that contain some-to-large amount(s) of noise has not been extensively discussed by the authors.

Questions

Q1: How does the model handle noisy datasets or texts with degraded quality? Q2: What were the computational resource requirements? Can this approach be extended to real-world problems in a different setting? Q3: How would LLMs affect or change the impact of this approach? Have the authors considered this aspect, and if the proposed method can do better than some existing LLMs? Q4: Can this approach be extended to multi-line text detection and recognition?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

L1: The efficiency of the proposed method in real-world datasets with noises has not been fully explored.

Summary

This paper presents a novel detection-based approach to text line recognition for both printed (OCR) and handwritten text (HTR), covering Latin, Chinese and cipher characters. Traditional detection-based methods have been largely neglected in HTR due to the difficulty of reading characters separately and the high cost of character-level annotation. The authors propose a solution to these challenges through three main insights: (i) using synthetic pre-training with diverse data for character localisation across different scripts; (ii) using modern transformer-based detectors to handle multiple character instances simultaneously, using a masking strategy to ensure consistency; and (iii) fine-tuning a pre-trained detection model with approximate character localisation using line-level annotations on real data, even with different alphabets.

Strengths

- Originality: The method is highly original, proposing a detection/classification approach to character recognition that differs from most state-of-the-art methods, which typically rely on autoregressive decoding from images of lines or pages. - Synthetic training data: The authors propose a way to train models using fully synthetic data, eliminating the need for actual character-level annotations. - Model efficiency: The model is relatively small (~40M parameters), requiring only 100k synthetic line samples for training. Error analysis: This approach allows for a better understanding of errors, distinguishing between detection and classification errors. - Computational cost: Characters can be independently predicted in parallel, reducing computational cost, especially without a language model. - Adaptability: The method can be easily adapted to any alphabet with minimal training data. - Code release: The authors are committed to releasing the code, facilitating further research and application.

Weaknesses

- Lack of new technical contributions: Despite the originality of the approach, the paper lacks novel technical contributions. The model, training strategy, fine-tuning with CTC, data augmentation and synthetic data generation are not novel. - Performance on handwritten documents: The method is not competitive on Latin handwritten documents and is outperformed by existing methods on Chinese handwritten documents. It does outperform on cipher recognition, but this is a more specialised and less researched area. - Scope of evaluation: The framework is only evaluated on perfectly segmented lines of text, raising questions about its performance on full pages. Full page processing would require additional steps such as text line detection and reading order retrieval, which could impact performance. - Impact of line detection: There are concerns about how the quality of line detection would affect character detection and recognition, particularly in cases where vertical or horizontal lines merge.

Questions

- Error cases: Can you provide examples of challenging real-world examples, such as rotated, upside-down or slanted lines; blank lines; vertically merged lines; strikethrough text; translucent paper; and mixed printed/handwritten characters? - Pre-training: How long was the model trained, on what hardware (GPUs), and why was pre-training limited to 100k lines of text? Was this value determined experimentally? - Input image size: What was the input image size used for training and evaluation? - Computational cost: How do you explain the difference in computational cost between DINO-DETR (25ms/line) and FasterDAN (7ms/line)? Why were comparisons made using a batch size of 1? Please provide a more comprehensive comparison, including batch sizes, CPU vs. GPU performance, impact of language modelling, input image sizes, and model parameters. - Language model decoding: What is the impact of a language model on inference speed? Can character recognition/classification still be parallelized with a language model? Can you decode on GPU with a KenLM language model, and what size of N-gram language model do you use? - Model release: Will the model be released to the public?

Rating

6

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

- Conclusion: The conclusion lacks depth and provides no insight into future improvements to the method. - Paper and writing: - Figure 2: The figure is unclear and needs to be made clearer. - Table 1: The table is misplaced, it appears on page 6 but is referenced on page 7.

Summary

This paper treats text line recognition as an object detection task and proposes a two-stage training approach based on DINO-DETR. In the first stage, synthetic data with bounding box information is used to predict the bounding boxes and categories of text, due to the absence of character-level annotations for text line datasets. In the second stage, real text line data is employed to sequentially fine-tune the classifier and the entire model. The proposed method achieves state-of-the-art performance in cipher text.

Strengths

- This paper is well-written and easy to follow. - The evaluation is well done and compares against several strong baselines.

Weaknesses

- Abalation study is easy. - Utilizes more engineering skills than methodological innovations.

Questions

All results in the paper were pre-trained on Latin text and fine-tuned on several datasets. What if we pre-trained and fine-tuned on data in the same language? For example, pre-training on synthetic Chinese text lines followed by fine-tuning on real Chinese datasets.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The proposed method performs well only on the cipher text recognition task.