PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition

Summary

This paper introduces a novel approach to reduce generation latency in Named Entity Recognition (NER) using Large Language Models (LLMs). The primary issue addressed is the high latency caused by the sequential decoding process in LLMs, which significantly lengthens the sequence by autoregressively generating all labels and mentions for NER. To tackle this, the authors propose Parallel Decoding in LLM for NER (PaDeLLM-NER), which integrates into existing generative model frameworks without requiring additional modules or architectural changes. PaDeLLM-NER enables simultaneous decoding of all mentions, effectively reducing generation latency. Experimental results show that PaDeLLM-NER can improve the inference speed than the traditional autoregressive approach.

Strengths

- The parallel decoding strategy is well-designed and experimental results prove the effectiveness. - The authors provide comprehensive experiments with different setting and with furthre analysis. - The paper is easy to follow.

Weaknesses

- The proposed method cannot improve the inference speed in scenarios where only one type of entity is predicted. - Since the method focuses on the inference efficiency of LLMs-based NER, it is better to report both inference speed and performance compared to zero-shot (Table 3) and supervised (Tables 4 and 5) methods. Notably, Table 3 only reports performance without considering the efficiency of different LLMs. Furthermore, why not report the performance of AutoReg_aug and AutoReg_struct in Table 3? - For better understanding of the training resource usage when compared with other methods, it is better to report the base language models used (SOTA methods) in Tables 4 and 5. - The writing of this paper could be further improved. For example, Line 219, “As per Ning et al...” appears to be a typo; the meanings of the underline (second performance) and bold (best performance) are not provided; and there is no explanation for why “*” indicates that results are not directly comparable in the Table 5 caption. - Comparing with fixed few-shot in-context learning of LLMs may also be worth considering, as caching the fixed prompt could improve the inference speed of LLMs.

Questions

Please see the weakness.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors provide one limitation section.

Summary

They create an NER system where an LLM first outputs the number of mentions there are of a given type (for all possible types). Then all mentions can be generated in parallel. This results in faster inference times as each generation is short, and they can be done in parallel.

Strengths

They compare to several different baseline on multiple NER datasets in multiple different settings. Their method is much faster than others.

Weaknesses

Their reformulation of NER as predicting (label, mention) pairs removes a critical component of classical NER, the actual alignment of the mention to the tokens. Polysemous words are often mentions in some context and not in others and it if often important to know which one was the actual mention, especially if it is used for things like editing downstream. The deduplication strategy is very aggressive and removes the possibility that some surface text is a label for multiple types in a single sentence. For example, "It is England vs. Italy on this sunny day in England", England is both a place (LOC) and a sports team (ORG) this would get filtered by their setup. The prose's definition of " prediction quality [...] that is on par" is rather loose, with their model being 6 points behind on average for zero-shot (table 3) and behind by a point of two on most supervised datasets.

Questions

How do you expect this to scale to NER datasets like Ontonotes where a there are 20+ different mention category? Similarly what about long documents that could have 10-30+ mentions of a given type? Did you see inconsistencies in the model outputs? For example a model that output that there is `1` person but then generated <mention 2> ${person name}?

Rating

9

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Summary

This paper presents PaDeLLM-NER, a novel approach for accelerating Named Entity Recognition (NER) inference in Large Language Models (LLMs) through parallel decoding. A reformulation of the NER task that enables parallel generation of label-mention pairs, significantly reducing inference latency. A two-step inference process involving mention count prediction and parallel mention generation. Extensive experiments demonstrated significant speedups (1.76x to 10.22x faster) compared to autoregressive approaches while maintaining or improving prediction quality across multiple datasets and two languages.

Strengths

The parallel decoding strategy for NER is innovative and addresses a significant bottleneck in LLM inference speed, which is important in some speed-sensitive applications. The authors conduct extensive experiments across multiple datasets, languages, and settings (zero-shot and supervised), proving the method's effectiveness. The reported speedups are substantial and could have a meaningful, practical impact on NER applications. The method is compatible with existing LLM architectures and can be integrated with other acceleration techniques. The methodology is well-explained with helpful diagrams and examples.

Weaknesses

Some details and corner cases are not well explained. For example, I didn't see the token location information in Figure 2. If the input has multiple and same mentions (e.g., "Donald Trump owns the Trump Organization" ), how does this framework distinguish with the same mentions? (e.g. Trump in the above example) In addition, it is not clear how the de-duplicate model processes the partially duplicated mentions. For example, in the above case, the "Trump organization" was recognized as ORG, and what if the person module predicted the "Trump" in the "Trump organization" as a person? Will the de-duplicate model filter this case?

Questions

No

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

No

Summary

This paper proposes an interesting extension of the parallel text generation paradigm, where the authors tackle the NER task and propose to generate the labels independently. For each label prediction, the proposed method first predicts the number of mentions and then predicts the exact entity. The results show that the proposed model performs reasonably well, while achieving faster inference.

Strengths

1. The proposed method is a pioneer work to accelerate LLM generation following the parallel generation paradigm. 2. We do observe significant speed-up empirically, which suggests the proposed method may be of value in real word applications.

Weaknesses

1. The importance of the two-step prediction for each entity is not justified. I feel there should be a baseline such that the multiple mentions can be predicted together in an autoregressive fashion. For example, I can predict "entity type: LOC Italy English" as a whole. 2. Fundamentally, parallel predictions should be weaker than autoregressive predictions due to the drop in dependency capturing. However, we observe from Table 4 that AR models are noticeably worse than the parallel approach. Since these results contradict common wisdom, there needs more effort to justify them. For example, the authors may need to reveal the full training/testing configurations of both the AR and parallel models, and there could be some more detailed error analysis to show how AR models are making more mistakes than the parallel approach. 3. The proposed approach may face difficulty when a word is used multiple times with different types. For example, in "Washington lives in Washington," the proposed approach may predict "LOC" and "PER" for both "Washington"; however, it can not align them because the parallel approach is ordered agnostic among entities. 4. The proposed method needs finetuning to adjust the LLMs, which can be difficult when it comes to very large LLMs.

Questions

1. Is the duplication issue, as mentioned in Figure 2, very common? Do you have statistics for this? 2. Do you know if the testing datasets are in the training data of the LLM?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors make earnest efforts to discuss the limitations. In addition, the previously mentioned Weakness 3 is another potential limitation. The authors may include further discussion in this regard.