We meant that we will clarify in the next version of the paper that these numbers are averaged.
Together with the clarifications we made in the original response, we will perform the following changes.
**i. We will change our current discussion of metrics (lines 232-237) to the following.**
> We use two sets of metrics to evaluate the performance of a tool for the binary function name recovery task.
> 1. Token-level Metrics. We evaluate the predicted function names following existing work in the reverse engineering domain [1]. The metrics first tokenize both a predicted name and the corresponding ground truth name, and calculate the precision, recall, and F1 score at the token-level. For each metric, we first compute the corresponding scores for individual function name predictions, and then average the scores across all functions.
> 2. Character-level Metrics. We adapt BLEU, METEOR, CHRF, ROUGE-L and ROUGE-LSum for the function name by tokenizing function names into characters and computing these metrics on character level, similar to [2,3]. We call them charBLEU, charMETEOR, charCHRF, charROUGE-L, and charROUGE-LSum. They provide a fine-grained evaluation of the function names and can avoid some limitations of tokenization. Similar to token-level metrics, we first compute the precision, recall, and the F1 score for individual functions, and then average the scores across all functions.
**ii. We will add the following sentence to Table 2 caption.**
> Note that we first calculate all metrics for individual functions, and then average the scores across all functions.
**iii. We will integrate the table in our original response (timestamp 07 Aug 2024, Q3) to Table 2 in the paper. The new columns present the results in SymLM precision, SymLM recall, SymLM F1, charBLEU, charMETEOR, charCHRF, charROUGE-L, and charROUGE-LSum.**
**iv. We will change the Appendix C.4 (lines 673-674) to the following**
> Formally, the token-level precision $P$, recall $R$, and F1 are defined as follows:
> $$
P(i) = \frac{\big\| T_{g}^{(i)} \cap T_{p}^{(i)} \big\|}{\big\| T_{p}^{(i)} \big\|}\quad
R(i) = \frac{\big\| T_{g}^{(i)} \cap T_{p}^{(i)} \big\|}{\big\| T_{g}^{(i)} \big\|} \quad
F1(i) = \frac{2 \times P(i) \times R(i)}{P(i) + R(i)},
$$
> where $T_{g}^{(i)}$ is the token set of the ground truth name for the $i$-th test case, and $T_{p}^{(i)}$ the token set of the $i$-th predicted name.
> The precision, recall, and F1 scores for the entire test set are the average scores of individual scores across all test cases. Formally,
> $$
P = \frac{1}{N}\sum_{i=1}^N P(i) \quad
R = \frac{1}{N}\sum_{i=1}^N R(i) \quad
F1 = \frac{1}{N}\sum_{i=1}^N F1(i),
$$
where $N$ is the number of test cases.
We are very grateful for your help in pointing out the places that cause confusion!
We hope the changes have clarified everything.
We will also go through another round of error checking (with fresh eyes) as suggested by the reviewer.
---
## Clarification on how we reuse the scripts
We want to clarify that in Table 2 of the original paper we show the results of charROUGE-L.
CharROUGE-LSum is *another* metric that we additionally added in the rebuttal to show that other character-level metrics are consistent with our original results.
We reused the script of SymLM from their Github repo and the open-source PyPI library “rouge” to compute the SymLM metrics and charROUGE-L metrics for individual function names in our Table 2.
[1] Jin et al. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. 2022 CCS.
[2] Lin et al. NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System. 2018 LREC.
[3] Shi et al. Natural Language to Code Translation with Execution. 2022 EMNLP.