Thanks for your constructive feedback. We respond to your questions one by one:
>W1 & Q1: Clarification on how the benchmark was constructed, including relationships, difficulty levels, and augmentation methods, and whether LLMs were involved.
Thank you for your insightful question. The benchmark construction begins with fundamental geometric elements: **points**, **lines**, **angles**, and **shapes**, where points serve as the most basic, indivisible concept underlying all geometric relationships.
We first identified key geometric relationships, including **line-based** (e.g., interactions between points and lines), **angle-based** (e.g., angular orientation and symmetry), and **shape-based** (e.g., positions of points or lines relative to a central shape). Using these, we manually created a base dataset of simple problems, expanding it slightly with LLM-generated examples. These problems typically require no additional reasoning, forming the **basic difficulty level**.
To construct the **advanced difficulty level**, we combined base relationships into reasoning chains (2–3 steps) to create more complex problems. This was primarily done using rule-based generation, with manual validation to ensure quality.
Finally, we applied two augmentation methods for diversity: (1) **Adding unrelated information**, where non-essential details were added (some generated using LLMs); and (2) **Re-labeling points**, which used rules to alter point labels without changing problem logic. Both methods preserved the original difficulty level since reasoning chain complexity remained unchanged.
>W2 & Q2: Using the LLaMA3-8B-Instruct model for fine-tuning is unconventional and suggesting providing results from fine-tuning the LLaMA3-8B base model.
Thank you for pointing this out. Since our former evaluation focused on instruction-tuned models, we fine-tuned the LLaMA3-8B-Instruct model for consistency. We have now added results from fine-tuning on the LLaMA-3-8B base model, as shown below:
|Model|Line-based (basic)|Line-based (advanced)|Angle-based (basic)|Angle-based (advanced)|Shape-based (basic)|Shape-based (advanced)|
|-|-|-|-|-|-|-|
|LLaMA3-8B-Instruct|63.16|42.11|52.17|13.04|26.92|38.46|
|FinetunedLLaMA3-8B-Instruct|34.14|40.56|32.50|13.75|39.87|40.51|
|FinetunedLLaMA3-8B-Base|**86.86**|**71.67**|63.16|**54.90**|**93.75**|**61.69**|
|GPT-4o|77.87|52.91|**66.67**|29.00|87.04|53.38|
Fine-tuning on the base model proves highly effective, significantly improving accuracy across all subsets, and surpassing the best-performing model (GPT-4o) in most categories. Additionally, comparing the two fine-tuned models reveals that the Finetuned LLaMA3-8B-Instruct model underperforms due to frequently selecting the uncertain option "cannot be inferred."
> W3: Including additional experimental metrics commonly used in math domains, such as pass@k, majority voting, or best-of-n, could improve the depth of analysis and provide a more comprehensive evaluation of model performance in this paper.
Thank you for the suggestion. In our main experiments, we used single greedy decoding (temperature set to 0) for all large models. To address your point, we conducted additional experiments on the GPT-3.5-turbo model using multiple sampling runs (temperature set to 1). We evaluated its performance using **pass@k** and **self-consistency with majority voting over k generations** (denoted as acc(SC=k)). The results are as follows:
|k|pass@k (lines-basic)|acc(SC=k)(lines-basic)|pass@k (lines-advanced)|acc(SC=k)(lines-advanced)|pass@k (angles-basic)|acc(SC=k)(angles-basic)|pass@k (angles-advanced)|acc(SC=k)(angles-advanced)|pass@k (shape-basic)|acc(SC=k)(shape-basic)|pass@k (shape-advanced)|acc(SC=k)(shape-advanced)|
|-----|----------------------|------------------------|-------------------------|---------------------------|-----------------------|-------------------------|--------------------------|----------------------------|-----------------------|-------------------------|--------------------------|----------------------------|
|1|64.29|64.29|42.46|42.46|42.11|42.11|22.41|22.41|68.75|68.75|38.31|38.31|
|3|71.43|60.71|62.30|45.24|63.16|52.63|37.93|22.41|93.75|75.00|59.74|38.96|
|5|75.00|64.29|72.22|44.44|68.42|42.11|55.17|25.86|100.00|75.00|68.83|43.51|
|10|75.00|64.29|83.33|45.24|84.21|57.89|70.69|32.76|100.00|75.00|77.92|42.21|
We observed that **pass@k** increases rapidly with larger k, indicating a growing likelihood of generating the correct answer through multiple attempts. However, the **majority voting accuracy** (acc(SC=k)) does not improve significantly, suggesting that the model's probability of generating correct answers remains low and that its reasoning over geometric tasks is inconsistent, resulting in dispersed outputs.