Clarification about pseudo-programs and soft reasoning
If it is helpful, here is a more specific comparison between the tasks we consider with pseudo-programs vs what previous works have done with regular programs.
The PAL (Program-aided Language Models) paper ([Gao et al 2023](https://arxiv.org/pdf/2211.10435)) uses Codex to generate fully executable Python programs to solve tasks in the following categories: Math reasoning, Symbolic reasoning, Algorithmic reasoning.
These involve tasks like GSM, where the program looks like this:
```
Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
money_initial = 23
bagels = 5
bagel_cost = 3
money_spent = bagels * bagel_cost
money_left = money_initial - money_spent
answer = money_left
```
The task thus boils down to 1. Parsing the problem into symbolic variable assignments, then 2. applying python-implemented internal functions like addition and subtraction.
This is the case for all the kinds of tasks they consider for PAL. E.g. for their algorithmic tasks like OBJECT COUNTING:
```
# Q: I have a chair, two potatoes, a cauliflower, a lettuce head, two tables, a cabbage, two onions, and three fridges. How many vegetables do I have?
vegetables_to_count = {
'potato': 2,
'cauliflower': 1,
'lettuce head': 1,
'cabbage': 1,
'onion': 2
}
answer = sum(vegetables_to_count.values())
```
The LM thus serves as a parser of the problem into a setting in which the solution falls out based on symbolic manipulation operations that Python easily handles. Thus, our definition of “exact reasoning” tasks are those for which a simple solution recipe contains only definitions and well-formed Pythonic operations in order to aggregate the definitions into an answer.
Now let’s consider some of the tasks we look at in our paper, like the CSQA problem from our rebuttal:
```
The teacher told all the students that listening was key, it was the main way they would gain what? (A) "empathy" (B) "anxiety" (C) "knowledge" (D) "falling down" (E) "hear things"]
```
What is the corresponding PAL program that could solve this question? It requires considering each option and the possible common sense relations between them and the situation in the question. One could code up a query to a symbolic ontology, but this is a much more complex solution than just generating code from an LM and feeding it to a Python executor. Instead, our tactic is to let the LM generate a pseudo-program that doesn’t actually get executed; it suggests to the LM a programmatic means to solve the CSQA instance, but circumvents the hard-to-code parts of the solution via underspecified function calls. Since it is underspecified, we use the LM to emulate the program execution because Python would not be able to.
```
# Step 1: Extract the main statement and options from the question.
statement, options = extract_statement_and_options(question)
# Step 2: Identify the key action and its purpose in the statement.
key_action, purpose = identify_key_action_and_purpose(statement)
# Step 3: Match the purpose with the most relevant option.
answer = match_purpose_with_option(purpose, options)
return {'statement': statement, 'key_action': key_action, 'purpose': purpose, 'answer': answer}
```
Each of these functions like ‘identify_key_action_and_purpose’ are undefined– they would be very hard to actually code up in pure Python. We face the same sort of challenge for tasks like sentiment analysis.
Thus, by “soft reasoning” we refer to NLP tasks that require nontrivial reasoning steps that you can’t write simple Python snippets to handle, such as ‘identify_key_action_and_purpose’.
We described this idea in L29-34 of the submission, but will expand it to make this more clear to future readers. As we state in the paper and in our rebuttal, ours is the first paper to handle these sorts of soft reasoning tasks using a code-based approach.
We hope this helps clear up your confusion. Please let us know if not, we are happy to do so while discussion period is still open.