Summary
# Problem:
The AI community explores AGI via the ‘language agent’ framework involving a complex workflow of LM prompting and tool usage. This framework can only be improved through through human experts’ manual engineering efforts (engineering/model-centric).
How to depart from this painstaking engineering-centric paradigm?
# Contributions:
The paper aims to switch to a data-centric paradigm where language agents have the ability to ‘autonomously learn and evolve in environments’ from the data they observe, all symbolic nodes being optimized jointly and autonomously rather than separately by domain experts. To this end, they propose the ‘agent symbolic learning’ framework where agents self-optimize using symbolic optimizers.
Symbolic optimizers are mimicking the connectionist learning algorithms of backpropagation and **gradient descent over the substrate of symbolic elements of the language agents, i.e. prompts, tools and the way they are stacked together** (as a graph of nodes), rather than over the substrate of numerical weights. They optimize a **text-based loss** that is the result of applying an LLM-as-a-judge framework onto evaluation of the whole forward pass/agent execution, and use backpropagation from the last to the first node along the computational graph of the agent execution.
**Each symbolic element is then optimized by asking an LLM to modify it based on the backpropagated feedback.**
The papers presents proof-of-concept experiments showing the value of their approach against static agent framework and other prompt/tool optimization methods (e.g. DSPy).
Strengths
# Strengths:
## Quality:
SQ1: the paper is of good quality, with seemingly high reproducibility.
## Clarity:
SC1: the paper is easy to read and overall well-written.
SC4: I really appreciate the insights of section 4.3 and would like to encourage the authors to expand on their writing by being more explicit about what real-world tasks they have considered and that are not necessarily well-represented in mainstream benchmarks.
## Originality:
SO1: Apart from the comtemporary approach of [1], this paper is as original as it gets. And, if I understand correctly, it improves over [1] by adding (i) the possibility of autonomously modifying the computational graph of the language agent, and (ii) the possibility of autonously updating tools or implement new ones.
## Significance:
SS1: I think the main contribution of the paper has the potential to be greatly significance for the community, but I find the current state of the paper to inadvertently occlude some valuable insights about limitations for instance.
Weaknesses
# Weaknesses:
## Quality:
WQ1: Section 3.1 offers an analogy between connectionist learning and agent symbolic learning, but it fails to mention and discuss where does the analogy breaks, such as for instance:
- in the fact that the loss function requires comparison to a groundtruth label which is not instantiated here in the agent symbolic learning ;
- in the fact that the lack of groundtruth labels makes the whole approach more akin to an unsupervised learning approach rather than a supervised learning approach (like connectionist learning) ;
- in the fact that the connectionist gradient is effective due to mathematical guarantees that ascertain that updating numerical weights in the gradient direction will lead to an optimization of the corresponding loss ; but there is no such guarantee when considering language gradient as they are only the result of an LLM output, which we know to be riddle with issues, such as hallucinations[2,3] and unfaithfulness[4,5] ;
- …
Adding this discussion would improve the quality and significance of the paper, by detailing to the reader possible limitations that could be addressed down the line, and it would greatly improve my appreciation of the paper towards proposing honest and thought-through research that does not build unnecessary hype.
WQ2: I would like to invite the authors to present results with open-source LMs, as it would be valuable for overall replicability purposes and also to understand better how the approach performs with different LMs.
WQ3: I would like to invite the authors to consider include an experiment that studies how the performance scale with the size of both the forward pass LM and the symbolic optimizer LM. I think it would provide great insights into the potential of the propose approach for the democratisation of LMs, for instance.
## Clarity:
WC1: In Section 3.2, in the paragraph explaining equation (1), ‘task requirements’ are mentioned without much details, and I believe it is the first time it is mentioned in the paper. I would like to invite the authors to define it as precisely as possible, possible in section 3.1 for problem formulation, as I now realise how important of an element it must be (Please clarify if I am mistaken?). I also suspect that it is an element that cannot be optimised in a data-centric way but is still dependent on manual engineering effort, along with the prompt $\mathcal{P}_\text{loss}$, right? If so, I mean to invite the authors to collate all of those elements that are not yet data-centric-ly optimisable within the propose agent symbolic learning framework and propose a discussion of how future works may try to address them?
WC2: In section 3.2, the language loss computation paragraph actually details supervised agent learning vs unsupervised agent learning scenarios. I think it would make better sense to introduce this in Problem Formulation Section 3.1, and clearly highlight in which scenario will the main paper present experiments. I was actually assuming until this paragraph that only the unsupervised agent learning scenario was performed given the mention of using LLM-as-a-judge framework. It might be valuable to nuance and explicitly detail what is meant by LLM-as-a-judge when it is used in Section 3.1 actually.
WC3: As much as possible, it would be really valuable to integrate the different prompt templates for loss and gradient, and gradient update computations into the main text of the paper, in a concise form, or at least point specifically at them and discuss some of their guiding principles further in depth, in order to give a more concrete understanding of their impact to the reader.
## Originality:
WO1: Optionally adding a comparison of the contemporary approach of [1] could improve the impact of the paper.
## Significance:
WS1: Following up on my concerns in SS1, I find that the paper does not sufficiently discusses the limitations of the proposed framework, which result in an impression of building up hype for a product rather than doing thought-through research. I would like to emphasise again that my appreciatetion of the work would be greatly improve if the authors could revise the paper by including a lot more insights about limitations and what does not work fully (yet) and/or how some critical elements may impact the performances of their proposed work. For instance, could it be possible to perform an ablation study on the quality of the task description or the quality of the different engineering-centric prompt templates for loss and gradient and gradient update computation, please?
Questions
Please refer to the different points above.
# References:
[1]: Yuksekgonul, Mert, et al. "TextGrad: Automatic" Differentiation" via Text." *arXiv preprint arXiv:2406.07496* (2024).
[2]: Banerjee, Agarwal, and Singla, ‘LLMs Will Always Hallucinate, and We Need to Live With This’.
[3]: Zhang, Muru, et al. "How language model hallucinations can snowball." *arXiv preprint arXiv:2305.13534* (2023).
[4]: Lanham et al., ‘Measuring Faithfulness in Chain-of-Thought Reasoning’.
[5]: Turpin et al., ‘Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting’.