Summary
The paper proposes a problem specification (OPTO) and a solution (optimizer OptoPrime) as well as a software framework (Trace) for agentic program optimization. The paper demonstrates high performance of the bundle above in solving 5 benchmarks. The paper demonstrates the ability to optimize 3 types of parameters: numeric, text prompts, and python code. The paper proposes Trace as an analogue of imperative-style AutoDiff engines like PyTorch for agentic programs relying on (black box) LLM inferences. The evaluated benchmarks cover a reinforcement learning setting (Battleship, Traffic control and Robot manipulation), scalar regression (Numerical optimization) and question answering (classification and more in BigBenchHard).
Strengths
[1] The authors do a great job in demonstrating the operational performance of Trace on 5 examples: Battleship, Numerical optimization, Traffic control, BigBenchHard, and Robot manipulation.
[2] Great to see Trace compared with both traditional optimizers and the recent LLM-based (OPRO) in Figure 5b.
[3] It is nice to see that the proposed method is designed to generalize to several types of payload (demonstrated 3: numeric values, text and code).
[4] Experiments in “5.3 Unifying Prompts and Functions Optimization” neatly showcase joint optimization of prompts and code which is a clear novelty, even though one may argue that a text prompt and a text of a piece of code are both textual information. Good to see that the optimization works even without the history like Yang 2024 (OPRO) or population like Pryzant 2023 (ProTeGi).
[5] The learning seems very sample-efficient with the reported scores achieved in just several single-sample iterations.
[6] After running exp/run_prompt_bigbench.py I confirm the reproducibility of the results on BBH.
Weaknesses
[1] The paper does not compare with “TEXTGRAD: Automatic Differentiation via Text” (arXiv:2406.07496v1) which is a concurrent work.
[2] The narrative goes from describing a piece of software (Trace) to the problem specification (OPTO) and then to the algorithmic solution (OptoPrime). I would expect the problem specification and the proposed algorithm to be explained first while the software details are discussed later or even in the Appendix.
[3] The experiment in 5.3 confirms that the proposed method works in principle, however due to the small scale of the optimized program (3 parameters), there is no clear signal that Trace will scale up for 10s or 100s of trainable parameters.
[4] The definition of a trace (line 126) is not given. I am familiar with the verb “to trace” with respect to a symbolic program, however it is unclear what “a trace” means. From a random web page (https://www.ituonline.com/tech-definitions/what-is-an-execution-trace/):
Definition: Execution Trace
An execution trace is a record of the sequence of operations executed by a program during its run. This trace includes function calls, variable values, and the flow of control among other details.
This definition is not mathematically clear.
[5] The work lacks the analysis of Figure A.12 where the learned prompt template does not make much sense.
[6] Running the baseline exp/run_prompt_bigbench_dspy.py fails.
[7] The paper does not demonstrate the optimization success in agentic programs with memory, i.e. when the optimized class like Predict below has internal state that changes from call to call:
```
@trace_class
class Predict(LLMCallable):
```
Questions
[1] On lines 79-80 you claim that “Remarkably, there is 80 no mention of Battleship nor details on how the functions reason and act should behave or adapt 81 in Fig. 2a”. However in Appendix H, Iteration 0 (Initialization) you provide relatively detailed instructions: “Given a map, analyze the board in a game. On map, O denotes misses, X denotes successes, and . denotes unknown positions.”. Since the Battleship game is a very well known one, ChatGPT could reproduce the code memorized from the public repositories. How do you assess the risk of this type of leak?
[2] Where can an example of prompt optimization of an LLM agent as per line 767 be found?
[3] I appreciate quoting “Complete” on Figure 2a as the example is not clear without `__call__()` and `select_coordinate()` that can only be found in the supplementary materials in exp/battleship_exp.py -> Policy2. I suggest replacing it with “An excerpt from”.
[4] In Figure 3 it is not clear how can g1, g2 and g3 be different since the connectivity of the graph is defined by the optimized torch-style module, specifically a chain of 2 optimizable nodes Reason and Act in the example.
[5] It would be valuable to know the USD cost of OpenAI API usage per experiment.
[6] On the technical side, python version is not mentioned, datasets and ray are not on the requirements list. However I appreciate providing OAI_CONFIG_LIST_sample.
Limitations
[1] The authors honestly admit that one of the main limitations of the current approach is LLM context length. Indeed, the scaling of the proposed algorithm that packs the entire trace into a single context for LLM inference inside OptoPrime.
[2] Scaling of the proposed framework to more learnable parameters and more sophisticated agentic programs is yet to be demonstrated.