Weaknesses
**Methodology**: The paper does not define what a task specification is concretely, nor does it define what it means for the demonstrations to be adequately summarized, in particular in Alg 1 the function is_summarized() is not defined, nor is its operationalization defined anywhere in the text of the paper. Specification traditionally refers to a formal statement whose semantics are well known and the satisfaction of the specification with respect to an output is computationally well-defined and consistent. By not having a concrete definition of specification, the evaluations set up in the latter part of the paper suffer from lack of diversity and a lack of quantification of task difficulty.
**Code as output language**: This work relies on the output code being interpretable, and this involves providing a set of primitive parametric functions to the LLM to use. The part of sensing the environment, and acting using perception feedback is something that is already programmed into the primitives. Coming up with an adequate and a competent set of parameterized primitives is challenging, and in this case is entirely the responsibility of the system designer.
**Claims of generalization** Tthe generalization shown is only generalizing towards named entity substitutions, not to complex control flows and temporal specifications. Specifically, all the tasks shown here are a sequence of subgoals where performing any subgoal out of order will not preclude the agent from completing the subsequent actions. Further, there is no reactivity in the task specifications, and there are no avoidance tasks. All of these task specifications notions are very common in robotics and planning problems that this system has not been evaluated on. Refer to [1] for a survey on robotics mission types (also relevant to symbolic planning). All the tasks here are limited to the visit or sequenced visit type. Further while the submission claims that demo2code can generalize for complex long-horizon tasks, the maximum task length is quite smaller than state of the art for symbolic planning. Automatically translating textual domain descriptions to a formal domain description followed by the use of automated planners has already shown more reliable performance on harder problems [5]. Further there is quite a bit of evidence that LLMs cannot plan beyond the simplest of domains [6], and this line of research is unacknowledged in the submission.
**Issues on evaluations of learning from demonstrations:** Generalization from demonstrations and language is a tricky subject. Usually demonstrations and language contain complementary sources of information. Therefore none of the system behaviors are incorrect in Figure 5, the core issue is that inductive learning is by definition an ill-posed problem, and many approaches to inductive learning have relied on Bayesian inference in the past. [2],[3]. Specifically, the issue of where to place the purple block (fig 5a) given the language description is underspecified, and the system had to forcibly ground the placement to any of the valid options to generate a trajectory. Committing to a valid assignment as was done by the lang2code model is one approach, and asking for resolution of referential ambiguity is another approach [4]. One might argue that learning to overconstrain the output based on a single demonstration is an example of overspecification.
[1] - Menghi, C., Tsigkanos, C., Pelliccione, P., Ghezzi, C. and Berger, T., 2019. Specification patterns for robotic missions. IEEE Transactions on Software Engineering, 47(10), pp.2208-2224.
[2] - Tenenbaum, J.B., 1999. A Bayesian framework for concept learning (Doctoral dissertation, Massachusetts Institute of Technology).
[3] - Shah, A., Kamath, P., Shah, J.A. and Li, S., 2018. Bayesian inference of temporal task specifications from demonstrations. Advances in Neural Information Processing Systems, 31.
[4] - Williams, Tom, Rafael C. Núñez, Gordon Briggs, Matthias Scheutz, Kamal Premaratne, and Manohar N. Murthi. "A dempster-shafer theoretic approach to understanding indirect speech acts." In Advances in Artificial Intelligence--IBERAMIA 2014: 14th Ibero-American Conference on AI, Santiago de Chile, Chile, November 24-27, 2014, Proceedings 14, pp. 141-153. Springer International Publishing, 2014.
[5] - Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J. and Stone, P., 2023. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477.
[6] - Valmeekam, K., Olmo, A., Sreedharan, S. and Kambhampati, S., 2022. Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change). arXiv preprint arXiv:2206.10498.