Summary
This paper proposes an algorithm to train an LLM-based agent *EAGLE*, that can align itself with existing domain-specific *latent embedding spaces* (e.g. embedding vectors in recommender systems, personalized advertising, and content creation) to discover novel content gaps and recommend new entities. It defines the problem setup as finding the optimal entity that maximizes the objective *utility function* $U(z; D)$, which is defined as the sum of user's and creator's utility, and the distance from all existing entities.
It formulates the setup as a Reinforcement learning optimization problem, in which MDP's state space consists of all possible entities, action space is defined by LLM-generated language prompts personalized to each user, the transition function is change in entity space obtained from the environment LLM, and the reward function is the *utility function* at horizon *H* and 0 otherwise. It considers several reference policies such as 1. Uniform 2. Greedy next step 3. *G-optimal* to constrain and regularize the learning objective. The *EAGLE* algorithm hence consists of first generating the candidate actions, training the reference policy, and then training the final policy given reference policy, pre-trained environment LLM, encoder and reward function.
The empirical experiments are performed on the dataset **Movielens-25M**, and the latent embeddings are the behavioral embeddings obtained via *matrix factorization* using *alternative least squares* trained on the user ratings. 100 candidate actions (50 generic, 50 personalized) are generated by prompting focusing on diverse aspects such as plot changes, character enhancements, and storyline. The quality is evaluated using a pool of human annotators who score the utility w.r.t to the Movielens user profile and the rater's user profile and the distance of generated movie from its anchor. The final metric used for evaluation is defined as the fraction of raters who preferred the generated movie to the original for each of the defined quality metrics. The proposed method is compared against the baseline method ELM (Embedding language model). It is observed that EAGLE outperforms ELM significantly on user and rater utility, but slightly decreases distance score. It also seems to be less sensitive to environment LLM used and improves scores primarily for poorly rated anchor movies, with a small decrease in perfectly rated movies. G-optimal design also seems to be more helpful for EAGLE compared to the reference policy.
Strengths
- The novel formulation efficiently incorporates existing domain-specific latent embeddings and leverages the generative capabilites of the pretrained LLMs to surface content gaps. This can be very helpful in providing text-based personalized recommendations to users in real-world recommender systems.
- It is computationally cheaper and relatively more data-efficient to train compared to previous methods as it doesn't require learning an explicit decoder.
Weaknesses
Some of the key design choices made by the proposed algorithm raise concerns on generalizibility to other real-world systems.
- Generating the action space requires significant prompt engineering effort with detailed criteria and in-context examples for each entity and the action set needs to be personalized to each user, and having personalized actions seems to be critical to the performance of EAGLE (Table 4). This raises serious concerns since the quality of the recommendation can be heavily biased towards the subset of the subjective criteria provided in the prompt, which may not always be possible to exhaustively define for each entity.
- The coverage of the latent embedding space would also be severely bottlenecked by the sampled candidate actions which may only explore a tiny portion of the latent embedding space in practice. It isn't clear how a practitioner would know if the action set criteria is diverse enough and how large of a hyperparameter *K* would result in a coverage that is sufficient.
While some of these issues are discussed in the limitation section, the experiments fail to provide a realistic picture to the algorithm since the proposed algorithm has far more domain-specific information compared to the baseline through the language prompt and in-context examples. One way to possibly address that may be reporting the efficacy of the algorithm with a simplified chosen language prompt in a zero-shot setting and stripping away any stylistic recommendations from the domain-specific information provided in the personalized actions prompt. While it would understandably perform worse in that setting, it would help demonstrate what portion of the performance boost is coming from the algorithm's exploration vs the initial user-specified selections, which isn't possible to conclude from the experiments in the paper.
Questions
1. Is it possible to quantitatively estimate the coverage of the latent ambient space given the generated candidate action space? If not, could you give a description on how you would recommend a practitioner to iterate on the prompts and hyperparameters. Qualitatively comment on the difficulty of the process and any assumptions that you made along the way.
Limitations
- The authors have addressed the limitations of their algorithm and the broader societal impacts.
- It could be further improved by discussing the summary of the practicality and generalizability aspects of the different components, as described in the above sections, in a separate paragraph of the Limitation section.