What's the Magic Word? A Control Theory of LLM Prompting

RE: Reviewer EeUn [1/2]

We thank reviewer EeUn for their insightful input. Here is how we have addressed the points raised: **Prior use of control theory for LLMs**: We added the following acknowledgement of Soatto et al’s work on LLM control theory: “To our knowledge, the only other work to date on the controllability of LLMs is Soatto et al. (2023). Soatto et al analyze the controllability of LLMs in terms of “meaningful sentences”, defined as the sigma-algebra generated by snippets of text written on the Internet. Their empirical analysis revolves around demonstrating that LLMs are capable of attributing meaning. The theoretical analysis of LLM controllability is limited to “meaningful sentences”, eliminating the possibility of out-of-distribution inputs and outputs. These restrictions render their results challenging to leverage toward a practical understanding of LLM controllability. We situate our work as a practically oriented exploration of LLM controllability. Motivated by challenges in developing LLM systems, we do not eliminate “meaningless sentences” from the state space or input space. We aim to establish a rigorous, general framework for understanding LLM systems and controllability that is amenable to the development of theory and practical engineering insights on systems design.” **Relevance of Section 4 (Self-Attention Proof)**: The point raised about whether there is a “natural topology” to justify our analysis is an interesting one. We have added the following discussion in Section 6 thanks to reviewer EeUn’s insightful comment: “Bounding the reachable set for self-attention is deeply related to the mechanism by which consistent representations are formed for multi-token generation. Steering a language model to generate a desired token sequence requires that the control input induce a token representation in the right-most token such that the next token prediction logits P (y∣u + x0 ) achieves a desired value. Moreover, generated tokens are fed back into the model, and their representations must be steered to control iterated generation. Self-attention is the primary mechanism by which the token representations exchange information, making the reachable set of output representations across multiple tokens in X0 for self-attention a fundamental part of LLM control theory.” While tokenization at the input may impose an arbitrary choice of topology, the controllability of logits on the output side is smooth. We are interested in manipulating the logit vector to maximize the probability of some desired sequence. Characterizing conditions for reaching outputs in this space aligns well with our analysis. That said, we welcome any insight into other ways to characterize the controllability of self-attention, particularly in the context of token embeddings with an arbitrary topology. We have also updated the proof to be in a theorem environment and have clarified the relationship between token representations in self-attention and tokens in LLM systems in Section 4. **Dynamical vs. Non-Dynamical Systems**: We have revised our framework in light of input from EeUn and other reviewers on the missing dynamical aspect of LLM systems. Particularly relevant is Definition 1, which is our new definition for LLM systems, and our commentary at the beginning of Section 3 clarifying the relationship differences between LLM systems and more conventional dynamical systems: “While LLMs are at times leveraged in a manner that masks the iterative aspects of generation, the reality is that token generation and externally imposed ``control input'' sequences are generated and processed sequentially, leading to non-trivial system dynamics. Several key differences remain between LLM-based systems and systems typically modeled through ordinary differential equations (ODEs), which have long been a cornerstone in the study of continuous-time dynamical systems: 1. **Discrete state and time**: LLM systems operate on sequences of discrete tokens over a discrete time set, in contrast to the continuous state spaces and time sets studied in classical control theory. 2. **Shift-and-Grow State Dynamics**: Whereas the system state in an ODE-based system has a fixed size over time, the system state x(t) for LLM systems grows as tokens are added to the state sequence. 3. **Mutual exclusion on control input token vs. generated token**: The LLM system state x(t) is written to one token at a time. The newest token is either drawn from the control input u(t) or is generated by the LLM by sampling x′ ∼ PLM (x′∣x(t)). This differs from traditional discrete stochastic systems, where the control sequence and internal dynamics generally affect the state synchronously.”

RE: Reviewer fWP6

We thank reviewer fWP6 for their support and insightful comments. **Generalizing self-attention results**: We hope the revised theorem/proof structure addresses reviewer fWP6’s concerns on presentation. We hope that the reviewers find **paragraph 2 of the Discussion section** insightful as to how/why the analytic result on self-attention might be generalizable: “Bounding the reachable set for self-attention is deeply related to the mechanism by which consistent representations are formed for multi-token generation. Steering a language model to generate a desired token sequence requires that the control input induce a token representation in the right-most token such that the next token prediction logits P (y∣u + x0 ) achieves a desired value. Moreover, generated tokens are fed back into the model, and their representations must be steered to control iterated generation. Self-attention is the primary mechanism by which the token representations exchange information, making the reachable set of output representations across multiple tokens in $\mathbf X_0$ for self-attention a fundamental part of LLM control theory.” We have clarified that the self-attention mechanism deals in token representations, and we have removed the unit magnitude ∣ui∣ ≤ 1 condition in favor of a generic maximum magnitude value that is propagated through our results ∣ui∣ ≤ Ωu, where Ωu may be any positive number. Ωu is largely a free parameter that can be adjusted as needed depending on input normalization or other mechanisms that may influence the input representations. Overall, we believe that the central result of demonstrating a limit to the reachable outputs of self-attention as a function of a controllable input representation and an imposed state representation is an exciting way to approach understanding the controllability of LLMs from a theoretical/first-principles perspective, and we thank reviewer fWP6 for their feedback for improving this part of our work. **Experimental Results not Exciting**: We have expanded the scope of our experimental results to probe the reachability of a larger space of tokens: in particular, we explore the reachability of outputs y based on the prior probability PLM (y∣x0) for some imposed initial state x0. We establish that the top 75 most likely next tokens, as estimated by the LLM itself, are reachable at least 85% of the time with prompts of k ≤ 10 tokens. Intriguingly, short prompt sequences can dramatically alter the likelihood of specific outputs, even making the least likely tokens become the most likely ones. The bottom right plot in Figure 1 shows a practically uniform relationship between how highly the LLM ranked a given output token y given x_0 and how likely we were to find a control input u — even a short one — that forced output token y to the top of the probability distribution. We hope this expanded scope underscores the potential of this control- theoretic framing for studying LLMs. **Connection between state controllability and k − ε**: In light of other reviewer’s advice on improving the formalization, we have unified our definitions of control in terms of the reachable set Ry(x0). Please see Section 3 and the supporting material in Appendix A for our new, more cohesive definition.

RE: Reviewer 8q5r

Hi 8q5r, Thank you for your attentive feedback. We appreciate your critiques! To your point about prepending a prefix like “after reading n words/tokens, output this particular word/token”—your intuition is exactly where ours was initially. Your suggestion can be made even stronger/more effective by straight repetition. Say we had “The brown fox jumped over the lazy ____“, and wanted to fill in the blank with ‘dog’. Just prefix: “The brown fox jumped over the lazy dog The quick brown fox jumped over the lazy dog The quick brown fox jumped over the lazy dog...” repeated enough times to guarantee the correct output. However, the problem with this strategy is that, to work realistically, it requires a large number of prefix tokens. Imagine we had a ‘budget’ and could only prepend 4 tokens as our control prompt. Can you think of an analogous noticeable example for 4 tokens? What about 2? We find there is some prompt length (in our paper we call it k), after which an instance is no longer controllable.In fact, our k-epsilon empirical results systematically investigate the relationship between prompt length and controllability. Epsilon represents the proportion of instances for which we didn’t find a ‘magic word’ (a control prompt steering the model to the correct final token). In Figure 1, you can see how epsilon drops significantly as the prompt length increases. Longer prompts are more controllable! As for how our definition and empirical analysis is entangled with the prompt optimization technique, we state in our discussion that our methods provide a lower bound for the k-epsilon controllability of Falcon-7b, Falcon-40b, and Llama-7b. Notably, this bound is met for k=1, since our technique ‘greedy back generation’ tests all possible single-token prompts and finds the best one. For k > 1, because it is impossible to explore the exponentially growing prompt space completely, there may be magic words we have not found. However, because we actually find each magic word (though there could be some we don’t find), the models are at least as controllable as we report. As for organization, we uploaded a new revision fixing many of the organizational issues. We hope you find it more acceptable. Again, we appreciate your feedback and concerns raised.

RE: Reviewer jCxR

Hi jCxR, Thank you for your time and consideration of our paper. Your concerns are well taken— from the perspective of an LLM engineer, a system designer, or a user, you are absolutely correct: we do not offer practical guidance. Instead, we take the perspective of scientists, treating LLMs as our object of study. We consider the LLM as a mathematical object for which the controllability can be determined. From this point of view, it was never our goal to improve factual correctness or the real abilities of LLMs, such as knowledge and reasoning. We are completely agnostic toward whether controllability is ‘good’ or ‘bad’, since we only aim to characterize and study it objectively and rigorously. Likewise, providing insights on prompt design or ways to stimulate LLM capabilities is outside of our scope. While it’s entirely possible that further research may use our rigorous foundation as a springboard for useful methodologies, our goal is strictly the objective scientific study. Your point about randomly generated weights is an astute observation. According to our analytical analysis, it is indeed the case that randomly generated weights are, at least in principle, as controllable as a fully trained network. Moreover, this result stems only from the mathematical structure of self-attention. Note that since self-attention is the only mechanism by which tokens can influence each other in an LLM, it is perhaps not so unexpected that the structure of self-attention should primarily characterize controllability. Again, we appreciate your feedback and comments, and have submitted a new revision addressing several issues. We hope you find it more acceptable!

RE: Reviewer 4Qb8

We thank reviewer 4Qb8 for support and their insight on issues of formalization and presentation. **Formalization**: We revised our definitions and cross-referenced our approach with multiple sources including Sontag, Kalman, and Ogata’s canonical control theory texts. Overall, we have shifted to a more reachability-first perspective — a point brought up in 4Qb8’s review. We have added a background section in Appendix A: “Abstract Systems an Control Theory Background” where we define generic systems, reachability, and controllability. We hope this will help communicate control theoretical concepts to newcomers properly. We also updated our formalization of LLM systems in Section 3. This update aims to align with the abstract systems and control definitions in the Appendix and to clarify the dynamical nature of LLM systems. We also added the following explanation in Section 3 about the key differences between LLM systems and more traditionally studied dynamical systems: “While LLMs are at times leveraged in a manner that masks the iterative aspects of generation, the reality is that token generation and externally imposed ``control input'' sequences are generated and processed sequentially, leading to non-trivial system dynamics. Several key differences remain between LLM-based systems and systems typically modeled through ordinary differential equations (ODEs), which have long been a cornerstone in the study of continuous-time dynamical systems: 1. **Discrete state and time**: LLM systems operate on sequences of discrete tokens over a discrete time set, in contrast to the continuous state spaces and time sets studied in classical control theory. 2. **Shift-and-Grow State Dynamics**: Whereas the system state in an ODE-based system has a fixed size over time, the system state x(t) for LLM systems grows as tokens are added to the state sequence. 3. **Mutual exclusion on control input token vs. generated token**: The LLM system state x(t) is written to one token at a time. The newest token is either drawn from the control input u(t) or is generated by the LLM by sampling x′ ∼ PLM (x′∣x(t)). This differs from traditional discrete stochastic systems, where the control sequence and internal dynamics generally affect the state synchronously.” __Input-output system and V^*__: While our empirical work revolves around “single-step” control (i.e., controlling the immediate next token generation), we have updated our LLM system definition and our characterization of LLM control theory to emphasize that LLM systems are, in fact, dynamical. Single-step control is merely a first step toward understanding the nature of more dynamical LLM system formulations with non-trivial internal dynamics (e.g., user interaction, chain-of-thought, retrieval-augmented generation, tool-wielding LLMs, etc). In particular, we acknowledge that the “shift-and- grow” state dynamics in V^* may be unfamiliar, but they are more realistic for how modern transformer-based LLMs function. This is particularly important for incorporating our analytic work on the reachable set for self-attention in Section 4. The growing state of LLMs is a fact of their operation, so to integrate our empirical work with our theoretical work, we maintain the state space V^*. Here is a paragraph from Section 6 discussing the relationship between the self-attention reachable set and multi-token generation: “Bounding the reachable set for self-attention is deeply related to the mechanism by which consistent representations are formed for multi-token generation. Steering a language model to generate a desired token sequence requires that the control input induce a token representation in the right-most token such that the next token prediction logits P (y∣u + x0 ) achieves a desired value. Moreover, generated tokens are fed back into the model, and their representations must be steered to control iterated generation. Self-attention is the primary mechanism by which the token representations exchange information, making the reachable set of output representations across multiple tokens in $\mathbf X_0$ for self-attention a fundamental part of LLM control theory.” Despite the growing state, we hope we have communicated why our LLM system formulation is still well-characterized as dynamical. We very much appreciate the feedback from reviewer 4Qb8 and we hope that our revised organization, structure, and expanded results address the concerns raised. As we are motivated by both the theoretical benefits of a control theoretic framework and the practical problems faced in LLM systems design, we hope that the reviewers enjoy the balance we seek to strike between the two.