Official Response to Reviewer Cmp4
> Method is ad-hoc and not theoretically motivated.
We have added a new Subsection 3.1 (“Logits Contain Residual Information”) providing empirical and theoretical motivation why language model logits may reveal information about past tokens.
In terms of the method, we feel we are following standard practice in modern deep learning using Transformers. The main architectural challenge is how to feed in a high-dimensional vector (32,000 numbers) into a transformer. Main contributions were collecting a large dataset of prompts, extracting next-token probability distributions, and demonstrating that inversion is possible and effective in a variety of settings. We are not sure what would make a deep learning approach more theoretically motivated.
> The formal methodology is difficult to understand. For example, it’s unclear how the prompt is actually decoded from the embedded output probability distribution, i.e., what happens after their proposed encoding.
We havve tried to clarify this subsection of the paper; please let us know what you still find confusing. The output vector is provided to the encoder of an encoder-decoder language model, where the decoder autoregressively predicts the prompt. This approach is roughly standard in most conditional NLP systems.
> I don’t understand what is going on in section 5. What does it mean to “control one logit?”
We consider the most common setting of LLM APIs, where we can provide a “logit bias” argument that upweights or downweights the relative importance of a token in the output. This feature is key to our algorithm. We have completely rewritten Section 5 and Algorithm 1 to be clearer and more contiguous with the rest of the paper, which may alleviate your concern.
> In the beginning of section 4 where it’s stated that you “train on samples from the conditional model”, what is “the conditional model”?
This line was simply meant to indicate that we train the inversion model on output from a (conditional) language model. It was not a critical point, and we’ve removed the statement entirely.
> Given that the vector v was projected from R^d to R^v (as part of the final linear layer of the generation model), the argument that projecting it back to R^d would lead to a loss of information feels a bit strange
This is a small notational inconsistency. The original dimensionality d is that of the victim language model (4096 for LLAMA) while the final dimensionality is the input embedding of the inverter language model (typically 768 for T5). So in this case, projecting directly from LLAMA to T5 space would lead to a loss of dimensionality with a factor of 4096/768≈5.3.
> Where is the theoretical discussion/experiments corresponding to inversion when only the text output is available? The methodology discussed in section 5 still assumes access to probabilities from the model, which are often not available in “models as a service” platforms.
We did consider this baseline, sorry for not making it more clear. For Llama2-chat, we train a model to predict the prompt input given language model output samples . This is shown in Table 1 as “Sample Inverter”. We add an additional paragraph of theoretical discussion in this matter to Section 3.2 (“Inverting from outputs”).
> In section 4, it mentions that there is a “fixed-length input sequence of 42 words.” Is this in reference to the sequence of embeddings that are fed to the model? Or is this alluding to how the embeddings are decoded into the expected input?
This is the sequence of embeddings that is fed into the encoder of the inversion model.
> How does the ability to reconstruct text change as a function of the pretrained model used to construct the distribution p(x | v)?
We consider this question through many ablations shown in Table 9. Notably, we conclude that we could have seen greatly improved inversion performance if we had trained an inverter with more parameters.