Thanks for your response! We’re glad to hear that the clarifications were helpful, and that you believe in the importance of our work.
We also value enabling others to build on our work. As part of this, we will release our code, included in the supplementary material. As suggested, we will also add details about how to choose tasks, metrics, and datasets, in order to aid future researchers. While we can’t upload revisions to our paper, we’re happy to share a rough outline of this information here.
Let us know if you’ve got any other questions, and if there’s anything we can do to help you raise your score!
---
### Task
Our path-patching approach is compatible with a variety of tasks. The chosen task should:
1. Have a clearly delimited set of correct and incorrect answers for each example.
2. Require only one forward pass of the model (as opposed to e.g. generation tasks which require multiple passes).
3. Be solvable by your model: if your model cannot solve the task, there may be no circuit. At minimum, the model should exhibit consistent behavior (even if it’s not exactly correct)
Keep in mind that the granularity of insights will depend on the granularity of the task chosen. Complex tasks like natural language inference could require different (sub-)circuits depending on the specific question; it might be hard to find one precise circuit responsible for the task. Smaller, simpler tasks will likely yield easier to interpret results.
For the purpose of this example, we’ll consider the task of fact retrieval. Each input will have an (ideally single-token) correct answer, which can be predicted with one forward pass. Moreover, it seems possible that facts are mostly stored / retrieved using the same circuit.
### Dataset
Path-patching requires two datasets: a normal and corrupted dataset. The normal dataset is just a collection of examples/inputs for the task; its examples should:
1. Clearly indicate the task at hand. LMs perform language modeling, and do not natively perform other tasks; they may leak probability to answers that are not correct or incorrect, but simply task-irrelevant. Your inputs should push as much probability as possible onto the task's output space.
2. Allow evaluation based on only the distribution over possible next tokens (generated via one forward pass)
3. Be representative of your task. Your choice of datasets effectively define the scope of the phenomenon you study—make sure the scopes of your datasets and your intended task match!
Each example from the normal dataset should have a corresponding corrupted example / input. The corrupted input should:
1. form a minimal pair with the normal input: they should differ minimally from each other (being the same length, and differing by only one or two tokens)
2. elicit a different model response, with a distinct correct answer, compared to the normal input
3. belong to the same sort of task. Remember that we locate the circuit by activating the same circuit with two different inputs.
For fact retrieval, a normal input could be "Paris is the capital of"; the corrupted counterpart could be "Rome is the capital of". Both of these examples are reasonable input for fact retrieval, but the two will elicit very different responses. Note that an input like "Paris is in" would be less appropriate, because it doesn't clearly indicate that the task is fact retrieval, or what fact should be retrieved.
### Metric
The metric is a function that takes in model logits and labels. It should:
1. Output a real number measuring model behavior / performance on the task.
2. Detect small changes in whether the model is behaving according to the normal or corrupted input, or somewhere in between. A continuous loss is thus preferable to metrics like 0-1 loss/accuracy.
One family of metrics used in previous work is the probability assigned to the correct answer(s), minus the probability of the incorrect answer(s) induced by the corrupted input. In the greater-than case, this is p(y>YY) - p(y<=YY). For fact retrieval, we would compute p(France) - p(Italy). This family of metrics the model is implicitly sensitive to the model generating off-task output as this generally takes away from the probability of correct answers. It is explicitly sensitive to the probability of the corrupted input’s answer.
Other metrics are possible. For example, if it is difficult to quantify task performance, but still possible to create minimal pairs, one could simply measure the KL-divergence between the original and (partially) patched / corrupted distributions. However, this is harder to interpret, and less targeted at your actual task of interest.