Response to Reviewer MA2R
> How is the $\Phi(x)$ computed in detail?
We will clarify how $\Phi(x)$ is computed for each of the Post-Hoc methods with the help of an example.
Let us say that the example $x$ outputs a prediction which has 5 output tokens. Let the output log probability associated with each output token be $p_i(x)$, where $0 <= i < 4$. Then, the Chow-Sum score is $s_{\rm sum}(x) = \sum_i p_i(x)$ and the Chow-Average score is $s_{\rm avg}(x) =\frac{1}{5} \sum_i p_i(x)$. Let the various quantiles be $s_{\rm quant}(x,\alpha_i)$ where $\alpha_i$ is in [0, 0.01, 0.02,... 0.1, 0.2, 0.3, …, 1.0]. We take 20 quantiles.
Let the decoder embedding from model 1’s final layer (averaged over all output tokens) be denoted as ${\rm Emb}_{\rm dec,final,avg,m1}(x) \in \mathbb{R}^{n_1}$.
Let the decoder embedding from model 2’s first layer (for the first output token) be denoted as ${\rm Emb}_{\rm dec,int,first,m2}(x) \in \mathbb{R}^{n_2}$.
**Post-Hoc-Quantile**
For the method Post-Hoc-Quantile, we concatenate the quantiles, Chow-Sum and Chow-Average scores:
$$ \Phi(x) = [ s_{\rm sum}(x), s_{\rm avg}(x), \{ s_{\rm quant}(x,\alpha_i) \} ] \in \mathbb{R}^{22} $$
**Post-Hoc-Embed-1**
For the method Post-Hoc-Embed-1, we concatenate the quantiles, Chow-Sum, Chow-Average scores and model 1’s decoder embeddings:
$$ \Phi(x) = [ s_{\rm sum}(x), s_{\rm avg}(x), \{ s_{\rm quant}(x,\alpha_i) \}, {\rm Emb}_{\rm dec,final,avg,m1}(x) ] \in \mathbb{R}^{22 + n_1} $$
**Post-Hoc-Embed-1+2**
For the method Post-Hoc-Embed-1+2, we concatenate the quantiles, Chow-Sum, Chow-Average scores, model 1’s decoder embeddings and model 2’s intermediate decoder embeddings:
$$ \Phi(x) = [ s_{\rm sum}(x), s_{\rm avg}(x), \{ s_{\rm quant}(x,\alpha_i) \}, {\rm Emb_{\rm dec,final,avg,m1}}(x), {\rm Emb_{\rm dec,int,first,m2}}(x) ] \in \mathbb{R}^{22 + n_1 + n_2} $$
We have also added this example in Appendix F.5.
> What is the performance of the post-hoc-quantile when predicting the golden deferring label?
Thanks for the interesting question. The final performance of a cascade is typically measured by the accuracy of the interleaved predictions from the small and large model. Nonetheless, the reviewer’s suggestion is indeed a useful diagnostic.
To that end, the table below shows the AUC-ROC score for predicting the golden binary deferring labels, which is 1 if model 2 is better than model 1, and 0 otherwise. We see that Chow-Sum and Chow-Average tend to be poorly predictive of this golden label. The recommended Post-Hoc deferral approaches yield significant gains in the AUC-ROC. We also added the table in Appendix F.6.
| Method |MNLI | TriviaQA | WMT DE -> FR|
|-|-|-|-|
|Chow-Sum|0.46|0.31|0.51|
|Chow-Average|0.53|0.53|0.56|
|Chow-Quantile-Best|0.73|0.52|0.62|
|Post-Hoc-Quantile|0.86|0.68|0.62|
|Post-Hoc-Embed-1|0.92|0.70|0.63|
|Post-Hoc-Embed-1+2|0.95|0.71|0.62|
> In figure2, figure3 and figure5, all figures use the deferring rate as the x-axis. I am curious about whether we could use the inference time cost as the x-axis.
Thanks for the suggestion. Note that the two are linearly related: for a deferral rate of $d%%$, the inference cost is $c_{\rm S} \cdot \frac{100 - d}{100} + (c_{\rm S} + c_{\rm L}) \cdot \frac{d}{100}$, where $c_{\rm S}, c_{\rm L}$ are the costs of the small and large model respectively.
We have also added a plot in Appendix F.2 where we plot the relative latency on the x-axis. In our experiments, we estimate the base model latency to be 36% that of the large model. (Please note that the latency estimate may vary depending on the dataset, because the number of output tokens varies. We estimate the latency based on 500 random examples on XSum dataset; this is intended to serve as a rough estimate of the relative speedup from cascading.)