Please check revision
Dear reviewer xpjw,
Thank you for your feedback! Please note that our original submission already had a brief limitations section in Appendix F. However, we have fleshed out the discussion of limitations by adding a concrete error analysis in Appendix A.4 (on image segmentation) and Appendix C.3 (on video-action segmentation), the two domains where we saw comparatively less improvement. Please check the revised PDF. We have also copied the text from those sections below:
**Appendix A.4**
> We manually inspect 50 random outputs from LAMPP on the ID setting and identify broadly the
following categories of errors, which are illustrated in Fig. 5.
> 1. Mislabelling: The ground truth labels are sometimes incorrect. For example, in Fig. 5(a),
the nightstand is mislabelled as a “table”. LAMPP is then wrongly penalize when it changes
the table to a nightstand.
> 2. Missegmentation: Object boundaries may be incorrectly identified, as in Fig. 5(b) and
Fig. 5(c). This can confuse LAMPP as it may be unlikely for two visually-similar items to
co-occur in the same room (e.g. beds and sofas), and LAMPP may try to “paint over” the
mis-segmented chunk with the wrong label.
> 3. Example inconsistent with LM priors: For a small number of examples, the LM prior may
be incorrect. These examples can be thought of as being sampled from the tail of the
distribution. For example, in Fig. 5(b), the LM believes that it is unlikely for beds and
sofas to co-occur in the same room. However, this particular example is of a furniture shop,
meaning that it is actually the case that the bed and sofa are next to each other.
> 4. Insufficient information to LM: Because we may only interact with LMs through text, the
LM loses out on certain visual signal that may help disambiguate between two semantically-
similar object labels. For example, in Fig. 5(c), LAMPP incorrectly converts the desk into
a table, likely because the LM does not have sufficient information to conclude that the
image is of an office setting (desks are more often used to in the context of work). Note that
“computer” and “printer” weren’t available labels for the segmentation model, but may have
be useful information for the LM.
**Appendix C.3**
> In this domain, GPT-3 often produced priors that were unaligned with the true distribution, leading
to errors in downstream prediction with LAMPP. As noted in C.1, we found that GPT-3 has a
proclivity to output actions in the order that they were presented in the prompt. To further probe
errors in GPT-3’s priors, we plot GPT-3’s distribution over actions for tasks against the ground-truth
distribution over actions in the real training and validation datasets. We found that, when these priors
were uncorrelated with real priors from the dataset, there tended to fall into the following categories:
> 1. Bias towards ordering in prompt: see discussion in Appendix C.1.
> 2. Task ambiguity: The LM and ground-truth demonstrations may prefer slightly different ways of accomplishing the same task. For example, in the “Grill steak” task (Fig. 6(a,b)), GPT-3 prefers to cut the steak before cooking (green line in Fig. 6(a) shows that the LM’s highest-probability initial action is “cut steak”), while the ground-truth distribution prefers cutting the steak after cooking (blue & orange lines in Fig. 6(b) shows that the datasets’ highest-probability action after “cut steak” is “taste steak”).
> 3. Action ambiguity: Note that, in this dataset, actions are typically named with a very short description, which may sometimes cause ambiguity. For example, in the “Change a tire” task Fig. 6(c), it is unclear whether actions involving the wheel is referring to the old wheel or the new wheel, and the action “start loose” is unclear.
> 4. Structural limitations of HSMM: Because we are modelling a sequence of scenes from videos
with an HSMM, the action segmentations may be imperfect – for example, in Fig. 6(b), the
HSMM learns that it is highly likely for the action after “cut steak” to also be “cut steak”
from the training datasets (with probability ∼ 0.5), perhaps because the cutting scene hasn’t
ended. The LM is unable to these transitions.
> 5. Wrong LM Priors: GPT-3 sometimes simply has incorrect priors about the action orderings
for tasks, for example putting a fairly high probability on “widthdraw wheel” after “tight
wheel” for task “Change a tire” in Fig. 6(c)
We understand that this is very short notice, but we would appreciate if you could take a look and let us know if you have remaining questions. If this has satisfactorily addressed your concerns, we would also appreciate if you could raise your score. Thank you!