Author's reply to the respected Reviewer PHih.
We thank the respected reviewer for their valuable time, and efforts in helping us to improve our work. We are grateful for the chance to engage deeply in the inner workings of APM.
In our reply to the respected reviewer Kzwl ```On memory/states and auto-regression in APM```, we mentioned substituting the word autoregression with sequential, because the trigger column $T$ unfolds to yield location aware columns, i.e. $T_{ij} = (T|p_{ij})$, where $p_{ij}$ is the generated hard-coded positional encoding as being used in transformers[3], and neural fields[2]. These $T_{ij}$ columns are responsible for sequential querying of the MLP.
Mathematically, the MLP predicts a location specific feature $f_{ij}=MLP(T_{ij})$ as a consequence of a forward-pass as each column gets pumped through the net. The MLP is queried ```sequentially``` with different columns $T_{ij}$ until the locations $ 1 \leq i \leq H$, $1 \leq j \leq W$ get exhausted, where $H,W$ are the dimensions of the input image $I$.
In the shared code in supplemental, MLP does not contain explicit recurrence, because the trigger column $T_{ij}$ carries the **entire** sequence $T$. The $p_{ij}$ in $T_{ij}$ guides the MLP to decode location-specific feature $f_{ij}$, which is a form of feature-expression as also previously hypothesized in GLOM [1] (our changes over Section 2.1 and Figure 3 in the GLOM paper).
We can also gain more depth by looking at Fig 3 of the GLOM [1]. The subtle differences are, 1) [a b] in that figure is the trigger column $T$ in APM, which carries the whole image $I$. 2)$x4$ is substituted by a positional encoding $p_{ij}$. 3) In addition to decoding location-specific RGB, we also decode higher dimensional features $f_{ij}$, for $ 1 \leq i \leq H$, $1 \leq j \leq W$ which allows APM to do field-based-decoding, which we leveraged for downstream-classification.
We have also added a detailed pseudo-code for operation of APM during test-time-training in
[Pseudocode](https://anonymous.4open.science/r/apm_rebuttal-F3D1/ttt_pseudocode.png).
The behaviour of APM is then akin to neural fields[2] (as suggested by the respected reviewer doB6), for eg, neural fields fire iid rays into the MLP shared across all input rays and decode location-specific rgb. Similarly, APM shares a MLP across all columns and decodes location-specific features. While neural fields work for 3D view synthesis, APM works for 2D image perception, with potential for extensions to other higher-dimensional input-percept, as also duly noted in Sec 11 of the GLOM paper[1].
As promised, we will also refine the paper-writing for better engagement with a prospective reader.
Yours sincerely,
Authors.
[1] Hinton, Geoffrey. "How to represent part-whole hierarchies in a neural network." Neural Computation 35.3 (2023): 413-452.
[2] Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106.
[3] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. Advances in neural information processing systems. 2017;30.