Speaking Your Language: Spatial Relationships in Interpretable Emergent Communication

Effective communication requires the ability to refer to specific parts of an observation in relation to others. While emergent communication literature shows success in developing various language properties, no research has shown the emergence of such positional references. This paper demonstrates how agents can communicate about spatial relationships within their observations. The results indicate that agents can develop a language capable of expressing the relationships between parts of their observation, achieving over 90% accuracy when trained in a referential game which requires such communication. Using a collocation measure, we demonstrate how the agents create such references. This analysis suggests that agents use a mixture of non-compositional and compositional messages to convey spatial relationships. We also show that the emergent language is interpretable by humans. The translation accuracy is tested by communicating with the receiver agent, where the receiver achieves over 78% accuracy using parts of this lexicon, confirming that the interpretation of the emergent language was successful.

Paper

References (31)

Scroll for more · 19 remaining

Similar papers

Peer review

Reviewer vZ266/10 · confidence 4/52024-07-01

Summary

This work investigates the presence of spatial deixis (e.g., spatial references in language dependent on the context of the utterance) in a signalling game within the paradigm of emergent communication. It begins by introducing a variant of the signalling game which requires the sender to communicate the relative position of some element in a sequence of integers visible to the receiver. Analysis of the semantics of the emergent communication is performed primarily through looking at normalized point-wise mutual information. These analyses show that the emergent communication regularly uses spatially-referent messages (or sub-message units), validating the presence of spatial deixis in the environment.

Strengths

In conjunction with standard criteria, there are three characteristics that are particularly important for emergent communication research: reusability (how easily can another researcher use the products of this research), generalizability (how much do the findings of this research apply broadly to our knowledge of emergent communication), and directedness (does this research contribute concretely to particular questions in emergent communication research). ### Quality - (major) The experimental design is of good quality; the methods are in line with the standards of the field. ### Clarity - (major) The language of this paper is easy to read and illustrates the central points effectively. ### Reusability - (major) The provided code appears to be of good quality (have not attempted to run it); this work would be very easy to reuse in subsequent experiments. ### Generalizability - (minor) The environment is relatively simple, with few confounding factors, making it easier to draw conclusions about more general tendencies in EC. ### Directedness - (major) This paper is directed toward an important goal in emergent communication: discovering/engineering more human-like features in EC, namely deixis or context-dependent reference. - (minor) Secondarily, this paper also demonstrates some degree of compositional semantics and syntactic features.

Weaknesses

### Quality - Nothing of note. ### Clarity - (minor) One or two more of the points in Section 5 need to be illustrated (likely with just a table); although the text is mostly clear, some tables which aggregate what is said there would make referencing the paper much easier. - (minor) It is a little bit confusing that your hypotheses are what you expect to be false; it might be clearer to state hypotheses in positive (even if the statistical test you are using is rejecting a null hypothesis of a random baseline). ### Reusability - Nothing of note. ### Generalizability - (major) It is not clearly the case that the environment addresses deixis in a way that applies to emergent communication environments more generally (see first question for more). - (major) There is not much discussion on how the deixis investigated in this paper is applicable to emergent communication more generally. ### Directedness - Nothing of note.

Questions

- The biggest question regarding the paper for me is how do we make the jump for the simple example of deixis presented in the empirical investigation of the paper to a more robust form of deixis. It is not wrong (and likely correct, in fact) to have started out with a toy problem, but I believe there either needs to be empirical work and/or some light theoretical work on what exactly is meant by "deixis" in this paper, how the environment investigated satisfies that definition, and how this is relevant for further environments. I might raise my score if the authors could respond with a slightly more formal characterization of deixis and how it maps both to the current environment and more sophisticated, "natural" environments (e.g., embodied multi-agent environment). - In the environment, are the integers actually represented as integers in the neural network or are they encoded as one-hot vectors? If they are OHVs, it is not clear that it is the case; if they are not OHVs, it seems like an odd design choice to feed a scalar into an NN when it is representing something categorical. - How are the various inputs to the receiver agent actually fed into the network. Are they just concatenated "temporally" and given as a sequence to the GRU? ### Comments - The fact that this paper is inducing a segmentation of emergent communication is (minor) contribution in and of itself, so I think it deserves a mention in the introduction. - The "td" variable needs to be introduced before the example; I am assuming it is the list of distractors plus the correct answer, but it should be stated explicitly when defining the vectors earlier. - Table 2: Don't reuse X; use other variables. - Include the actual URL for the (Anonymous) GitHub so that it is obvious it is a link. - I don't understand the point being made at Line 337. - I do not think it is appropriate to specifically mention "SVO" as an interpretation of the language since there is no clear way to distinguish between nouns, verbs, subjects, or objects; I think it is fine to say that there is syntactic structure, but I a skeptical of there being evidence to make any claim further than that here.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

2

Limitations

N/A.

Reviewer JyuC7/10 · confidence 4/52024-07-12

Summary

This paper proposes a new communication game in the emergent communication framework to analyze the emergence of _deictic reference, i.e. expressions akin to demonstratives like "this" and "that". These are important expressions in natural language and especially in this emergence literature, since their meaning is context-dependent and "functional", i.e. cannot be reduced to objective properties of the object of reference. The paper also introduces an application of normalized pointwise information to the analysis of the emergent communication protocol in order to identify holistic messages (where a message refers to an entire meaning) and compositional ones (where certain n-grams and/or positions refer to specific "components" of the meaning). Both of these are welcome contributions and will be of interest to many people working on emergent communication. The core idea in their game is to use integers within longer sequences as the object of reference, provide a _partial observation_ of the true context to the sender (so that absolute positional information cannot be used) and to _mask out_ the target object in a sequence (so that the integer itself cannot be used); what remains as possible information to convey are things like "two to the right of 13".

Strengths

* A carefully designed emergent communication scenario which requires something like spatial deixis to emerge for successful communication. This is an important component of human language that goes beyond what has been done in existing literature. * Interesting and useful application of NPMI for the analysis of (non-)compositionality of the resulting messages. * Engages well with existing literature to situate the new contribution of this paper. * Results also show a robustness to things like random seed, which is not always the case in emergent communication.

Weaknesses

* Some experimental details could be more carefully reported and some analyses could be more systematic/quantitative (see questions below). * The artificial messages used to validate their NPMI metric does not yield results as strong as one would like (as discussed in the Limitations section); this makes it not entirely clear that the metric does what its intended to do.

Questions

* Why did you choose a fixed-length of 3 for the messages (as opposed to either a single token, or variable-length)? * Line 92 and 114: should "target integers" and "targets" both be singular? There's one target integer, correct? Or is the plural here just over a batch of examples? If the latter, the wording is a bit confusing since the worked case in the paper is just one example ("batch size 1" so to speak). * Can $PMI_c$ and $PMI_{nc}$ be seen as one metric, with the latter a special case of the former (i.e. for the full tri-grams)? The discussion just before Section 5 seems to suggest so, so I would encourage more elaboration on whether these are really two separate degrees or not. For instance: does high $nc$ entail low $c$, and vice versa? * "The analysis provided in this section is based on the messages collected from the test dataset after the training has finished". What was the train/test split here? Appendix A provides model / optimizer hyper-parameters, but what are the game/environment/data choices? * Can the observations in Section 5.1 and 5.2 be made more quantitative? I would appreciate a more detailed analysis of the types of composition observed, their frequency, and other factors like that. * H2 and Table 1: while the Comp-P case is above chance, if the NPMI method correctly identified "genuinely compositional" messages, we would expect nearly perfect accuracy in this case, right? * Very minor typographic point: I think that the "n" in "$n$-gram" and similarly in the main text should be in math mode.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Reviewer VwFJ5/10 · confidence 4/52024-07-15

Summary

The authors design a referential game environment intended to motivate the emergence of spatial references, cast in the form of a task where the target integer must be selected from an integer sequence. The character vocabulary for the message is smaller in size than the set of integers in the list, and this necessitates an alternative to directly specifying the target integers. Using a traditional GRU-based speaker/listener architecture, the model achieves high task accuracy. Using existing information theoretic measures the authors are able to roughly decode the semantics of the messages and show some degree of compositionality in the messages.

Strengths

- The experimental design is simple but I think straightforward and correct for what the authors want to test. - Similarly, it appears from the analysis in sections such as Table 2 that the resulting messages do seem to exhibit a variety of communication strategies, including the desired type in some messages (compositional positional) - The approach of decoding the meaning and segmentation of the messages via NPMI (though I would have also liked to see some discussion of where this would/would not be appropriate in terms of a general evaluation metric for EC. It seems some strong expectation over what the emergent language needs to say may be necessary? In this case, the presence of the integers, for instance)

Weaknesses

- Overall I find the biggest weakness to be in the scope of the paper and the degree to which the design of the environment caters to the type of messages the authors want to elicit here. It comes across as a toy problem, and through the lens of the field as a whole, I think it raises the question of whether there is sufficient novelty in making such small and targetted tweaks to the referential game formula. This might be best highlighted by revisiting the motivating example, such as "a blue vase with intricate motifs on the table". Why refer to this as "the vase over there" or "that vase near you"? There are pressures in the referential game to draw out these spatial references, but these feel artificial and devoid of broader understanding about linguistic pressures when we compare them to the pragmatic concerns that would motivate a spatial reference (and what type of spatial reference) in the motivating example. I also think claims are over-stated. The authors claim this is the first paper in EC to have syntax and make comparisons to SVO ordering. This comes across as a flimsy attempt to connect to human language and to signal a degree of progress in the complexity of ECs, but a lengthier discussion is warranted. Syntax can't be treated as something that does or does not exist, but rather, discussion of what formal language class the emergent language falls into would be relevant, and nothing here would necessitate a CFG or the degree of syntax that is meaningful when it comes to discussing natural language. Despite the simplicity I do see value in this work but I would have liked to have seen a less contrived environment with a more difficult learning problem / substantial scope in the necessary semantics to feel comparable to the degree of contributions typical of a paper at this venue. I think it would be far more appropriate at a more targetted venue where it can also recieve appropriate attention and discussion.

Questions

- There are some fairly trivial solutions to this problem. It seems compositional integer style gets at this -- are there cases where compositional integer would fail? I'm not seeing the need to learn a spatial solution to this problem when it seems that a two-character code could cover all possible target integers. Does this vary as the length of sequence or size of alphabet are increased/decreased? - Similarly, is there any reason to motivate the choice of vocabulary size with respect to latin alphabet? The chunks of the messages are more akin to words than characters. To me it just read as an attempt to have a connection to human language, but that relationship was not meaningful. - It is mentioned that the hyperparameters for MI are optimized for translation accuracy. I can make some guesses as to what might be done here, but it wasn't clear to me exactly what is being compared here.

Rating

5

Confidence

4

Soundness

4

Presentation

3

Contribution

2

Limitations

Some limitations have been discussed, though limitations of the experimental design (such as 3 character messages, "nouns" being integers and therefore being able to reference without features, etc.) are not. The authors mention some doubts in previous work regarding the value of referential game -- I think this is precisely one of the ways that these doubts emerge, i.e., "we try very hard to setup a task where success must be achieved in this way, and the model finds it". These sorts of limitations may not be addressable in the scope of the paper, but they are also not mentioned. There are no negative societal impacts from such work.

Reviewer YdNg5/10 · confidence 4/52024-07-17

Summary

This paper shows that emergent communication can learn spatial references. They first create a modified referential game which requires the agents to communicate by messages that indicate the relative position of a number. The proposed agent architecture shows that the GRU-based agents can achieve good performance. The analysis uses NPMI to identify the meaning of the ngrams in the message (i.e., the correlation between the ngram and the referred positions). This paper further shows that the mapping generated by NPMI is correct by generating additional datasets based on the identified dictionaries to show that both non-compositional and compositional messages carry the intended meanings.

Strengths

- This paper proposes a novel spatial game to study the emergence of spatial references. - This paper shows that NPMI is an effective measure to decompose the messages by finding correlations with the intended meanings.

Weaknesses

- The paper is not very easy to follow especially the definition and design of different types of sequences, examples of the messages, and how the hypotheses are tested. The presentation can still be improved. - It is unclear how much the test set overlaps with the training set when measuring the accuracy. There is no control of generalization tests such as varying full sequence length or observation of certain patterns of sequences. So, it is hard to understand if the learned messages are effective or memorization of part of sequences in training. For example, does the ngram that means “leftmost” can effectively communicate in a longer sequence? - The design of the game put high communication pressure on the agents. The agents need to develop messages conveying relative positions in order to succeed. How does the success relate to the communication protocol, for example, when the message length is longer, is it still necessary to develop messages that convey relative positions? It is unclear about the role of channel bandwidth, effective communication, and developed messages. - The test in Compositional-NP is to generate the dataset by removing the positional component of the message. This is an extreme case of H2. In reality, the message is most likely to be corrupted rather than removed. To reject the null hypothesis, it will be more convincing to have a corrupted message version.

Questions

- In the experiment, the observation is always fixed length which makes the communication easier. What happens when the observation is longer? I can imagine if the longer sequence contains the same number at different positions, it will introduce some ambiguity in the messages.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

- The experiment is based on one-dimensional sequences, the types of spatial references are limited in this case. It will be helpful if the author can discuss how it can extend to more complex types of spatial references.

Reviewer vZ262024-08-09

Reply

Regarding the response to the "generalization" point of the review starting with "Let O be the observation available to an agent", I think this is a step in the right direction but still needs more development (although it is a little hard to judge the proposed formalization out of context). I think the main shortcoming of the sketch of formalization proposed is that it is still not quite clear what "deixis" is in a sort of environment-agnostic way, which is what I was getting at in my review (although potentially unclearly). For example, $O_p$ is a point in the observation serving as the origin, but what does "point in the observation" mean in some environment-independent way? Maybe one could start off by specifying that the environment has some structure which supports deictic expressions in the first place (e.g., space, time, succession, 1st-2nd-3rd person distinction). Additionally, there needs to be some way that the context in which the potentially deictic expression is grounded; that is, we can't just have some arbitrary $O_p$, there needs to be some way in which $O_p$ is derivable from the context of the utterance. With these in mind, we could maybe define deixis in an EC environment along the lines of the following: - The observations of the environment support some notion of position (or something more general, if you can think of it). - Using this notion of position, we can take some original point, apply a transformation (i.e., moving a distance in space), and derive the target point. - The original point is derived from the context of the utterance. I am not saying this the best or only way to do, but hopefully this better illustrates what I meant by "generalization". I think the account offered in the rebuttal is on the right track, but maybe a couple extra steps could be taken to make it environment agnostic. I am moving recommendation from a 5 to a 6 because think the proposed changes are close to making this paper more widely applicable to other EC environments. Nevertheless, I probably will move my rating to a 7 because I am not able to see a fully-developed formalization in context. --- As an aside, regarding OHVs vs scalars, it seems odd that OHVs would perform worse since it essentially pre-processes the scalar values into something the neural networks can work with---maybe this has something to do with over-parameterization when switching from scalars to OHVs, but I don't think this all that important (aside from making it clear that scalars are being used since one might naturally assume integers are represented as OHVs), as it does not invalidate the experiments. Everything else not mentioned here I found was addressed suitably by the rebuttal.

Authorsrebuttal2024-08-12

Thank you for your continued engagement with our paper! We improve the formalisation of deixis below, making it more environment-agnostic. To do so, we expand the observation to be an $m$-dimensional $R^n$ tensor, which can represent any possible properties of the observation. ## Formalisation of deixis Let $O$ represent an abstract observation that an agent perceives from its environment, $O \in R^{n \times n \times n ... n = n^m}$. The $m$ dimensions can represent the spatial, temporal, or other positions. For example $R^{n^3}$ could represent a 3-D observatrion. Let $O_p$ be the reference point in $O$, which coordinates are represented by an $n$-tuple of real numbers $(x_1,x_2 ... x_m)$ and $O_t$ be the target point in $O$, with its coordinates represented by an $n$-tuple of real numbers $(y_1, y_2 ... y_m)$. Then, the relative distance function $d(O_p,O_t)$ returns an $n$-tuple of real numbers $(z_1,z_2 … z_m)$, such that $z_i = x_i - y_i$. This relative distance function allows for unambiguous identification of the target object $O_t$, given that the position of $O_p$ is known. A deictic expression can be defined as a mapping of the value of $d(O_p,O_t)$, the reference point $O_p$, and their context $O$, to a specific linguistic or symbolic expression that describes the relationship between $O_p$ and $O_t$. This mapping can be represented as: $(O,d(O_p,O_t),O_p) \rightarrow \text{Expression}(O,d(O_p,O_t),O_p)$ where the resulting expression $\text{Expression}(O,d(O_p,O_t),O_p)$ is a description of the reference point $O_p$ and its relative distances to the target point $O_t$, given the context $O$. ### Examples Consider two objects, $O_p$ and $O_t$, located at Cartesian coordinates $(x_1, x_2)$ and $(y_1, y_2)$, respectively. The relative distance function $d(O_p, O_t) = (z_1, z_2)$ captures the differences in both dimensions. In this context, suppose the expression "2 to the left and 3 up" is used. This would correspond to $z_1 = -2$ (indicating 2 to the left) and $z_2 = 3$ (indicating 3 up) relative to the reference point $O_p$. However, without knowing the exact position of $O_p$, this information alone is insufficient to identify $O_t$. To resolve this, the reference point $O_p$ must either be explicitly included in the deictic expression or implicitly understood by all interlocutors. For example, if it is agreed that the reference point is at $(5, 4)$, where $(5,4)$ could be an abstraction of any object, the expression "2 to the left and 3 up" clearly locates $O_t$ at $(3, 7)$. Without such an agreement, a full deictic expression is necessary, such as "2 to the left and 3 up from $(5, 4)$". Depending on the context $O$ and the need for clarity between agents, some information provided by the relative distance function $d$ may be omitted. For example, if the target object $O_t$ is the only item of a given type in the environment to the left of the reference point $O_p$, simply stating "to the left" might be sufficient without specifying the exact distances or the $y$ direction. Similarly, if the agents are conversing about objects on a flat surface, the height difference may be irrelevant and ignored, simplifying the expression further. The examples above could also be rewritten to showcase the ability to specify any other deictic basis, by simply changing the underlying meaning of a given dimension. The expression "5 minutes after 10:00" specifies a temporal deixis by explicitly locating an event $O_t$ relative to a reference time $O_p$. In a 3D spatial context, an expression like "4 to the left, 2 forward, and 1 down" captures the exact relative position of an object $O\_t$ in Euclidean space. This formalization will be included in the revised version of our paper.

Reviewer vZ262024-08-13

Thank you for providing this. It is pretty close to what I had in mind in writing my prior comment, and I think it will facilitate applying the results of this paper more broadly.

Reviewer JyuC2024-08-12

Thank you for the detailed and helpful reply! I am already a fan of this paper, and these responses clarify any last remaining questions that I had.

Reviewer YdNg2024-08-12

Thanks for the reply

I thank the authors for the further explanation. It is good to see it generalize to a longer sequence. However, this is not the only form of generalization; also, non-overlapping training and test sets do not mean the messages generalize to different combinations of deixis. For example, seeing "4 to the left" and "3 to the right" and generalizing to "3 to the left". These generalization tests will need to reflect what a formal deixis definition this paper uses. Regarding the design to avoid repetition, I understand the difficulty in identifying the referent position, but this restriction can limit the learned messages to simple ones. So, it will be helpful to discuss this limitation in the design too. Overall, I appreciate the authors' response and it addressed some of my questions. I will increase my score to 5 and suggest a further discussion on the generalization that reflects the definition of deixis.

Reviewer VwFJ2024-08-13

Regarding the simple environment, I agree and disagree. Of course there is prudence in starting from a simple, well-controlled environment, as presented in the paper. But of course the risk is that it will be difficult to scale the approach up or reconcile it with existing work. Frankly this is an all too common problem in the EC literature, which is rampant with one-off papers which each show some semblance of the target structures in emergent languages, and which are never incorporated or built upon ever again, and cited only in passing as yet another linguistic phenomena which now has a toy world which can give rise to some simple form of it. The question should probably be, is it too much to ask for both? Is it too much to expect that an EC paper at least show some consideration for what comes next, or to attempt to show both a contrived toy scenario and a more sophisticated one which explores the limitations of the approach in the face of a more realistic and more difficult learning problem? To this end, I greatly appreciate the prodding by Reviewer vZ26, and the author response, to formalize a more general notion of deixis, such that we could at least have some understanding of what scaling this work up might entail. With that in mind, with the inclusion of this formalisation I am willing to increase my score to 5, on the accept side of the decision boundary, as I think the work as a whole might represent an adequately thorough investigation of this specific issue. This is a difficult decision -- I am still a bit concerned about the potential lack of impact from the work, especially when submitted to a competitive and general interest venue, and where the additions to the draft (which seek to formalize the broader phenomenon of deixis) are still a work in progress.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC