Online Relational Inference for Evolving Multi-agent Interacting Systems

Follow-up from Authors

Dear Reviewer arYB, Following our previous response, we have carefully addressed all the concerns raised during the review process. We trust that our answers have adequately resolved the issues highlighted. If our answers meet your expectations, we kindly ask for your consideration in updating the final scores accordingly. Please don't hesitate to reach out if any further clarification or discussion is needed. Thank you for your time and consideration. Best regards, Authors

Response to Reviewer mc8m (2)

**Q1. [Alignment in Train and Test Data]** As stated in the above answer, in online learning, the training and testing steps are **NOT** separated. Therefore, we do not explicitly train the model on springs systems, stop training, and then test the model on charged systems. Rather, given a series of streaming observations, the model repeatedly performs online learning (no separate supervised training). Our goal is to study, whether a randomly initialized model, successfully learn a mixed dynamical system where the interaction dynamics changes with time. In our setup, the dynamics continuously changes from spring system to charge particles (and vice-versa) every 3-k iteration, as demonstrated in Figure 3(b) (main paper). The first such period of 6-k iteration, the ORI does exactly what the reviewer commented (“Such as trained on springs but then tested on charged particles”). In this period, an un-trained (randomly initialized model), only observes and learns from spring dynamics data for the first 3k iteration. At the end of 3k iterations, the (unknown) input dynamics for generating observations (data) suddenly switches to charge system. In other words, during the duration of 3k to 6k iterations, an ORI model ONLY learned spring system is being adapted on *unseen* dynamics of charge particles. As ORI continuously learns this new dynamics, it can still achieve 96.7% accuracy at the end of 6k iteration. Therefore, the first 6k iteration in Figure 3(b) exactly shows the case where the model is “trained on springs but then tested on charged particles” and performs very well. As the model continues to observe more periods of such 6k-iterations (mixed dynamics), it learns to adapt even faster to the dynamics shift. This is evident from the result in the second period of mixed-dynamics (6k-12k iterations) the model reaches 97% accuracy after 3.7k-iteration, instead of 6k-iteration in the first period. In summary, ORI can successfully adapt to the “unseen” dynamics (“trained on springs but then tested on charged particles”) without learning from mixed-dynamics data. We also see that once ORI learns from sequences of mixed dynamics (springs->charged, charged->springs), it adapts to the dynamics change faster. We will add this discussion in the main paper to clarify our contributions. **Q3. [Error Accumulation in AdaRelation]** Yes, the past adjacency matrix is also learned. We assume that, the reviewer is referring to the error in the relational accuracy. However, as we observed from experimental results in Figure 3, ORI with AdaRelation indeed showcases higher accuracy than ORI without AdaRelation (i.e., constant learning rates), which does not employ the past “learned” adjacency matrix. For example, in Figure 3(a), ORI with AdaRelation shows average accuracy of 84.8% while ORI with constant learning rates, such as relation lr 100 and relation lr 200, show 83.4% and 73.8%, respectively. Also, in Figure 3(b), the average accuracy of AdaRelation, relation lr 20, and relation lr 50 are 91.3%, 86.5%, and 90.7% respectively. We would like to clarify in AdaRelation (equation (2) & (3)), that the deviation between the current “learned” adjacency matrix and the past “learned” one is employed only for “adjusting the learning rate”, not the gradient (i.e., dL_mse/dI). We agree that unstable learning rates may influence the gradient as well in the future time steps, as in any gradient-based optimization methods, but as our adaptive learning rate in AdaRelation is bounded by equation (4), the errors due to adaptation of the learning rate are not infinitely accumulated.

Response to Reviewer mc8m (3)

**Q6. [Initial Adjacency Matrix filled with 0.5]** It’s unfortunate that the reviewer did not find our code. We have already provided the link to our codebase in the abstract of the initial manuscript. Please check the attached manuscript. The role of the adjacency matrix is to provide the graph’s edge structure to message passing layers in GNN. Instead of “vanilla message passing” where the associated edges are simply represented by binary, we consider “weighted message passing”, with the edges having continuous values. ORI performs weighted message passing from all nodes with continuous weights. Our “initial” adjacency matrix is filled with 0.5 to start with a scenario where ORI performs equally weighted message passing. As ORI performs online learning from the observations, the weights of adjacency matrix evolve according to the underlying (unknown) graph structure and dynamics. Please note, the “actual” adjacency matrices used to generate an experimental sequence is a binary one. Hence, to compute the relation accuracy, i.e., the similarity between “learned” and “actual” adjacency matrix, we threshold the “learned” matrix by 0.5 to create a binary matrix before comparing with the “actual” matrix. We will clarify ORI does weighted message passing in the main paper. Note, we do not claim “weighted message passing” as a new contribution of this paper. NRI also provides such functionality (in line 203 in train.py of NRI codebase, their edges can be either binary or continuous by a function argument “hard”). **Q5. [Intuition behind the Deviation D_I(t)]** The deviation is a function of |I(t)-I(t-w)|, where both I(t) and I(t-w) are the "learned" adjacency matrix, not the actual one. Hence, the significant change in the “actual” adjacency matrix used for generating the observed time series, may not necessarily lead to large values of the deviation. This depends on the how quickly ORI learns the “new” adjacency matrix as discussed below. We would like to remind that actual adjacency matrix is *not* available to the model for supervision. Consider a scenario when the actual adjacency matrix significantly changes between the time step ‘t-w’ and ‘t’. Assume ORI has “learned” the “actual” adjacency matrix at time step “t-w”. Now, at time step “t”, ORI can respond in two possible ways. First, ORI may quickly identify the new adjacency matrix at the time step “t”. In this case, I(t) and I(t-w) will be related to the new and previous “actual” adjacency matrices, respectively. Assuming the two “actual” adjacency matrices are significantly different, the deviation between two “learned” adjacency matrices will also be large. Hence, based on equation (3), the learning rate will decrease. This will make the learned adjacency matrix stable, thereby helping ORI to stay at the new “learned” adjacency matrix at time “t”, which is desirable as that is also the actual adjacency matrix. Let us now consider the second case where ORI does not quickly learn the new “actual” adjacency matrix and hence, the “learned” adjacency matrix at time “t” stays close to the one “learned” at time step “t-w”. In other words |I(t) - I(t-w)| remains low even if the “actual” adjacency matrices have changes. In this case, following equation (2), the learning rate increases to rapidly update the “learned” adjacency matrix, which is desirable to quickly move the “learned” matrix to the “actual” one. In summary, the equation (2) and (3) appropriately updates the learning rate when “actual” adjacency matrix changes, even without any knowledge/supervision of that “actual” matrices. **Q8. [Biased Learning without Trajectory Mirror]** Thank you for the suggestion. We will include it in the main paper. The coordinates of springs and charged systems are -5≤x≤5 and -5≤y≤5. The coordinates of CMU MoCap is -5.1≤x≤11.6, 0.0≤y≤29.5, and -35.2≤z≤58.2. These coordinates are same as the setup in NRI and dNRI, as we mentioned in line 220 and 227: “The synthetic datasets were generated using the open-source code from NRI” and “processed using dNRI’s open-source code”.

Follow-up from Authors

Dear Reviewer mc8m, Following our previous response, we have carefully addressed all the concerns raised during the review process. We trust that our answers have adequately resolved the issues highlighted. If our answers meet your expectations, we kindly ask for your consideration in updating the final scores accordingly. Please don't hesitate to reach out if any further clarification or discussion is needed. Thank you for your time and consideration. Best regards, Authors

Response to Reviewer mc8m

Dear Reviewer mc8m, We sincerely appreciate the reviewer's reply and additional questions! We are glad to know that the reviewer could find our code and provide follow-up questions. **[Where does the 0.5 appear in the code?]** initial edges are defined by “torch.ones / 2” (i.e., 0.5). For NRI, please check the line 334 in NRI/train.py, “edges = nn.Parameter( torch.ones((1, args.num_atoms*(args.num_atoms-1), 2), requires_grad=True) / 2 )”. For NRI-MPM, please check the line 385 in NRI-MPM/instructors/XNRI.py, “self.edges = nn.Parameter( torch.ones((n_atoms*(n_atoms-1),1,2), requires_grad=True) / 2 )”. We will leave the comments in the code later. **[what is the 'es' in the online_load_nri() of MPM]** We believe the reviewer is referring to ‘es’ in the line 148 in NRI-MPM/run.py. ‘es’ is to provide the node index of senders and receivers for all "possible" edges during message passing, hence *not related to the ground-truth adjacency matrix*. Please note that, we **never** use the ground-truth adjacency matrix to train the model. For example, for systems with 10 agents, ‘es’ is (2,90) matrix, where 90 represents the number of all "possible" edges excluding self-interaction (i.e., number of agents * (number of agents - 1)), and 2 represents sender and receiver node. This contains the same elements regardless of datasets: [[0, 0, 0, 0, 0, 0, 0, 0, 0, - e.g., node that can send message (0-th node) 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 9], [1, 2, 3, 4, 5, 6, 7, 8, 9, - e.g., nodes that can receive message from 0-th node 0, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8]] The usage of this matrix is to know the node indices of "possible" sender and receiver (e.g., for k-th edge, es[0][k] is sender and es[1][k] is receiver). We will also clarify this in the code later. As the discussion period ends soon, again, please don't hesitate to reach out if any further clarification or discussion is needed. We are always glad to answer the concerns of the reviewer. Best regards, Authors

Response to Reviewer fSnF

Thank you for your positive feedback and for recognizing the potential of our approach in inferring irregularly evolving graphs. We appreciate your suggestion and will include these discussions in the revised paper. We are grateful for your decision to maintain your current rating. Thank you once again for your thorough review. Best regards, Authors