Reinforcement learning presents an attractive paradigm to reason about several distinct aspects of sequential decision making, such as specifying complex goals, planning future observations and actions, and critiquing their utilities. However, the combined integration of these capabilities poses competing algorithmic challenges in retaining maximal expressivity while allowing for flexibility in modeling choices for efficient learning and inference. We present Decision Stacks, a generative framework that decomposes goal-conditioned policy agents into 3 generative modules. These modules simulate the temporal evolution of observations, rewards, and actions via independent generative models that can be learned in parallel via teacher forcing. Our framework guarantees both expressivity and flexibility in designing individual modules to account for key factors such as architectural bias, optimization objective and dynamics, transferrability across domains, and inference speed. Our empirical results demonstrate the effectiveness of Decision Stacks for offline policy optimization for several MDP and POMDP environments, outperforming existing methods and enabling flexible generative decision making.
Paper
Similar papers
Peer review
Summary
This paper focuses on solving offline RL with generative models. Concretely, it proposes a method to modularize the joint distribution of time-induced trajectories and use separate generative models to represent observation module, reward module, and action module. Evaluations are conducted on D4RL benchmark with MDP and POMDP environments. Extensive comparisons against prior works are included.
Strengths
- The paper is fairly well presented and easy to follow. - The proposed method is extensively evaluated against multiple related approaches.
Weaknesses
- Novelty of the proposed method is limited. The improved performance can also be attributed to other confounding factors, such as larger models due to reward and action modules being factored out. - It is counter-intuitive to ignore the canonical time-induced casual ordering in favor of different token types. More intuitions and theoretical analysis (if applicable) are encouraged to provide. - Lack of evidences to support the claims on modular expressivity (L145). Are there any experiments showing it can transfer to new environments? - The empirical results, especially those on D4RL locomotion tasks, are not significant enough to justify the extra compute introduced. - Just swapping each module with different model architectures/modeling strategies is not enough to gain insights of the proposed method. More ablations and analysis are necessary.
Questions
- Why DS's performance on POMDP Hopper task with medium-expert data (Table 3) is even better than that on fully observable setting (Table 2)? - How does DS compare to more straightforward baseline, such as the one with separate heads for three modules but shares the same backbone?
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
- The design choice of ignoring temporal causality is not well motivated. Case studies or even proof-of-concept experiments would be helpful to justify it. - The performance improvement could be attributed to confounding factors such as larger model sizes. More experiments with controlled variables (e.g., same amount of model parameters) would help to support claims made in the paper.
Summary
This paper highlights a drawback in prior frameworks, such as Decision Transformer and Diffuser, where the absence of modular hierarchies among different tokens results in limited expressivity and flexibility. To overcome this issue, the paper introduces Decision Stacks (DS), a modular algorithm designed for learning goal-conditioned policies using offline datasets. The proposed method parameterizes three generative model-based modules for future observation prediction, reward estimation, and action generation, respectively. Through various offline evaluations in both MDP and POMDP environments, this paper demonstrates that DS outperforms previous approaches by generating superior plans.
Strengths
This paper presents a modular probabilistic framework, utilizing deep generative models to establish token-level hierarchies in trajectory generation. The proposed algorithm, Decision Stacks (DS), incorporates independent generative models for simulating the temporal evolution of observations, rewards, and actions, allowing for parallel learning and enabling flexible generative decision making. The algorithm is both straightforward and effective, particularly in the POMDP setting, where DS surpasses other baseline methods by a significant margin. Additionally, the experiments are conducted meticulously, and the visualizations of example rollouts in the Maze2D-medium-v1 environment are clear.
Weaknesses
One major concern regarding the proposed algorithm DS is the high complexity of the model, as it requires training three generative models. However, in most experiments such as offline RL with an MDP setting, the performance improvement compared to other baselines that use only one generative model is not significant. Furthermore, the paper lacks an explanation of the specific dimensions that were excluded to construct the POMDP setting. Additionally, as depicted in Figure 1, the generation of observation, reward, and action sequences follows a sequential order, and there exist dependencies among the three generative models. However, the paper does not provide clear explanations on how training can be parallelized and how the choice of generative models affects performance.
Questions
(1)How is the handcoded controller implemented in 4.1 and 4.2, respectively? What role does it play? In Table 1 and Figure 1, why do Decision Diffuser (DD) and DS exhibit similar performance when this controller is added? DD does not adopt a modular structure but utilizes the same diffusion-based observation model as DS. Does this indirectly suggest that the modular structure has minimal impact as long as a good generative model is chosen? (2)During the experiment, why was a diffusion model used for the observation model while transformer models were used for the action and reward models? Why do different choices of generative models lead to the results shown in Table 4? It is hoped that the author can provide corresponding reasonable explanations instead of simple combination attempts. typoes, e.g., "seggregate" --> "segregate".
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Methods such as Decision Transformer [1], which reduce reinforcement learning to a prediction task, have gained popularity partly due to their simplicity. In contrast, DS trades a high model complexity for relatively modest expressivity. It would be more convincing in terms of expressivity if experiments were conducted in stochastic environments[2] that yield better results. Furthermore, the paper lacks rational analysis in certain aspects, such as the selection of generative models to generate higher quality trajectories. [1] Chen L, Lu K, Rajeswaran A, et al. Decision transformer: Reinforcement learning via sequence modeling[J]. Advances in neural information processing systems, 2021, 34: 15084-15097. [2] Paster K, McIlraith S, Ba J. You Can't Count on Luck: Why Decision Transformers Fail in Stochastic Environments[J]. arXiv preprint arXiv:2205.15967, 2022.
Reminder
Thank you again for your valuable feedback on our paper. We've carefully considered your comments and conducted new experiments to address the weaknesses and questions you highlighted. We'd like to note that the discussion phase is drawing to a close, and we hope you've had a chance to review our response. Given the limited time, are there any other questions or concerns you'd like us to address? We would greatly appreciate your support in this regard.
Summary
This paper proposes to tackle the problem decision making using a stack of different generative models. The first generative model constructs a conditional distribution over observations. A subsequent generative model constructs a conditional distribution over rewards, with a final generative model constructs a distribution over actions. The authors illustrate the efficacy of this decomposed approach across a suite of different tasks.
Strengths
- I enjoyed reading the paper -- it was quite clear and easy to read. The motivation of the paper to decompose the generative modeling objective into a set of component modules is sound. - The paper follows a formulation of offline reinforcement learning as probabilistic inference that is likely to relevant to a large audience at NeurIPS with the increasing popularity of generative models - The approach performs well across a set of different environments, outperforming existing baselines across 3 separate tasks.
Weaknesses
- The results illustrated in the approach are a bit toy -- with the largest gains in the Maze2D environment. The Mujoco control environment has somewhat limited gains and the constructed POMDP environment seems a bit artificial. It may be that the Mujoco control environments have already saturated in performance and it may be interesting to try more complex planning tasks such as RLBench or other robotic manipulation environments. - While I understand the motivation to decompose trajectory synthesis into a set of modular components, I'm not sure I completely understand the particular decomposition of first observations, then rewards, and then actions. It seems like decomposing rewards first may be more natural then observations. - It might also be interesting to explore the extent to which decomposed submodules can enable compositional generalization or more efficient multitask learning by encoding structure in the greneration procedure. For instance -- perhaps the observation model can be used across a set of different tasks. There are a couple of typos in the paper listed below: - L133 typo auotregressive -> autoregressive - In L303 Dai et al should be Du et al. The method does not use a state model as a pretrained text2image model but rather a learned text2video model. - The table before section 4.2 is misformatted and extends to the margin of the paper
Questions
I had a couple questions about the underlying approach that might be interesting to discuss: - In equation 6, we sequentially sample from each factored distribution to sample from equation 3. In practice, would we instead want to sample marginalized distribution over actions across all possible observations and rewards given the goal? - It might be interesting to think a bit about how the individual decomposed modules can also provide feedback with respect to each other. For instance a planned set of states may induce a poor distribution over rewards. Having the conditional reward model then provide some feedback to regenerate the planned set of states may then be interesting. [1] may be an interesting read [1] Composing Ensembles of Pretrained Networks via Iterative Consensus
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Yes
Summary
This paper proposes to disentangle the different modalities (reward, observations, and actions) utilized in offline goal-conditioned reinforcement learning instead of the standard temporal single-module structure. The paper contributes three independent generative modules that have the benefit of parallelizable training. The paper includes empirical results showing the proposed algorithm's comparable or superior performance to baselines on several MDP and POMDP environments.
Strengths
1. The core idea to utilize a different decomposition of tokens to enable independent training of generative models appears interesting and novel. I enjoyed the simplicity of the approach, accompanied by the good empirical results. 2. Overall, I found the paper to be well-written and clear. I liked that the authors are clearly familiar with the relevant related literature and used diagrams throughout to help illustrate concepts. 3. Given the good empirical results, the idea of semantic decomposition (noted in strength 1) could be generally impactful for communities interested in transformers + RL. 4. I appreciate the extensive experiments performed to support the claims of flexibility and performance.
Weaknesses
In general, I liked this paper. A few things that I think could improve it: 1. I would have liked to see the training cost comparison to the other algorithms. 2. In general, I would also like to see a more extensive discussion of the experimental results, as noted in questions 4-6 in the Questions section below. 3. I feel like some of the claims are a bit strong. For example, how does the work "guarantee" expressivity and flexibility? This language is often used when we have an assured property of the system or algorithm, but I don't see any rigorous theoretical evaluation of this claim. More minor: 1. Missing some references, including work in compositional offline RL (as mentioned in the introduction, first paragraph) [Mendez et al., 2022] and a reference to the POMDP formalism [Kaelbling, Littman, and Cassandra, 1998]. 2. The focus on MDP and POMDP environments was somewhat unclear to me. I think including some motivation or intuition in the Introduction as to why this approach would work better than previous works in both types of environments would make this more clear. ----------------------------------------------------------------------------- [Mendez et al., 2022] Mendez, Jorge A., Harm van Seijen, and Eric Eaton. "Modular lifelong reinforcement learning via neural composition." arXiv preprint arXiv:2207.00429 (2022). [Kaelbling, Littman, and Cassandra, 1998] Kaelbling, Leslie Pack, Michael L. Littman, and Anthony R. Cassandra. "Planning and acting in partially observable stochastic domains." Artificial intelligence 101.1-2 (1998): 99-134.
Questions
1. Recent work [Carroll, et al. 2022] incorporates dynamics learning in some training setups and demonstrates improved performance over vanilla Decision Transformer. Is this not included in the experimental comparisons due to the different training scheme & architecture choice? 2. Why do only some of the algorithms have a reported error (e.g., in Tables 1&2)? Also, is the value standard error, standard deviation? 3. Could the authors please clarify what is meant by the expert-normalized scores in Section 4.2? 4. What is the training cost of the proposed algorithm (in terms of both samples and wall clock time)? How does it compare to the baselines? 5. Why is the architectural flexibility investigated using Hopper-medium-v2 despite it not being used in the other experiments (e.g., those presented in Table 3)? 6. Why does DS work better on medium but not medium expert or medium replay HalfCheetah (Table 3)? 7. How were the two dimensions of the state chosen to exclude (Section 4.3)? What is the observation representation in these experiments? --------------------------------------------------------------------------------- [Carroll et al., 2022] Carroll, Micah, et al. "Uni [mask]: Unified inference in sequential decision problems." Advances in neural information processing systems 35 (2022): 35365-35378.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have a small section in the Conclusion focusing on the limitations. I felt like this was sufficient. However, I can always appreciate a lengthier discussion section, especially when one's proposed algorithm performs similarly to other algorithms in some settings.
Summary
The paper proposes using different generative models (transformer or diffusion) rather than the same model (like in trajectory transformer/decision diffuser) for observation prediction, reward estimation, and action prediction in “model-based” offline RL. They demonstrate that this flexibility improves performance in offline RL in a POMDP setting where two dimensions of the state vector are removed for each environment.
Strengths
1. The paper is well written and easy to read. 2. The consistent increase on POMDPs is encouraging. And the ablation with different modules is very interesting.
Weaknesses
1. Similar performance to DD on D4RL Gym tasks. 2. Major similarities to previous work (trajectory transformer, decision diffuser) and the authors acknowledge this and highlight the reward modeling as novel. While it is interesting, a (temporal difference) variant already exists in the trajectory transformer paper [Janner et al 2021]. Namely, the trajectory transformer with Q function (a form of reward modeling) guided planning which can be found in Section 4.2 of [Janner et al 2021], under “Combining with Q functions”. 3. Does it matter which two observation dimensions are removed? An explanation of which dimensions were removed and why is missing and a sensitivity analysis to the dimensions removed are missing as well. 4. Is there any intuition for why the conditioning in Equations (4), (5), (6) were chosen? 5. Is there heuristic that could be used to choose the different modules with any online interaction? Such a heuristic would be necessary in offline RL.
Questions
Please see weaknesses.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
2 fair
Limitations
Yes.
Response to the authors
> DS introduces a novel token-first ordering of the trajectory components, instead of conventional timestep-first arrangements. Using a single model to model them together requires adjusting for modality-specific encoders and decoders. Does this simply shift the responsibility from the encoder to the tokenizer? > We did a new sensitivity analysis on dimension occlusions. This is really helpful. But is there perhaps a reason for occluding two dimensions only and not more? What happens with more occluded dimensions? > We tested this choice empirically as well in early experiments, and found our choice to significantly outperform the counterpart, which is shown in Table 3 in the rebuttal PDF. This is helpful as well for future users of DS. > In offline RL, datasets guide the choice of module architecture based on data type. I apologize for the typo. I meant to ask "any heuristic that could be used to choose the different modules **without** any online interaction?". I'm wondering about this because it is already particularly difficult to choose hyperparameters for a single model without any online interaction. Most papers today seem to evaluate online (indirectly violating offline RL requirements). Does having multiple architectures complicate this further?
question responses and a broader sensitivity analysis.
Thank you for the prompt response and careful reading of our rebuttal! We appreciate the new insights from your questions and address them below. >...simply shift the responsibility from the encoder to the tokenizer? Please apologize for our confusion but if we understand the question correctly, the comment is suggesting that our modules are different only in the use of different tokenizations? We’d argue that our modules are not simply employing distinct tokenizers. 1. As highlighted in the paper, splitting different modules explicitly avoids parameter sharing and thus provides more flexibility in training architectures, objectives, and optimization algorithms (e.g., transformers vs diffusion models). Such differentiation becomes important when considering the different natures of modalities. For instance, while states for robotic control can be represented as smooth signals in the form of images, the corresponding actions, such as joint torques, tend to be high-frequency and less smooth. 2. Using distinct modules in DS enables clean and effective generalization to new environments with overlapping semantics. In our rebuttal pdf (Table 2), we showed that the observation and reward models in DS, can be effectively reused across diverse environments with different action spaces. Doing so with non-modular approaches will involve finetuning the entire model. 3. Another advantage of our approach, as pointed out in our rebuttal, is that using distinct modules can allow us to directly use pretrained perception models. E.g., text2video models a language command (goal) to consecutive video frames (observation sequence). >What happens with more occluded dimensions? Thank you for pointing out the need for a broader sensitivity analysis! We conducted additional experiments and included the results below. We found that DS still continues to perform the best in aggregate performance. | Occluded dimension | BC | DT | TT | DD | Diffuser | DS(ours) | |---------------------------------------|----------|---------- |---------- |----------|----------|---------- | | Occlude first 2 dim of velocity | 43.3±2.2 | 59.0±2.3 | **68.7±8.3** | 52.3±5.5 | 52.5±7.4 | 68.0±3.4 | | Occlude middle 2 dim of velocity | 34.0±0.9 | 73.1±4.6 | 76.7±7.9 | 31.6±5.1 | 24.2±3.1 | **81.1±6.9** | | Occlude last 2 dim of velocity | 51.1±1.3 | 105.2±1.0| 61.6±3.3 | 32.7±2.4 | 70.7±4.7 | **110.9±0.4**| | Occlude last 4 dim of velocity | 2.5±0.1 | 53.6±4.8 | 59.3±6.8 | 9.0±2.6 | 5.8±1.7 | **69.0±9.9** | | Occlude last 6 dim of velocity | 3.4±0.1 | 30.4±4.9 | **57.9±5.2** | 3.8±8.1 | 2.9±0.2 | 33.0±4.7| | Occlude first 2 dim of position | 14.6±2.4 | **40.4±3.2** | 15.0±4.1 | 8.7±9.3 | 29.5±1.7 | 38.9±6.2 | | Occlude first 5 all dim of position | 10.4±1.0 | 27.6±5.3 | 38.9±4.1 | 10.7±0.8 | 12.4±3.7 | **39.0±2.5** | | Average | 22.7±1.1 | 55.6±3.7 | 54.0±5.7 | 21.3±4.8 | 28.3±3.2 | **62.8±4.9** | > any heuristic that could be used to choose the different modules without any online interaction? Thank you for bringing up this important practical question! In addressing the challenge of choosing different modules in an offline RL setting without online interaction, we propose the following heuristics: 1. One potential heuristic for fine-tuning each module is to measure the module's validation likelihoods. These can be computed on a held-out set of offline trajectories, and crucially, they don't require online interactions. This approach facilitates a data-driven method to compare and select module architectures. 2. Similarly, we can use a 2-sample test (e.g., Wasserstein distance) to compare the generated sequences with the dataset sequences. Given DS's modular design, we believe we can effectively utilize traditional unimodal definitions for these tests. There's no need to make adjustments to accommodate multiple modalities, streamlining the evaluation process. 3. DS's modular design enables us to utilize models trained on similar data and modalities for design decisions. As each module is focused on a single modality, it's feasible to apply configurations from models previously pretrained on similar modalities. This can serve as a foundation, which can then be fine-tuned further based on specific project requirements. While we acknowledge that these heuristics may not guarantee an optimal performance correlation, they can significantly narrow down the search space. Finally, the modularity of DS eliminates some hyperparameters that are often encountered in non-modular designs. For instance, traditional designs may require optimization of a weighted average of per-variable loss terms, where the weights become additional hyperparameters to tune. For DS, each component can be individually inspected and optimized, as opposed to a non-modular system where issues might be more intertwined and challenging to pinpoint.
Reminder
Thank you again for your valuable feedback on our paper. We've carefully considered your comments and conducted new experiments to address the new questions you highlighted. We'd like to note that the discussion phase is drawing to a close, and we hope you've had a chance to review our response. Given the limited time, are there any other questions or concerns you'd like us to address? We would greatly appreciate your support in this regard.
Thank you for the new experiments
Thank you for your response and new results. I am happy to raise my score from 3 to 5. With respect to my point about tokenizers, I simply meant to state that a token-first ordering of the trajectory components as proposed by DS may not necessarily be an improvement over the conventional timestep-first arrangement since now you require modality-specific tokenizers (such as BPE for language, patch tokenizer for images) rather than modality specific encoders (CNN for images, nn.Embedding for language).
Thank you
Thank you for acknowledging our work and the constructive and insightful feedback during the rebuttal discussions!
Reminder
Thank you again for your valuable feedback on our paper. We've carefully considered your comments and conducted new experiments to address the weaknesses and questions you highlighted. We'd like to note that the discussion phase is drawing to a close, and we hope you've had a chance to review our response. Given the limited time, are there any other questions or concerns you'd like us to address? We would greatly appreciate your support in this regard.
Thanks authors for attempting to address my questions. However, I'm still concerned about the following aspects. - Uniqueness of the proposed method (also mentioned by reviewer Xfrm); - Lack of rational justification of design choices, including increased model complexity (also mentioned by reviewers Xfrm Q4 and xgrf) and the neglection of temporal dependency and casual dependency between three random variables (also mentioned by reviewer xgrf); - The relatively simple and toyish evaluation tasks and superficial construction of the POMDP setting (also mentioned by reviewer E8pR); - Absence of arguments or proofs that support claims made in this paper (also mentioned by reviewer vCY4). For example: 1) L5 and L148 "maximal expressivity". It's possible that there is other modeling strategy that can be more expressive. Authors neither theoretically justify this, nor empirically demonstrate this. 2) L10 "our framework guarantees both expressivity and flexibility". The word choice of "guarantee" is inappropriate due to the lack of rigorous proof. - Lack of in-depth analysis into the flexibility provided by this model. Conducting experiments on all the Cartesian product {Observation Models} $\times$ {Action Models} $\times$ {Reward Models} is respectful, insightful conclusions/takeaways are missed though. Furthermore, I have the following questions regarding authors' reply: - Regarding table 2, which one from the three maze environments is modified? Are the transferred reward model and observation model kept frozen during training (otherwise the seemingly good transfer results could be attributed to the joint training in new tasks)? With the transferred reward model and observation model, does the experiment achieve better sample efficiency? Modifications on the action space are far-fetched to be called new tasks. - If the better performance on Hopper POMDP may be explained by the limited number of rollouts, wouldn't this suggest that experiment results reported in this work may not be statistically significant, also due to the limited evaluation rollouts? - FLOPS is usually used to measure computation. Regarding table 4, what are FLOPS values for DS and baselines?
response (1/2)
We thank the reviewer for the response. >Uniqueness of the proposed method (also mentioned by reviewer Xfrm); We have addressed this aspect in our original rebuttal. Given the subjective nature of this follow-up comment, can you elaborate what is the remaining issue? Also, note the comment by reviewer Xfrm is in a different context in comparison with a specific form for trajectory transformer, which we also addressed in our rebuttal response for the reviewer. >Lack of rational justification of design choices, including increased model complexity (also mentioned by reviewers Xfrm Q4 and xgrf) and the neglection of temporal dependency and casual dependency between three random variables (also mentioned by reviewer xgrf); We have addressed these questions in our rebuttal. To reiterate, we respectfully refute the comment that there is an increase in training time. We also elaborated on why temporal dependency is not necessary due to the chain rule and also ignores the distinct modality attributes. For both, we also included empirical evidence validating our design choices. We would be grateful if you can specify why our response in the original rebuttal does not address your original question. >The relatively simple and toyish evaluation tasks and superficial construction of the POMDP setting (also mentioned by reviewer E8pR); We respectfully disagree that our evaluation tasks are “toyish”. D4RL is the standard benchmark used in all major offline RL papers in the last few years [CQL, DT, IQL, Diffuser, DD] . Maze2D task was used for benchmarking planning by Diffuser (published at NeurIPS 2021) and DD (published at ICLR 2023, Oral), which are two of the closest baselines to our method. We've undertaken a comprehensive sensitivity analysis that varies the occluded dimensions for the POMDPs scenario as in Table 1 in the rebuttal PDF. Additionally, it's worth noting that our POMDPs construction on Mujuco tasks, specifically the case of exclusion of all velocity dimensions, was used as the main experimental setting in https://arxiv.org/pdf/1906.09510.pdf (UAI 2019) and also used in https://arxiv.org/pdf/1604.06778.pdf (ICML 2016) sec 3.3. >Absence of arguments or proofs that support claims made in this paper (also mentioned by reviewer vCY4). For example: L5 and L148 "maximal expressivity". It's possible that there is other modeling strategy that can be more expressive. Authors neither theoretically justify this, nor empirically demonstrate this. L10 "our framework guarantees both expressivity and flexibility". The word choice of "guarantee" is inappropriate due to the lack of rigorous proof. To reiterate our response to reviewer vCY4, maximal expressivity is guaranteed from our autoregressive factorization in Eq. 3. That is, since autoregressive models are derived from chain rule of probability, under idealized assumptions on model expressivity and dataset size, they can learn any data distribution. For flexibility, we refer to the property that the modular design allows us to use any architecture and training objective for the observation, reward, and action modules. As we mentioned to reviewer vCY4, we will edit and clarify the language in the final version. >Lack of in-depth analysis into the flexibility provided by this model. Conducting experiments on all the Cartesian product … is respectful, insightful conclusions/takeaways are missed though. Below we list the key insights from DS’s flexibility: Flexibility in architecture parameterization: Through our architectural flexibility experiments, we've showcased that each of the three modules can be parameterized using any appropriate architecture. Compositional transfer: To further illustrate the versatility of DS, we've conducted compositional generalization experiments. These demonstrate that observation and reward models can be effectively reused (while remaining frozen) for environments with varying action spaces. If you are looking for a specific kind of insight, please let us know and we’d be happy to discuss it.
response (2/2)
>Regarding table 2, which one from the three maze environments is modified? Are the transferred reward model and observation model kept frozen during training (otherwise the seemingly good transfer results could be attributed to the joint training in new tasks)? With the transferred reward model and observation model, does the experiment achieve better sample efficiency? Modifications on the action space are far-fetched to be called new tasks. Regarding Table 2, the experiments were carried out in the Maze2d-Umaze-v2 environment. Yes, both the reward and observation models remain frozen during transfer to different action spaces. The one observation model is shared across 6 environments; each of the two reward models is reused across 3 action spaces. By reusing these models across varying reward and action spaces, we can sidestep the need for their retraining, leading to more modular training compared to finetuning the whole model and better performance. >If the better performance on Hopper POMDP may be explained by the limited number of rollouts, wouldn't this suggest that experiment results reported in this work may not be statistically significant, also due to the limited evaluation rollouts? We used a similar or more number of rollouts than prior studies and this should not be diminishing the statistical significance of our method. Additionally, our broader sensitivity analysis consistently demonstrated the better performance of DS across multiple POMDP environments, showcasing its robust performance. >FLOPS is usually used to measure computation. Regarding table 4, what are FLOPS values for DS and baselines? Thank you for the suggestion. Our FLOPs are comparable to those of the closest baseline DD (3618 vs. 3287). However, we also note that FLOPs primarily account for computational operations and tend to overlook other integral aspects of real-world processing speed such as data transfer, I/O operations, and varied latencies [1]. As a result, even algorithms with matching FLOPs can deliver different performances across diverse hardware architectures, especially when certain hardware elements are optimized for specific operations. That is why we reported wall-clock time in our original experiments. This metric offers a comprehensive view of real-world performance, capturing both computational and other related delays, thus providing a direct insight into algorithmic efficiency. Also we emphasize that in our comparisons, we did not intentionally select any baseline to appear smaller, either in size or FLOPs. Every method is tailored to harness the maximum potential of the architecture it operates on and the architectures have been tuned extensively by the original authors of each baseline method. In fact, a method's inability to deliver optimal performance on larger architectures often signals limitations in scalability. [1] Bartoldson et al. Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities. JMLR. https://arxiv.org/abs/2210.06640
A broader sensitivity analysis on the construction of the POMDP setting
Dear reviewer ywNv, Thank you for engaging with us during the discussion phase! In our discussions with reviewer Xfrm, we conducted a broader sensitivity analysis of the excluded dimension within the POMDP experiments, as detailed in the table below. We believe this result helps address the question you raised below: > superficial construction of the POMDP setting By testing varying numbers of occluded dimensions and different semantics, such as velocity and position, we found that DS consistently outperforms other baselines in overall performance. | Occluded dimension | BC | DT | TT | DD | Diffuser | DS(ours) | |---------------------------------------|----------|---------- |---------- |----------|----------|---------- | | Occlude first 2 dim of velocity | 43.3±2.2 | 59.0±2.3 | **68.7±8.3** | 52.3±5.5 | 52.5±7.4 | 68.0±3.4 | | Occlude middle 2 dim of velocity | 34.0±0.9 | 73.1±4.6 | 76.7±7.9 | 31.6±5.1 | 24.2±3.1 | **81.1±6.9** | | Occlude last 2 dim of velocity | 51.1±1.3 | 105.2±1.0| 61.6±3.3 | 32.7±2.4 | 70.7±4.7 | **110.9±0.4**| | Occlude last 4 dim of velocity | 2.5±0.1 | 53.6±4.8 | 59.3±6.8 | 9.0±2.6 | 5.8±1.7 | **69.0±9.9** | | Occlude last 6 dim of velocity | 3.4±0.1 | 30.4±4.9 | **57.9±5.2** | 3.8±8.1 | 2.9±0.2 | 33.0±4.7| | Occlude first 2 dim of position | 14.6±2.4 | **40.4±3.2** | 15.0±4.1 | 8.7±9.3 | 29.5±1.7 | 38.9±6.2 | | Occlude first 5 all dim of position | 10.4±1.0 | 27.6±5.3 | 38.9±4.1 | 10.7±0.8 | 12.4±3.7 | **39.0±2.5** | | Average | 22.7±1.1 | 55.6±3.7 | 54.0±5.7 | 21.3±4.8 | 28.3±3.2 | **62.8±4.9** | As it approaches the discussion deadline, we sincerely hope you've had an opportunity to review our response. May we kindly inquire if there are any additional questions or concerns you'd like us to address? We truly value your feedback and insights. Thank you!
Thanks
Thanks for the clarifications! I think this paper is quite interesting and have raised my score to a 7.
Thank you
Thank you for acknowledging our work and the constructive and insightful feedback during the rebuttal discussions!
Reminder 2 + additional experiments on POMDP settings
Thank you again for your valuable feedback on our paper. We've carefully considered your comments and conducted new experiments to address the weaknesses and questions you highlighted. In our discussions with reviewer Xfrm, we conducted a broader sensitivity analysis of the excluded dimension within the POMDP experiments beyond our original rebuttal as well, as detailed in the table below. We believe these findings also help address the question you raised. > Q2: the paper lacks explanation regarding specific dimensions excluded in the POMDP setting is missing. By testing varying numbers of occluded dimensions and different semantics, such as velocity and position, we found that DS consistently outperforms in overall performance. | Occluded dimension | BC | DT | TT | DD | Diffuser | DS(ours) | |---------------------------------------|----------|---------- |---------- |----------|----------|---------- | | Occlude first 2 dim of velocity | 43.3±2.2 | 59.0±2.3 | **68.7±8.3** | 52.3±5.5 | 52.5±7.4 | 68.0±3.4 | | Occlude middle 2 dim of velocity | 34.0±0.9 | 73.1±4.6 | 76.7±7.9 | 31.6±5.1 | 24.2±3.1 | **81.1±6.9** | | Occlude last 2 dim of velocity | 51.1±1.3 | 105.2±1.0| 61.6±3.3 | 32.7±2.4 | 70.7±4.7 | **110.9±0.4**| | Occlude last 4 dim of velocity | 2.5±0.1 | 53.6±4.8 | 59.3±6.8 | 9.0±2.6 | 5.8±1.7 | **69.0±9.9** | | Occlude last 6 dim of velocity | 3.4±0.1 | 30.4±4.9 | **57.9±5.2** | 3.8±8.1 | 2.9±0.2 | 33.0±4.7| | Occlude first 2 dim of position | 14.6±2.4 | **40.4±3.2** | 15.0±4.1 | 8.7±9.3 | 29.5±1.7 | 38.9±6.2 | | Occlude first 5 all dim of position | 10.4±1.0 | 27.6±5.3 | 38.9±4.1 | 10.7±0.8 | 12.4±3.7 | **39.0±2.5** | | Average | 22.7±1.1 | 55.6±3.7 | 54.0±5.7 | 21.3±4.8 | 28.3±3.2 | **62.8±4.9** | We'd like to note that the discussion phase is drawing to a close, and we hope you've had a chance to review our response. Given the limited time, please let us know if there are there any other questions or concerns you'd like us to address.
Thank you for your response and new experimental results. （1）Regarding question 4, the author has provided a relatively brief analysis of data modalities. However, there seems to be a lack of in-depth analysis and discussion about the specific structural advantages of the diffusion model and the transformer in handling these data modalities. Furthermore, there appears to be no in-depth discussion regarding the results generated by different model combinations in Table 4. （2）Tackling serialized trajectory information aids in inferring missing state information, which is a common practice in POMDPs. As mentioned in the Limitations section, the expressiveness in stochastic environments may be more challenging and persuasive compared to POMDP settings.
Decision
Accept (poster)