Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance
Multi-objective learning (MOL) problems often arise in emerging machine learning problems when there are multiple learning criteria, data modalities, or learning tasks. Different from single-objective learning, one of the critical challenges in MOL is the potential conflict among different objectives during the iterative optimization process. Recent works have developed various dynamic weighting algorithms for MOL such as MGDA and its variants, where the central idea is to find an update direction that avoids conflicts among objectives. Albeit its appealing intuition, empirical studies show that dynamic weighting methods may not always outperform static ones. To understand this theory-practical gap, we focus on a new stochastic variant of MGDA - the Multi-objective gradient with Double sampling (MoDo) algorithm, and study the generalization performance of the dynamic weighting-based MoDo and its interplay with optimization through the lens of algorithm stability. Perhaps surprisingly, we find that the key rationale behind MGDA -- updating along conflict-avoidant direction - may hinder dynamic weighting algorithms from achieving the optimal ${\cal O}(1/\sqrt{n})$ population risk, where $n$ is the number of training samples. We further demonstrate the impact of the variability of dynamic weights on the three-way trade-off among optimization, generalization, and conflict avoidance that is unique in MOL. We showcase the generality of our theoretical framework by analyzing other existing stochastic MOL algorithms under the framework. Experiments on various multi-task learning benchmarks are performed to demonstrate the practical applicability. Code is available at https://github.com/heshandevaka/Trade-Off-MOL.
Paper
Similar papers
Peer review
Summary
This paper has two contributions: (1) MoDo algorithm which is a variant of MGDA with a double sampling to obtain an unbiased stochastic estimate of the gradient problem. (2) A solid theoretically analysis on the error of multi-objective optimization.
Strengths
This paper has a very detailed analysis on the optimization error and generalization error of multi-objective optimization.
Weaknesses
In the experiments, authors only compare MoDo with MGDA, but there are many other algorithms, like CAGrad, GradNorm, Uncertainty Weight. This baseline is not enough to demonstrate the effectiveness of the proposed method. Overall, MoDo is better than MGDA. However, in two objectives out of three (table 2), MGDA is better than MoDo, also not convincing enough on the effectiveness of the proposed method.
Questions
Could you summarize and simply explain the advantages of MoDo over MGDA here, smaller error or more efficient? Why MoDo has an unbiased gradient estimation? line 142
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Authors only discuss the situation of using Pareto optimal to get the aggregated gradient. However, there are other methods other than Pareto to balance multi-objective optimization like balancing gradient magnitudes like AdaTask, GradNorm.
Summary
This work considers the multi-objective learning problem. The classic idea of dynamic weighting in MOL is to take gradients from each objective and to weight them using a fixed procedure to avoid conflicts between different objectives. Empirically, however, there often seems to be performance degradation when using these methods. They discover that this is due to a tradeoff between optimization, generalization, and conflict avoidance. They propose a new algorithm, MoDo, that interpolates between static weighting and dynamic weighting and find parameters that can control this tradeoff effectively.
Strengths
Overall, this is a strong work that provides a lot of insight into dynamic weighting in multi-objective optimization. The highlights are as follows: - The paper is very intuitively written and easy to follow. Specifically, the three-way tradeoff is clear both intuitively and quantitatively. In addition, Figures 1 and 2 are very well done and extremely insightful. - The proposed MoDo algorithm is very simple and intuitive while brilliantly highlighting the three-way tradeoff inherent in dynamic weighting algorithms. I found the remark about being in the early stopping regime for generalization error to diminish (T = o(n)) interesting, and it is also empirically highlighted later in the work. - The tradeoffs for both strongly convex and nonconvex cases are analyzed - The findings in this work translate to a practical explanation of the behavior of dynamic weighting algorithms
Weaknesses
- While MoDo is a great theoretically inspired algorithm that controls the aforementioned tradeoffs, I am not sure how well it performs empirically. While there is an empirical result, the tasks seem rather simple. In addition, as Table 2 shows, MoDo does not consistently outperform MGDA. In practice, I feel handling the conflict avoidance tradeoff shouldn't matter much as long as the accuracy for the task is good. Therefore, I am not sure how applicable the algorithm would be in practical situations. - As the authors mention in their limitations section, the analysis is only about one specific algorithm in one specific setting, which makes it unclear how general the principles in this work would apply when the assumptions are relaxed. I feel this given this work is meant to analyze a practical phenomenon, this is a substantial weakness, as it is unlikely that Assumptions 1-3 are all true in real-world settings (such as the image classification setting the authors test on) - There is no intuition of the proof in the main text, and while this is common in optimization papers, it would be useful to learn about the key insights towards the proof.
Questions
- Can you compare the theoretical tradeoffs in Table 1 with the empirical findings in Figure 4? How well do they match with each other? - How can knowledge of this three-way tradeoff be used to design better algorithms in multi-task learning in the future? - How can we tune the parameters in practice? How sensitive is the algorithm to the parameters?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Great!
Summary
This paper studies three-way trade-off in multi-objective learning: 1) optimization error caused by sampling and stochastic training; 2) generalization error that measures the difference between source and target sets; 3) conflict-avoidance direction error that is the bias between the calculated direction and the right one. The authors propose the MoDo algorithm to optimize this trade-off in one algorithm. Plenty of theoretical analysis supports the claims.
Strengths
1. This paper considers the multi-objective algorithmic design in a bigger picture by compositing three-way trade-off together, and provides solid analysis to solve it. 2. This paper is well written. In particular, Table 1 and Figure 2 help to understand the contribution better.
Weaknesses
1. The proposed algorithm needs to compute three batches of gradients to run an iteration, which is much less computationally efficient. 2. Benchmarks are too limited that only contain static and MGDA (2018 proposed), experiments have not compared with other typical methods like PCGrad and CAGrad, which have better empirical performance. 3. Several claims are out-of-date. In 34-35 "Unfortunately, the reason behind this empirical performance degradation is not fully understood and remains an open question" is not accurate, since [1] has proved that the vanilla MGDA algorithm and also PCGrad and CAGrad will not converge to Pareto optimal. In 268 "This can be overcome by increasing the batch size during optimization [25]" is wrong, because the assumption on Lipschitzness for $\lambda$ with respect to gradients has been proved to be wrong (Proposition 2 in [1]). Also, it can be proved that even a stochastic error of the gradient can cause a significant bias in the direction, so only increasing the batch size does not work. [1] S Zhou, W Zhang, J Jiang, W Zhong, J Gu, W Zhu - Advances in Neural Information Processing Systems, 2022
Questions
1. Why use double sampling? Can MoCo completely solve the problem? Or is it possible to reduce the sample complexity? 2. Does the proposed theory can help improve PCGrad and CAGrad?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
Sample efficiency of the proposed algorithm.
Summary
This paper studies the multi-objective optimization, and in particular, focus on the generalization and stability analysis. By decomposing the Pareto stationarity error into the generalization and optimization error, the authors then analyze and upper-bound these two errors respectively. The distance to the conflict-avoidant direction, which optimizes al objectives jointly, is also analyzed. A stochastic variant of MGDA named MoDo has been developed and analyzed. Based on the theoretical derivations, the authors find a three-way tradeoff among the optimization error, generalization error and the CA error. Some implications on the parameter selection and illustrations are provided.
Strengths
1. The studied topic on multi-objective optimization has received increasing attention thanks to the important applications like multi-task learning. Studying the generalization, stability and the tradeoff is important and under-explored. 2. Although generalization has been studied for multi-objective optimization, stability has not been explored. This work seems to be the first one to fill this gap. 3. The tradeoff among optimization and generalization is not surprising, but their tradeoffs with the CA direction error seem to be new given the multi-objective structure.
Weaknesses
1. How is (4a) equivalent to (4b)? Can the authors provide some more details? 2. Before (4a), it says that CA direction maximize the minimum descent of all objectives. But (4a) seems to say that the direction minimizes something. 3. The upper bounds in the nonconvex case may not be very tight given the large exponential dependence, and hence may not be able to exactly capture the generalization and tradeoff behaviors. Is it possible to provide a lower bound in this case? Or what challenges in getting a tight bound? 4. The analysis uses Frobenius norm, which may be large in practice. Instead, spectrum norm may be more proper. Can the authors comment on why use Frobenius norm rather than spectrum norm? Overall, I appreciate the studied problem and the analysis in this work, but I am also open to other reviewers’ comments.
Questions
See Weakness part.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
See Weakness part.
Summary of Theoretical Contributions
### Summary of Theoretical Contributions We provide the **first ever known stability-based generalization of MOL**. This is a non-trivial extension of the results in single objective learning [12], which cannot be directly applied to MOL due to several challenges as follows. **T1.** The **measure of Pareto Stationary (PS) risk** defined in gradients (line 108, Eq.(2)) overcomes a key *challenge* brought by the classical function value-based risk measures -- the **unnecessarily small step size choice**. Specifically, it requires $1/t$ step size decay in the nonconvex case, otherwise the generalization error will depend exponentially on $T$. But such choice leads to very slow convergence of the optimization error. This is addressed by the definitions of gradient based measures and sampling determined MOL algorithms, which yields a stability bound in ${\cal O}(T/n)$ without any step size decay. See more discussions in line 175-177 below Theorem 1. **T2.** The **generalization bound using the algorithmic stability approach** (Theorems 1,2) is the first on generalization and stability analysis for MOL, and is non-trivial compared to SGD because it involves two coupled sequences. The *challenge* is that the **classical contraction property that is often used to derive stability in the SC case does not hold**. This is addressed by controlling the change of $\lambda_ t$ by the step size $\gamma$ to derive a tighter bound. See more discussions in Appendix B.4, line 660-672. **T3.** The **optimization error bound** (Theorem 3) is also improved with **either relaxed assumption or improved convergence rate**. More details are summarized in **General Response-Q1**. This is achieved by 1) instead of bounding the approximation error to $\lambda^ *(x_ t)$, we bound that to the update direction $\nabla F_ S(x_ t)\lambda^ *(x_ t)$ as a whole, and 2) instead of using the descent lemma of $F_ S(x_ t)\lambda_ t$ with a dynamic weight, we use that of $F_ S(x_ t)\lambda$ with a fixed weight, thereby improving the tightness of the bound. See more discussions in Appendix C.3. We would like to further explain the technical challenges and how we address them in the proof if the reviewers are interested. We look forward to further engagement with the reviewers and area chair! Sincerely, Authors
Looking forward to reviewers' feedback
Dear AC and reviewers, We would like to start by thanking all reviewers for the positive feedback and constructive comments given in the initial reviews. While the discussion period has started several days ago, we have not received any feedback based on our responses. We would like to use the interactive feature of OpenReview to engage the reviewers with the discussion. In particular, in our responses, we believe we have provided a1) summary of our theoretical contribution; a2) additional clarifications on algorithm designs; and a3) new numerical results with more baselines; to fully address the reviewer's concerns. We hope our responses convince the reviewers about the merits of this work. If the reviewer has any other suggestions or comments, please don't hesitate to let us know! Best Regards, Authors
Thanks for the comments and clarification. Previously I was on the 5-6 borderline, but after the clarifications I still maintain that 6 is a better choice, as I still believe this work is technically solid and would have moderate impact. My main concern is still that it is not clear how these insights can be applied for future advances in multi-objective optimization.
Response to the concern of how to apply the insights in this paper
## How these insights can be applied for future advances in multi-objective optimization Thank you very much for your quick response. It is an excellent suggestion, and in fact our ongoing work to apply the insights in this work for future advances in multi-objective optimization (MOO). Here's a breakdown of how the insights can be applied in each of the areas. **1. Theoretical Applications:** - **Analyzing other MOO algorithms.** This cannot be achieved by simply combine prior works solely on optimization and generalization analysis because they are often focusing on different settings. Applying this theoretical framework, we could also analyze the three types of errors for other dynamic weighting MOO algorithms such as MoCo, CAGrad, and PCGrad. Specifically, for generalization, Propositions 2, 3 still hold. Since the stochastic implementations of PCGrad and CAGrad are sampling determined (Definition 3), the bound in the NC case (Theorem 1) holds. Combining the optimization and generalization error bounds, we can find better hyperparameters in PCGrad and CAGrad to minimize the test risks. - **Studying benefits of MOO algorithms over static weighting.** Prior theoretical works of MOO mostly focus on optimization convergence to Pareto stationarity. However, this can also be guaranteed by static weighting. Therefore, the theoretical benefits of MOO algorithms over static weighting remain open. Our analysis addresses this critical question by demonstrating the advantages of MOO algorithms in CA distance reduction. This provides a justification for their use in cases where CA distance reduction is crucial. **2. Practical Applications:** - **Hyperparameter choice:** Our theory is suitable for analyzing the effect of hyperparameters such as step size and number of iterations on the three errors, and the total testing risk. This allows us to find better hyperparameters to minimize the test risks. For example, when performing grid search of hyperparameters for MoDo, we focus on a range with relatively small $\gamma$. - **Algorithm choice:** Comparing error bounds of different algorithms allows for a more informed algorithm selection process based on the nature of the problem. The choice of algorithm depends on which error dominates the performance. For example, if in a problem the CA distance is a major factor that prevents the algorithm to achieve good performance, then we choose to use the algorithm with the smallest CA distance. And if the generalization error dominates, we could choose e.g., MoDo with a small step size $\gamma$ or static weighting to obtain better testing performance. - **Algorithm design:** Our theory could inspire the development of new MOO algorithms that strike a better balance between among the three types of errors. For example, it suggests large drift of $\lambda$ could degrade the test error. Therefore, future algorithms could be designed to update $\lambda$ not at each iteration, but only when there is significant conflict in gradients, e.g., the angle between two gradients is larger than 90 degrees. This could potentially improve both the test error and the efficiency of the MOO algorithm. --- Following your excellent suggestion, we will also incorporate this in our discussion of future work in the revision.
Thanks for the response
I thank the reviewer for the detailed response. Based on the global response, could I ask two more questions? 1. Can you elaborate how MoDo achieves the $\epsilon^{-2}$ sample complexity from Theorem 3? Why is there a huge improvement over the SOTA MoCo without bounded function assumption? 2. The Frobenius norm may be still a little bit large. Can the authors show the dependence on the number of tasks in the bounds since it may matter in the MTL?
Response to additional questions
Thank you very much for engaging in the discussion! The answers to your questions are as follows. >**Q1-1.** Can you elaborate how MoDo achieves ${\cal O}(\epsilon^{-2})$ sample complexity from Theorem 3? For a fair comparison with prior works such as MoCo, we refer to the result provided in Appendix C.3 in page 39, Eq.(183) without taking the square root, which states the optimization error is bounded as $$\frac{1}{T}\sum_{t=1}^T \mathbb{E}_ A || \nabla F_ S(x_ t) \lambda_ t^ *(x_ t) ||^2 = O(\frac{1}{\alpha T} + \alpha + \gamma)$$ where choosing $\alpha = \Theta(T^ {-\frac{1}{2}}), \gamma= \Theta(T^ {-\frac{1}{2}})$, we can obtain the optimal rate $$\frac{1}{T}\sum_{t=1}^T \mathbb{E}_ A || \nabla F_ S(x_ t) \lambda_ t^ *(x_ t) ||^2 = O(T^ {-\frac{1}{2}})$$ which implies $O(\epsilon^ {-2})$ sample complexity. >**Q1-2.** Why is there a huge improvement over the SOTA MoCo without bounded function assumption? This is mainly due to the difference of the MoDo algorithm with double / independent sampling, and the difference in our proof techniques compared to those in MoCo. Some key ideas of the proof techniques are summarized in **Summary of Theoretical Contributions-T3**. To be more specific, in our proof we adopt the following steps: **S1.** We first use the descent lemma for $F_ S(x)\lambda$ using a fixed $\lambda$, see Eq.(176) in our Appendix. This allows cancellation of the function values when taking the telescope sum, which cannot be achieved using a dynamic $\lambda_ t$. As a result, the bounded function value assumption needs to be introduced in prior works if using a dynamic $\lambda_ t$ for the descent lemma, see e.g., Eq.(75) in the MoCo paper [8]. **S2.** Then, the inner product term $\mathbb{E}_ A\langle\nabla F_ S(x_ t) \lambda, \nabla F_ S(x_ t) \lambda_ {t+1}\rangle$ in the inequality Eq.(177) is related with the gradient norm $\mathbb{E}_ A||\nabla F_ S(x_ t) \lambda_ t||^2$ by the property from the update of $\lambda_ t$ in Lemma 16, Eq.(155), which is a nice property with bias reduction due to double sampling in the subproblem. **S3.** Finally, taking telescope sum, the optimization error measured in gradient norm $\mathbb{E}_ A||\nabla F_ S(x_ t) \lambda_ t||^2$ can be bounded without introducing additional assumptions. Comparing the result without the bounded function value assumption in MoCo, Theorem 2, the inner product term in their Eq.(70), $\langle\nabla f_ m(x_ k), \nabla F(x_ k) \lambda_ k^ *-Y_ k \lambda_ k\rangle$, is bounded by Cauchy-Schwartz inequality, and then by triangle inequality, which boils down to bounding the moving average gradient error $||Y_ k - \nabla F(x_ k)||$, the dynamic weight approximation error $||\lambda_ k -\lambda_ {\rho,k}^ *||$, etc. This may not be as tight as our **S2** directly using the property in Lemma 16 from double sampling in the subproblem. >**Q2.** The Frobenius norm may be still a little bit large. Can the authors show the dependence on the number of tasks in the bounds since it may matter in the MTL? Sure, thanks for the insightful suggestion! We have restated below the bounds with explicit dependence on the number of tasks $M$. **Theorem 1:** $\epsilon_F^2 = O\Big(\frac{MT}{n}\Big)$, $\mathbb{E}_ {A,S}[R_ {\rm gen}(A(S))] = O(M^ {\frac{1}{2}} T^ {\frac{1}{2}} n^ {-\frac{1}{2}} )$, since $G^2 = O(M)$. **Theorem 2:** $\epsilon_F^2 = O\Big(\frac{M}{n}(\alpha + \frac{M}{n} + M\gamma)\Big)$, and $\mathbb{E}_ {A,S}[R_{\rm gen}(A(S))] = O(M^{\frac{1}{2}} n^ {-\frac{1}{2}} )$, when $M\leq n$ and $M\gamma \leq 1$. **Theorem 3 (without square root):** $\frac{1}{T}\sum_{t=1}^T \mathbb{E}_A || \nabla F_S(x_t) \lambda_t^*(x_t) ||^2 = O(\frac{1}{\alpha T} + \alpha + M^{\frac{3}{2}}\gamma)$ where $\gamma$ can be controlled to ensure $M^{\frac{3}{2}}\gamma$ to be small. We will also include discussion of the dependence on $M$ in our revision, and further improve the bound using other matrix norms in our future work.
Thanks for the clarification
I thank the authors for the clarification. I increase my score to 7. Best, Reviewers
Thanks for your constructive comments
Thank you again for engaging in the discussion and providing constructive comments! We will incorporate the suggestion in our revision.
A kind request of your feedback
Dear Reviewer UWFs, Thank you very much for your review. While the discussion period has started several days ago, we have not received your feedback on our response. We believe we have addressed all your concerns including: - Comparison with other baselines - Advantages of MoDo over MGDA We kindly request your feedback on whether our response resolves your concerns. Your additional comments would be invaluable to us! Sincerely, Authors
A kind request of your feedback
Dear Reviewer LuAm, Thank you very much for your review. While the discussion period has started several days ago, we have not received your feedback on our response. We believe we have addressed all your concerns including: - Comparison with other baselines - Sample efficiency of MoDo - More precise and detailed discussion and comparison of the pioneering contribution of [47] in stochastic multi-objective optimization We kindly request your feedback on whether our response resolves your concerns. Your additional comments would be invaluable to us! Sincerely, Authors
following questions
Thanks for the reply of the authors! I feel my original rating is too low but I need to confirm some points: (1) The motivation of stochastic MOO algorithms? Why we want stochastic MOO? because the true gradient or full gradient is hard to get (as claimed in MoCo)? I have this question because MGDA and CAGrad has been introduced and worked before these stochastic MOO algorithms, I wonder how did MGDA or CAGrad to get objective gradients? maybe they just use the gradient of a mini-batch? hard to believe MGDA or CAGrad was using a full gradient (the gradient of the whole dataset)? yeah, this is the github (https://github.com/isl-org/MultiObjectiveOptimization/blob/master/multi_task/train_multi_task.py) of MGDA from the original authors, seems MGDA is using mini-batch gradient (2) In your "general response", you mentioned "vanilla mini-batch MGDA does not converge", may I ask where does this come from? any proof or empirically results? note: I am not the author of MGDA, just wonder how does this multi-gradient method really work in practice
Response to following questions
Dear Reviewer UWFs, Thanks a lot for your prompt reply! Below are our answers. >**Q1.** The motivation of stochastic MOO algorithms? Why we want stochastic MOO? because the true gradient or full gradient is hard to get (as claimed in MoCo)? The motivation is that full-batch gradient requires large memory or computation and is impractical to get for large-scale problems. As the reviewer correctly pointed out, most of the implementations of practical multi-task learning algorithms use stochastic (mini-batch) gradient instead of the true / full gradient. This can be observed in the github of MGDA you mentioned, and also the github of CAGrad. However, the algorithm design and its analysis work only under the deterministic setting, which generates a significant gap. Note that this gap is not purely theoretical but also of practical relevance as evidenced in your Q2 below. >**Q2.** In your "general response", you mentioned "vanilla mini-batch MGDA does not converge", may I ask where does this come from? any proof or empirically results? note: I am not the author of MGDA, just wonder how does this multi-gradient method really work in practice Yes, this has been proved in recent works [47, Theorem 1] and [25, Section 4], which states that "There is a stochastic convex optimization problem for which MGDA, PCGrad, CAGrad do not converge to the Pareto optimal solution." Numerical examples to demonstrate this non-convergence phenomenon can be also found in Figure 1 of [47]，Figure 2 of [25], and Figure 3 of [8]. However, prior works of MOO optimization convergence analysis (e.g. the analysis provided in CAGrad) analyzed the deterministic version of the algorithms, but implemented the stochastic mini-batch gradient. This motivates a recent line of works that design different stochastic variants of MGDA algorithms [8,25,47], and provide their theoretical optimization convergence analysis. [25] Suyun Liu and Luis Nunes Vicente. "The Stochastic Multi-gradient Algorithm for Multi-objective Optimization and its Application to Supervised Machine Learning." Annals of Operations Research, 2021. [47] Shiji Zhou et al. "On the Convergence of Stochastic Multi-Objective Gradient Manipulation and Beyond." NeurIPS 2022 [8] Heshan Fernando et al. "Mitigating gradient bias in multi-objective learning: A provably convergent stochastic approach." ICLR 2023 --- Thank you very much for engaging in the discussion! We really appreciate your feedback and hope our answer resolves your questions. We would be happy to answer your following questions if there are any.
rating raised from 4 to 5
Thanks for the reply! With the better understanding of this paper, I raised the rating
Thank you for your detailed comments
Thank you very much for your detailed comments! We will make sure to incorporate your comments in the revision.
Thanks for the rebuttal.
Although the review did not engage, I'll carefully read and consider it during the decision period. AC
Thanks for the assistance
Dear Area Chair, Thank you very much for your efforts in the review process. We would be happy to provide any additional information if asked. Sincerely, Authors
Decision
Accept (poster)