Quantifying uncertainty is important for actionable predictions in real-world applications. A crucial part of predictive uncertainty quantification is the estimation of epistemic uncertainty, which is defined as an integral of the product between a divergence function and the posterior. Current methods such as Deep Ensembles or MC dropout underperform at estimating the epistemic uncertainty, since they primarily consider the posterior when sampling models. We suggest Quantification of Uncertainty with Adversarial Models (QUAM) to better estimate the epistemic uncertainty. QUAM identifies regions where the whole product under the integral is large, not just the posterior. Consequently, QUAM has lower approximation error of the epistemic uncertainty compared to previous methods. Models for which the product is large correspond to adversarial models (not adversarial examples!). Adversarial models have both a high posterior as well as a high divergence between their predictions and that of a reference model. Our experiments show that QUAM excels in capturing epistemic uncertainty for deep learning models and outperforms previous methods on challenging tasks in the vision domain.
Paper
Similar papers
Peer review
Summary
The authors propose a new method for estimating epistemic uncertainty. They propose to adversarially search for modes of the posterior distribution. Empirically, they demonstrate that their uncertainty performs well on OOD detection.
Strengths
The empirical results of this approach on OOD detection are very promising.
Weaknesses
- The method performs well on ood detection. Is the resulting uncertainty also well calibrated? - l60 : The argument as to why ensembles and BNNs underperform is not clear to me. The authors say that they miss important posterior modes. Caen the authors elaborate on this? - L142: can the authors demonstrate that the current methods underestimate uncertainty? Just because some methods dont explore all modes doesn mean that they underestimate uncertainty. E.g. they may overestimate the width of other modes. - Lack of comparison with other ensemble methods, e.g. [1]. [1] in particular shows similar results on the two moon example. This makes me wonder whether the main impact of the method comes from the mixture of gaussians, as done in [1], or from the adversarial search. I guess it would be important to quantify that. [1] Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
Questions
see weaknesses
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
N/A
Summary
The paper introduces Quantification of Uncertainty with Adversarial Models (QUAM), a novel approach for epistemic uncertainty estimation in deep learning. Well-known uncertainty quantification approaches, such as Deep Ensembles or variational inference, underestimate the epistemic uncertainty by sampling from posterior modes found by gradient descent, which might prevent some modes to be found due to the shape of the loss landscape. To overcome this limitation, QUAM introduces adversarial models: plausible models with a high posterior that differ in the prediction from a reference one. Mixture importance sampling, with the adversarial models as modes, is then used to estimate the epistemic uncertainty. Experiments on both synthetic and vision data confirmed the validity of the approach in improving epistemic uncertainty estimation.
Strengths
The paper is very well-written, all the design choices are motivated by references to existing work, and the experiments are reproducible and robust. The topic is of great significance since in relates to important topics such as Out-Of-Distribution detection and Responsible AI, as also stated by the authors in the Societal Impact Statement. Overall, I really liked reading this work and I rate it as a strong accept.
Weaknesses
I have no major remarks against this version of the paper.
Questions
I have just two minor observations: - The notation used in equation 1 (like the mutual information) was introduced only in the Appendix. - Figure C.1 (Appendix): maybe wrong colour assignment of the adversarial model (yellow instead of blue).
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
The authors addressed potential negative societal impact of their work in the Appendix.
Summary
This paper introduces Quantification of Uncertainty with Adversarial Models (QUAM). Building on the claim that previous epistemic uncertainty estimation methods (e.g. MC dropout, Deep Ensembles) underestimate the epistemic uncertainty by only considering the posterior distribution when sampling models, the authors introduce an Adversarial Model Search algorithm to identify models located within a posterior mode but with different predictive distributions compared to the reference model.
Strengths
Previous epistemic uncertainty estimation methods (i) require changing the training procedure to account for uncertainty estimation by e.g. adding dropout layers or training multiple models, (ii) underestimate the epistemic uncertainty by only sampling models from the posterior distribution. The proposed method QUAM addresses (i) by searching for adversarial models at test time for each test sample, thus working with any pre-trained model. Moreover, it solves (ii) by designing an adversarial model search algorithm that allows to cover more posterior modes by identifying models located within a posterior mode but with different predictive distributions compared to the reference model.
Weaknesses
## Lack of clarity Although the method is interesting and addresses some important limitations of previous epistemic uncertainty estimation approaches, the paper writing should be revised. Too often important details are omitted in the main paper or disclosed too late. In particular, 1. **Definition of adversarial models.** the definition of adversarial models introduced in this paper is completely new and has nothing to do with previous definitions of generative adversarial models [a, b] or adversarial attacks [c]. However, the first definition of what the authors mean by adversarial models is provided only at page 6, making it very hard to understand the paper before reading line 226. This paper would **greatly benefit** from earlier definitions of adversarial models in both abstract and introduction. Moreover, writing "(not adversarial examples!)" is the least elegant solution. 2. **Missing details.** Implementation details are solely reported in the supplement, but some details are essential in the main paper for better understanding. In particular, the number of adversarial models, number of MC samples, and the number of models in the deep ensemble should be clearly stated in the main paper. 3. **Settings (a) and (b)** It is not clear from the writing how the settings (a) and (b) defined in Section 2 relate to the experiments and whether the proposed QUAM presents any significant advantage on either task in particular compared to previous methods. ## Unsupported claims Some claims are unsupported. They should either be supported by quantitative or theoretical analysis in the paper or some reference to other papers should be provided. In particular, 1. The claim that previous epistemic uncertainty estimation methods underestimate the uncertainty (line 59, line 143) is not backed up by empirical results or references to other papers. 2. "Adversarial models are characterized by a large value of the integrand of the integral that defines he epistemic uncertainty" (line 65). Without a definition of adversarial models in the introduction, this claim is not possible at all to understand. ## Missing ablation on number of adversarial models The main paper does not specify the number of adversarial models learned through algorithm 1. According to the supplement, it seems that 10 are used. The paper would benefit from an ablation on the number of adversarial models ## Missing explanation of why the search algorithm does not always converge to the same solution The search algorithm is not constrained to avoid finding always the same solution. Since it is an optimization problem, it is likely that similar solutions to the same problem are found if starting from the same initialization. The paper would benefit from an explanation of why the search does not converge always to the same adversarial model. Perhaps, enforcing that solutions must be different could improve the performance. ## Missing ablation on trade-off inference speed vs. uncertainty estimation performance at different number of samples As pointed out by the authors, the proposed method sacrifices inference speed for uncertainty estimation performance. Although this undermines the practicality of the proposed approach, the theoretical insights gained from QUAM could outweigh this limitation. However, an ablation on the trade-off between inference speed and uncertainty estimation performance at different number of samples for MC dropout, Deep Ensembles and QUAM is needed. I would expect MC dropout to be significantly more efficient for low number of samples, still obtaining reasonable performance at low inference time. Moreover, QUAM should have a higher uncertainty estimation performance upper bound than the competitors, if computational resources allow to tolerate the high inference time. This comparison would allow practitioners to understand which method to use based on their computation budget and needs. ## References [a] Wang, Kunfeng, et al. "Generative adversarial networks: introduction and outlook." IEEE/CAA Journal of Automatica Sinica 4.4 (2017): 588-598. [b] Li, Guofu, et al. "Security matters: A survey on adversarial machine learning." arXiv preprint arXiv:1810.07339 (2018). [c] Chakraborty, Anirban, et al. "Adversarial attacks and defences: A survey." arXiv preprint arXiv:1810.00069 (2018).
Questions
Overall, I find the paper unclear and lacking some important details and ablations. I believe the contribution to be relevant, but the paper would benefit from rewriting for clarity and from additional experiments based on the comments in the Weaknesses section. I would change my opinion if the authors would convincingly address my comments in the Weaknesses section and, most importantly, optimize the paper for clarity. Providing the definition of adversarial models only at page 6 makes very hard understanding the first 5 pages.
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
1 poor
Contribution
3 good
Limitations
The authors pointed out the limitations of their work.
Summary
This paper is about uncertainty estimation using adversarial models (not examples!). The authors propose a new uncertainty estimation method, called QUAM, which performs a search of an adversarial model, which is one that fits the training set but has predictions far away from the predefined model, with the idea that a point is uncertain if multiple models explain it very differently, while still fitting the original training set. QUAM has the potential to estimate epistemic uncertainty better than other uncertainty models, and experimental results point in this direction. Contributions are: - The QUAM framework for uncertainty quantification using adversarial model search. - The new concept of an adversarial model, different from an adversarial example, which is a model (set of weights) that explains the training set well while having maximum difference with the orignal model in a new test point. - The proposed method can estimate epistemic uncertainty for any model, including a pretrained model, which is an advantage over usual uncertainty methods that require modifications to the training process or model retraining.
Strengths
- The paper is very well written and easy to understand. - The paper touches an important topic, uncertainty estimation methods fail in samples far from the training set, usually producing overconfident uncertainties that are not useful to detect out of distribution samples. This is well known from Ovadia et al and other papers in the literature. - This paper defines a new kind of meta-model, the adversarial model for uncertainty quantification, where to make a prediction in a new test point, new model parameters are found that fit the training set while providing alternative explanations for the new test point, producing higher quality epistemic uncertainty. This is of course very computationally expensive and it is mentioned as a limitation in the conclusions. - Experimental results show that QUAM outperforms the baselines on the task of out of distribution detection on MNIST vs FMNIST/KMINST/EMNIST/OMNIGLIT and ImageNet-1K vs Imagenet-O/ImageNet-A. - I believe the proposed method QUAM is a good contribution to the field of uncertainty quantification, as it proposed a new framework for uncertainty estimation that is model agnostic and provides high quality epistemic uncertainty estimates.
Weaknesses
- I have some serious doubts about the concept/interpretation of aleatoric uncertainty in this paper. In the literature and widely agreed concepts, aleatoric uncertainty is about the data, it is a property of the data, like stochasticity in measuring processes or noise, but the paper makes claims that "aleatoric uncertainty is the stochasticity of the model and epistemic uncertainty is the uncertainty about model parameters." (Lines 39/40), here the claim is made that aleatoric uncertainty is about stochasticity of the model, which I do not believe it is correct. Just as an example, there are non-stochastic models (like ensembles) that can estimate aleatoric uncertainty without using stochastic components. And I mention previously, aleatoric uncertainty is mainly a property of the data, not of the model, even though a model can estimate the aleatoric uncertainty of the data. I believe here the paper should clarify or simply fix the definition. ~~- Another issue I have is about selection of baselines, in the two moons setting, DUQ [1] is a strong baseline that actually approximates epistemic uncertainty much closer than QUAM in the two moons dataset (see Figure 1 of the DUQ paper), and I believe this also points that an issue in epistemic uncertainty estimation is the model's inductive bias. DUQ uses a RBF layer to predict classes, based on distance to class centroids, unlike a separating hyperplane in a highly dimensional space, so epistemic uncertainty is very different. I believe in this paper DUQ and other single network uncertainty estimation method (like Direct Uncertainty Estimation, DDU), should also be compared.~~
Questions
Questions: - How were the baselines selected? The baseline UQ methods are not state of the art for epistemic uncertainty estimation. I already mentioned in weaknesses that DUQ has better epistemic uncertainty estimation in the two moons dataset. - In Figure 3f, QUAM has artifacts or non-smoothness in the uncertainty, in the top-left and bottom-right corners, while other models do not have such artifacts, is there an explanation or intuition about this? Post-rebutal, both of these questions have been answered to my satisfaction by the authors. Suggestions: - The DE (LL) setting is the same as the concept of sub-ensembles [2], maybe it could be mentioned or cited. - In Tables 1 and 2, the aleatoric uncertainty of the model is claimed to be used, but I am not sure what this means, as most models actually output predictive uncertainty, which is the combination of aleatoric and epistemic uncertainty. Even a model without any uncertainty quantification (i.e without using MC-Dropout, Ensembling, BNNs, etc) still has a non-zero degree of aleatoric and epistemic uncertainty, so I believe this claim should be changed to predictive uncertainty and not aleatoric uncertainty. [2]: Deep Sub-Ensembles for Fast Uncertainty Estimation in Image Classification by Matias Valdenegro-Toro. arXiv 1910.08168.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
The paper properly discusses the limitations of the proposed method in the conclusion sections, including the fact that for each new test point, a model search has to be performed which is computationally very expensive.
The authors provided a convincing rebuttal, and I appreciate their efforts towards improved clarity of their submission. - **Clarity:** 1. _Definition of adversarial models:_ I thank the authors for acknowledging the importance of an earlier definition of adversarial models, and for adding it to the introduction. 2. _Missing details:_ This is an important step for clarity and reproducibility. - **Unsupported Claims:** The reply provided by the authors and supporting references seem convincing. - **Number of adversarial models and convergence:** I thank the authors for pointing out Sec. C. 1 in the appendix. Sec. C.1 reports fundamental details that should at least partially be mentioned in the main paper. The observation that "directly maximizing the KL divergence always leads to similar solutions to the optimization problem" was an obvious limitation by just reading the main paper, and it was not addressed there. To ensure that this paper complies with the clarity standards of this conference, I hope that the authors will carefully discuss this in the main paper. - **Ablation on inference speed vs. performance:** The reply provided by the authors, the new proposal for tuning the computation cost of QUAM and the results provided in Figure 1 are convincing. I thank the authors for providing them and I believe that they will enrich the paper. Overall, the authors addressed most of my concerns. However, my initially negative opinion was also greatly influenced by the initial poor quality of presentation and clarity. The authors will have to work on a significant rewriting to make the paper suitable for the standards of this conference. I am leaning towards increasing my rating, the extent of which will also depend on the discussion with other reviewers.
Thank you again for your helpful and constructive suggestions. We rewrote the manuscript accordingly, which resulted in a significant improvement of clarity and quality of presentation. We fully agree with the reviewer that the clarity and good presentation of research results are very important. To summarize, we did the following main improvements of the manuscript: - **Number of adversarial models and convergence:** We added a new subsection on the practical implementation at the end of section 3 in the main paper. This subsection is based upon section C.1 from the appendix. Most importantly, we now discuss the problem that direct KL optimization always leads to the same solution as explained in Sec. C.1. We further explain that we want to find different regions with large contributions to the epistemic uncertainty integral in equation (1) and (2). This is achieved by minimizing the cross-entropy towards one out of all possible classes at a time. Indeed, the more of those regions QUAM identifies, the more effective is mixture importance sampling. We agree with the reviewer that it is essential to elaborate on this problem and how we tackled it in the main paper, as it is essential to our method. - **Definition of adversarial models:** We added Definition 1 to the introduction (paragraph starting at line 58). Consequently, we revised the informal definition of adversarial models previously given in this paragraph accordingly. Furthermore, we contrast the definition of adversarial models from other concepts with ‘adversarial’ in their naming, such as “adversarial examples”, “adversarial training”, “generative adversarial networks”, or “adversarial model-based RL”. - **Unsupported claims:** We added the additional references [1], [2] and [3] and extended discussion of why prior methods underestimate epistemic uncertainty to the respective claims (line 59, line 143) and more explicitly refer to our empirical evidence. - **Missing details:** We moved information about the most crucial hyperparameters (e.g. ensemble size, # of forward passes for MC dropout, # adversarial model searches, …) from the appendix to the experimental section in the main paper. - **Ablation on inference speed vs. performance:** We added the new ablation on inference speed vs. performance to the main paper. - **Additional experiments requested by other reviewers:** We added the new results for MoLA to Table 1 in the main paper. Furthermore, we added the results for calibration to the appendix. Would the reviewer see the need for any additional significant changes? We would gladly consider and address those. Finally, we thank the reviewer for acknowledging our convincing results, as well as our efforts towards improving the clarity and quality of presentation of our paper and hope that they adjust their assessment of our work accordingly. --- [1] Wilson, A. G., & Izmailov, P. (2020). Bayesian deep learning and a probabilistic perspective of generalization. NeurIPS. [2] D'Angelo, F., & Fortuin, V. (2021). Repulsive deep ensembles are bayesian. NeurIPS. [3] Parker-Holder, J., & ... & Foerster, J. (2020). Ridge rider: Finding diverse solutions by following eigenvectors of the hessian. NeurIPS.
Thank you or addressing my points. Further comments below: **Calibration** The plot 2 e) in the additional PDF looks odd. Why does QUAM almost exclusively predict 100% confidence? How would the method compare on a dataset where calibration is less saturated? **Empirical evidence that prior methods underestimate epistemic uncertainty** I dont think that fig. 2/3 are sufficient empirical evidence that ensembles under estimate uncertainty. In particular, other works have found that Ensembles are better calibrated than MCD. It seems there is rather sth. of with the trained ensemble in this work. The fact that all ensembles members reach the same solution indicates that there is not suficient randomness in the process? Do the authors only vary the model initilization?
We thank the reviewer for his additional comments and questions. **Calibration** We agree with the reviewer, that the results depicted in Figure 2 e) in the PDF attached to the global answer looks different to other considered methods. Our explanation of them is as follows. The calibration results for QUAM were obtained using the same hyperparameters as for misclassification detection. Naturally, this is a discriminative task, where low uncertainty (thus high confidence) should be assigned to the correctly classified samples and high uncertainty (thus low confidence) to the misclassified samples. QUAM excels at this, as stated in Table C.2 in the appendix (IN-Misclass.). Since the majority of samples are correctly classified, we expect that QUAM yields high confidence for those samples. Future work could shed more light on the nature of these observations. We note that, while related, quantifying the uncertainty defined by Equations (1) and (2) in our paper is not equivalent to attaining a well calibrated prediction. *We would like to emphasize that QUAM was not explicitly designed to improve calibration, but does so empirically.* We acknowledge that future work would need to investigate calibration of those methods for different datasets. However, we argue that ImageNet is a favorable dataset for calibration, as the accuracy of the classifier is at ~84% top-1 accuracy, thus also lower accuracy bins (to calculate the ECE) contain enough samples to get robust statistical estimates. **Empirical evidence that prior methods underestimate epistemic uncertainty** To the best of our knowledge, it is not clear cut whether MCD or Deep Ensembles are better calibrated (see e.g. Figure 2 test performance in [1]). The members of the Deep Ensemble in Figures 2 and 3 of our paper are indeed the same architecture trained from different random initializations, exactly as proposed in the original paper [2] and generally used as a baseline in e.g. [1], [3], [4] or [5]. Given that [4] (Figure 1) and [5] (Figure 1) report qualitatively the same results for Deep Ensembles as in Figure 3 of our paper, it suggests that the experiment was implemented correctly. Also, equivalent results for the same experiment with Deep Ensembles have been reported in [3] (Figure 1), which was suggested by the reviewer. Further empirical evidence that MCD and ensembles of MCD models (a combination of Deep Ensembles and MCD thereof) underestimate epistemic uncertainty for regions far from training samples (in the latent space of a variational autoencoder trained on MNIST) is given in Figures 1 and 6 in [6]. Finally, note that we varied the model architecture by using different sizes of externally obtained, pre-trained, and verified EfficientNets [7] to perform the ImageNet experiments, as reported in Table 2 of our paper. QUAM empirically outperforms this ensemble in estimating the uncertainty as well. We hope that our response clarifies the remaining questions and gives rise to a more positive assessment of our work. --- [1] Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., ... & Snoek, J. (2019). Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32. [2] Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30. [3] Eschenhagen, R., Daxberger, E., Hennig, P., & Kristiadi, A. (2021). Mixtures of Laplace approximations for improved post-hoc uncertainty in deep learning. arXiv preprint arXiv:2111.03577. [4] Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T., & Lakshminarayanan, B. (2020). Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33, 7498-7512. [5] Van Amersfoort, J., Smith, L., Teh, Y. W., & Gal, Y. (2020, November). Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning (pp. 9690-9700). PMLR. [6] Smith, L., & Gal, Y. (2018). Understanding measures of uncertainty for adversarial example detection. The Conference on Uncertainty in Artificial Intelligence (UAI). [7] Tan, M., & Le, Q. (2019, May). Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning (pp. 6105-6114). PMLR.
good rebuttal
Thank you for the detailed rebuttal. I agree with point 2. About point 3, wouldn't it make more sense to then predict all baselines the same way, even if they have similar artifacts? At least to make a sensible comparison. About point 1, aleatoric uncertainty, I think here I have the biggest disagreement, I think your description is a bit conflicting because the conditional distribution has mixed aleatoric and epistemic uncertainty in it (called predictive uncertainty), and obtaining only aleatoric uncertainty from it is difficult without additional methods, that is why model stochasticity does not model aleatoric uncertainty exclusively. Since this point is not really relevant from your paper, I suggest to actually remove it since it could mislead future readers. I will update my review accordingly.
We thank the reviewer for updating the review and the rating of our work. Regarding point 3, we agree with the reviewer that predicting all baselines the same way on a per-test-sample basis would make a better comparison to QUAM. However, previous work such as [1], [2] or [3] conducted similar experiments on the two-moon dataset, applying the baselines on all test samples at once. We decided to stay close to the setup of prior work, but we can also change the baselines to operate on a per-test-sample basis. Moreover, for a sufficiently high number of sampled models, the artifacts will smoothen out, so we could also just do more adversarial model searches for QUAM. Either way, we will elaborate on the origin of artifacts being different models sampled for individual test points. We fully agree with the reviewer, the debated point about aleatoric uncertainty is not really relevant for our paper and change the respective section accordingly. --- [1] Eschenhagen, R., Daxberger, E., Hennig, P., & Kristiadi, A. (2021). Mixtures of Laplace approximations for improved post-hoc uncertainty in deep learning. arXiv preprint arXiv:2111.03577. [2] Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T., & Lakshminarayanan, B. (2020). Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33, 7498-7512. [3] Van Amersfoort, J., Smith, L., Teh, Y. W., & Gal, Y. (2020, November). Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning (pp. 9690-9700). PMLR.
I thank the authors for having fixed my minor observations. Since all the reviewers acknowledged the contribution of adversarial models to the field, and the authors addressed most of the reviewers' concerns on clarity, missing related works and missing ablation studies, I confirm my positive assessment of the work.
Discussion with authors ends tomorrow
Dear reviewers, This is just a reminder that discussions with the authors will end tomorrow (Aug. 21). Many of you have engaged active discussions with the authors. If you have not, please read authors’ rebuttal and other reviews, and engage with the authors for any concerns you may still have. Please remember to update your ratings after the discussions. Best, Your AC
Decision
Accept (poster)