We study a generalization of classical active learning to real-world settings with concrete prediction targets where sampling is restricted to an accessible region of the domain, while prediction targets may lie outside this region. We analyze a family of decision rules that sample adaptively to minimize uncertainty about prediction targets. We are the first to show, under general regularity assumptions, that such decision rules converge uniformly to the smallest possible uncertainty obtainable from the accessible data. We demonstrate their strong sample efficiency in two key applications: active fine-tuning of large neural networks and safe Bayesian optimization, where they achieve state-of-the-art performance.
Paper
Similar papers
Peer review
Summary
This paper considers active learning with Gaussian Process in a transductive setting where the learner wants to optimize the model performance on a target space A while it can only sample from a sample space S. It proposes a few algorithms that sequentially choose the example that minimizes a few variations of "posterior uncertainty" of the model on A. Assuming the measure for such uncertainty is submodular, it provides bounds on the convergence rate of the proposed algorithm. Empirically, it evaluates the proposed methods on two scenarios: fine-tuning and safe Bayesian optimization.
Strengths
- This paper considers a practically relevant setup of active learning. - The paper is mostly written clearly. - The proposed methods are sound. It gives both theoretical guarantees and empirical evaluation.
Weaknesses
- My major concern is novelty and significance: - The main idea of the proposed query strategy is to minimize posterior uncertainty. This is a well-known method in active learning, and it is an especially straightforward choice in a Bayesian setting. - For the theoretical results, I'm not very familiar with Gaussian Process literature, but I'm not convinced that the theoretical guarantees in Section 3.1 and 3.2 are nontrivial. It would help readers appreciate the results if the authors could clarify the challenges of obtaining such results, tightness of the bounds, how strong it is compared to other baseline sample strategies, or any new insights from such bounds. -------- I've read author feedback. Now I agree that the theoretical part offers interesting techniques and results (which roughly matches known results in similar problems), but I personally think the results are still a bit unsatisfactory (specifically, the dependency on |S|, the RHS of the bound in Thm 3.3 can grow with n increases). In addition, since this paper claims the theory part is one of the major contributions, it would be better if there are more explanations, discussions, and comparisons for the results. Apart from that, I still tend to think the proposed AL approach is a well-known idea in active learning and applying it to transductive learning is a bit natural, so the contribution of this part is not significant enough for a top conference. And I agree with reviewer aHJz that "few-shot fine-tuning of large neural networks" is a bit oversold.
Questions
In addition to issues in the Weaknesses section, a few questions: - For Section 4, - is there any existing Bayesian active learning algorithms that can be used as baselines in this setup? - it is mentioned in line 213-214 that "we will focus here on .. $A \cap S = \emptyset$", but in Figure 3 and line 243 it is comparing samples from $P_A$. How can the algorithm sample from $P_A$ if the sample space S does not intersect with A? What is the exact setup of the experiments? - How do you choose hyperparameters for the baselines? - For section 5, I'm not familiar with this task, but in my opinion, it does not provide enough details and explanations. For example - What exactly is your sample strategy? In particular, how do you ensure the chosen samples are safe? In line 276 it says "a natural choice for the target space of safe BO is A_n", but it looks like A_n can be unsafe. - In line 317-318, "VTL minimizes marginal variances, .. which are decisive for expanding the safe set". Why?
Rating
5
Confidence
2
Soundness
3
Presentation
2
Contribution
2
Limitations
N/A
Discussion Period
Dear Reviewer, We hope this message finds you well. We have noticed that our detailed rebuttal, addressing each of the concerns raised in your review, has not yet received any feedback. We understand the demanding nature of the review process and appreciate the time and effort invested in evaluating our work. We kindly urge you to consider our responses to your questions, as we believe they adeptly address your concerns. With some days left, we hope we can still have a fruitful rebuttal period. Thank you, Authors
Since one of the main claimed contributions is the theory part, I would like to take a deeper look. I'm not very familiar with theory in gaussian process or bayesian learning, so it would take some time. Questions and comments so far: - Given there are a lot of notations involved, it would be clearer if you could provide a list of notations for reference. - It is still unclear to me how strong/tight your results are. It would be easier if you could refer to some similar upper or lower bounds. - For Thm 3.2, is C a universal constant, or it depends on other things? Specifically, how does it scale with the size of A, S, or X, and what happens if they're infinite (I saw your comment to Reviewer WV1c13, but could you give some quantitative characterization)? - For thm 3.3 - how realistic are the assumptions (bounded norm for f*, sublinear \gamma_n)? - It looks \beta_n is increasing as n increases, won't this make the bound very weak? - Why isn't there a term that characterizes how close f* can be approximated by the model class (Gaussian process)? - For Lemma C.16 and the comments thereafter, how can you conclude b_\epsilon is universal constant? doesn't it depends on all other terms in inequality (15)?
We thank the reviewer for taking a close look at our novel convergence guarantees. > It is still unclear to me how strong/tight your results are. It would be easier if you could refer to some similar upper or lower bounds. One can show that $\sigma_n^2(\boldsymbol{x}) \geq \frac{1}{n+1} \sigma_0^2(\boldsymbol{x}) \in \Omega(1/n)$ which is achieved if the point $\boldsymbol{x}$ is sampled repeatedly (and no other points have to be generalized to). Please see eq. (7.26) of [1]. The derivation is along the lines of [2, Section 2]. In particular, our eq. (2) is tight up to $\gamma_n$. Our results in the agnostic kernelized setting (Theorem 3.3) match regret bounds of kernelized bandit algorithms in the same setting. For example, [3, Theorem 4] prove for the celebrated and widely used GP-UCB algorithm that $R_n / n \leq \widetilde{O}(\frac{\gamma_n + \|\|{f^*}\|\|_k \sqrt{\gamma_n}}{\sqrt{n}})$ where $R_n$ is the cumulative regret up to round $n$ and $\widetilde{O}(\cdot)$ subsumes log-factors. A similar result can be found in [4, Theorem 3] (ICML 2020 Test of Time Award winner). Our results match the rates of results in this widely cited and often applied line of work on kernelized bandits. > For Thm 3.2, is $C$ a universal constant, or it depends on other things? Specifically, how does it scale with the size of $\mathcal{A}$, $\mathcal{S}$, or $\mathcal{X}$, and what happens if they're infinite (I saw your comment to Reviewer WV1c13, but could you give some quantitative characterization)? You can find the constant $C$ defined precisely in the proof of Theorem 3.2 (cf. lines 1227 and 1229). $C$ is independent of $n$. For a finite discretization of a continuous domain, our results hold as stated in the main body of the paper. We generalize our analysis to continuous domains when the RKHS is finite-dimensional in Section C.6.4. With this generalization, intuitively, the dimensionality of the RKHS assumes the role of the size of $\mathcal{S}$ from the finite setting. > For thm 3.3, how realistic are the assumptions (bounded norm for f*, sublinear \gamma_n)? $f^*$ has bounded norm iff it is element in the corresponding RKHS. We remark that for standard kernels (such as RBF/squared exponential or Matérn kernels), the space of functions is extremely large; containing, e.g., the smooth or the continuous functions defined over $\mathcal{X}$. Our dependence on $\|\|f^*\|\|$ and $\gamma_n$ are standard compared to the large literature on kernelized bandits [e.g., 3] as discussed above. We provide the magnitudes of $\gamma_n$ for common kernels in Table 3 (last page of Appendix). $\gamma_n$ is logarithmic in $n$ for many common kernels. We further remark that $\gamma_n$ is a very standard measure of "information capacity" / "function class complexity" in the literature on kernelized bandits. [3] is merely one of the many works that fundamentally rely on this notion. > It looks \beta_n is increasing as n increases, won't this make the bound very weak? Yes, $\beta_n(\delta)$ is increasing with $n$, however, only very slowly at the rate $\sqrt{\gamma_n}$. This dependence on $\beta_n$ is standard in the literature on kernelized bandits [e.g., 3, Theorem 2]. > Why isn't there a term that characterizes how close f* can be approximated by the model class (Gaussian process)? The term $\|\| f^* \|\|$ measures "how easy" it is to find $f^*$ within the model class (the RKHS) with $\|\| f^* \|\| = \infty$ if $f^*$ is not at all in the model class. The term $\gamma_n$ measures the "size" / "capacity" of the model class. > For Lemma C.16 and the comments thereafter, how can you conclude b_\epsilon is universal constant? doesn't it depends on all other terms in inequality (15)? We mean "universal" with respect to $n$ and $\boldsymbol{x}$. We are very sorry for the confusion and have updated this line to point this out explicitly. Thank you for highlighting this! --- In summary, we would like to mention: 1. Our results match rates of prior work from the widely recognized and large literature on kernelized bandits. We use the same measures of size and complexity of model class as most works from kernelized bandits. 2. Our dependence on $\gamma_{\mathcal{A}, \mathcal{S}}(n)$ can be dramatically better than $\gamma_n$, since this only measures the information capacity of prediction targets $\mathcal{A}$ as opposed to of the entire domain $\mathcal{X}$. We hope to have addressed the reviewer's questions adeptly. We are happy to answer any other questions that may help with the interpretation of our results! --- 1. Williams and Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006. 2. Opper and Vivarelli. General bounds on bayes errors for regression with gaussian processes. NeurIPS, 11, 1998. 3. Chowdhury and Gopalan. On kernelized multi-armed bandits. ICML, 2017. 4. Srinivas et al. Gaussian process optimization in the bandit setting: No regret and experimental design. ICML, 2010.
Brief List of Notation
As requested, the following includes a brief list of notations that are defined in the appendix. Kindly let us know if you have a question about any other symbol. | Symbol | Reference | |-|-| | $\sigma^2$ | maximum initial variance (line 1015) | | $\tilde{\sigma}^2$ | maximum [initial variance + noise variance] (line 1016) | | $\Delta_{\mathcal{A}}(\boldsymbol{x} \mid X)$ | marginal gain (Lemma C.2) | | $\Gamma_n$ | maximal marginal gain in iteration $n+1$ (line 1084) | | $\alpha_{\mathcal{A}, \mathcal{S}}(n)$ and $\alpha_n$ | task complexity (definition C.6) | | $\bar{\kappa}(X \mid D)$ | information ratio (eq. (8)) | | $\kappa_{\mathcal{A}}(n)$ | submodularity ratio (definition C.10) | | $b_{\epsilon}$ | bound on size of approx. Markov boundary (eq. (16)) | The definitions of $\alpha, \bar{\kappa}, \kappa$ are all to generalize to instances where Assumption 3.1 is satisfied approximately. Under Assumption 3.1, all can be bounded trivially: $\alpha \leq 1$ and $\bar{\kappa}, \kappa \geq 1$.
Thanks for the explanation and the added list of notation! I think it would make the paper clearer if some of them are added to the future version (e.g. list of notation, how the results or parameters compared to existing literature, and what they would be in some examples like Table 3). I still have a few questions: - Tightness of Thm 3.2: - Is this notion of "constant" C also standard in literature? I'm a bit concerned about it since it involves many factors ($\sigma^2$, $|S|$, $1/\lambda_\min(K)$) which do not look negligible to me. In particular, it looks not very strong to me since it scales linearly in $|S|$, and it involves maximum initial variance for every example in A (if $\sigma_0^2(x)$ is mostly small but extremely big for some points, the bound of $\sigma_n^2(x)$ would be quite big, even for x such that $\sigma_0^2(x)$ is small). And perhaps it can be clearer if you can explain exactly what it is (up to universal constants) in some specific examples. - I would expect there to be a smoother transition between (1) and (2) as x moves into $A\cap S$. Is there any explanation why it goes from $1/\sqrt{n}$ to $1/n$ directly? - Tightness of Thm 3.3: For results that you cited (e.g. [3, Theorem 4]), the regret diminishes as $n \to \infty$, but for your bound, the irreducible part gets even larger. Can you comment on this? - Proof: For the inequality following line 1225, how is the $\bar{\kappa}_n$ term bounded?
We thank the reviewer for their continued look into our new theoretical results. We will add a version of this table of notation to the camera-ready version as well as a brief discussion on related results from other problem settings. Thank you for this suggestion and improving our work in this regard. > Is this notion of "constant" $C$ also standard in literature? I'm a bit concerned about it since it involves many factors which do not look negligible to me. In particular, it looks not very strong to me since it scales linearly in $|\mathcal{S}|$, and it involves maximum initial variance for every example in A. And perhaps it can be clearer if you can explain exactly what it is (up to universal constants) in some specific examples. The dependence on the initial variance is standard in kernelized analysis of bandits. Usually it is assumed that the kernel is such that $k(\boldsymbol{x},\boldsymbol{x}) \leq 1$ which alleviates this. In our analysis, the dependence on $|\mathcal{S}|$ is required to achieve convergence to the irreducible variance (kindly see the discussion below). We would be happy to make this dependence evident in the main theorem statement if the reviewer thinks this is important. A linear dependence on the size of $|\mathcal{S}|$ is standard in works on safe exploration (i.e., where "extrapolation" is required due to restrictions on the domain) [see, e.g., 1]. We further thank the reviewer for the suggestion, and will update the appendix to clarify whenever we mention "universal constant" what this term is a constant with respect to. > I would expect there to be a smoother transition between (1) and (2) as x moves into . Is there any explanation why it goes from to directly? The difference in rates is indeed interesting. In our analysis this stems from using a "nested" learning problem that learns the function over all of $\mathcal{S}$. This appears to be required in our view to achieve the *very strong* convergence to the *irreducible* variance $\eta_{\mathcal{S}}^2(\boldsymbol{x})$, since there is a slight mismatch between the original learning problem wrt. $\mathcal{A}$ and a different learning problem wrt. $\mathcal{S}$. It might be possible to obtain tighter results in the special case $\mathcal{S} \subseteq \mathcal{A}$ (i.e., when the two learning problems are not misaligned). We are inclined to leave this study as an interesting direction for future work. > Tightness of Thm 3.3: For results that you cited (e.g. [3, Theorem 4]), the regret diminishes as $n\to\infty$, but for your bound, the irreducible part gets even larger. Can you comment on this? We would like to point out that in the referenced works on bandits no "extrapolation" is required since *all* points can be observed directly. These works therefore do not have a notion of "irreducible error". However, we agree while the slight increase of this error in the agnostic setting is likely an artifact of us performing the analysis in a Bayesian setting, in this Bayesian setting this error is tight. > Proof: For the inequality following line 1225, how is the $\bar{\kappa}_n$ term bounded? It follows from Assumption 3.1 (submodularity) that $\bar{\kappa}_n \geq 1$. We have updated the proof to make this more explicit. --- We would further like to highlight that due to the large set of instances of *transductive* active learning, our results constitute new results in multiple settings: 1. The *extrapolation* to points outside $\mathcal{S}$ which unlocks the application to Safe BO, leading to a new sota algorithm. 2. *Directed* learning where $\mathcal{A} \subseteq \mathcal{S}$. Here the analysis simplifies and we prove the rate $\gamma_{\mathcal{A},\mathcal{S}}(n)/n$. Finally, we would like to mention that beyond our theoretical contributions, the application of transductive active learning to two new applications, Safe BO and the discussed fine-tuning setting, are also key contributions of our work. In both of these settings our findings show that our proposed approaches to transductive active learning substantially improve upon the current state of the art. We hope that upon reevaluation these substantial contributions are acknowledged by the reviewer by increasing their score. --- 1. Sui, Yanan, et al. "Safe exploration for optimization with Gaussian processes." International conference on machine learning. PMLR, 2015.
Thanks again for the detailed explanation. I will adjust my review.
Summary
The paper investigates the generalization of active learning to scenarios with specific prediction targets and limited information due to constrained sample spaces, offering a flexible framework adaptable to various domains such as recommender systems, molecular design, and robotics. It introduces novel generalization bounds of independent interest for active learning. The practical performance of the model relies on its ability to capture correlations between points (learning the latent structure) and accurately estimate uncertainty. The study demonstrates that sampling relevant and diverse points significantly improves performance across various applications, surpassing the state-of-the-art methods.
Strengths
The paper is well written and clearly structured with novel contributions in generalization bounds. It provides sufficient theoretical analysis for specific acquisition functions in active learning with experimental results on some datasets.
Weaknesses
The authors use MNIST and CIFAR100 as experiments on active few-shot tuning. First, image classification is a well-studied topic and lack of novelty. Image classification uses convolutional networks for classifier while current LLMs uses auto-regressive models therefore most of theoretical assumptions and results might not be able to extend. Even though the paper is well-written, the claims that the results shall be able to extend to prompt optimization is groundless. It would be more interesting to study the prompt tuning/efficient data selection for fine-tuning LLMs. Second, the baselines for active learning is not state-of-the-art. For instance, there are more state-of-the-art baselines than badge, typicluster and probcover, for AL domains in image classification. Moreover, there has been work in using influence function to select data to fine tune LLMs or image classifications that worth comparing.
Questions
N/A
Rating
4
Confidence
4
Soundness
3
Presentation
4
Contribution
3
Limitations
N/A
Ask for Clarification
We thank the reviewer for their review. To address the points raised in our rebuttal effectively, we kindly ask for some clarification and additional references. > For instance, there are more state-of-the-art baselines than badge, typicluster and probcover, for AL domains in image classification. We would greatly appreciate it if the reviewer could point us to those state-of-the-art baselines. We would be more than happy to compare against additional approaches. To the best of our knowledge, BADGE, TypiClust, and ProbCover are some of the most widely recognized and best-performing AL methods for neural network training to date. > Moreover, there has been work in using influence function to select data to fine tune LLMs or image classifications that worth comparing. As we are not fully familiar with this specific line of work, we would be grateful if the reviewer could provide references that they believe are relevant for comparison. Thank you in advance for your assistance in providing these clarifications. We believe that this will facilitate a more productive rebuttal process.
1. There are some active learning papers published in vision venues. [1] Li, Jingyao, et al. "Bal: Balancing diversity and novelty for active learning." IEEE Transactions on Pattern Analysis and Machine Intelligence (2023). [2] Parvaneh, Amin, et al. "Active learning by feature mixing." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. 2. Below are references for influence function to finetune language models or image classification tasks. [1] Kwon, Yongchan, et al. "Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models." arXiv preprint arXiv:2310.00902 (2023). [2] Liu, Zhuoming, et al. "Influence selection for active learning." Proceedings of the IEEE/CVF international conference on computer vision. 2021. [3] Zhao, Zhuokai, Yibo Jiang, and Yuxin Chen. "Direct Acquisition Optimization for Low-Budget Active Learning." arXiv preprint arXiv:2402.06045 (2024). [4] Xia, Mengzhou, et al. "Less: Selecting influential data for targeted instruction tuning." arXiv preprint arXiv:2402.04333 (2024).
Discussion Period
Dear Reviewer, We hope this message finds you well. We have noticed that our detailed rebuttal, addressing each of the concerns raised in your review, has not yet received any feedback. We understand the demanding nature of the review process and appreciate the time and effort invested in evaluating our work. We kindly urge you to consider our responses to your questions, as we believe they adeptly address your concerns. With some days left, we hope we can still have a fruitful rebuttal period. Thank you, Authors
Response to reviewer’s comment
Dear reviewer, Thanks for your active engagement in the rebuttal period. We have a follow up question on your comment: “However, I think fine-tuning model weights for image classification task would not serve as a novel contribution.” First, while we might be of a different opinion, we respect your opinion regarding the contribution of our fine tuning experiments. However, we would like to mention that there are some very important other contributions of our paper that should also be considered in the evaluation (below is a non exhaustive list): 1. We formulate the problem of TAL which results in natural decision rules/acquisition functions of ITL, VTL. 2. We are the first to give uniform convergence rates for the posterior uncertainty in regions beyond the sample space. This implies a new generalisation bound for RKHS functions. Note that our assumptions are very general and common in active learning literature. 3. We evaluate our method on two distinct domains, (i) fine-tuning of large neural networks and (ii) safe BO. In both cases, our method outperforms the baselines. We hope this clarifies our contributions further. Moreover, even though the reviewer thinks our contribution for the fine tuning case are not novel, our work has other significant contributions. Given these contributions, we kindly ask the reviewer to reconsider their evaluation of our work. Looking forward to the reviewer’s response.
Dear reviewer, we would like to thank you for your efforts in reviewing our submission and pointing us to the related work on influence functions. We will include the above comparison to this line of work in our updated version. Regarding our citation of [1], our intention was to mention that in future work, the methods of this submission could potentially be extended to the setting described in [1]. We did *not* intend to suggest that this submission is an extension to [1]. We agree with the reviewer that our methods cannot be applied trivially (due to the different learning paradigms), however, selecting relevant and diverse data seems also to be important for the fine-tuning of LLMs (as argued in the limitations of [2]). We will clarify this in the updated version, and thank the reviewer for highlighting this. With only one day left, we would appreciate it if the reviewer can express their thoughts about our latest comment and potentially raise their score. [1] Hardt, M. and Sun, Y. Test-time training on nearest neighbors for large language models. ICLR, 2024. [2] Xia, Mengzhou, et al. ”Less: Selecting influential data for targeted instruction tuning.” In ICML, 2024.
Summary
This paper introduces and considers approaches for the generalized (transductive) active learning problem, where the space of prediction targets and samples are not necessarily the same. The authors propose methods ITL and VTL to select samples in order to minimize the posterior uncertainty about the target function with the target space $A$. They prove favorable properties of these methods and compare their performance to other active learning techniques on empirical settings, including batch active learning and safe Bayesian optimization.
Strengths
- The paper is well-written and organized, with ample thorough discussions. - The problem of transductive active learning is novel to the best of my knowledge and may be of interest to the broader ML community. - Compelling empirical results are provided that support the theoretical results of the paper.
Weaknesses
- The paper is quite dense and long, with a total of 68 pages when appendices are considered.
Questions
1. How does the computational cost of the methods fare in practice to other methods?
Rating
7
Confidence
2
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes
Summary
This paper introduces transductive active learning based on the uncertainty of GP when the labeling space and target space can be different. They have assumptions such as submodularity of information gain and information capacity’s sub-linearity in GP, resulting in a bound for variance of GP’s posterior with the proposed labeling algorithm. Under a specific setup, this can go to 0, and it cannot avoid the irreducible error when the deviation between labeling space and target space is larger. The various properties of the proposed algorithm(ITL and VTL) are examined by simple examples showing labeling in the target space as much as possible. The proposed algorithm's applications are interesting, including few-shot learning and safe BO (rarely considered in conventional active learning). These parts show the algorithm’s flexibility and applicability well. The results are promising.
Strengths
Among many advantages, the theoretical aspect is better because it can provide the criteria for the use of kernels that are strongly connected to the tuning of the algorithms. Also, the applications are impressive and enlarging the area of active learning. The use of GP is solid because it can provide a solid mathematical background and reliable uncertainty in many cases, compared to variational Bayes approaches.
Weaknesses
This paper gives too little attention to conventional active learning. Although the novelty of the new area is essential in this paper, traditional active learning tasks are still popular. In this view, this paper does not provide strong evidence for the superiority compared to conventional active learning. At first glance, the proposed active learning algorithm seems too specific. In theoretical aspects, it is unsatisfactory to consider only the spaces. More clarification of theoretical aspects in the paper can be helpful to strengthen the paper. $gamma_n$’s sub-linearity can be clarified as the mathematical formula. In GP, the kernel is essential, and the experiments can reveal what kernel is best or robust. The discussion about kernels is too short.
Questions
Q1: What’s the difference between $\gamma_n$ and $\gamma_{\mathcal{A},\mathcal{S}}(n).$ These notations can be improved. Q2: If only the target space is (uncountable) separable space, can the theorem have any change?
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Not well-discussed
Thank you for your response. I read the reviews and the responses. I lean towards keeping my score.
We would like to thank you for your efforts in reviewing our work and for recognizing its relevance to the broader ML community.
Hi I have read all the rebuttals. I do appreciate the authors efforts in literature review and agree that the references the authors list are more comprehensive. However, I think fine-tuning model weights for image classification task would not serve as a novel contribution and the learning paradigm for fine tuning deep neural networks is different from fine tuning LLMs as LLMs are generative models (the authors cite [1] and argue this paper could be an extension for the future work for fine-tuning the model for each prompt.) I think the contribution below is not novel and a bit vague. "We apply the transductive active learning framework to batch-wise active few-shot fine tuning of large neural networks and to safe Bayesian optimization. We empirically show, in both cases, that ITL and VTL outperform the state-of-the-art. [1] Hardt, M. and Sun, Y. Test-time training on nearest neighbors for large language models. ICLR, 2024. I would lean towards keeping my score.
Reply to the Rebuttal
Thanks for your reply. Almost all my concerns are resolved. I only want the results applied in the conventional AL learning setups, which can be expected shown in other chances. I'll keep my score.
Decision
Accept (poster)