Collaborative Learning via Prediction Consensus

Summary

To facilitate information exchange among agents and improve prediction accuracy on a shared target domain, this paper proposes a decentralized learning algorithm based on prediction consensus inspired from social science for leveraging each agent’s predictions. To be concrete, each agent firstly shares its prediction and weighs these predictions according to the trust scores, then these proxy labels are used to augment local model training via distillation. The authors show the trust measure based on cosine similarity can attain trust matrix with ideal properties, which could facilitate effective consensus. The efficacy is empirically demonstrated by showing that the proposed method is better than classical baselines under some heterogeneous conditions.

Strengths

This paper considers a collaborative learning setting where each agent owns their own data and agents want to collaborate to improve their predictive performance over a target domain. The authors propose a decentralized algorithm based on distillation and trust weighting scheme, which is useful in cases where data sharing and model sharing are not allowed. The key ingredient, which is called trust, is inspired from social science, which is an interesting topic. The theoretical analysis seems sound although I didn’t check all the details. The authors also conduct sufficient experiments to demonstrate the effectiveness of the proposed method. Overall, the paper is a well-written.

Weaknesses

In the collaborative setting, each agent owns their own data and wants to improve its predictive accuracy while keeping its data and model private, which is a significant problem. However, exchanging predictions may leak valuable information. Is there a more private way to share data between agents? The goal of agents is to improve the prediction performance on a shared target domain. Is it possible that ensemble algorithms like bagging work better here? I think the motivation and necessity of collective prediction could be improved. I didn’t clearly figure out why the proposed method could keep communication at a minimum as claimed in the paper. Could the authors give more detailed explanations? The explanation of co-training in line 144 is not that convincing. As far as I know, dispersed and heterogeneous data on multiple agents cannot be simply viewed as multi-view data. Could the author give more explanations?

Questions

please see weaknesses

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No

Summary

This paper proposes a collaborative learning method that leverages unlabeled auxiliary data to facilitate the exchange of expertise among agents. The method adaptively weights the influence of each collaborator on the pseudo-labels until a consensus on how to label the auxiliary data is reached. The authors demonstrate that their collaboration scheme significantly boosts individual model's performance with respect to the global distribution, compared to local training. They also show that their method is particularly effective in scenarios where the intrinsic beliefs of individuals counterbalance the averaging process and yield a diversity of opinions.

Strengths

1. The proposed approach utilizes unlabeled auxiliary data to enhance the exchange of expertise among agents, resulting in a significant improvement in individual model performance when compared to local training with respect to the global distribution. 2. The method dynamically assigns weights to each collaborator's influence on the pseudo-labels, iteratively reaching a consensus on how to label the auxiliary data. This adaptive weighting effectively detects and mitigates the negative impact of poor models on the collective performance. 3. The authors demonstrate the method's efficacy, particularly in scenarios where individuals hold diverse beliefs that counterbalance the averaging process. This diversity of opinions leads to improved performance in heterogeneous environments, showcasing the method's potential in such situations.

Weaknesses

1. The paper lacks a comprehensive comparison with other recent state-of-the-art collaborative learning methods, making it challenging to evaluate the relative performance of the proposed method. Including such comparisons would enhance the understanding of its strengths and weaknesses. 2. The paper would benefit from a more in-depth analysis of the computational complexity of the proposed method. This analysis would shed light on its scalability to large-scale datasets or complex models, which is crucial for practical implementation. 3. The paper assumes the trustworthiness and non-malicious nature of all agents, which may not be realistic in real-world scenarios. To address this limitation, a more robust trust weighting scheme capable of handling malicious agents should be considered, ensuring the method's applicability in diverse environments.

Questions

1. Can you provide more details on the computational complexity of the proposed method? How does it scale to large-scale datasets or complex models? 2. How does the proposed method compare to other state-of-the-art collaborative learning methods in terms of performance? Can you provide a comprehensive comparison? 3. How robust is the trust weighting scheme to malicious agents? Have you considered scenarios where some agents may intentionally provide incorrect information? 4. Have you tested the proposed method on real-world datasets? If so, can you provide some examples and discuss the results? 5. How sensitive is the proposed method to the choice of hyperparameters? Have you conducted a sensitivity analysis to assess the impact of different hyperparameters on performance?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The paper briefly mentions some potential limitations of the proposed method, such as the lack of a comprehensive comparison with other state-of-the-art methods and the assumption of trustworthy and non-malicious agents. However, the authors do not provide a detailed discussion of these limitations or potential negative impacts of their work. While the paper does provide some insights into the strengths and weaknesses of the proposed method, a more thorough analysis of the limitations and potential negative impacts would be desirable.

Summary

The paper proposes a collaborative learning approach that leverages unlabeled auxiliary data to improve individual models through consensus. The trust weighting scheme adapts to each collaborator's influence, leading to a consensus on how to label the auxiliary data. The authors demonstrate that this collaboration scheme significantly boosts individual model performance and effectively mitigates the negative impact of bad models on the collective. Overall, the paper makes a valuable contribution to the field of collaborative learning and presents a promising approach for improving model performance through consensus.

Strengths

- The paper proposes a novel approach for collaborative learning through consensus on unlabeled auxiliary data. - The collaboration scheme significantly boosts individual model performance and mitigates the negative impact of bad models on the collective. - The paper provides a thorough description of the algorithm and its implementation. - The experimental results demonstrate the effectiveness of the proposed approach.

Weaknesses

- The paper could benefit from a more detailed analysis of the impact of different trust weighting schemes on the consensus. - The paper could benefit from a more detailed discussion of the assumptions and limitations of the trust-based iterative pseudo-labeling process. - The paper could provide more insights into the computational complexity of the proposed algorithm and potential scalability issues.

Questions

- How does the proposed approach scale to larger datasets and more complex model architectures, and can the authors provide more insights into the computational complexity of the algorithm?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

In this paper, the authors consider a collaborative learning setting where agents want to improve their predictive performance on a shared target domain. The paper proposes a novel algorithm based on prediction-consensus, which effectively addresses statistical and model heterogeneity in the learning process. The algorithm works by having agents pseudo-label data from the target domain. The pseudo-labels are then used to compute a trust weighting scheme, which determines how much each agent's opinion should be weighted when reaching a consensus on how to label the unlabeled data. Theoretical results that show consensus can be reached via the algorithm and justify the conditions for good consensus to be achieved. Overall, the paper is a significant contribution to the field of collaborative learning. The proposed algorithm is a promising approach for improving the predictive performance of individual models in the presence of heterogeneity.

Strengths

* The paper proposes a novel algorithm for collaborative learning that is based on prediction-consensus. * The paper is well-written and easy to follow. The authors do a good job of explaining the motivation for the work, the proposed algorithm, and the experimental results. * The theoretical results in the paper are sound. The authors provide a rigorous analysis of the proposed algorithm and show that it can reach consensus under certain conditions.

Weaknesses

* The experimental results in the paper are somewhat weak. The authors only report results on a few datasets and with a few model architectures. And the datasets seem over-simplified. It would be helpful to see more experimental results on a wider variety of datasets and with a wider variety of model architectures. This would help to give a better sense of the generalizability of the results. * Again on the experimental results, it would be great to study the effects of the hyperparameter lambda and distance function D. It would be helpful to see how the performance of the algorithm is affected by different hyperparameter settings.

Questions

* Assumption 1 states that there is no concept shift between the local data distributions. However, it seems to be a quite strong assumption. If there is no concept shift, then the problem of collaborative learning becomes much easier. * Depending on the choice of lambda, is it possible that the algorithm converges to a point where although the local models reach consensus, the consensus is not ideal?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.