Summary
The paper presents a new model for MDTC, built on the previous shared-private feature extraction architecture. The innovation includes 1) modelling the parameter of domain-specific feature extractors as a Gaussian random variable, and for each domain, the parameter is drawn from the distribution. This is why the model is called stochastic adversarial network, or SAN, 2) domain label smoothing 3) pseudo labelling regularization. The authors show some empirical successes on some datasets.
Strengths
The paper demonstrates that the authors are well aware of the challenges in MDTC and are familiar with various tools in deep learning (such as reparametrization trick, label smoothing, pseudo labelling etc).
Weaknesses
I have some concerns about this work.
1. Assuming the design of proposed model is sensible (in fact I have doubts on this; see 2), the work heuristically puts together a bunch of well-known techniques to improve performance. Works of primarily such a nature, although potentially valuable in practice, do not possess enough novelty that justifies a publication in ICLR.
2. I have doubts on the proposed approach in the "stochastic" part. Let us track the parameter $W_1$ of the domain-specific feature extractor for domain 1. In the beginning it is drawn from the prescribed Gaussian, say, its value is $W_1^{(0)}$, and after the first iteration, the Gaussian parameter gets updated (using the reparametrization trick) -- well, whether Gaussian parameter is updated or not is not critical here. Then in the next iteration, $W_1$ is drawn again, let us call it $W_1^{(1)}$. If this understanding is correct, then $W_1^{(0)}$ and $W_1^{(1)}$ can be very different. That is, along the training process, $W_1$ will randomly hop everywhere as long as the Gaussian variance is not vanishing. How would such a scheme work at all? Bringing the parameter $W_2$ of the second domain-specific extractor into the picture would show an even more absurd picture: at each iteration $t$, $W_1^{(t)}$ and $W_2^{(t)}$ are random variables following the same Gaussian distribution. How would $W_1$ and $W_2$ track their respective domain specific features? If this structure were to work, it would have to be the case where the Gaussian variance is very small (which might be the case as shown in Figure 3 of the appendix). In that case, all domain-specific extractors are more or less the same, i.e, all equal to the Gaussian mean, only subject to some tiny *domain-nonspecific* random perturbation. That would defeat the entire purpose of having domain specific feature extractors. -- I could misunderstood the paper and I am willing to hear the authors' defence on this. In your defence, please also show the initial and final values of the Gaussian mean vector $\mu$ (say, in terms of its L1-norm divided by its dimension), I would like compare it with $\sigma$.
Questions
See weakness 2.
Additional question: The authors say that the conventional shared-private adversarial scheme will have "exponential increase" in model parameters as new domains emerge? Why is it exponential?
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.