Interest in generative models has grown tremendously in the past decade.\nHowever, their training performance can be adversely affected by contamination,\nwhere outliers are encoded in the representation of the model. This results in\nthe generation of noisy data. In this paper, we introduce weighted conjugate\nfeature duality in the framework of Restricted Kernel Machines (RKMs). The RKM\nformulation allows for an easy integration of methods from classical robust\nstatistics. This formulation is used to fine-tune the latent space of\ngenerative RKMs using a weighting function based on the Minimum Covariance\nDeterminant, which is a highly robust estimator of multivariate location and\nscatter. Experiments show that the weighted RKM is capable of generating clean\nimages when contamination is present in the training data. We further show that\nthe robust method also preserves uncorrelated feature learning through\nqualitative and quantitative experiments on standard datasets.\n