Robust model fitting is a core algorithm in a large number of computer vision\napplications. Solving this problem efficiently for datasets highly contaminated\nwith outliers is, however, still challenging due to the underlying\ncomputational complexity. Recent literature has focused on learning-based\nalgorithms. However, most approaches are supervised which require a large\namount of labelled training data. In this paper, we introduce a novel\nunsupervised learning framework that learns to directly solve robust model\nfitting. Unlike other methods, our work is agnostic to the underlying input\nfeatures, and can be easily generalized to a wide variety of LP-type problems\nwith quasi-convex residuals. We empirically show that our method outperforms\nexisting unsupervised learning approaches, and achieves competitive results\ncompared to traditional methods on several important computer vision problems.\n