Unsupervised learning has recently made exceptional progress because of the\ndevelopment of more effective contrastive learning methods. However, CNNs are\nprone to depend on low-level features that humans deem non-semantic. This\ndependency has been conjectured to induce a lack of robustness to image\nperturbations or domain shift. In this paper, we show that by generating\ncarefully designed negative samples, contrastive learning can learn more robust\nrepresentations with less dependence on such features. Contrastive learning\nutilizes positive pairs that preserve semantic information while perturbing\nsuperficial features in the training images. Similarly, we propose to generate\nnegative samples in a reversed way, where only the superfluous instead of the\nsemantic features are preserved. We develop two methods, texture-based and\npatch-based augmentations, to generate negative samples. These samples achieve\nbetter generalization, especially under out-of-domain settings. We also analyze\nour method and the generated texture-based samples, showing that texture\nfeatures are indispensable in classifying particular ImageNet classes and\nespecially finer classes. We also show that model bias favors texture and shape\nfeatures differently under different test settings. Our code, trained models,\nand ImageNet-Texture dataset can be found at\nhttps://github.com/SongweiGe/Contrastive-Learning-with-Non-Semantic-Negatives.\n