State-of-the-art neural networks are vulnerable to adversarial examples; they\ncan easily misclassify inputs that are imperceptibly different than their\ntraining and test data. In this work, we establish that the use of\ncross-entropy loss function and the low-rank features of the training data have\nresponsibility for the existence of these inputs. Based on this observation, we\nsuggest that addressing adversarial examples requires rethinking the use of\ncross-entropy loss function and looking for an alternative that is more suited\nfor minimization with low-rank features. In this direction, we present a\ntraining scheme called differential training, which uses a loss function\ndefined on the differences between the features of points from opposite\nclasses. We show that differential training can ensure a large margin between\nthe decision boundary of the neural network and the points in the training\ndataset. This larger margin increases the amount of perturbation needed to flip\nthe prediction of the classifier and makes it harder to find an adversarial\nexample with small perturbations. We test differential training on a binary\nclassification task with CIFAR-10 dataset and demonstrate that it radically\nreduces the ratio of images for which an adversarial example could be found --\nnot only in the training dataset, but in the test dataset as well.\n