Evading adversarial example detection defenses requires finding adversarial\nexamples that must simultaneously (a) be misclassified by the model and (b) be\ndetected as non-adversarial. We find that existing attacks that attempt to\nsatisfy multiple simultaneous constraints often over-optimize against one\nconstraint at the cost of satisfying another. We introduce Orthogonal Projected\nGradient Descent, an improved attack technique to generate adversarial examples\nthat avoids this problem by orthogonalizing the gradients when running standard\ngradient-based attacks. We use our technique to evade four state-of-the-art\ndetection defenses, reducing their accuracy to 0% while maintaining a 0%\ndetection rate.\n