Attribute prediction from face images in the wild is a challenging problem. To automatically describe face attributes from face containing images, traditionally one needs to cascade three technical blocks --- face localization, facial feature extraction, and classification --- in a pipeline. As a typical classification problem, face attribute prediction has been addressed by using deep learning networks. Current state-of-the-art performance was achieved by using two cascaded CNNs, which were specifically trained to learn face localization and facial attribute prediction. In this paper we experiment in an alternative way of exploring the power of deep representations from the networks: we employ off-the-shelf CNNs trained for face recognition tasks to do facial feature extraction, combined with conventional face localization techniques. Recognizing that the describable face attributes are diverse, we select representations from different levels of the CNNs and investigate their utilities for attribute classification. Experiments on two large datasets, LFWA and CeleA, show that the performance is totally comparable to state-of-the-art approach. Our findings suggest two potentially important questions in using CNNs for face attribute prediction: 1) How to maximally leverage the power of CNN representations. 2) How to best combine traditional computer vision techniques with deep learning networks.