Machine-learning algorithms have shown outstanding image recognition or\nclassification performance for computer vision applications. However, the\ncompute and energy requirement for implementing such classifier models for\nlarge-scale problems is quite high. In this paper, we propose Feature Driven\nSelective Classification (FALCON) inspired by the biological visual attention\nmechanism in the brain to optimize the energy-efficiency of machine-learning\nclassifiers. We use the consensus in the characteristic features\n(color/texture) across images in a dataset to decompose the original\nclassification problem and construct a tree of classifiers (nodes) with a\ngeneric-to-specific transition in the classification hierarchy. The initial\nnodes of the tree separate the instances based on feature information and\nselectively enable the latter nodes to perform object specific classification.\nThe proposed methodology allows selective activation of only those branches and\nnodes of the classification tree that are relevant to the input while keeping\nthe remaining nodes idle. Additionally, we propose a programmable and scalable\nNeuromorphic Engine (NeuE) that utilizes arrays of specialized neural\ncomputational elements to execute the FALCON based classifier models for\ndiverse datasets. The structure of FALCON facilitates the reuse of nodes while\nscaling up from small classification problems to larger ones thus allowing us\nto construct classifier implementations that are significantly more efficient.\nWe evaluate our approach for a 12-object classification task on the Caltech101\ndataset and 10-object task on CIFAR-10 dataset by constructing FALCON models on\nthe NeuE platform in 45nm technology. Our results demonstrate significant\nimprovement in energy-efficiency and training time for minimal loss in output\nquality.\n