The accuracy of deep convolutional neural networks (CNNs) generally improves\nwhen fueled with high resolution images. However, this often comes at a high\ncomputational cost and high memory footprint. Inspired by the fact that not all\nregions in an image are task-relevant, we propose a novel framework that\nperforms efficient image classification by processing a sequence of relatively\nsmall inputs, which are strategically selected from the original image with\nreinforcement learning. Such a dynamic decision process naturally facilitates\nadaptive inference at test time, i.e., it can be terminated once the model is\nsufficiently confident about its prediction and thus avoids further redundant\ncomputation. Notably, our framework is general and flexible as it is compatible\nwith most of the state-of-the-art light-weighted CNNs (such as MobileNets,\nEfficientNets and RegNets), which can be conveniently deployed as the backbone\nfeature extractor. Experiments on ImageNet show that our method consistently\nimproves the computational efficiency of a wide variety of deep models. For\nexample, it further reduces the average latency of the highly efficient\nMobileNet-V3 on an iPhone XS Max by 20% without sacrificing accuracy. Code and\npre-trained models are available at\nhttps://github.com/blackfeather-wang/GFNet-Pytorch.\n