Abstract Even though convolutional neural networks (CNN) hasachieved near-human performance in various computer vi-sion tasks, its ability to tolerate scale variations is lim-ited. The popular practise is making the model biggerfirst, and then train it with data augmentation using ex-tensive scale-jittering. In this paper, we propose a scale-invariant convolutional neural network (SiCNN), a modeldesigned to incorporate multi-scale feature exaction andclassification into the network structure. SiCNN uses amulti-column architecture, with each column focusing on aparticular scale. Unlike previous multi-column strategies,these columns share the same set of filter parameters by ascale transformation among them. This design deals withscale variation without blowing up the model size. Experi-mental results show that SiCNN detects features at variousscales, and the classification result exhibits strong robust-ness against object scale variations. 1. Introduction Many classical computer vision tasks have enjoyed agreat breakthrough, primarily due to the large amount oftraining data and the application of deep convolution neu-ral networks (CNN) [8]. In the most recent ILSVRC 2014competition [11], CNN-based solutions have achieved near-human accuracies in image classification, localization anddetection tasks [14,16].Accompanying this progress are studies trying to under-stand what CNN has learnt internally and what contributeto its success [2,13,17]. By design, layers within the net-work have progressively larger receptive field sizes, allow-ing them to learn more complex features. Another key pointis the shift-invariance property, that a pattern in the inputcan be recognized regardless of its position [9]. Pooling lay-ers contribute resilience to slight deformation as well smallscale change [12].However, it is evident that CNN deals with shift-variancefar better than scale-invariance [3]. Not dealing with scale-invariance well poses a direct conflict to the design philos-ophy of CNN, in that higher layers may see and thus cap-tures features of certain plain patterns simply because theyare larger at the input, not because they are more complex.In other words, there is no alignments between in the posi-tion of a filter and the complexity it captures. What is more,there are other invariance that CNN does not deal with inter-nally. Examples include rotations and flips (since featuresof natural objects are mostly symmetric).A brutal force solution would be to make the networklarger by introducing more filters to cope with scale vari-ations of the same feature, accompanied by scale-jitteringthe input images, often by order of magnitude. This is, infact, the popular practice today [1,8,14]. It is true even forproposals that directly deal with this problem. For example,[3] drives the CNN with crops of different size and positionswith three differnt scales, and then uses VLAD pooling toproduce a feature summary of the patches.We explore a radically different approach that is alsosimple. Observing that filters that detect the same patternbut with different scales bear strong relationship, we adopt amulti-column design and designate each column to special-ize on certain scales. We call our system SiCNN (Scale-invariant CNN). Unlike a conventional multi-column CNN,filters in SiCNN are strongly regulated among columns.The goal is to make the network resilient to scale variancewithout blowing up number of free parameters, and thus re-duce the need of jittering the input.We performed detailed analysis and verified that SiCNNexhibits the desired behavior. For example, the column thatdeals with larger scale is indeed activated by input patternswith the larger scaling factor, and the system as a wholebecomes less sensitive to scale variance. On unaugmentedCIFAR-10 dataset [7], our method produces the best re-1