When designing Convolutional Neural Networks (CNNs), one must select the\nsize\\break of the convolutional kernels before training. Recent works show CNNs\nbenefit from different kernel sizes at different layers, but exploring all\npossible combinations is unfeasible in practice. A more efficient approach is\nto learn the kernel size during training. However, existing works that learn\nthe kernel size have a limited bandwidth. These approaches scale kernels by\ndilation, and thus the detail they can describe is limited. In this work, we\npropose FlexConv, a novel convolutional operation with which high bandwidth\nconvolutional kernels of learnable kernel size can be learned at a fixed\nparameter cost. FlexNets model long-term dependencies without the use of\npooling, achieve state-of-the-art performance on several sequential datasets,\noutperform recent works with learned kernel sizes, and are competitive with\nmuch deeper ResNets on image benchmark datasets. Additionally, FlexNets can be\ndeployed at higher resolutions than those seen during training. To avoid\naliasing, we propose a novel kernel parameterization with which the frequency\nof the kernels can be analytically controlled. Our novel kernel\nparameterization shows higher descriptive power and faster convergence speed\nthan existing parameterizations. This leads to important improvements in\nclassification accuracy.\n