Inverse problems play a central role for many classical computer vision and\nimage processing tasks. Many inverse problems are ill-posed, and hence require\na prior to regularize the solution space. However, many of the existing priors,\nlike total variation, are based on ad-hoc assumptions that have difficulties to\nrepresent the actual distribution of natural images. Thus, a key challenge in\nresearch on image processing is to find better suited priors to represent\nnatural images.\n In this work, we propose the Adaptive Quantile Sparse Image (AQuaSI) prior.\nIt is based on a quantile filter, can be used as a joint filter on guidance\ndata, and be readily plugged into a wide range of numerical optimization\nalgorithms. We demonstrate the efficacy of the proposed prior in joint\nRGB/depth upsampling, on RGB/NIR image restoration, and in a comparison with\nrelated regularization by denoising approaches.\n