In this paper, we propose a novel method for protecting convolutional neural\nnetwork (CNN) models with a secret key set so that unauthorized users without\nthe correct key set cannot access trained models. The method enables us to\nprotect not only from copyright infringement but also the functionality of a\nmodel from unauthorized access without any noticeable overhead. We introduce\nthree block-wise transformations with a secret key set to generate learnable\ntransformed images: pixel shuffling, negative/positive transformation, and FFX\nencryption. Protected models are trained by using transformed images. The\nresults of experiments with the CIFAR and ImageNet datasets show that the\nperformance of a protected model was close to that of non-protected models when\nthe key set was correct, while the accuracy severely dropped when an incorrect\nkey set was given. The protected model was also demonstrated to be robust\nagainst various attacks. Compared with the state-of-the-art model protection\nwith passports, the proposed method does not have any additional layers in the\nnetwork, and therefore, there is no overhead during training and inference\nprocesses.\n