An analysis of the transfer learning of convolutional neural networks for artistic images

Transfer learning from huge natural image datasets, fine-tuning of deep\nneural networks and the use of the corresponding pre-trained networks have\nbecome de facto the core of art analysis applications. Nevertheless, the\neffects of transfer learning are still poorly understood. In this paper, we\nfirst use techniques for visualizing the network internal representations in\norder to provide clues to the understanding of what the network has learned on\nartistic images. Then, we provide a quantitative analysis of the changes\nintroduced by the learning process thanks to metrics in both the feature and\nparameter spaces, as well as metrics computed on the set of maximal activation\nimages. These analyses are performed on several variations of the transfer\nlearning procedure. In particular, we observed that the network could\nspecialize some pre-trained filters to the new image modality and also that\nhigher layers tend to concentrate classes. Finally, we have shown that a double\nfine-tuning involving a medium-size artistic dataset can improve the\nclassification on smaller datasets, even when the task changes.\n