Hard Example Generation by Texture Synthesis for Cross-domain Shape Similarity Learning

Image-based 3D shape retrieval (IBSR) aims to find the corresponding 3D shape\nof a given 2D image from a large 3D shape database. The common routine is to\nmap 2D images and 3D shapes into an embedding space and define (or learn) a\nshape similarity measure. While metric learning with some adaptation techniques\nseems to be a natural solution to shape similarity learning, the performance is\noften unsatisfactory for fine-grained shape retrieval. In the paper, we\nidentify the source of the poor performance and propose a practical solution to\nthis problem. We find that the shape difference between a negative pair is\nentangled with the texture gap, making metric learning ineffective in pushing\naway negative pairs. To tackle this issue, we develop a geometry-focused\nmulti-view metric learning framework empowered by texture synthesis. The\nsynthesis of textures for 3D shape models creates hard triplets, which suppress\nthe adverse effects of rich texture in 2D images, thereby push the network to\nfocus more on discovering geometric characteristics. Our approach shows\nstate-of-the-art performance on a recently released large-scale 3D-FUTURE[1]\nrepository, as well as three widely studied benchmarks, including Pix3D[2],\nStanford Cars[3], and Comp Cars[4]. Codes will be made publicly available at:\nhttps://github.com/3D-FRONT-FUTURE/IBSR-texture\n