Although linguistic typology has a long history, computational approaches\nhave only recently gained popularity. The use of distributed representations in\ncomputational linguistics has also become increasingly popular. A recent\ndevelopment is to learn distributed representations of language, such that\ntypologically similar languages are spatially close to one another. Although\nempirical successes have been shown for such language representations, they\nhave not been subjected to much typological probing. In this paper, we first\nlook at whether this type of language representations are empirically useful\nfor model transfer between Uralic languages in deep neural networks. We then\ninvestigate which typological features are encoded in these representations by\nattempting to predict features in the World Atlas of Language Structures, at\nvarious stages of fine-tuning of the representations. We focus on Uralic\nlanguages, and find that some typological traits can be automatically inferred\nwith accuracies well above a strong baseline.\n