Unstructured text provides decision-makers with a rich data source in many\ndomains, ranging from product reviews in retail to nursing notes in healthcare.\nTo leverage this information, words are typically translated into word\nembeddings -- vectors that encode the semantic relationships between words --\nthrough unsupervised learning algorithms such as matrix factorization. However,\nlearning word embeddings from new domains with limited training data can be\nchallenging, because the meaning/usage may be different in the new domain,\ne.g., the word ``positive'' typically has positive sentiment, but often has\nnegative sentiment in medical notes since it may imply that a patient tested\npositive for a disease. In practice, we expect that only a small number of\ndomain-specific words may have new meanings. We propose an intuitive two-stage\nestimator that exploits this structure via a group-sparse penalty to\nefficiently transfer learn domain-specific word embeddings by combining\nlarge-scale text corpora (such as Wikipedia) with limited domain-specific text\ndata. We bound the generalization error of our transfer learning estimator,\nproving that it can achieve high accuracy with substantially less\ndomain-specific data when only a small number of embeddings are altered between\ndomains. Furthermore, we prove that all local minima identified by our\nnonconvex objective function are statistically indistinguishable from the\nglobal minimum under standard regularization conditions, implying that our\nestimator can be computed efficiently. Our results provide the first bounds on\ngroup-sparse matrix factorization, which may be of independent interest. We\nempirically evaluate our approach compared to state-of-the-art fine-tuning\nheuristics from natural language processing.\n