During 2020, social media chatter has been largely dominated by the COVID-19 pandemic. In this paper, we study the extent of polarization of COVID-19 discourse on Twitter in the U.S. First, we propose Retweet-BERT, a scalable and highly accurate model for estimating user polarity by leveraging language features and network structures. Then, by analyzing the user polarity predicted by Retweet-BERT, we provide new insights into the characterization of partisan users. Right-leaning users, we find, are noticeably more vocal and active in the production and consumption of COVID-19 information. Our analysis also shows that most of the highly influential users are partisan, which may contribute to further polarization. Crucially, we provide empirical evidence that political echo chambers are prevalent, exacerbating the exposure to information in line with pre-existing users' views. Our findings have broader implications in developing effective public health campaigns and promoting the circulation of factual information online.