BERT-based models are currently used for solving nearly all Natural Language\nProcessing (NLP) tasks and most often achieve state-of-the-art results.\nTherefore, the NLP community conducts extensive research on understanding these\nmodels, but above all on designing effective and efficient training procedures.\nSeveral ablation studies investigating how to train BERT-like models have been\ncarried out, but the vast majority of them concerned only the English language.\nA training procedure designed for English does not have to be universal and\napplicable to other especially typologically different languages. Therefore,\nthis paper presents the first ablation study focused on Polish, which, unlike\nthe isolating English language, is a fusional language. We design and\nthoroughly evaluate a pretraining procedure of transferring knowledge from\nmultilingual to monolingual BERT-based models. In addition to multilingual\nmodel initialization, other factors that possibly influence pretraining are\nalso explored, i.e. training objective, corpus size, BPE-Dropout, and\npretraining length. Based on the proposed procedure, a Polish BERT-based\nlanguage model -- HerBERT -- is trained. This model achieves state-of-the-art\nresults on multiple downstream tasks.\n