Code-Switching (CS) remains a challenge for Automatic Speech Recognition\n(ASR), especially character-based models. With the combined choice of\ncharacters from multiple languages, the outcome from character-based models\nsuffers from phoneme duplication, resulting in language-inconsistent spellings.\nWe propose Contextualized Connectionist Temporal Classification (CCTC) loss to\nencourage spelling consistencies of a character-based non-autoregressive ASR\nwhich allows for faster inference. The CCTC loss conditions the main prediction\non the predicted contexts to ensure language consistency in the spellings. In\ncontrast to existing CTC-based approaches, CCTC loss does not require\nframe-level alignments, since the context ground truth is obtained from the\nmodel's estimated path. Compared to the same model trained with regular CTC\nloss, our method consistently improved the ASR performance on both CS and\nmonolingual corpora.\n