We introduce a new procedure to neuralize unsupervised Hidden Markov Models\nin the continuous case. This provides higher flexibility to solve problems with\nunderlying latent variables. This approach is evaluated on both synthetic and\nreal data. On top of generating likely model parameters with comparable\nperformances to off-the-shelf neural architecture (LSTMs, GRUs,..), the\nobtained results are easily interpretable.\n