There has been an increasing surge of interest on development of advanced\nReinforcement Learning (RL) systems as intelligent approaches to learn optimal\ncontrol policies directly from smart agents' interactions with the environment.\nObjectives: In a model-free RL method with continuous state-space, typically,\nthe value function of the states needs to be approximated. In this regard, Deep\nNeural Networks (DNNs) provide an attractive modeling mechanism to approximate\nthe value function using sample transitions. DNN-based solutions, however,\nsuffer from high sensitivity to parameter selection, are prone to overfitting,\nand are not very sample efficient. A Kalman-based methodology, on the other\nhand, could be used as an efficient alternative. Such an approach, however,\ncommonly requires a-priori information about the system (such as noise\nstatistics) to perform efficiently. The main objective of this paper is to\naddress this issue. Methods: As a remedy to the aforementioned problems, this\npaper proposes an innovative Multiple Model Kalman Temporal Difference (MM-KTD)\nframework, which adapts the parameters of the filter using the observed states\nand rewards. Moreover, an active learning method is proposed to enhance the\nsampling efficiency of the system. More specifically, the estimated uncertainty\nof the value functions are exploited to form the behaviour policy leading to\nmore visits to less certain values, therefore, improving the overall learning\nsample efficiency. As a result, the proposed MM-KTD framework can learn the\noptimal policy with significantly reduced number of samples as compared to its\nDNN-based counterparts. Results: To evaluate performance of the proposed MM-KTD\nframework, we have performed a comprehensive set of experiments based on three\nRL benchmarks. Experimental results show superiority of the MM-KTD framework in\ncomparison to its state-of-the-art counterparts.\n