In this paper, we propose a novel approach for learning from data using rule-based fuzzy inference systems where the model parameters are estimated using Bayesian inference and Markov Chain Monte Carlo techniques. We show the applicability of the method for regression and classification tasks using synthetic datasets and also a real world example in the financial services industry. Then, we demonstrate how the method can be extended for knowledge extraction to select the individual rules in a Bayesian way which best explains the given data. Finally, we discuss the advantages and pitfalls of using this method over state-of-the-art techniques and highlight the specific class of problems where this would be useful.