We develop a stochastic approximation-type algorithm to solve finite state/action, infinite-horizon, risk-aware Markov decision processes. Our algorithm has two loops. The inner loop computes the risk by solving a stochastic saddle-point problem. The outer loop performs <inline-formula><tex-math notation="LaTeX">$Q-$</tex-math></inline-formula> learning to compute an optimal risk-aware policy. Several widely investigated risk measures (e.g., conditional value-at-risk, optimized certainty equivalent, and absolute semideviation) are covered by our algorithm. Almost sure convergence and the convergence rate of the algorithm are established. For an error tolerance <inline-formula><tex-math notation="LaTeX">$\epsilon >0$</tex-math></inline-formula> for optimal <inline-formula><tex-math notation="LaTeX">$Q$</tex-math></inline-formula>-value estimation gap and learning rate <inline-formula><tex-math notation="LaTeX">$k\in (1/2,\,1]$</tex-math></inline-formula>, the overall convergence rate of our algorithm is <inline-formula><tex-math notation="LaTeX">$\Omega ((\ln (1/\delta \epsilon)/\epsilon ^{2})^{1/k}+(\ln (1/\epsilon))^{1/(1-k)})$</tex-math></inline-formula> with probability at least <inline-formula><tex-math notation="LaTeX">$1-\delta$</tex-math></inline-formula>.