Wireless systems are becoming more and more complicated. As a consequence, the expressions of objective function or constraints of many optimization problems are hard or even impossible to derive. In this paper, we propose a model-free framework to learn the mapping from environment parameters to the solutions of generic constrained optimization problems without the labels generated by numerically finding the optimal solution. We use neural networks respectively for parameterizing the policy to be optimized, the Lagrange multiplier function associated with instantaneous constraint, and approximating the unavailable objective function or constraints. We provide learning algorithms to train all the neural networks simultaneously. We reveal the connections of the proposed framework with reinforcement learning, which is a widely recognized tool for model-free problems. Numerical and simulation results demonstrate the efficiency of model-free learning by taking a well-known power control problem as an example.