Among the various machine learning methods solving partial differential equations, the Random Feature Method (RFM) stands out due to its accuracy and efficiency. In this paper, we demonstrate that the approximation error of RFM exhibits spectral convergence when it is applied to the second-order elliptic equations in one dimension, provided that the solution belongs to Gevrey classes or Sobolev spaces. We highlight the significant impact of incorporating the Partition of Unity Method (PUM) to enhance the convergence of RFM by establishing the convergence rate in terms of the maximum patch size. Furthermore, we reveal that the singular values of the random feature matrix (RFMtx) decay exponentially, while its condition number increases exponentially as the number of the features grows. We also theoretically illustrate that PUM may mitigate the excessive decay of the singular values of RFMtx.