We consider the optimization problem with a generally quadratic matrix constraint of the form XTAX=J$$ {X}^T AX=J $$ , where A$$ A $$ is a given nonsingular, symmetric n×n$$ n\times n $$ matrix and J$$ J $$ is a given k×k$$ k\times k $$ symmetric matrix, with k≤n$$ k\le n $$ , satisfying J2=Ik$$ {J}^2={I}_k $$ . Since the feasible set constitutes a differentiable manifold, called the indefinite Stiefel manifold, we approach this problem within the framework of Riemannian optimization. Namely, we first equip the manifold with a Riemannian metric and construct the associated geometric structure, then propose a retraction based on the Cayley transform, and finally suggest a Riemannian gradient descent method using the attained materials, whose global convergence is guaranteed. Our results not only cover the known cases, the orthogonal and generalized Stiefel manifolds, but also provide a Riemannian optimization solution for other constrained problems that have not been investigated. As applications, we consider, via trace minimization, several eigenvalue problems of symmetric positive definite matrix pencils, including the linear response eigenvalue problem, a matrix least square problem, a general framework for the Procrustes problem, and constrained matrix equations. The presented numerical results justify the theoretical findings.