Bessel Function Analysis of Nesterov's ODE in $N$-Player Quadratic Games

We analyze Nesterov's accelerated gradient descent (NAGD) for Nash equilibrium seeking in $N$-player quadratic games. While the continuous-time NAGD dynamics - the Su-Boyd-Candès ODE - are well understood for convex optimization, their behavior with non-symmetric pseudo-gradient matrices arising in games has not been characterized precisely. We establish spectral characterizations via Bessel function modal analysis: the equilibrium is unstable whenever any eigenvalue of the pseudo-gradient matrix $G$ lies outside $\mathbb{R}_{\geq 0}$, and all trajectories converge when every eigenvalue lies in $\mathbb{R}_{\geq 0}$ and $G$ is diagonalizable. Remarkably, complex eigenvalues with positive real parts, which ensure stability for first-order gradient dynamics, induce exponential instability in NAGD. This reveals that the momentum mechanism enabling $O(1/t^2)$ convergence in optimization can be detrimental for equilibrium seeking in non-potential games.