Recent Advances in Statistical Foundations of Large Language Models: A Review of Probabilistic and Optimization Perspectives

Abstract

Large Language Models (LLMs) have transformed the landscape of artificial intelligence by demonstrating remarkable capabilities in natural language understanding, generation, reasoning, and multimodal interaction. This review examines recent advances in the statistical foundations of LLMs, with particular emphasis on probabilistic modeling and optimization perspectives that underpin their development and performance. The study synthesizes contemporary research on statistical learning theories, autoregressive probabilistic frameworks, Bayesian interpretations, scaling laws, and uncertainty estimation methods that guide the behavior of transformer-based architectures. Furthermore, the review explores optimization mechanisms including stochastic gradient descent, adaptive optimization algorithms, regularization strategies, reinforcement learning from human feedback, and parameter-efficient fine-tuning techniques that contribute to improved generalization and computational efficiency. Attention is also given to emerging topics such as sparse modeling, interpretability, alignment optimization, and statistical robustness in high-dimensional learning environments. The paper critically evaluates challenges associated with overparameterization, convergence instability, bias propagation, hallucination, and energy-intensive training processes. By integrating probabilistic reasoning with optimization theory, the review highlights how statistical principles continue to shape the evolution of scalable and reliable language models. The study concludes that future progress in LLM research depends on the development of more theoretically grounded, computationally efficient, and ethically aligned statistical methodologies capable of supporting trustworthy artificial intelligence systems across diverse application domains.