Joint User Pairing and Beamforming Design of Multi-STAR-RISs-Aided NOMA in the Indoor Environment via Multi-Agent Reinforcement Learning

To increase the quality of the 6G / B5G network, conventional cellular networks based on terrestrial base stations are geographically and economically restricted. Meanwhile, Non-Orthogonal Multiple Access (NOMA) allows multiple users to share the same resources, which improves the spectral efficiency of the system and has the advantage of supporting a larger number of users. Additionally, by intelligently manipulating the phase and amplitude of both the reflected and transmitted signals, Simultaneously Transmitting and Reflecting RISs (STAR-RISs) can achieve improved coverage, increased spectral efficiency, and enhanced communication reliability. However, STAR-RISs must simultaneously optimize the amplitude and phase shift corresponding to reflection and transmission, which makes existing terrestrial networks more complicated and is considered a major challenge. Motivated by the above, we study the joint user pairing for NOMA and the beamforming design of Multi-STAR-RISs in an indoor environment. Then, we formulate the optimization problem with the objective of maximizing the total throughput of mobile users (MUs) by jointly optimizing the decoding order, user pairing, active beamforming, and passive beamforming. However, the formulated problem is a mixed-integer non-linear programming (MINLP). To address this challenge, we first introduce the decoding order for NOMA networks. Next, we decompose the original problem into two subproblems, namely: 1) MU pairing and 2) Beamforming optimization under the optimal decoding order. For the first subproblem, we employ correlation-based K-means clustering to solve the user pairing problem. Then, to jointly deal with beamforming vector optimizations, we propose Multi-Agent Proximal Policy Optimization (MAPPO), which can make quick decisions in the given environment owing to its low complexity. Finally, simulation results prove that our proposed MAPPO algorithm is superior to Proximal Policy Optimization (PPO) and Advanced Actor-Critic (A2C) by a maximum of 1% and 6%, respectively. Furthermore, the proposed algorithm converges 1.5 times faster than the typical PPO algorithm.