A Fundamental Discussion on Reward-Based Reinforcement Learning for Uplink Resource Allocation in NTN

Abstract

The integrated usage of non-terrestrial network (NTN) nodes like high altitude platform (HAP), and low earth orbit (LEO) satellite plays a significant supplemental role in the upcoming sixth generation (6G) network to provide connectivity services for massive mobile users. Various network functions can be performed intelligently using potential artificial intelligence (AI) tools. Resilient and effective network channel accessibility and resource management, however, pose significant challenges due to the mobility, scarce resources, and complexity of integrated terrestrial networks (TNs) and NTNs, and massive mobile users on ground networks. Considering that many services demand real-time uplink (UL) data transmission, in this work, we consider the uplink of cooperated air-space segments (HAPs and LEO satellites) for large-sized data transmission where HAPs transmit data simultaneously to the LEO satellites through multiple sub-carriers using non-orthogonal multiple access (NOMA) approach, and the LEO satellite receives the superimposed signal data from HAPs. To maximize the fair allocation of resources among HAPs while reflecting the LEO satellite's load and utility, the problem is formulated as nonlinear mixed-integer programming (MINLP). Given that NTN channels are dynamic, we advocate a novel deep reward-based multiple-agent reinforcement learning for dynamic channel allocation. Furthermore, to handle the short visibility of the LEO satellites in learning, we employ a prioritized experience reply (PER) through a priority sampling approach (PSA). Numerical results reveal that our proposed approach significantly outperforms learning-based and conventional baseline models.