Deep Intrinsic Surprise-Regularized Control (DISRC): A Biologically Inspired Mechanism for Efficient Deep Q-Learning in Sparse Environments

Deep reinforcement learning (DRL) has driven major advances in autonomous control. Still, standard Deep QNetwork (DQN) agents tend to rely on fixed learning rates and uniform update scaling, even as updates are modulated by temporal-difference (TD) error. This rigidity limits data efficiency and destabilizes convergence, especially in sparse-reward settings where feedback is infrequent. While methods like curiositydriven exploration and prioritized replay indirectly influence learning dynamics, few approaches directly regulate update magnitudes using intrinsic signals. We introduce Deep Intrinsic Surprise-Regularized Control (DISRC), a biologically inspired augmentation to DQN that dynamically scales Q-updates based on latent-space surprise. DISRC encodes states via a LayerNormbased encoder and computes a deviation-based surprise score relative to a moving latent setpoint. Each update is then scaled in proportion to both TD error and surprise intensity, promoting plasticity during early exploration and stability as familiarity increases. We evaluate DISRC on two sparse-reward MiniGrid environments, which included MiniGrid-DoorKey- $8 \times 8$-v0 and MiniGrid-LavaCrossingS9N1-v0, under identical settings as a vanilla DQN baseline. In DoorKey, DISRC reached the first successful episode (reward $>0.8$) 33% faster than the vanilla DQN baseline (79 vs. 118 episodes), with lower reward standard deviation (0.25 vs. 0.34) and higher reward area under the curve (AUC: 596.42 vs. 534.90). These metrics reflect faster, more consistent learning - critical for sparse, delayed reward settings. In LavaCrossing, DISRC achieved a higher final reward (0.95 vs. 0.93) and the highest AUC of all agents (957.04), though it converged more gradually. These preliminary results establish DISRC as a novel and effective mechanism for regulating learning intensity in off-policy agents, improving both efficiency and stability in sparse-reward domains. This positions DISRC as a step toward real-world decision systems, such as autonomous robots or clinical planners, where agents must adapt under sparse, delayed feedback. By treating surprise as an intrinsic learning signal, DISRC enables agents to modulate updates based on expectation violations, enhancing decision quality when conventional value-based methods fall short.