Distributed fog computing environments demand efficient resource management to support real-time Internet of Things (IoT) applications. This paper proposes a fully distributed load-balancing framework based on multi-agent reinforcement learning (MARL), where independent agents learn to manage heterogeneous fog resources without centralized control or inter-agent coordination. The agents jointly optimize a global objective that minimizes workload waiting delay (reduces overall fog queue accumulation) while ensuring fair resource utilization. We evaluated agents’ dynamic adaptation to unpredictable load bursts through transfer learning in simulated fog environments with heterogeneous, unbalanced, and geographically distributed fog nodes. Compared to centralized RL, learning localized policies within smaller collaboration regions allows our distributed agents to achieve superior performance (up to <inline-formula> <tex-math notation="LaTeX">$\mathbf {70.1\%}$ </tex-math></inline-formula> reduction in average waiting delay), reduces state-action space, accelerates convergence (<inline-formula> <tex-math notation="LaTeX">$6\times $ </tex-math></inline-formula> faster), and scales efficiently as the network grows. In addition, we analyze the impact of realistic interval-based state observation (using a protocol like Gossip) to evaluate the trade-off between performance and practical deployment constraints. Compared to the unrealistic assumption of real-time state availability before every decision, a Gossip interval of 3 seconds in our simulations reduces the state observation overhead by a factor of <inline-formula> <tex-math notation="LaTeX">$\mathbf {9.5\times }$ </tex-math></inline-formula>, ensuring the solution is viable for real-world deployment.