Multi-agent Reinforcement Learning (MARL) offers new opportunities in the cyber defense domain. We propose a hierarchical MARL architecture that decomposes defense strategies into specialized sub-tasks like network investigation and host recovery. A master defense policy coordinates these sub-tasks, enabling efficient adaptation to shifting attacker strategies with minimal fine-tuning. Evaluation in the CybORG CAGE 4 cyber defense environment shows that our hierarchical learning approach achieves high performance in terms of convergence speed, episodic return, and several interpretable metrics relevant to cybersecurity.