Federated Learning (FL) has been considered as an appealing framework to\ntackle data privacy issues of mobile devices compared to conventional Machine\nLearning (ML). Using Edge Servers (ESs) as intermediaries to perform model\naggregation in proximity can reduce the transmission overhead, and it enables\ngreat potentials in low-latency FL, where the hierarchical architecture of FL\n(HFL) has been attracted more attention. Designing a proper client selection\npolicy can significantly improve training performance, and it has been\nextensively used in FL studies. However, to the best of our knowledge, there\nare no studies focusing on HFL. In addition, client selection for HFL faces\nmore challenges than conventional FL, e.g., the time-varying connection of\nclient-ES pairs and the limited budget of the Network Operator (NO). In this\npaper, we investigate a client selection problem for HFL, where the NO learns\nthe number of successful participating clients to improve the training\nperformance (i.e., select as many clients in each round) as well as under the\nlimited budget on each ES. An online policy, called Context-aware Online Client\nSelection (COCS), is developed based on Contextual Combinatorial Multi-Armed\nBandit (CC-MAB). COCS observes the side-information (context) of local\ncomputing and transmission of client-ES pairs and makes client selection\ndecisions to maximize NO's utility given a limited budget. Theoretically, COCS\nachieves a sublinear regret compared to an Oracle policy on both strongly\nconvex and non-convex HFL. Simulation results also support the efficiency of\nthe proposed COCS policy on real-world datasets.\n