FLuID: Mitigating Stragglers in Federated Learning using Invariant Dropout

Summary

The paper proposes (Federated Learning using Invariant Dropout) FLuiD to address the straggler problem in FL. Due to the presence of system heterogeneity, straggling nodes (nodes with resource constraints) become a bottleneck in the FL training process. In this paper, the authors propose a FLuID to address this problem. First, the identification of invariant neurons, neurons that quickly optimize and remain relatively stable for the remainder of the training. Second, the straggling devices participating in the training. This information will be used to “prune” the global model and send a subset model to the straggling devices in order to efficiently utilize the available computation and communication resources.

Strengths

+ The straggler experiments included setups conducted using mobile devices. + The experiments show the proposed invariant dropout speeding up execution time relative to ordered dropout and random dropout.

Weaknesses

- It appears most of the experiments involved only 5 devices. While it is commendable that the experiments involved hardware, it is not clear to claim that the accuracy margins would hold as the number of clients increases, as is expected in practical FL environments. The scaled up experiments involving 50 to 100 clients can easily be scaled up to 1000 - 3000 clients in FEMNIST for instance. This might require an FL simulation environment, but it would give a good sense of how the proposed scheme yields improved training time and accuracy. - FL usually requires client sampling in each round. This is necessary as the system scales, and not all clients can participate in each round. I didn’t see the client sample ratio discussed. Thus I assume, in your experiments, all clients are participating in each training round. If that’s the case, how will client sampling in each round affect the dropout scheme? It is not clear if the invariant dropout scheme will maintain its performance margin if client sampling is incorporated into the training process.

Questions

- How the proposed approach is scalable with increasing the number of clients (clients can easily be scaled up to 1000 - 3000 clients in FEMNIST for instance.)? - Does the invariant dropout scheme will maintain its performance if client sampling is incorporated into the training? Clarification on this and further experiments would be helpful. - The authors should enhance the plot labels in Fig. 5

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

- Scalability is a major limitation that is not discussed and experimented well. - Sampling during the training process needs to be addressed clearly.

Summary

The paper focuses on the issue of mitigating stragglers in a heterogenous FL environment through dynamic load balancing, introducing a technique called Invariant Dropout and an adaptive training framework called FLuID. Invariant Dropout dynamically creates customized sub models that include only the neurons exhibiting significant changes over a certain threshold. Through experimental evaluation, the papers shows that this approach mitigates performance overheads caused by stragglers while also achieving higher accuracy compared to the state-of-the-art technique - Ordered Dropout.

Strengths

- Up to 18% speedup in performance - Up to 1.4 pp improvement in accuracy over state-of-the-art Ordered Dropout - Evaluation is detailed - on 3 models and datasets and compared with two techniques: Random and Ordered - Improves on previous work which have drawback of incurring training bias, creating performance centric sub models, or entirely reconstructing the sub model

Weaknesses

- Accuracy gains compared to Ordered and Random across datasets and models, while statistically significant may not justify the additional complexity of this system. - There is scope to improve the presentation - In the lines 40-47 2 sentences are repeated - Figure 2b has no label on the x axis

Questions

N/A

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

In addition to the weaknesses mentioned above, the authors note two limitations - Overhead to handle stragglers and maintain system performance which may increase with changes in straggler performance - Currently the system only uses pre-defined sub model sizes mapped to straggler performance which keeps the framework lightweight, but for varied edge devices, fine grained sub model determination will be required. The impact of this fine grained approach on the overhead needs to be measured to check for its suitability on different edge devices. If the overhead increases, the performance improvements noted here may diminish.

Summary

FLuID authors tackle a straggler problem in Federated Learning, where a central model is trained across a set of heterogeneous devices. This problem is particularly challenging when performance capabilities at training time are actually variable. This variable heterogeneity at training time requires a mechanism that can change the model sent to a client per round, in order to create a control mechanism that mitigates the clients’ straggler effect. FLuID achieves this with a proposed Invariant Dropout method. The Invariant Dropout technique quite simply refers to identifying neurons the updates for which have become minimal (close to invariant) over some number of rounds. This observation is used to decide to exclude such neurons (or “drop” them) from training for a given client. Neurons can be ranked based on the extent to which they are “invariant”, and a line can be drawn as a function of computation budget to determine how many neurons to drop. FLuID builds on several technical challenges: 1. identifying invariant neurons 2. identifying stragglers 3. determining a subset of the model weights to send to each client/straggler FLuID is able to match or exceed accuracy across several datasets. Performance includes real-world mobile client evaluation. Straggler effects are mitigated using Invariant Dropout.

Strengths

+ FLUid tackles a quintessential problem of stragglers in FL, which very practical and pervasive + FLUid avoids using asynchronous aggregation techniques, which jeopardize model convergence. It sticks with synchronous aggregation + the dropout technique is well known in DNN training and has been shown to improve performance, when random dropout is used + The dynamic nature of invariant dropout is effective and can adapt to dynamically changing client compute (or communication) capacity. Support for this dynamic heterogeneity is rare in literature + the paper is well written + the experimental methodology is well executed

Weaknesses

* the idea is very simple, some could consider the adaptation of dropout from previous literature to FL setting incremental? I’m certainly familiar with this idea and surprised this hasn’t been published yet. * this paper compares to different types of dropout as baselines, but doesn’t consider other published baselines, such as SuperFed [1]: * https://arxiv.org/abs/2301.10879 [1] SuperFed, https://arxiv.org/abs/2301.10879, 26 Jan 2023

Questions

Questions: * It would be interesting to get your perspective on comparison between FLuID and SuperFed [1]. SuperFed proposes sending clients subgraphs of the central model, which is isomorphic to structured dropout. SuperFed does NOT track neuron invariance, of course, but the idea of using structured dropout to send a smaller model to the client is there. In its conclusion, SuperFed mentions “adjusting .. for client resource constraints (e.g., compute, bandwidth, memory capacity)”. [1] SuperFed, https://arxiv.org/abs/2301.10879, 26 Jan 2023

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

yes, credit to authors for pointing out limitations: minimal overhead to handle stragglers and degradation of system perf in the worst case when the dynamics of client capacity variability causes instability.

Summary

The paper presents a framework, FLuID, for cross-device federated learning, where some of the clients are “stragglers”. The training time of these stragglers is significantly higher, hence they dictate the overall training time. FLuID uses Invariant Dropout to dynamically reduce the stragglers’ training time, hence alleviating their overall FL training time degradation.

Strengths

The authors tackle a very important problem in cross-device federated learning. Clients are usually edge and mobile devices that are running several processes and applications at the same time. Therefore, the training time varies as compared to dedicated servers used in cross-silo FL setups. Therefore, proposing a dynamic approach to identify the stragglers and the appropriate dropout rate is of major importance in this domain.

Weaknesses

While this paper targets a very important and relevant problem in the FL domain, I have concerns regarding the proposed framework, as detailed below: * How FLuID measures the training time of the stragglers, and how can it distinguish between stragglers due to high training time, and high upload/download latency? FLuID targets the former, while the latter requires a different solution than proposed in this paper. * Some of the design decisions/details lack further explanations. For instance, the setting of T_target and th (see more about it in the Questions section). * The number of invariants is bounded (at any given training round for any given model), while “Speedup” (as defined in the paper in line 197) is generally unbounded (depending on the stragglers’ training times). Therefore, FLuID might result in an aggressive dropout rate, and thus diminishing return. * The evaluation is insufficient: (1) It includes small and relatively “easy” datasets to learn, as well as small ML models. (2) The obtained accuracy improvement is quite negligible. (3) Some of the evaluation setup information is missing. Mostly the distribution of the stragglers' training time, see also question (1) in the Questions section.

Questions

(1) Lines 196-198: “T_target is set equal to the training time of the ‘next slowest client’ … Thus speedup ensures optimal utilization of available clients”, can you explain this argument? How does such a setting of T_target ensure optimal utilization of the clients, regardless of the stragglers’ training time distribution? (2) Line 200: how does the server measure the training time of clients? The server is aware of the end-to-end latency of given clients that generally depends on the client’s (1) downlink bandwidth and latency, (2) training time, and (3) uplink bandwidth and latency. (3) Line 11 in Algorithm 1 is unclear: what’s “inv”? (4) The discussion about setting th in lines 209-219 is unclear, is it a heuristic? Does it have any theoretical properties/guarantees?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

None