Privacy-Preserving Federated Learning for Medical Image Classification with Selective Homomorphic Encryption

Federated learning lets hospitals train shared diagnostic models without exchanging patient images, yet the updates they exchange each round can be inverted to reconstruct training images. Homomorphic encryption (HE) protects these updates, but encrypting an entire model with CKKS inflates communication and computation to impractical levels for cross-silo medical use. We present PASHE-FL, which exploits the structure of personalized federated learning: the client-specific classifier head stays local and is never uploaded, so encryption need only cover the shared backbone. The server ranks backbone coordinates by importance from the public global model and selects the same top-ρ set for all clients, avoiding mask negotiation; these are encrypted with CKKS, the remaining coordinates are quantized, and the encrypted fraction is annealed over training. On four medical image-classification tasks, PASHE-FL matches the personalized FedPer baseline within about one accuracy point while cutting per-round uplink by roughly 7.1–7.6× and encryption time by about 8× relative to full-model HE; this accuracy comes from personalization, not encryption. Under a gradient-inversion attack, encrypting only the top 5–10% most important coordinates collapses reconstruction quality, whereas encrypting random coordinates does not. PASHE-FL offers an empirical privacy–cost trade-off under the stated threat model rather than a formal privacy guarantee.