[Continuously updating] Last Update: 2023/3/11
Abstract#
Federated Learning is an emerging machine learning technique that allows training a global model using local data while protecting data privacy. It reduces communication costs, supports distributed devices, and improves model accuracy and generalization performance. This article introduces the concepts, common algorithms, advantages, disadvantages, and applications of Federated Learning.
Concepts#
Federated Learning is a distributed machine learning technique that aggregates local data from distributed devices to train a global model while preserving the privacy of each device's data. In Federated Learning, each device performs local model training and uploads the local model parameters to a central server for global model aggregation. By keeping the local data on the devices, Federated Learning protects data privacy, reduces communication costs, and improves model generalization performance.
Common Federated Learning Algorithms#
Aggregation Algorithms#
FedAvg#
FedAvg is a Federated Learning algorithm that performs local model training on devices and aggregates the local model parameters on a server to create a global model. It avoids the risk of uploading raw data and reduces communication costs. FedAvg is one of the most widely used Federated Learning algorithms.
For more details, see here:
https://www.royc30ne.com/fedavg/
FedProx#
FedProx is an improved version of the FedAvg algorithm. It introduces a regularization term to balance the weights between local and global models. FedProx improves the accuracy and generalization performance of the global model and handles data imbalance and device heterogeneity issues better.
FedOpt#
FedOpt is an algorithm proposed for optimization problems in Federated Learning. It uses a recursive approach for model training and introduces new optimization techniques such as momentum and adaptive learning rate to improve training effectiveness and convergence speed.
Attack Algorithms#
Data Poisoning Attack#
This is an attack method that manipulates or injects malicious samples into the local data of participants to affect the training effectiveness of the global model.
Model Replacement Attack#
This is an attack method that replaces or modifies the local model parameters of participants to affect the training effectiveness of the global model or leak information from other participants.
Gradient Leakage Attack#
This is an attack method that uses the gradient information returned by the server to infer data features or label information of other participants.
Defense Algorithms#
Gradient Clipping#
This is a simple and effective defense method that reduces the impact of abnormal or malicious updates by limiting the gradient norm submitted by each participant to the server.
Krum#
The basic idea of the Krum algorithm is to select the most representative result from the predictions of multiple models as the final prediction result. The most representative result should have high prediction accuracy and significant differences from the predictions of other models, thereby reducing the error of the ensemble model.
Bulyan#
Bulyan is an improved algorithm based on Krum, which adds Trimmed Mean to Krum.
Advantages and Disadvantages of Federated Learning#
Advantages#
(1) Data Privacy Protection: Federated Learning allows model training using local data without sharing the raw data, thus protecting data privacy.
(2) Reduced Communication Costs: Federated Learning enables local model training on devices, reducing the communication costs of uploading data.
(3) Support for Distributed Devices: Federated Learning supports distributed devices, allowing each device to perform local model training, reducing the burden on the central server, and improving the scalability and applicability of the algorithm.
(4) Improved Model Accuracy and Generalization Performance: Federated Learning utilizes more local data for model training, thereby improving model accuracy and generalization performance.
Disadvantages#
(1) Data Imbalance Issue: Due to the differences in local datasets on devices, the quality and accuracy of the trained models on each device may vary, affecting the accuracy and generalization performance of the global model.
(2) Device Heterogeneity Issue: Due to different hardware configurations and software environments of devices, there may be differences in local model parameters, affecting the convergence performance of the global model.
(3) Privacy Leakage Issue: Although Federated Learning protects data privacy, there are still potential privacy leakage issues, such as the potential exposure of privacy information through the global model parameters.
Applications#
Federated Learning technology has been widely applied in various fields such as healthcare, finance, and intelligent transportation. Among them, the healthcare field is one of the most widely used areas for Federated Learning. For example, Federated Learning can be applied to medical image analysis, medical diagnosis, and personalized treatment. In addition, Federated Learning technology can also be applied to financial risk assessment, intelligent traffic prediction, and other fields.
Recommended Papers (Continuously Updating)#
Comprehensive Survey of Federated Learning: Advances and Open Problems in Federated Learning
FedAvg: Communication-Efficient Learning of Deep Networks from Decentralized Data
Krum: Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent
Bulyan: A Little Is Enough: Circumventing Defenses For Distributed Learning