Skip to content

(SIGKDD 2022) Connected Low-Loss Subspace Learning for a Personalization in Federated Learning (https://arxiv.org/abs/2109.07628)

Notifications You must be signed in to change notification settings

vaseline555/SuPerFed

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

77 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

(SuPerFed) Connecting Low-Loss Subspace for Personalized Federated Learning Generic badge

  • Note: for more handy and research-friendly implementation, please check this repository. SuPerFed_Overview This repository is an official implementation of the SIGKDD 2022 paper Connecting Low-Loss Subspace for Personalized Federated Learning by Seok-Ju (Adam) Hahn, Minwoo Jeong and Junghye lee.
    [ PAPER | POSTER | SLIDE | VIDEO ]

PWC
PWC
PWC
PWC
PWC
PWC

Abstract

Due to the curse of statistical heterogeneity across clients, adopting a personalized federated learning method has become an essential choice for the successful deployment of federated learning-based services. Among diverse branches of personalization techniques, a model mixture-based personalization method is preferred as each client has their own personalized model as a result of federated learning. It usually requires a local model and a federated model, but this approach is either limited to partial parameter exchange or requires additional local updates, each of which is helpless to novel clients and burdensome to the client's computational capacity. As the existence of a connected subspace containing diverse low-loss solutions between two or more independent deep networks has been discovered, we combined this interesting property with the model mixture-based personalized federated learning method for improved performance of personalization. We proposed SuPerFed, a personalized federated learning method that induces an explicit connection between the optima of the local and the federated model in weight space for boosting each other. Through extensive experiments on several benchmark datasets, we demonstrated that our method achieves consistent gains in both personalization performance and robustness to problematic scenarios possible in realistic services.

Dependencies

  • Please install the required packages first by executing a command pip install -r requirements.txt.
torch >= 1.9
torchvision >= 0.10
numpy >= 1.20
tensorboard >= 2.8
matplotlib >= 3.5
scipy >= 1.20

Commands

  • For detailed description of command arguments, please check main.py or run command python main.py -h.
python3 main.py --exp_name superfed-mm_mnist_patho_50 \
--dataset MNIST --is_small --in_channels 1 --num_classes 10 \
--algorithm superfed-mm --model_name TwoNN --fc_type LinesLinear \
--C 0.1 --K 50 --R 500 --E 10 --B 10 --L 0.6 --nu 1 --mu 0.01 \
--split_type pathological --shard_size 600 \
--eval_every 50
python3 main.py --exp_name superfed-lm_cifar100_diri_1 \
--dataset CIFAR100 --is_small --in_channels 3 --num_classes 100 \
--algorithm superfed-lm --model_name ResNet9 --fc_type LinesLinear --conv_type LinesConv --bn_type LinesBN \
--C 0.05 --K 100 --R 500 --E 5 --B 20 --L 0.6 --nu 2 --mu 0.01 --lr 0.01 \
--split_type dirichlet --alpha 1.0 --n_jobs 20 \
--eval_every 50
python3 main.py --exp_name superfed-mm_shakespeare \
--dataset Shakespeare \
--algorithm superfed-mm --model_name NextCharLM --fc_type LinesLinear --lstm_type LinesLSTM --embedding_type LinesEmbedding \
--C 0.008 --R 500 --E 5 --B 50 --lr 0.8 --nu 1.5 --mu 0.01 --L 0.45 --n_jobs 10 --lr_decay 0.995 \
--split_type realistic \
--eval_every 50

Cite in BibTeX

@inproceedings{SuPerFed,
  author = {Hahn, Seok-Ju and Jeong, Minwoo and Lee, Junghye},
  title = {Connecting Low-Loss Subspace for Personalized Federated Learning},
  year = {2022},
  isbn = {9781450393850},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  doi = {10.1145/3534678.3539254},
  booktitle = {Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  pages = {505–515},
  numpages = {11},
  keywords = {label noise, federated learning, personalization, personalized federated learning, non-iid data, mode connectivity},
  location = {Washington DC, USA},
  series = {KDD '22},
  archivePrefix = {arXiv},
  arxivId = {2109.07628}
}

Contact

Feel free to contact the first author (Adam; seokjuhahn@unist.ac.kr) or leave an issue if face with a problem when using this implementation. Thank you! 😃

About

(SIGKDD 2022) Connected Low-Loss Subspace Learning for a Personalization in Federated Learning (https://arxiv.org/abs/2109.07628)

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published