mlsys-text-P-Federated.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"bHZ54p4iIfem"},"source":["---\n",">「団結は強さだ… チームワークとコラボレーションがあれば、すごいことが成し遂げられる。」\n",">\n","> Mattie Stepanek\n","---"]},{"cell_type":"markdown","metadata":{"id":"L9HXMz2Zvn8D"},"source":["# 連合学習が生まれた背景\n","\n","AIには大量のデータが必要であるが、データを集めるには様々な障害がある\n","- 契約上の問題\n","  - 契約で情報の二次配布ができないなど\n","- プライバシの問題\n","  - 学習に用いるデータに個人を特定可能な情報が含まれている場合、オプトインを取得するなど容易に利用することができない\n","- 通信・ストレージの問題\n","  - データを大量にかつ一か所に集めるには、それに見合ったストレージ量や通信容量が必要\n","- 規制\n","  - General Data Protection Regulation （GDPR）  \n","  一般データ保護規則\n","  - California Consumer Privacy Act of 2018（CCPA）  \n","  カリフォルニア州消費者プライバシ法\n","  - 様々な規制が提示されている\n","\n","これらを解決するためには、プライバシを確保しながら分断されたデータでも利用可能な機械学習方法が必要である"]},{"cell_type":"markdown","source":["# 連合学習(Federated Learning)とは\n","\n","複数のデータ所有者が協力して1つの機械学習モデルを構築することを一般に連合学習と呼ぶ\n","\n","<img src=\"http://class.west.sd.keio.ac.jp/dataai/text/flfig.png\" width=500>\n","\n","以下のような特徴・必要事項がある\n","- 参加者の生データそのものは交換しない\n","  - 連合学習に参加し、他の参加者が所有するデータを復元することが困難\n","  - 生データそのものが交換されれば、それぞれ独立したモデルの融合となる\n","- 参加者自身が所有するデータのみで学習させるモデルよりも連合学習の方が高くなる\n","  - ローカルのみの方が高性能であれば連合学習は不要である\n","- モデルの性能はすべてのデータを集約した際の性能に近くなる\n","  - 連合学習の方が高性能ならデータを集約する必要はない\n","\n","なお、連合学習の他、モデルを分割する分散学習も存在し、それらの特徴は、情報通信ネットワークを用いて、学習の計算過程やパラメータ更新などを交換することである\n","- 従って、クライアント・サーバモデルを前提とする場合が多い\n","  - 様々なリモートで情報を交換するため、セキュリティ上の問題が発生する\n","    - サーバが公開してはならないクライアントの情報を公開する\n","    - クライアントが粗悪な学習を行った結果、全体の性能も低下する\n","\n"],"metadata":{"id":"aSrLcM-UZyyl"}},{"cell_type":"markdown","metadata":{"id":"gd-vX3cavOCt"},"source":["# セキュリティ\n","\n","機械学習のセキュリティは大きく２つの考え方がある\n","\n","- セキュア機械学習  \n","攻撃者は機械学習システムを破壊する\n","  - 整合性：モデルの出力を改変し誤った出力とする\n","  - 可用性：モデルパラメータを使用できないレベルに改変する\n","- プライバシ保護機械学習(連合学習のターゲット)  \n","  攻撃者は機械学習システムのプライバシを侵害する\n","  - プライバシ：データ提供者の個人情報を特定する\n","  - 機密性：モデル情報を流出させる"]},{"cell_type":"markdown","source":["\n","## MLに関連するセキュリティ\n","\n","MLに関連するセキュリティについて、連合学習とは少し観点が異なるが、ここでまとめておく\n","\n","|  | プライバシ保護ML | セキュアML |\n","| :---: | :--- | :--- |\n","| セキュリティ違反 | - プライバシ | - 整合性 |\n","|  | - 機密性 | - 可用性 |\n","| 攻撃 | - 再構築攻撃 | - モデルポイズニング攻撃 |\n","|  | - モデル反転攻撃 | - 敵対的攻撃 |\n","|  | - メンバーシップ推論攻撃 | - 中間者攻撃 |\n","| 防御手法 | - MPC | - 防御的蒸留 |\n","|  | - 準同型暗号 | - 敵対的学習 |\n","|  | - 差分プライバシ | - 正則化 |\n","\n","\n","- 再構築攻撃: ランダムに初期化した乱数データを入力して勾配情報を獲得、勾配を基に窃取したいデータが属するクラスに対する入力データの誤差が小さくなる方向に入力データを変更することで特定に近づける\n","- モデル反転攻撃: モデルの入力と出力を反転させる手法で、モデルが入手できている場合は直接勾配降下法で入力データ特定に向けた最適化が可能\n","- メンバーシップ推論攻撃: 学習データ分布を予測し、「学習データに含まれていると思われるデータ」と「含まれていないと思われるデータ」を用意、その分類スコアを獲得、さらにそのスコアを入力としてメンバーかどうかを独自に構築した攻撃用ネットワークで学習させることで、どのような分類の入力が利用されたかを特定する\n","- モデルポイズニング攻撃: モデルが誤った推論を行うように、誤った入力を投入させる\n","- 敵対的攻撃: 本来のデータと区別がつきにくい敵対例に対して、正しいデータが獲得できないようなデータを提示することで、その解決法としてReLUのような微分不可能な点を含まない活性化関数を利用することが挙げられている\n","\n","<img src=\"http://class.west.sd.keio.ac.jp/dataai/text/adversarial_example.png\" width=500>\n","\n","- MPC(Multi-party Computation): 複数人感での通信\n","- 準同型暗号: 暗号化したまま計算や検索ができる暗号化方式\n","- 差分プライバシ: ある個人のデータを含むデータセットについて、それと隣接(ある特定の部分しか違わない)データセットと区別できないようにデータを加工する匿名化手法で、一般にデータにラプラスノイズを加えて(ラプラスメカニズム)匿名化する\n","- 防御的蒸留: 学習済みモデルの入力と出力をよりシンプルなアーキテクチャを持ったモデルで学習させて比較する\n","- 敵対的学習: 正しいラベルと紐づけた敵対的サンプルを学習データに加えて学習を行う防御手法\n","- 正則化: 情報を追加する、ペナルティを与えるなどした情報の加工により過学習を防ぐ\n","\n","その他、勾配マスキングと呼ばれる、学習時に入力データに対する微分値を最小化（難読化）することで、敵対的サンプルの効果を低減させる防御手法がある"],"metadata":{"id":"pIiBZgFMjDSr"}},{"cell_type":"markdown","source":["## 攻撃について"],"metadata":{"id":"mRojpN-mjX-K"}},{"cell_type":"markdown","source":["### 想定される攻撃者像\n","\n","- 正直者  \n","基本的に無害\n","- “semi-honest”, “honest-but-curious”, “半正直”な攻撃者  \n","  - プロトコルやルールは守る\n","  - 自身が取得できない情報を不正に知ろうとする\n","- 悪意のある攻撃者\n","  - プロトコルやルールから逸脱する\n","  - システム破壊を試みる"],"metadata":{"id":"zWk-tjzDjbQp"}},{"cell_type":"markdown","source":["### 機械学習システムへのプライバシ脅威モデル\n","- 再構築攻撃\n","  - 学習や推論時に生データや特徴ベクトルを再構築  \n","ex. 勾配情報から学習データを復元\n","- モデル反転攻撃  \n","  - 様々なデータをモデルに入力して学習データやモデル構造を再構築する\n","- メンバーシップ推論攻撃\n","  - 特定の標本が学習データに含まれているかを調べる\n","- 属性推論攻撃\n","  - 公開データの中から個人(標本)を識別することで、個人の属性を入手する\n","    - 強力な背景知識があれば匿名化されていても個人を特定可能となる場合がある\n","- モデルポイズニング攻撃\n","  - 学習処理を解析、攻撃してモデルの更新をコントロールする\n","    - ex. 自分の更新だけ強く反映させる\n","  - 連合学習は特にこの点での脆弱性に対する対応が必要となる"],"metadata":{"id":"vCC1IxwlGvTQ"}},{"cell_type":"markdown","source":["# 分散深層学習"],"metadata":{"id":"7QR6_px4HOjv"}},{"cell_type":"markdown","source":["## 分散機械学習の目的\n","\n","スケーラビリティの獲得\n","\n","- 機械学習をスケールさせる\n","  - 大規模化しても性能が追従して向上してほしい\n","- 分散処理\n","  - メモリ不足に対応する\n","- データを水平分割し複数ノードで同時に並列して学習を行う\n","  - 学習時間を短縮する\n","- プライバシ保護\n","  - データ所有者がデータ公開を望まない\n","- データ公開ではない方法で協調し学習を達成したい\n","  - 分散処理\n"],"metadata":{"id":"PAEgfJhpqM8_"}},{"cell_type":"markdown","source":["## 対応手法\n","- データ並列\n","  - 巨大な学習データを水平分割し並列学習を行う\n","- ノード間の学習を集約する\n","  - モデル並列\n","    - 巨大なモデルを分割し複数GPUを利用して学習する\n","    - パイプライン化してスループットを向上させる\n","- グラフ並列\n","  - Graph Neural Network(GNN)について並列化する\n","    - データの関連をネットワークで表現したグラフデータに深層学習を取り入れたモデル\n","    - RNNはunfoldして直列につながったGNNとみなすことができ、これをより複雑なネットワークにする\n","    - グラフとしての入力データからデータグループ間の結合を獲得して学習を進める\n","- タスク並列\n","  - タスクを分割して並列実行する\n","  - 複数GPUやGPU・CPU協調など\n","\n"],"metadata":{"id":"qBtDKZFNsIey"}},{"cell_type":"markdown","source":["プライバシ保護が目的の分散機械学習\n","\n","何を秘匿したいのか（どれか，もしくは全部）\n","入力の学習データ\n","出力の予測ラベル\n","モデル情報\n","メタデータ（データと所有者の関係性とか）\n","データの分割\n","水平分割\n","標本は異なるが，属性は同じ\n","ex. 全ての参加者が同じビジネスをしている\n","垂直分割\n","標本は同じだが，属性が異なる\n","ex. 参加者は異なるビジネスをしている\n","\n"],"metadata":{"id":"8bVbaV3vqRSa"}},{"cell_type":"markdown","source":["## プライバシ保護分散機械学習技術\n","\n","- 匿名化\n","  - 差分プライバシなどで匿名化されたデータを使用する\n","    - 比較的実装が容易\n","    - データのみ匿名化される\n","    - 匿名化によりモデル性能が低下する\n","- 秘密計算による暗号化\n","  - 秘密計算などを用いて分散機械学習を行う\n","  - 計算効率が低下する\n","- プライバシ保護勾配降下法\n","  - 代数的アプローチ\n","    - 行列計算の代数的性質を利用する\n","    - その内容は秘密計算に近しい\n","  - 疎な勾配更新\n","    - 勾配の一部のみを共有する\n","  - 勾配を通信する際に差分プライバシを施す\n","  - 勾配を量子化する\n","- Naïveな連合学習\n","  - FedAvg\n","\n","すべての問題を解決する手法はなく次の点でトレードオフとなる\n","- モデル性能\n","- 計算コスト\n","- 通信コスト\n","\n"],"metadata":{"id":"aUHFETI5qWJs"}},{"cell_type":"markdown","source":["## Naïveな連合学習\n","\n","FedAvgがもっともよく知られている\n","\n","- 想定環境\n","  - データが水平に分割されている\n","  - 全ての参加者で同じタスクを行う\n","- FedAvgの方法\n","  - 参加者がそれぞれのモデルとデータでモデルの更新を行う\n","  - 参加者は勾配(モデルの更新情報)をコーディネータ(サーバ)に送る\n","  - コーディネータは集約した勾配もしくはモデルの加重平均を用いてグローバルモデルを更新する\n","  - コーディネータはグローバルモデルを参加者に配布する\n","  - 上記を繰り返す\n","\n","- FedAvgにおけるプライバシの定義\n","  - 参加者が保持する生データが開示されなければ良い\n","    - ただし、勾配から生データ情報をおおよそ復元可能であることが実証されている\n","  - コーディネータは不正をしない\n","    - 実際にはコーディネータが攻撃者になり勾配を公開する可能性が否定できない\n","\n"],"metadata":{"id":"CrOmJXR6qaaA"}},{"cell_type":"markdown","source":["# 水平連合学習\n","\n","<img src=\"http://class.west.sd.keio.ac.jp/dataai/text/DataDistribution_Horizontal.png\" width=300>\n","\n","- 特徴\n","  - データが水平に分割される\n","    - 赤い部分が水平に区切られる\n","  - 標本は異なるが、属性は同じ\n","\n","- 具体例(次ような場合)\n","  - 参加者は地方銀行で似たカラムのDBを保持\n","  - 参加者はそれぞれの地域でサービスを提供\n","  - サービスの利用者(標本)は異なる\n","\n","- 水平連合学習のアルゴリズム\n","  - 先述のFedAvg\n","  - サーバクライアント型の連合学習\n","  - FedAvgでは何を平均化するのか？\n","    - モデル平均も勾配平均も両方ともFedAvgと呼ばれる\n","    - 参加者間でモデル初期値が同じであり、ローカルモデルの更新が行われると必ずモデル集約が行われる場合は、モデル平均と勾配平均が等価となる\n","\n","- 水平連合学習の課題\n","  - 設計者がデータを見ることができない\n","  - クライアントのデータが秘匿されているのでテスト環境を用意するのが難しい\n","  - ハイパーパラメータのチューニングのためなどに何回も実験することが困難\n","  - 参加者集める方法の検討が必要\n","  - 参加者が同じビジネスをしている場合は競合と協力することになるため契約上の問題が発生する可能性がある\n","  - インセンティブ設計が必要\n","  - 参加者の不正行為を防ぐ手法が必要\n"],"metadata":{"id":"gpAGtR1wqhMY"}},{"cell_type":"markdown","source":["-- 垂直連合学習\n","\n","<img src=\"http://class.west.sd.keio.ac.jp/dataai/text/DataDistribution_Vertical.png\" width=300>\n","\n","- 特徴\n","  - データが垂直に分割される\n","  - 標本は同じであるが、属性は異なる\n","\n","- 具体例(次のような場合)\n","  - 参加者は銀行と小売店で保持しているDBのカラムは異なる\n","  - 同じ個人がそれぞれのサービスを利用\n","  - 標本は同じ\n","\n","- 垂直連合学習のアルゴリズム\n","  - 連合線形回帰\n","    - 連合学習で線形回帰を実現\n","  - 準同型暗号を使用して参加者が保有するデータのプライバシを守る\n","    - SecureBoost\n","  - 連合学習でGBDTを実現\n","      - GBDTとはGradient(勾配)、Boosting(ブースティング) 、Decision(決定)、Tree(決定木であるDicision Tree)を組み合わせた手法のこと\n","\n","- 垂直連合学習の課題\n","  - 参加者の相互依存が大きい\n","  - 学習の途中で中間結果をやり取りすることが多い\n","  - 参加者ごとに持っているデータの属性が異なる\n","    - 参加者の離脱により連合学習が破綻しうる\n","    - 耐障害性が大事\n","\n","連合学習のプライバシ保護や攻撃に関する研究のほとんどが水平連合学習である"],"metadata":{"id":"oIRR6nNQqt7N"}},{"cell_type":"markdown","source":["# Federated Optimization（連合最適化）\n","\n","次のような分散最適化がある\n","- Non-IID: ユーザごとに持つデータの分布が異なる\n","- Unbalanced: ユーザによってデータ量が異なる\n","- Massively distributed:ユーザ数に比べて、ユーザごとのサンプルサイズが極めて少ない場合\n","- Limited communication: 通信量を削減する"],"metadata":{"id":"1MOQIbfgqy7i"}},{"cell_type":"markdown","source":["## Personalized Federated Learning\n","\n","- モデルをクライアントごとにPersonalizeする\n","- クライアント間のデータ分布の差異に対応\n","- さまざまな手法が提案されている\n","  - MAML問題の分散型手法\n","  - ヘッセ行列を近似する\n","  - モデルを組み合わせる手法\n","  - 共有部分とPersonalize部分に分けてそれぞれ学習させる手法\n","  - グローバルな部分をFedAvg，ローカルな部分をローカルで学習させる手法\n","  - 各クライアントがローカルな特徴抽出器とグローバルな出力を持つ手法\n","  - グローバルモデルとローカルモデルを異なる正則化で学習させる手法\n","  - データ分布が似ているクライクライアントをグループ化する手法\n","  - データ分布が近いクライアント同士で連携する手法"],"metadata":{"id":"UNvQ_n-d9itM"}},{"cell_type":"markdown","source":["Federated Learning(FL)を実際に実装する\n","\n","FLはネットワークでモデルを結合するため、普通に実装するとネットワークプログラミングが必要となる\n","\n","PyTorchには分散学習に関するコマンドが備わっているが、ネットワークプログラミングの知識は別途必要となる\n","\n","そこで、ここではFLライブラリを用いず、PyTorchの基本機能を用いて概念を理解する\n","- 通信部分は含んでおらず、仮想環境（リモート環境）を利用するような実装にはなっていない"],"metadata":{"id":"TuBgAgQbSw_V"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"QwGLl2JnOq3w"},"outputs":[],"source":["import copy\n","from logging import Formatter, StreamHandler, getLogger\n","import os\n","import random\n","from typing import Dict, List\n","\n","import numpy as np\n","import torch\n","from torch import nn\n","from torch import optim\n","from torch.utils.data import DataLoader\n","from torch.utils.data.dataset import Subset\n","import torchvision\n","import torchvision.transforms as transforms"]},{"cell_type":"markdown","source":["パラメータを設定する"],"metadata":{"id":"RVZ_pceMnNl-"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"KhtPaTqPOq3y"},"outputs":[],"source":["config = {\n","    \"batch_size\": 128,\n","    \"device\": \"cuda:0\",\n","    \"epochs\": 5,\n","    \"learning_rate\": 0.0001,\n","    \"num_clients\": 5,\n","    \"rounds\": 5,\n","    \"seed\": 42,\n","}"]},{"cell_type":"markdown","source":["ログの取得および乱数のシードに関する設定を行う"],"metadata":{"id":"1WbFOBajneyA"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"WBp1xNHLOq3y"},"outputs":[],"source":["def get_logger(name=\"EXP\"):\n","    log_fmt = Formatter(f\"%(asctime)s [{name}][%(levelname)s] %(message)s \")\n","    logger = getLogger(__name__)\n","    handler = StreamHandler()\n","    handler.setLevel(\"INFO\")\n","    handler.setFormatter(log_fmt)\n","    logger.setLevel(\"INFO\")\n","    logger.addHandler(handler)\n","    logger.propagate = False\n","    return logger\n","\n","def seed_everything(seed=1234):\n","    random.seed(seed)\n","    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n","    np.random.seed(seed)\n","    torch.manual_seed(seed)\n","    torch.cuda.manual_seed(seed)\n","    torch.backends.cudnn.deterministic = True\n","    torch.backends.cudnn.benchmark = True"]},{"cell_type":"markdown","source":["専用のデータローダを設定する"],"metadata":{"id":"tB16C2nonlWw"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"Jo1CG5W_Oq3z"},"outputs":[],"source":["class FedAvgRetriever():\n","    def __init__(self, config: Dict) -> None:\n","        self.config = config\n","\n","        # init CIFAR-10\n","        transform = transforms.Compose([\n","            transforms.ToTensor(),\n","            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n","        ])\n","        self.train_set = torchvision.datasets.CIFAR10(\n","            root='./data', train=True, download=True, transform=transform)\n","        self.test_set = torchvision.datasets.CIFAR10(\n","            root='./data', train=False, download=True, transform=transform)\n","\n","    def get(self, num_clients: int) -> List[Dict[str, DataLoader]]:\n","        train_size = len(self.train_set) // num_clients\n","        test_size = len(self.test_set) // num_clients\n","\n","        dataloaders = []\n","        for i in range(num_clients):\n","            train_idx = range(i * train_size, (i + 1) * train_size)\n","            test_idx = range(i * test_size, (i + 1) * test_size)\n","\n","            train_set = Subset(self.train_set, train_idx)\n","            test_set = Subset(self.test_set, test_idx)\n","\n","            train_loader = DataLoader(\n","                train_set, batch_size=self.config[\"batch_size\"], shuffle=True, num_workers=4)\n","            test_loader = DataLoader(\n","                test_set, batch_size=self.config[\"batch_size\"], shuffle=False, num_workers=4)\n","            dataloaders.append({\n","                \"train\": train_loader,\n","                \"test\": test_loader\n","            })\n","\n","        return dataloaders"]},{"cell_type":"markdown","source":["シンプルなMNISTの分類モデルを記述する\n","- ここでは精度は問わずシンプルな設計とする"],"metadata":{"id":"vKIK2jJcn4yA"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"dspaQqXpOq3z"},"outputs":[],"source":["class CNNModel(nn.Module):\n","    def __init__(self):\n","        super(CNNModel, self).__init__()\n","        self.relu = nn.ReLU()\n","        self.pool = nn.MaxPool2d(2)\n","\n","        self.conv1 = nn.Conv2d(3, 16, 3)\n","        self.conv2 = nn.Conv2d(16, 128, 3)\n","\n","        self.fc1 = nn.Linear(128, 64)\n","        self.fc2 = nn.Linear(64, 10)\n","\n","    def forward(self, x):\n","        x = self.conv1(x)\n","        x = self.relu(x)\n","        x = self.pool(x)\n","        x = self.conv2(x)\n","        x = self.relu(x)\n","\n","        x_size = x.size()\n","        x = x.reshape(x_size[0], -1, x_size[2] ** 2).mean(2)\n","\n","        x = self.fc1(x)\n","        x = self.relu(x)\n","        x = self.fc2(x)\n","        return x"]},{"cell_type":"markdown","source":["クライアントモデルを記述する"],"metadata":{"id":"3mp8bXfQsyAU"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"IQeORnO9Oq3z"},"outputs":[],"source":["class FedAvgClient():\n","    def __init__(self, config: Dict, logger, dataloaders: Dict[str, DataLoader]) -> None:\n","        self.config = config\n","        self.train_loader = dataloaders[\"train\"]\n","        self.test_loader = dataloaders[\"test\"]\n","\n","        self.model: CNNModel = None  # type: ignore\n","\n","    def set_global_model(self, global_model: CNNModel):\n","        del self.model\n","        self.model = copy.deepcopy(global_model)\n","\n","    def train_local_model(self) -> CNNModel:\n","        criterion = nn.CrossEntropyLoss()\n","        optimizer = optim.Adam(self.model.parameters(),\n","                               lr=self.config[\"learning_rate\"])\n","\n","        def _train_loop():\n","            running_loss = 0\n","            for i, (X, y) in enumerate(self.train_loader):\n","                X = X.to(self.config[\"device\"])\n","                y = y.to(self.config[\"device\"])\n","                optimizer.zero_grad()\n","                outputs = self.model(X)\n","                loss = criterion(outputs, y)\n","                running_loss += loss.item()\n","                loss.backward()\n","                optimizer.step()\n","            return running_loss / len(self.train_loader)\n","\n","        def _test_loop():\n","            labels = []\n","            preds = []\n","            running_loss = 0\n","            with torch.no_grad():\n","                for i, (X, y) in enumerate(self.test_loader):\n","                    X = X.to(self.config[\"device\"])\n","                    y = y.to(self.config[\"device\"])\n","                    output = self.model(X)\n","                    loss = criterion(output, y)\n","                    running_loss += loss.item()\n","                    labels.append(y.detach().cpu())\n","                    preds.append(output.detach().cpu())\n","            labels = torch.cat(labels, dim=0)\n","            preds = torch.cat(preds, dim=0).argmax(1)\n","            acc = torch.sum(preds == labels) / len(labels)\n","            return running_loss / len(self.test_loader), acc.item()\n","\n","        self.model.to(self.config[\"device\"])\n","        for epoch in range(self.config[\"epochs\"]):\n","            self.model.train()\n","            train_loss = _train_loop()\n","            self.model.eval()\n","            test_loss, acc = _test_loop()\n","\n","            logger.info(f\"train_loss: {train_loss}, test_loss: {test_loss}, acc: {acc}\")\n","\n","        return self.model"]},{"cell_type":"markdown","source":["サーバ側を記述する\n","- モデルのパラメータを取得し、その平均をとって配布する\n","- モデルパラメータの取得と設定はすでに学んだ方法である"],"metadata":{"id":"zMMMKFqQtF-g"}},{"cell_type":"code","source":["class FedAvgServer():\n","    def __init__(self, config: Dict) -> None:\n","        self.config = config\n","\n","    def init_global_model(self) -> CNNModel:\n","        global_model = CNNModel()\n","        return global_model\n","\n","    def aggregate(self, local_models: List[CNNModel]) -> CNNModel:\n","        global_model = CNNModel().to(self.config[\"device\"])\n","\n","        # conv 1\n","        global_model.conv1.weight.data = torch.zeros_like(\n","            global_model.conv1.weight.data)\n","        for local_model in local_models:\n","            global_model.conv1.weight.data += local_model.conv1.weight.data\n","        global_model.conv1.weight.data /= len(local_models)\n","\n","        # conv 2\n","        global_model.conv2.weight.data = torch.zeros_like(\n","            global_model.conv2.weight.data)\n","        for local_model in local_models:\n","            global_model.conv2.weight.data += local_model.conv2.weight.data\n","        global_model.conv2.weight.data /= len(local_models)\n","\n","        # fc1\n","        global_model.fc1.weight.data = torch.zeros_like(\n","            global_model.fc1.weight.data)\n","        for local_model in local_models:\n","            global_model.fc1.weight.data += local_model.fc1.weight.data\n","        global_model.fc1.weight.data /= len(local_models)\n","\n","        # fc2\n","        global_model.fc2.weight.data = torch.zeros_like(\n","            global_model.fc2.weight.data)\n","        for local_model in local_models:\n","            global_model.fc2.weight.data += local_model.fc2.weight.data\n","        global_model.fc2.weight.data /= len(local_models)\n","\n","        return global_model"],"metadata":{"id":"5X1lhlRQtJqg"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["実際に学習を行う\n","- ここでは、クライアント数は5としている"],"metadata":{"id":"anFMxHe4tUow"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"RaLzThPEOq30","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1692459334309,"user_tz":-540,"elapsed":418004,"user":{"displayName":"西宏章","userId":"00237858890977261979"}},"outputId":"3a02be13-46b4-4522-f074-90129a9a59ca"},"outputs":[{"output_type":"stream","name":"stderr","text":["2023-08-19 15:28:24,617 [EXP][INFO] FedAvg start \n"]},{"output_type":"stream","name":"stdout","text":["Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./data/cifar-10-python.tar.gz\n"]},{"output_type":"stream","name":"stderr","text":["100%|██████████| 170498071/170498071 [00:13<00:00, 12188829.60it/s]\n"]},{"output_type":"stream","name":"stdout","text":["Extracting ./data/cifar-10-python.tar.gz to ./data\n","Files already downloaded and verified\n"]},{"output_type":"stream","name":"stderr","text":["/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n","  warnings.warn(_create_warning_msg(\n","2023-08-19 15:28:43,316 [EXP][INFO] Round: 0 start \n","2023-08-19 15:28:43,317 [EXP][INFO] train client: 0 \n","2023-08-19 15:28:49,862 [EXP][INFO] train_loss: 2.29698503470119, test_loss: 2.2834156304597855, acc: 0.12849999964237213 \n","2023-08-19 15:28:52,674 [EXP][INFO] train_loss: 2.2623536405684073, test_loss: 2.231324553489685, acc: 0.19300000369548798 \n","2023-08-19 15:28:55,464 [EXP][INFO] train_loss: 2.1914314348486403, test_loss: 2.1465223729610443, acc: 0.21050000190734863 \n","2023-08-19 15:28:58,241 [EXP][INFO] train_loss: 2.112358381476583, test_loss: 2.0831671357154846, acc: 0.22599999606609344 \n","2023-08-19 15:29:02,077 [EXP][INFO] train_loss: 2.0663266378112985, test_loss: 2.04762352257967, acc: 0.24650000035762787 \n","2023-08-19 15:29:02,083 [EXP][INFO] train client: 1 \n","2023-08-19 15:29:04,946 [EXP][INFO] train_loss: 2.2969547555416443, test_loss: 2.2827677577733994, acc: 0.1185000017285347 \n","2023-08-19 15:29:07,820 [EXP][INFO] train_loss: 2.265543264678762, test_loss: 2.2365413159132004, acc: 0.15600000321865082 \n","2023-08-19 15:29:10,602 [EXP][INFO] train_loss: 2.200262124025369, test_loss: 2.158265084028244, acc: 0.18700000643730164 \n","2023-08-19 15:29:14,441 [EXP][INFO] train_loss: 2.122735065749929, test_loss: 2.0904496014118195, acc: 0.2070000022649765 \n","2023-08-19 15:29:17,367 [EXP][INFO] train_loss: 2.0719903662234924, test_loss: 2.0527751967310905, acc: 0.2240000069141388 \n","2023-08-19 15:29:17,372 [EXP][INFO] train client: 2 \n","2023-08-19 15:29:20,154 [EXP][INFO] train_loss: 2.295381971552402, test_loss: 2.283901497721672, acc: 0.10750000178813934 \n","2023-08-19 15:29:22,921 [EXP][INFO] train_loss: 2.2619625677036335, test_loss: 2.2357766777276993, acc: 0.15449999272823334 \n","2023-08-19 15:29:27,224 [EXP][INFO] train_loss: 2.193174024171467, test_loss: 2.155044600367546, acc: 0.20250000059604645 \n","2023-08-19 15:29:30,255 [EXP][INFO] train_loss: 2.1134957784338844, test_loss: 2.0916464626789093, acc: 0.21549999713897705 \n","2023-08-19 15:29:33,073 [EXP][INFO] train_loss: 2.056852825080292, test_loss: 2.0488977134227753, acc: 0.2460000067949295 \n","2023-08-19 15:29:33,077 [EXP][INFO] train client: 3 \n","2023-08-19 15:29:35,860 [EXP][INFO] train_loss: 2.2988234320773353, test_loss: 2.284667730331421, acc: 0.12250000238418579 \n","2023-08-19 15:29:39,348 [EXP][INFO] train_loss: 2.2691726443133775, test_loss: 2.235634297132492, acc: 0.18700000643730164 \n","2023-08-19 15:29:42,622 [EXP][INFO] train_loss: 2.203475647334811, test_loss: 2.1445652842521667, acc: 0.22550000250339508 \n","2023-08-19 15:29:45,467 [EXP][INFO] train_loss: 2.1218627238575416, test_loss: 2.07366743683815, acc: 0.23199999332427979 \n","2023-08-19 15:29:48,312 [EXP][INFO] train_loss: 2.070873983298676, test_loss: 2.0355022847652435, acc: 0.2485000044107437 \n","2023-08-19 15:29:48,314 [EXP][INFO] train client: 4 \n","2023-08-19 15:29:51,711 [EXP][INFO] train_loss: 2.2961907386779785, test_loss: 2.2828347086906433, acc: 0.13249999284744263 \n","2023-08-19 15:29:54,910 [EXP][INFO] train_loss: 2.261481034604809, test_loss: 2.23083533346653, acc: 0.17749999463558197 \n","2023-08-19 15:29:57,813 [EXP][INFO] train_loss: 2.1960067145432096, test_loss: 2.1434985995292664, acc: 0.2084999978542328 \n","2023-08-19 15:30:00,673 [EXP][INFO] train_loss: 2.122477087793471, test_loss: 2.073565199971199, acc: 0.24449999630451202 \n","2023-08-19 15:30:03,975 [EXP][INFO] train_loss: 2.067606266540817, test_loss: 2.0277160555124283, acc: 0.25600001215934753 \n","2023-08-19 15:30:03,981 [EXP][INFO] Aggregate global model \n","2023-08-19 15:30:03,988 [EXP][INFO] Distribute global model \n","2023-08-19 15:30:03,998 [EXP][INFO] Round: 1 start \n","2023-08-19 15:30:03,999 [EXP][INFO] train client: 0 \n","2023-08-19 15:30:07,471 [EXP][INFO] train_loss: 2.056095649924459, test_loss: 2.0417901128530502, acc: 0.25099998712539673 \n","2023-08-19 15:30:10,311 [EXP][INFO] train_loss: 2.033187401445606, test_loss: 2.0252261757850647, acc: 0.2644999921321869 \n","2023-08-19 15:30:13,240 [EXP][INFO] train_loss: 2.0159195465377615, test_loss: 2.0130587741732597, acc: 0.2639999985694885 \n","2023-08-19 15:30:16,594 [EXP][INFO] train_loss: 1.9997472717792173, test_loss: 1.9954518377780914, acc: 0.27300000190734863 \n","2023-08-19 15:30:19,965 [EXP][INFO] train_loss: 1.9841318039954463, test_loss: 1.9818083718419075, acc: 0.2784999907016754 \n","2023-08-19 15:30:19,969 [EXP][INFO] train client: 1 \n","2023-08-19 15:30:22,848 [EXP][INFO] train_loss: 2.0620275011545495, test_loss: 2.0516035929322243, acc: 0.23399999737739563 \n","2023-08-19 15:30:25,728 [EXP][INFO] train_loss: 2.040223690527904, test_loss: 2.034414194524288, acc: 0.24250000715255737 \n","2023-08-19 15:30:28,979 [EXP][INFO] train_loss: 2.024984035310866, test_loss: 2.018737815320492, acc: 0.24150000512599945 \n","2023-08-19 15:30:32,361 [EXP][INFO] train_loss: 2.01164923438543, test_loss: 2.0044284611940384, acc: 0.24899999797344208 \n","2023-08-19 15:30:35,201 [EXP][INFO] train_loss: 1.9971907787685153, test_loss: 1.989967241883278, acc: 0.2694999873638153 \n","2023-08-19 15:30:35,204 [EXP][INFO] train client: 2 \n","2023-08-19 15:30:38,024 [EXP][INFO] train_loss: 2.0474460124969482, test_loss: 2.046607494354248, acc: 0.2524999976158142 \n","2023-08-19 15:30:41,136 [EXP][INFO] train_loss: 2.0218982485276236, test_loss: 2.026445634663105, acc: 0.2549999952316284 \n","2023-08-19 15:30:44,705 [EXP][INFO] train_loss: 2.0058107149751883, test_loss: 2.007997713983059, acc: 0.2865000069141388 \n","2023-08-19 15:30:47,508 [EXP][INFO] train_loss: 1.988010693200027, test_loss: 1.9945377483963966, acc: 0.27799999713897705 \n","2023-08-19 15:30:50,358 [EXP][INFO] train_loss: 1.9737177740169476, test_loss: 1.9770801812410355, acc: 0.2705000042915344 \n","2023-08-19 15:30:50,361 [EXP][INFO] train client: 3 \n","2023-08-19 15:30:53,380 [EXP][INFO] train_loss: 2.063877695723425, test_loss: 2.0261541679501534, acc: 0.24449999630451202 \n","2023-08-19 15:30:57,056 [EXP][INFO] train_loss: 2.0385638882842243, test_loss: 2.0088801085948944, acc: 0.26100000739097595 \n","2023-08-19 15:30:59,934 [EXP][INFO] train_loss: 2.0253996833970276, test_loss: 1.9933618605136871, acc: 0.2605000138282776 \n","2023-08-19 15:31:02,811 [EXP][INFO] train_loss: 2.015147652807115, test_loss: 1.9784535244107246, acc: 0.26499998569488525 \n","2023-08-19 15:31:05,743 [EXP][INFO] train_loss: 1.9983147065850753, test_loss: 1.9631916731595993, acc: 0.27000001072883606 \n","2023-08-19 15:31:05,747 [EXP][INFO] train client: 4 \n","2023-08-19 15:31:09,550 [EXP][INFO] train_loss: 2.0609196258496634, test_loss: 2.018537975847721, acc: 0.2549999952316284 \n","2023-08-19 15:31:12,400 [EXP][INFO] train_loss: 2.0371444859082186, test_loss: 2.00210127979517, acc: 0.24449999630451202 \n","2023-08-19 15:31:15,298 [EXP][INFO] train_loss: 2.02171904829484, test_loss: 1.9876553490757942, acc: 0.2605000138282776 \n","2023-08-19 15:31:18,144 [EXP][INFO] train_loss: 2.0074264097817336, test_loss: 1.9740401655435562, acc: 0.2605000138282776 \n","2023-08-19 15:31:22,157 [EXP][INFO] train_loss: 1.9883738182768036, test_loss: 1.9577223286032677, acc: 0.265500009059906 \n","2023-08-19 15:31:22,159 [EXP][INFO] Aggregate global model \n","2023-08-19 15:31:22,167 [EXP][INFO] Distribute global model \n","2023-08-19 15:31:22,173 [EXP][INFO] Round: 2 start \n","2023-08-19 15:31:22,174 [EXP][INFO] train client: 0 \n","2023-08-19 15:31:24,986 [EXP][INFO] train_loss: 2.0140087830869455, test_loss: 2.0050382390618324, acc: 0.26249998807907104 \n","2023-08-19 15:31:27,856 [EXP][INFO] train_loss: 1.989828971367848, test_loss: 1.9893902838230133, acc: 0.27149999141693115 \n","2023-08-19 15:31:30,742 [EXP][INFO] train_loss: 1.9735883429080625, test_loss: 1.9761405810713768, acc: 0.2840000092983246 \n","2023-08-19 15:31:34,666 [EXP][INFO] train_loss: 1.9599335329442085, test_loss: 1.963020846247673, acc: 0.2815000116825104 \n","2023-08-19 15:31:37,496 [EXP][INFO] train_loss: 1.9472006182127362, test_loss: 1.9518435150384903, acc: 0.2824999988079071 \n","2023-08-19 15:31:37,500 [EXP][INFO] train client: 1 \n","2023-08-19 15:31:40,444 [EXP][INFO] train_loss: 2.0238457284396207, test_loss: 2.0070073679089546, acc: 0.25949999690055847 \n","2023-08-19 15:31:43,311 [EXP][INFO] train_loss: 1.9989170409456085, test_loss: 1.9902749583125114, acc: 0.2590000033378601 \n","2023-08-19 15:31:47,203 [EXP][INFO] train_loss: 1.9871167351927939, test_loss: 1.9763717129826546, acc: 0.26350000500679016 \n","2023-08-19 15:31:50,075 [EXP][INFO] train_loss: 1.970614164690428, test_loss: 1.9675609767436981, acc: 0.2644999921321869 \n","2023-08-19 15:31:52,916 [EXP][INFO] train_loss: 1.961304403558562, test_loss: 1.9516838490962982, acc: 0.2775000035762787 \n","2023-08-19 15:31:52,921 [EXP][INFO] train client: 2 \n","2023-08-19 15:31:55,753 [EXP][INFO] train_loss: 2.0038577559628066, test_loss: 2.000324785709381, acc: 0.26499998569488525 \n","2023-08-19 15:31:59,451 [EXP][INFO] train_loss: 1.97788537001308, test_loss: 1.9862018823623657, acc: 0.27799999713897705 \n","2023-08-19 15:32:02,394 [EXP][INFO] train_loss: 1.9647523255287846, test_loss: 1.9702641293406487, acc: 0.2865000069141388 \n","2023-08-19 15:32:05,717 [EXP][INFO] train_loss: 1.9493777253959752, test_loss: 1.9563117995858192, acc: 0.2854999899864197 \n","2023-08-19 15:32:09,192 [EXP][INFO] train_loss: 1.9398726526694963, test_loss: 1.943826675415039, acc: 0.2955000102519989 \n","2023-08-19 15:32:09,197 [EXP][INFO] train client: 3 \n","2023-08-19 15:32:13,134 [EXP][INFO] train_loss: 2.0233754447743864, test_loss: 1.9810544326901436, acc: 0.265500009059906 \n","2023-08-19 15:32:16,021 [EXP][INFO] train_loss: 1.9998370756076862, test_loss: 1.9671121761202812, acc: 0.2775000035762787 \n","2023-08-19 15:32:18,912 [EXP][INFO] train_loss: 1.987758072116707, test_loss: 1.9547612071037292, acc: 0.27000001072883606 \n","2023-08-19 15:32:21,778 [EXP][INFO] train_loss: 1.9760831337940843, test_loss: 1.9396707713603973, acc: 0.2944999933242798 \n","2023-08-19 15:32:25,679 [EXP][INFO] train_loss: 1.9638308271577087, test_loss: 1.929007612168789, acc: 0.27399998903274536 \n","2023-08-19 15:32:25,683 [EXP][INFO] train client: 4 \n","2023-08-19 15:32:28,611 [EXP][INFO] train_loss: 2.0186341397369962, test_loss: 1.9743024334311485, acc: 0.26649999618530273 \n","2023-08-19 15:32:31,488 [EXP][INFO] train_loss: 1.9908760952044138, test_loss: 1.96155696362257, acc: 0.2669999897480011 \n","2023-08-19 15:32:34,366 [EXP][INFO] train_loss: 1.9837321284451062, test_loss: 1.9498715251684189, acc: 0.25850000977516174 \n","2023-08-19 15:32:38,127 [EXP][INFO] train_loss: 1.9719152903255028, test_loss: 1.9394303485751152, acc: 0.2745000123977661 \n","2023-08-19 15:32:41,223 [EXP][INFO] train_loss: 1.9592153694056258, test_loss: 1.9317759200930595, acc: 0.2815000116825104 \n","2023-08-19 15:32:41,224 [EXP][INFO] Aggregate global model \n","2023-08-19 15:32:41,232 [EXP][INFO] Distribute global model \n","2023-08-19 15:32:41,239 [EXP][INFO] Round: 3 start \n","2023-08-19 15:32:41,240 [EXP][INFO] train client: 0 \n","2023-08-19 15:32:44,153 [EXP][INFO] train_loss: 1.9531313633616967, test_loss: 1.9533831775188446, acc: 0.28700000047683716 \n","2023-08-19 15:32:47,138 [EXP][INFO] train_loss: 1.9362058292461346, test_loss: 1.9410492405295372, acc: 0.2800000011920929 \n","2023-08-19 15:32:51,050 [EXP][INFO] train_loss: 1.9260662643215325, test_loss: 1.9312954470515251, acc: 0.29499998688697815 \n","2023-08-19 15:32:54,086 [EXP][INFO] train_loss: 1.9182495452180695, test_loss: 1.92277592420578, acc: 0.29249998927116394 \n","2023-08-19 15:32:56,974 [EXP][INFO] train_loss: 1.9097387534153611, test_loss: 1.911210097372532, acc: 0.3019999861717224 \n","2023-08-19 15:32:56,978 [EXP][INFO] train client: 1 \n","2023-08-19 15:32:59,920 [EXP][INFO] train_loss: 1.9706663949580132, test_loss: 1.9522471353411674, acc: 0.26750001311302185 \n","2023-08-19 15:33:03,754 [EXP][INFO] train_loss: 1.9529085370558728, test_loss: 1.943982519209385, acc: 0.28299999237060547 \n","2023-08-19 15:33:06,881 [EXP][INFO] train_loss: 1.944637552092347, test_loss: 1.9331250563263893, acc: 0.2809999883174896 \n","2023-08-19 15:33:09,854 [EXP][INFO] train_loss: 1.93566193610807, test_loss: 1.9240497425198555, acc: 0.2840000092983246 \n","2023-08-19 15:33:12,842 [EXP][INFO] train_loss: 1.9234874761557277, test_loss: 1.9168970361351967, acc: 0.2874999940395355 \n","2023-08-19 15:33:12,845 [EXP][INFO] train client: 2 \n","2023-08-19 15:33:16,847 [EXP][INFO] train_loss: 1.9457679519170448, test_loss: 1.9440011009573936, acc: 0.29100000858306885 \n","2023-08-19 15:33:19,847 [EXP][INFO] train_loss: 1.9255676903302157, test_loss: 1.9320867508649826, acc: 0.2980000078678131 \n","2023-08-19 15:33:22,830 [EXP][INFO] train_loss: 1.916139915019651, test_loss: 1.9216115921735764, acc: 0.29649999737739563 \n","2023-08-19 15:33:25,772 [EXP][INFO] train_loss: 1.9080292725864845, test_loss: 1.9118556529283524, acc: 0.296999990940094 \n","2023-08-19 15:33:29,667 [EXP][INFO] train_loss: 1.8988281126263775, test_loss: 1.904650904238224, acc: 0.3050000071525574 \n","2023-08-19 15:33:29,673 [EXP][INFO] train client: 3 \n","2023-08-19 15:33:32,784 [EXP][INFO] train_loss: 1.9666554671299608, test_loss: 1.9252692982554436, acc: 0.2865000069141388 \n","2023-08-19 15:33:35,736 [EXP][INFO] train_loss: 1.961793961404245, test_loss: 1.9147208482027054, acc: 0.30399999022483826 \n","2023-08-19 15:33:38,732 [EXP][INFO] train_loss: 1.9423235247406778, test_loss: 1.9085543006658554, acc: 0.289000004529953 \n","2023-08-19 15:33:42,706 [EXP][INFO] train_loss: 1.9389430676834494, test_loss: 1.895670011639595, acc: 0.31349998712539673 \n","2023-08-19 15:33:45,825 [EXP][INFO] train_loss: 1.9269460077527203, test_loss: 1.886974386870861, acc: 0.3154999911785126 \n","2023-08-19 15:33:45,830 [EXP][INFO] train client: 4 \n","2023-08-19 15:33:48,750 [EXP][INFO] train_loss: 1.957737943794154, test_loss: 1.9253881871700287, acc: 0.28700000047683716 \n","2023-08-19 15:33:51,716 [EXP][INFO] train_loss: 1.945895188971411, test_loss: 1.913097657263279, acc: 0.28949999809265137 \n","2023-08-19 15:33:55,640 [EXP][INFO] train_loss: 1.9348428445526316, test_loss: 1.904701478779316, acc: 0.28700000047683716 \n","2023-08-19 15:33:58,652 [EXP][INFO] train_loss: 1.9238520890851565, test_loss: 1.8957650735974312, acc: 0.28850001096725464 \n","2023-08-19 15:34:01,588 [EXP][INFO] train_loss: 1.9158143921743465, test_loss: 1.8854341730475426, acc: 0.30550000071525574 \n","2023-08-19 15:34:01,592 [EXP][INFO] Aggregate global model \n","2023-08-19 15:34:01,598 [EXP][INFO] Distribute global model \n","2023-08-19 15:34:01,605 [EXP][INFO] Round: 4 start \n","2023-08-19 15:34:01,606 [EXP][INFO] train client: 0 \n","2023-08-19 15:34:04,574 [EXP][INFO] train_loss: 1.9346223496183563, test_loss: 1.9208421930670738, acc: 0.2874999940395355 \n","2023-08-19 15:34:08,498 [EXP][INFO] train_loss: 1.9096770030033738, test_loss: 1.9140776917338371, acc: 0.2944999933242798 \n","2023-08-19 15:34:11,590 [EXP][INFO] train_loss: 1.9032344305062596, test_loss: 1.904666356742382, acc: 0.2955000102519989 \n","2023-08-19 15:34:14,553 [EXP][INFO] train_loss: 1.8933612560924096, test_loss: 1.8977190554141998, acc: 0.2944999933242798 \n","2023-08-19 15:34:17,565 [EXP][INFO] train_loss: 1.8886597609218163, test_loss: 1.8887428268790245, acc: 0.29499998688697815 \n","2023-08-19 15:34:17,569 [EXP][INFO] train client: 1 \n","2023-08-19 15:34:21,571 [EXP][INFO] train_loss: 1.9493921859354912, test_loss: 1.9322999343276024, acc: 0.2745000123977661 \n","2023-08-19 15:34:24,534 [EXP][INFO] train_loss: 1.9249186606346806, test_loss: 1.923433668911457, acc: 0.27799999713897705 \n","2023-08-19 15:34:27,478 [EXP][INFO] train_loss: 1.917476249646537, test_loss: 1.9158058166503906, acc: 0.2750000059604645 \n","2023-08-19 15:34:30,482 [EXP][INFO] train_loss: 1.9098865744433826, test_loss: 1.9069753363728523, acc: 0.29100000858306885 \n","2023-08-19 15:34:34,502 [EXP][INFO] train_loss: 1.9033046704304368, test_loss: 1.8979234099388123, acc: 0.28200000524520874 \n","2023-08-19 15:34:34,509 [EXP][INFO] train client: 2 \n","2023-08-19 15:34:37,513 [EXP][INFO] train_loss: 1.920441746711731, test_loss: 1.9167771488428116, acc: 0.2904999852180481 \n","2023-08-19 15:34:40,565 [EXP][INFO] train_loss: 1.9012920735757561, test_loss: 1.9055271744728088, acc: 0.30399999022483826 \n","2023-08-19 15:34:43,568 [EXP][INFO] train_loss: 1.8954899416694158, test_loss: 1.90816081315279, acc: 0.2994999885559082 \n","2023-08-19 15:34:47,636 [EXP][INFO] train_loss: 1.8815287698673298, test_loss: 1.889672502875328, acc: 0.31049999594688416 \n","2023-08-19 15:34:50,574 [EXP][INFO] train_loss: 1.875487234019026, test_loss: 1.8830309957265854, acc: 0.3100000023841858 \n","2023-08-19 15:34:50,576 [EXP][INFO] train client: 3 \n","2023-08-19 15:34:53,538 [EXP][INFO] train_loss: 1.959277699265299, test_loss: 1.9035794585943222, acc: 0.28600001335144043 \n","2023-08-19 15:34:56,510 [EXP][INFO] train_loss: 1.9342118290406238, test_loss: 1.8939088955521584, acc: 0.3009999990463257 \n","2023-08-19 15:35:00,596 [EXP][INFO] train_loss: 1.92575090142745, test_loss: 1.8874038755893707, acc: 0.3100000023841858 \n","2023-08-19 15:35:03,558 [EXP][INFO] train_loss: 1.9136072925374479, test_loss: 1.8811371326446533, acc: 0.30300000309944153 \n","2023-08-19 15:35:06,488 [EXP][INFO] train_loss: 1.914542240432546, test_loss: 1.8772813007235527, acc: 0.30300000309944153 \n","2023-08-19 15:35:06,490 [EXP][INFO] train client: 4 \n","2023-08-19 15:35:09,419 [EXP][INFO] train_loss: 1.9532752550100978, test_loss: 1.894366815686226, acc: 0.28949999809265137 \n","2023-08-19 15:35:13,600 [EXP][INFO] train_loss: 1.9217965497246272, test_loss: 1.8872556686401367, acc: 0.29100000858306885 \n","2023-08-19 15:35:16,631 [EXP][INFO] train_loss: 1.9182978959023198, test_loss: 1.8825026527047157, acc: 0.3070000112056732 \n","2023-08-19 15:35:19,594 [EXP][INFO] train_loss: 1.905833268467384, test_loss: 1.8741002976894379, acc: 0.3084999918937683 \n","2023-08-19 15:35:22,583 [EXP][INFO] train_loss: 1.9000905058051967, test_loss: 1.8681481257081032, acc: 0.30149999260902405 \n","2023-08-19 15:35:22,586 [EXP][INFO] Aggregate global model \n","2023-08-19 15:35:22,594 [EXP][INFO] Distribute global model \n","2023-08-19 15:35:22,601 [EXP][INFO] FedAvg finished \n"]}],"source":["seed_everything(config[\"seed\"])\n","logger = get_logger()\n","\n","logger.info(\"FedAvg start\")\n","\n","# init dataset\n","retriever = FedAvgRetriever(config)\n","dataloaders = retriever.get(config[\"num_clients\"])\n","\n","# init server and clients\n","server = FedAvgServer(config)\n","clients = [FedAvgClient(config, logger, dataloaders[i])\n","            for i in range(config[\"num_clients\"])]\n","\n","# init FedAvg\n","global_model = server.init_global_model()\n","for client in clients:\n","    client.set_global_model(global_model)\n","\n","# round loop\n","for round in range(config[\"rounds\"]):\n","\n","    logger.info(f\"Round: {round} start\")\n","\n","    local_models = []\n","    for i, client in enumerate(clients):\n","        logger.info(f\"train client: {i}\")\n","        local_model = client.train_local_model()\n","        local_models.append(local_model)\n","\n","    logger.info(f\"Aggregate global model\")\n","    global_model = server.aggregate(local_models)\n","    logger.info(f\"Distribute global model\")\n","    for client in clients:\n","        client.set_global_model(global_model)\n","\n","logger.info(\"FedAvg finished\")"]},{"cell_type":"markdown","source":["# 課題1\n","\n","精度を改善してみよう"],"metadata":{"id":"pvZYMdV7zbi7"}},{"cell_type":"markdown","source":["# 課題2\n","\n","3つ以上のFLに改造してみよう"],"metadata":{"id":"SVEz51A4zwwd"}}],"metadata":{"accelerator":"GPU","colab":{"provenance":[{"file_id":"https://github.com/keioNishi/lec-dataai/blob/main/dataai-text-J-Diffusion.ipynb","timestamp":1661617425151},{"file_id":"https://github.com/huggingface/notebooks/blob/main/diffusers/stable_diffusion.ipynb","timestamp":1661425026567}],"gpuType":"T4","toc_visible":true},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":0}