save model in global rank 0 in multinode #1357
Conversation
|
Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
✅ Deploy Preview for pytorch-examples-preview canceled.
|
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
There was a problem hiding this comment.
Pull Request Overview
This PR focuses on ensuring that model snapshots are saved only on the global rank 0 node in a multi-node distributed training setup, thereby preventing multiple nodes from redundantly saving models. Key changes include updating the sampler epoch setting in the training loop and modifying the condition for saving model snapshots by replacing self.local_rank with self.global_rank.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| distributed/minGPT-ddp/mingpt/trainer.py | Added a conditional check to update the epoch for the sampler only during training. |
| distributed/ddp-tutorial-series/multinode.py | Updated snapshot saving condition to check self.global_rank instead of self.local_rank. |
Comments suppressed due to low confidence (1)
distributed/ddp-tutorial-series/multinode.py:74
- Ensure that self.global_rank is properly initialized and consistent across all nodes to guarantee that only the intended process saves the model snapshot.
if self.global_rank == 0 and epoch % self.save_every == 0:
No description provided.