[REQUEST] Extend offload_states
to support models with cpu-based optimizer
#6596
Labels
enhancement
New feature or request
offload_states
to support models with cpu-based optimizer
#6596
Is your feature request related to a problem? Please describe.
The issue is related to #5620 and #6011. The new
offload_states
API works only withFusedAdam
GPU optimizer. Currently there is no way to offload a trainable model that is using a CPU-based optimizer likeDeepSpeedCPUAdam
.Describe the solution you'd like
Extend #6011 to support offloading of a model configured with CPU-based
DeepSpeedCPUAdam
optimizer.Thanks,
The text was updated successfully, but these errors were encountered: