ValueError : Attemting to unscale fp16 Gradients

[ADKoishi]

Fine-tuning the u-net with LoRA disabled and fp16 AMP will trigger a “[ValueError : Attemting to unscale fp16 Gradients]” error.
Obviously, this is caused by the dtype of the parameters and gradients in the u-net module being fp16, and the scaler failed to scale those gradients.
This could be fixed with:
[train_with_rm.py]
pipeline.vae.to(accelerator.device, dtype=inference_dtype)
pipeline.text_encoder.to(accelerator.device, dtype=inference_dtype)
[NEW] -> unet_dtype = inference_dtype if config.use_lora else torch.float32
pipeline.unet.to(accelerator.device, dtype=unet_dtype)
🤔 Is there any better solutions?

[AHHHZ975]

I had the same issue and this resolved it.