GPU requirements?

[Itto1992]

Hi! Thank you for your nice work.

I'd like to try this repo on my personal computer.
Could you tell me how many machine resources are needed to train RWKV-LoRA?
In particular, how many VRAMs are needed to fine-tune RWKV-13B?

Thank you in advance.

[Blealtan]

For RWKV-14B, with aggressive settings (e.g. ctxlen <= 1024, bsz <=2, grad_cp on) it might be possible on 40GB VRAM (I remember I have successfully run, but not really sure). Otherwise you will need 80GB. All numbers refer to a single card.

[Itto1992]

@Blealtan
Thank you for your prompt and informative response!
Your explanation helps me understand the GPU requirements for training RWKV-LoRA, especially for RWKV-14B.
I appreciate your assistance and the great work you've put into this project.
Have a nice day!