Training SD3 Lora underfit discuss

[inspire-boy]

What is a good parameter configuration for character training (regardless of datasets)? I tried different parameters and the result didn't look like the reference character.

[sayakpaul]

Have you tried ablating on the following things?

• Training the text encoders? The support is being added here [Sd3 Dreambooth LoRA] Add text encoder training for the clip encoders huggingface/diffusers#8630.
• Learning rates?
• Increasing the number of instance images that you have?
• Trying out different captions per instance images?

Some of the above points come from this blog post:
https://huggingface.co/blog/sdxl_lora_advanced_script

[bghira]

out of those things i think the caption is the most important, other than LR.

if you have a caption that kinda already does what you want, or at least has the correct species/object "genre" eg. a coffee mug or a human male in their 30s.

if you try and train on a celebrity name that looks like a person for example. it will do really well.

but. as an experiment i tried actually training celebrities into SD3 using their real names and they only ever looked like cartoonish versions of themselves. something might be wrong with the model, eg. if it's severely undertrained there is no amount of LoRA to fix it.

[sayakpaul]

eg. if it's severely undertrained there is no amount of LoRA to fix it.

100 percent agree.

And then the data aspect is quite important too. Like the instance images -- they must capture different poses and orientations in order for the model to learn geometric properties better. But of course, all of these come into effect when the underlying model is trained hard enough.

[leonary]

What is a good parameter configuration for character training (regardless of datasets)? I tried different parameters and the result didn't look like the reference character.

So far, all scripts have faced underfitting problems on more than dozens of datasets, including SimpleTuner, OneTrainer, and Diffuser. Text encoder training is being supported, such as OneTrainer, but OneTrainer still has problems now.
In addition, the basic model has no problem with underfitting some concepts, which should not affect the effect of subsequent fine-tuning. SD3 shows serious underfitting on characters. Although I can't figure out why they have so little data on characters, SD3 has extremely excellent logic and detail rendering capabilities, so the fine-tuning effect will be extremely amazing after the training scripts of the community are improved.
In short, no matter how you set up LR, no matter how many times you learn a single image, no matter how wonderful your labels are, you will not see a good fitting effect in the current various training scripts. I am very grateful for the work of the training script developers, and I hope that the training scripts of SD3 can quickly become more usable.