SDXL seems to not train self_attn layers in Text Encoders

[Nekotekina]

Hello, I noticed that the recent version only trains MLP in Text Encoders, whereas existing LoRAs or LoRAs trained with GUI version of kohya-ss (that uses older version) seem to train all layers. Is it a mistake on my side? I couldn't find any option to control it.

This is what usually gets trained:

lora_te1_text_model_encoder_layers_0_mlp_fc1.alpha
lora_te1_text_model_encoder_layers_0_mlp_fc1.lora_down.weight
lora_te1_text_model_encoder_layers_0_mlp_fc1.lora_up.weight
lora_te1_text_model_encoder_layers_0_mlp_fc2.alpha
lora_te1_text_model_encoder_layers_0_mlp_fc2.lora_down.weight
lora_te1_text_model_encoder_layers_0_mlp_fc2.lora_up.weight
lora_te1_text_model_encoder_layers_0_self_attn_k_proj.alpha
lora_te1_text_model_encoder_layers_0_self_attn_k_proj.lora_down.weight
lora_te1_text_model_encoder_layers_0_self_attn_k_proj.lora_up.weight
lora_te1_text_model_encoder_layers_0_self_attn_out_proj.alpha
lora_te1_text_model_encoder_layers_0_self_attn_out_proj.lora_down.weight
lora_te1_text_model_encoder_layers_0_self_attn_out_proj.lora_up.weight
lora_te1_text_model_encoder_layers_0_self_attn_q_proj.alpha
lora_te1_text_model_encoder_layers_0_self_attn_q_proj.lora_down.weight
lora_te1_text_model_encoder_layers_0_self_attn_q_proj.lora_up.weight
lora_te1_text_model_encoder_layers_0_self_attn_v_proj.alpha
lora_te1_text_model_encoder_layers_0_self_attn_v_proj.lora_down.weight
lora_te1_text_model_encoder_layers_0_self_attn_v_proj.lora_up.weight

This is what I see in my attempts to use newest version of sdxl_train_network.py:

lora_te1_text_model_encoder_layers_0_mlp_fc1.alpha
lora_te1_text_model_encoder_layers_0_mlp_fc1.lora_down.weight
lora_te1_text_model_encoder_layers_0_mlp_fc1.lora_up.weight
lora_te1_text_model_encoder_layers_0_mlp_fc2.alpha
lora_te1_text_model_encoder_layers_0_mlp_fc2.lora_down.weight
lora_te1_text_model_encoder_layers_0_mlp_fc2.lora_up.weight

[AbstractEyes]

I've been primarily using locon for full finetunes or unet, and lokr for attention as they've yielded the best results without the need for regularization.
Only problem is you have to delete the metadata_cache and let it verify the cache, if you want them to work after moving folders or directories after drive caching latents; because there's some odd quirk with locon not using the correct directory when finding images; and I haven't bothered to fix it.
Until this problem is fixed that's an option.

[Nekotekina]

@AbstractEyes Hello, is it really a problem what I'm observing? The TE still gets trained. Maybe I can fix it myself.

[kohya-ss]

Thank you for reporting. I will check it sooner.