Recommended setup for Flux-dev full fine-tune

[PCanavelli]

Hey everyone,

I'm setting up a full fine-tune for Flux-dev using SimpleTuner, and I'd like to know if there's a tried-and-tested config for this. I've found many conflicting claims regarding both settings and hardware choices, and considering how resource-intensive Flux is, I'd rather not waste too many attempts before settling on a recipe that works.

Is there anyone around who would care to share example setups that worked in their case?

More generally: any tip learned from experience is welcome.

[bghira]

i'd be curious to know why you're doing full-rank instead of an adapter. especially since LoKr scales to millions of images, there's really no need to go down that path? it's a distilled model, the difficulties in tuning it extend beyond the mere number of trainable parameters

[PCanavelli]

Main reason is that I have a use case where I need to run several LoRAs on top of my checkpoint at inference time with no possibility to cherry-pick results.

Based on limited experiments with Flux + tons more with SD15 / SDXL, the number of failed inferences (e.g. distorted limbs, failure to match prompt) increases rapidly with the number of adapters loaded simultaneously. Fiddling with their strength and rank never helped.

That's why I'd rather minimise the number of adapters by training the checkpoint on data that are common to most inferences.

[bghira]

it needs a LOT of data and you have to keep in mind the balance required is not = "the same number of images for each concept" as not every concept is equally difficult to learn

the lessons from SD15 and SDXL don't really apply. they are Unet models based on CLIP and this is a multimodal diffusion transformer with T5 embeds and CLIP as a pooled (weak) conditioning that has been guidance-distilled. we train it in a quantised state, and SDXL would fry if you do this.

there's a lot to drop by the wayside when you move to train Flux. it is a 12B parameter model vs SDXL's 2.6B. this alone makes it just an entirely different world.

using LoKr will probably get you further than full-rank tuning. and LoRA is just an ancient shitty mess at this point. there is a reason its' not the default