After 500+ LoRAs made, here is the secret

(a reprint of my article posted on reddit)

Well, you wanted it, here it is:

The quality of dataset is 95% of everything. The rest 5% is not to ruin it with bad parameters.

Yeah, I know, GASP! No seriously, folks are searching for secret parameters or secret sauce - but this is the whole deal.

And I mean crystal clean dataset. Yes, I know, thousands of items (maybe tens of thousands), generated or scrubbed from internet, who has time to look at it. I see it in "pro" dataset. Look at some random items, and soon you will spot a garbage - because it was obviously generated or scrubbed and never really checked. What's a few rotten eggs, right? Well, it will spoil the whole bunch as grandma Pam said.

Once I started manually checking the dataset and removing or changing the garbage the quality jumped 10-fold. Yes, it takes a huge amount of time - but no matter of parameters or tricks will fix this, sorry.

The training parameters are there not to ruin it - not make it better, so you don't have to chase the perfect LR 2.5647e-4 it doesn't exist. You kind of aim for the right direction and if dataset is great, most of the time you'll get there.

Some more notes:

• 13b can go only THAT far. There is no way you can create 100% solid finetuning on 13b. You will get close - but like with a child, sometimes it will spill a cup of milk in your lap. 33b is the way. Sadly training 33b on home hardware with 24GB is basically useless because you really have to tone down the parameters - to what I said before - basically ruining it. 48GB at least for 33b so you can crank it up.

• IMHO gradient accumulation will LOWER the quality if you can do more than a few batches. There may be sweet spot somewehere, but IDK. Sure batch 1 and GA 32 will be better than batch 1 and GA 1, but that's not the point, that's a bandaid Edit: It could prevent overfitting though and hence help with generalization. It depends what is the goal and how diverse the dataset is.

• size of dataset matters when you are finetuning on base, but matters less when finetuning on well finetuned model. - in fact sometimes less is better in that case or you may be ruining a good previous finetuning.

• alpha = 2x rank seems like something that came from the old times when people had potato VRAM at most and wanted to get there fast. I really don't feel like it makes much sense - it multiplies the weights and that's it. Making things louder, makes also noise louder.

• my favorite scheduler is warmup, hold for 1 epoch then cosine down for the next 1- x epochs.

• rank is literally how many trainable parameters you get - you don't have to try to find some other meaning (style vs knowledge). It's like an image taken with 1Mpixel vs 16Mpixel. You always get the whole image, but on 1Mpixel the details are very mushy - while you can still see the big subject, you better not expect the details will be fine. The problem of course is - do you have enough diverse training data to fill those parameters with? If not, you'd be creating very specific model that would have hard time to generalize. Lowring rank will help with generalizations, but also the mundane details will be lost.

Anything else?

Oh, OK, I was talking about LORA for LLM, but it surely applies to SD as well. In fact it's all the same thing (and hence PEFT can be used for both and the same rules apply)