LLaMa-13B LoRA Alpaca-25

This repo contains a low-rank adaptation (LoRA) finetuned model of LLaMa-13B on 25% of the Alpaca cleaned dataset. This version of the weights was trained with the following hyperparameters:

• Epochs: 3
• Cutoff length: 512
• Learning rate: 3e-4
• Lora r: 8
• Lora alpha: 16
• Lora dropout: 0.05
• Lora target modules: q_proj, k_proj