Model

"bigscience/bloomz-560m"

Dataset

"Abirate/english_quotes" (English quotes)

Training procedure

PEFT using LoRA

lora configuration: - r=1, - lora_alpha=3, - target_modules=["query_key_value"], - lora_dropout=0.05, - bias="none", - task_type="CAUSAL_LM"

trainable params: 98,304 || all params: 559,312,896 || trainable%: 0.01757585078102687 None

training with Pytorch using the folowing parameters: - learning_rate= 1e-2 - num_train_epochs=10

Framework versions