Llama-2-7b-chat-hf-romanian

This model is a fine-tuned version of Llama-2-7b-chat-hf on the _ dataset.

Training procedure

Trained with LoRA in 4-bit (bitsandbytes enables this process) using the rank of low rank matrices equal to 64 and targeting all linear layers.

Training hyperparameters

For training, the following parameters were used:

• num_train_epochs=3
• per_device_train_batch_size=1
• gradient_accumulation_steps=2
• optim="paged_adamw_32bit"
• save_steps=0
• logging_steps=10
• learning_rate=2e-4
• weight_decay=0.001
• fp16=False
• bf16=False
• max_grad_norm=0.3
• max_steps=-1
• warmup_ratio=0.03
• group_by_length=True
• lr_scheduler_type="cosine"

Training results

Framework versions

• transformers: 4.34.0
• accelerate: 0.23.0
• peft: 0.5.0
• sentencepiece: 0.1.99
• bitsandbytes: 0.41.1
• torch: 2.0.1+cu118
• datasets: 2.14.5