Model Card for opt350m-codealpaca20k

Model Description

An opt-350m model trained on the CodeAlpaca 20k dataset using quantization and Progressive Embedding Fine-Tuning (PEFT). The resulting model is designed to understand and generate code-related responses based on the prompts provided.

original model car

Model Architecture

• Base Model: facebook/opt-350m
• Fine-tuning: Parameter-Efficient Fine-Tuning (PEFT)

Training Data

The model was trained on the lucasmccabe-lmi/CodeAlpaca-20k dataset. This dataset contains code-related prompts and their corresponding outputs. Script used for training is avaiable here

Training Procedure

Quantization Configuration:

• Quantization Type: 4-bit
• Compute Dtype: float16
• Double Quant: Enabled

PEFT Configuration:

• Lora Alpha: 16
• Lora Dropout: 0.5
• Bias: None
• Task Type: CAUSAL_LM
• Target Modules: q_proj, v_proj, k_proj

Training Arguments:

• Batch Size: 4 (per device)
• Gradient Accumulation Steps: 2
• Number of Epochs: 10
• Optimizer: adamw_bnb_8bit
• Learning Rate: 2e-5
• Max Gradient Norm: 0.3
• Warmup Ratio: 0.03
• Learning Rate Scheduler: Cosine
• Logging Steps: 10
• Save Steps: 250
• FP16 Precision: Enabled

Training information from wandb

• train/total_flos: 72,761,086,854,758,400
• train/train_loss: 1.5557164267259171
• train/train_runtime: 5892.7285 seconds
• train/train_steps_per_second: 4.248
• _runtime: 5891.33976650238 seconds
• _timestamp: 1,695,390,058.0198596
• train/epoch: 10
• train/global_step: 25,030
• train/learning_rate: 8.371592860045851e-12
• train/train_samples_per_second: 33.977
• _step: 2,503
• _wandb.runtime: 5890 seconds
• train/loss: 1.4114

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt350m")
model = AutoModelForCausalLM.from_pretrained("harpomaxx/opt350m-codealpaca20k)

prompt = "Question: [Your code-related question here] ### Answer: "
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(inputs)
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(decoded_output)

License

OpenRail