phi_1.5_alpaca_python_100k

This model is a fine-tuned version of microsoft/phi-1_5

Model description

Please check the original repo for more information about the model.

Intended uses & limitations

The intended use for this model falls under the microsoft research license.

Training and evaluation data

This model is fine-tuned on:

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 4
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
training_steps: 1000

Framework versions

Transformers 4.35.0.dev0
Pytorch 2.2.0.dev20231009
Datasets 2.14.5
Tokenizers 0.14.0

Code Example

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("macadeliccc/phi_1.5_python_instruct_650k", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-1_5", trust_remote_code=True)
inputs = tokenizer('''```python
def print_prime(n):
   """
   Print all primes between 1 and n
   """''', return_tensors="pt", return_attention_mask=False)

outputs = model.generate(**inputs, max_length=200)
text = tokenizer.batch_decode(outputs)[0]
print(text)

This is a modified version of the microsoft example code for this fine-tune. For more information please visit the original repo.

Citations

@article{textbooks2,
  title={Textbooks Are All You Need II: \textbf{phi-1.5} technical report},
  author={Li, Yuanzhi and Bubeck, S{\'e}bastien and Eldan, Ronen and Del Giorno, Allie and Gunasekar, Suriya and Lee, Yin Tat},
  journal={arXiv preprint arXiv:2309.05463},
  year={2023}
}

@misc{gunasekar2023textbooks,
      title={Textbooks Are All You Need}, 
      author={Suriya Gunasekar and Yi Zhang and Jyoti Aneja and Caio César Teodoro Mendes and Allie Del Giorno and Sivakanth Gopi and Mojan Javaheripi and Piero Kauffmann and Gustavo de Rosa and Olli Saarikivi and Adil Salim and Shital Shah and Harkirat Singh Behl and Xin Wang and Sébastien Bubeck and Ronen Eldan and Adam Tauman Kalai and Yin Tat Lee and Yuanzhi Li},
      year={2023},
      eprint={2306.11644},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}