<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

gpt-neo-1.3B-sft-peft-oasst1history

This model is a fine-tuned version of EleutherAI/gpt-neo-1.3B on a reworked version of the oasst1 dataset. This version has the same base data, but it is trained to use the thread history as recommended by the OpenAssistant GitHub repository. It achieves the following results on the evaluation set:

• Loss: 1.1245

Model description

GPT-based model from EleutherAI/gpt-neo-1.3B. Supervised fine tuned on the oasst1 history format.

Intended uses & limitations

This model is supervised fine tuned with the OpenAssistant/oasst1 dataset. Therefore, it is not ready for deployment! It should be aligned using techniques of reinforcement learning from human feedback (RLHF) and domain adaptation before using it in production!

Training and evaluation data

This model uses the OpenAssistant/oasst1 dataset. The dataset is formated in different steps of a chat thread history to work with contextual information as recommended here.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• training_steps: 1200

Training results

Framework versions

• Transformers 4.31.0.dev0
• Pytorch 2.0.1+cu118
• Datasets 2.13.1
• Tokenizers 0.13.3