LoRA weights only and trained for research - nothing from the foundation model. Trained using Anthropics HH dataset which can be found here https://huggingface.co/datasets/Anthropic/hh-rlhf

Sample usage

import torch
import os
import transformers
from peft import PeftModel
from transformers import LlamaTokenizer, LlamaForCausalLM

model_path = "decapoda-research/llama-13b-hf"
peft_path = 'serpdotai/llama-hh-lora-13B'
tokenizer_path = 'decapoda-research/llama-13b-hf'

model = LlamaForCausalLM.from_pretrained(model_path, load_in_8bit=True, device_map="auto") # or something like {"": 0}
model = PeftModel.from_pretrained(model, peft_path, torch_dtype=torch.float16, device_map="auto") # or something like {"": 0}
tokenizer = LlamaTokenizer.from_pretrained(tokenizer_path)

batch = tokenizer("\n\nUser: Are you sentient?\n\nAssistant:", return_tensors="pt")

with torch.no_grad():
    out = model.generate(
        input_ids=batch["input_ids"].cuda(),
        attention_mask=batch["attention_mask"].cuda(),
        max_length=100,
        do_sample=True,
        top_k=50,
        top_p=1.0,
        temperature=1.0,
        use_cache=False
    )
print(tokenizer.decode(out[0]))

The model will continue the conversation between the user and itself. If you want to use as a chatbot you can alter the generate method to include stop sequences for 'User:' and 'Assistant:' or strip off anything past the assistant's original response before returning.

Trained for 2 epochs with a sequence length of 640, mini-batch size of 3, gradient accumulation of 5, on 8 A6000s for an effective batch size of 120.

Training settings:

LoRA config: