Training procedure

The following bitsandbytes quantization config was used during training:

• load_in_8bit: True
• load_in_4bit: False
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: fp4
• bnb_4bit_use_double_quant: False
• bnb_4bit_compute_dtype: float32

Framework versions

• PEFT 0.4.0.dev0

import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

peft_model_id = "Abinaya/opt-1.3-b-lora"

config = PeftConfig.from_pretrained("Abinaya/opt-1.3b-lora-summary")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b")
model = PeftModel.from_pretrained(model, "Abinaya/opt-1.3b-lora-summary")

tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)

# Load the Lora model
model = PeftModel.from_pretrained(model, peft_model_id)

For inference to get summary

batch = tokenizer("Natural language processing is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data", return_tensors='pt')

with torch.cuda.amp.autocast():
  output_tokens = model.generate(**batch, max_new_tokens=50)

print('\n\n', tokenizer.decode(output_tokens[0], skip_special_tokens=True))