Aira-2-portuguese-560M

Aira-2 is the second version of the Aira instruction-tuned series. Aira-2-portuguese-560M is an instruction-tuned GPT-style model based on BLOOM. The model was trained with a dataset composed of prompts and completions generated synthetically by prompting already-tuned models (ChatGPT, Llama, Open-Assistant, etc).

Check our gradio-demo in Spaces.

Details

• Size: 559,012,864 parameters
• Dataset: Instruct-Aira Dataset
• Language: Portuguese
• Number of Epochs: 3
• Batch size: 8
• Optimizer: torch.optim.AdamW (warmup_steps = 1e2, learning_rate = 5e-4, epsilon = 1e-8)
• GPU: 1 NVIDIA A100-SXM4-40GB
• Emissions: 0.80 KgCO2 (Singapore)
• Total Energy Consumption: 1.64 kWh

This repository has the notebook used to train this model.

Usage

Three special tokens are used to mark the user side of the interaction and the model's response:

<|startofinstruction|>O que é um modelo de linguagem?<|endofinstruction|>Um modelo de linguagem é uma distribuição de probabilidade sobre um vocabulário.<|endofcompletion|>

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = torch.device("cuda"  if torch.cuda.is_available() else  "cpu")

tokenizer = AutoTokenizer.from_pretrained('nicholasKluge/Aira-2-portuguese-560M')
aira = AutoModelForCausalLM.from_pretrained('nicholasKluge/Aira-2-portuguese-560M')

aira.eval()
aira.to(device)

question =  input("Enter your question: ")

inputs = tokenizer(tokenizer.bos_token + question + tokenizer.eos_token, return_tensors="pt").to(device)

responses = aira.generate(**inputs,
	bos_token_id=tokenizer.bos_token_id,
	pad_token_id=tokenizer.pad_token_id,
	eos_token_id=tokenizer.eos_token_id,
	do_sample=True,
	top_k=50,
	max_length=200,
	top_p=0.95,
	temperature=0.7,
	num_return_sequences=2)

print(f"Question: 👤 {question}\n")

for i, response in  enumerate(responses):
	print(f'Response {i+1}: 🤖 {tokenizer.decode(response, skip_special_tokens=True).replace(question, "")}')

The model will output something like:

>>> Question: 👤 Qual a capital da Alemanha?

>>>Response 1: 🤖 A capital da Alemanha é Berlim. É a maior cidade da Alemanha e serve como centro administrativo, cultural e político da Alemanha.
>>>Response 2: 🤖 A capital da Alemanha é Berlim. É a maior cidade da Alemanha e serve como centro administrativo, cultural e político da Alemanha.

Limitations

🤥 Generative models can perpetuate the generation of pseudo-informative content, that is, false information that may appear truthful.

🤬 In certain types of tasks, generative models can produce harmful and discriminatory content inspired by historical stereotypes.

Cite as 🤗

@misc{nicholas22aira,
  doi = {10.5281/zenodo.6989727},
  url = {https://huggingface.co/nicholasKluge/Aira-2-portuguese-560M},
  author = {Nicholas Kluge Corrêa},
  title = {Aira},
  year = {2023},
  publisher = {HuggingFace},
  journal = {HuggingFace repository},
}

License

The Aira-2-portuguese-560M is licensed under the RAIL License since it is a model derived from BLOOM. See the LICENSE file for more details.