ruGPT-3.5-13B Converted to GGML Format / ruGPT-3.5-13B Конвертированная в формат GGML

Model Description / Описание модели

English

This repository contains a GGML-formatted version of the ruGPT-3.5-13B model originally hosted on Hugging Face. The model has 13 billion parameters and was initially trained on a 300GB dataset from various domains. It was further fine-tuned on 100GB of code and legal documents. The model understands both Russian and English.

Dataset Details

Training Data: 300GB from various domains
Fine-tuning Data: 100GB of code and legal documents
Technical Specs: Trained using Deepspeed and Megatron libraries on 300B tokens dataset for 3 epochs, around 45 days on 512 V100 GPUs. Fine-tuned for 1 epoch with a sequence length of 2048, around 20 days on 200 A100 GPUs.
Perplexity: Around 8.8 for Russian language

Usage

For QuantizationType.Q4_0 and ContainerType.GGJT

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_0-ggjt.bin")
print(model.generate("The meaning of life is ").text)

For QuantizationType.Q4_0 and ContainerType.GGML

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_0.bin")
print(model.generate("The meaning of life is ").text)

For QuantizationType.Q4_1 and ContainerType.GGJT

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_1-ggjt.bin")
print(model.generate("The meaning of life is ").text)

For QuantizationType.Q4_1 and ContainerType.GGML

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_1.bin")
print(model.generate("The meaning of life is ").text)

For QuantizationType.Q5_0 and ContainerType.GGJT

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_0-ggjt.bin")
print(model.generate("The meaning of life is ").text)

For QuantizationType.Q5_0 and ContainerType.GGML

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_0.bin")
print(model.generate("The meaning of life is ").text)

For QuantizationType.Q5_1 and ContainerType.GGJT

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_1-ggjt.bin")
print(model.generate("The meaning of life is ").text)

For QuantizationType.Q5_1 and ContainerType.GGML

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_1.bin")
print(model.generate("The meaning of life is ").text)

For QuantizationType.Q8_0 and ContainerType.GGJT

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q8_0-ggjt.bin")
print(model.generate("The meaning of life is ").text)

For QuantizationType.Q8_0 and ContainerType.GGML

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q8_0.bin")
print(model.generate("The meaning of life is ").text)

f16 Version

# f16 Version
from llm_rs import AutoModel

model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-f16.bin")
print(model.generate("Смысл жизни в ").text)

Compatibility

While this model is intended to be compatible with any GGML-compatible UI, it has not been extensively tested in such environments. Use at your own risk.

Русский

Этот репозиторий содержит версию модели ruGPT-3.5-13B в формате GGML. Модель имеет 13 миллиардов параметров и изначально обучалась на 300ГБ данных из различных доменов. Далее она была дообучена на 100ГБ кода и юридических документов. Модель понимает как русский, так и английский языки.

Детали набора данных

Тренировочные данные: 300ГБ из различных доменов
Данные для дообучения: 100ГБ кода и юридических документов
Технические характеристики: Обучена с использованием библиотек Deepspeed и Megatron на наборе данных из 300 миллиардов токенов за 3 эпохи, примерно 45 дней на 512 GPU V100. Дообучена 1 эпоху с длиной последовательности 2048, примерно 20 дней на 200 GPU A100.
Перплексия: Около 8,8 для русского языка

Использование

Для QuantizationType.Q4_0 и ContainerType.GGJT

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_0-ggjt.bin")
print(model.generate("Смысл жизни в ").text)

Для QuantizationType.Q4_0 и ContainerType.GGML

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_0.bin")
print(model.generate("Смысл жизни в ").text)

Для QuantizationType.Q4_1 и ContainerType.GGJT

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_1-ggjt.bin")
print(model.generate("Смысл жизни в ").text)

Для QuantizationType.Q4_1 и ContainerType.GGML

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_1.bin")
print(model.generate("Смысл жизни в ").text)

Для QuantizationType.Q5_0 и ContainerType.GGJT

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_0-ggjt.bin")
print(model.generate("Смысл жизни в ").text)

Для QuantizationType.Q5_0 и ContainerType.GGML

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_0.bin")
print(model.generate("Смысл жизни в ").text)

Для QuantizationType.Q5_1 и ContainerType.GGJT

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_1-ggjt.bin")
print(model.generate("Смысл жизни в ").text)

Для QuantizationType.Q5_1 и ContainerType.GGML

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_1.bin")
print(model.generate("Смысл жизни в ").text)

Для QuantizationType.Q8_0 и ContainerType.GGJT

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q8_0-ggjt.bin")
print(model.generate("Смысл жизни в ").text)

Для QuantizationType.Q8_0 и ContainerType.GGML

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q8_0.bin")
print(model.generate("Смысл жизни в ").text)

Версия f16

from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-f16.bin")
print(model.generate("Смысл жизни в ").text)

Совместимость

Хотя эта модель предназначена для совместимости с любым GGML-совместимым интерфейсом, она не была тщательно протестирована в таких средах. Используйте на свой страх и риск.