Model Card for bellman-7b-1k

The goal is to see how far you can get with a non-english finetune of base llama2.

The name comes from the Swedish bard and poet Carl Mikael Bellman who lived in the 1700s. As with any bard, what this model says should be taken with a grain of salt. Even though it has the best of intentions.

Model Details

Model Description

WIP!

Swedish finetune of NousResearch/Llama-2-7b-chat-hf

Full qlora finetune using "jeremyc/Alpaca-Lora-GPT4-Swedish"

Sadly only trained on 1024 context length. If this turns out well, I'll aim for another one with 4k.

Currently at: 100% of the total dataset. 1 epoch.

Model Sources [optional]

<!-- Provide the basic links for the model. -->

Uses

This is an experimental finetune. Its use should mainly be for entertainment purposes, until it has been tested further. It's trained as an instruct model and could function as an assistant.

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

[More Information Needed]

Out-of-Scope Use

The model should not be used for medical, economical or juridical advice.

Bias, Risks, and Limitations

The model has no additional alignment tuning and inherits any bias from the base model or dataset.

In addition, the dataset used has been machine translated and this affects the models knowledge of the Swedish language.

Since it's only 7b parameters, it can get confused and mix up facts.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Prompt format: [INST] Your input. [/INST] Model response.

Training Details

Training Data

https://huggingface.co/datasets/jeremyc/Alpaca-Lora-GPT4-Swedish

[More Information Needed]

Training Procedure

Trained on Google Colab, on a V100 GPU

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

[More Information Needed]

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[More Information Needed]

Results

[More Information Needed]

Summary

Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]