Model Card
We provide a low-rank adapter for an instruction-tuned 12B-parameter GPT3-style language model.
Prompting
For instructions that do not require extraneous inputs, the recommended prompt is:
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
< -- instruction goes here --- >
### Response:
For instructions that do require extraneous inputs, the recommended prompt is:
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
< -- instruction goes here -- >
### Input:
< -- extraneous input goes here -- >
### Response:
Since the model performs causal language modeling, the model's response to the prompt is the text completing the sequence beginning with the prompt.
Instruction-Tuning
This model was instruction-tuned on an internally-curated Alpaca-style dataset.
- Epochs: 3
- Batch size: 128
- Cutoff length: 2048
- Learning rate: 3.2e-5, cosine decay
- LoRA r: 8
- LoRA alpha: 16
- LoRA dropout: 0.0
- LoRA target modules: 'query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'
- License: The instruction-tuning data is subject to the Creative Commons 4.0 license.
Base Model
This model was instruction-tuned from a 12B variant from the Pythia family.
- Repository: EleutherAI/pythia-12b
- Paper: arxiv:2304.01373
- License: The base model is subject to the Apache 2.0 license.
- Model type: Transformer-based Language Model
Licensing Information
We release this adapter under the Creative Commons NonCommercial (CC BY-NC 4.0) license.
