Baize v2 13b GPTQ
GPTQ 4bit quantized version of Baize V2 13b by Project Baize. <br>Please check the original repo for more information about this model!
This model was quantized using GPTQ-for-LLaMa with the settings:
python llama.py ./input/baize-v2-13b c4 --wbits 4 --true-sequential --groupsize 128 --save_safetensors ./output/baize-v2-13b-4bit-128g.safetensors
<hr>
Available Sizes:
<hr>
Usage with Oobabooga's Text Generation WebUI:
- In the Model tab, enter "
GamaTech/baize-v2-13b-GPTQ
" and click Download. - Once the download is complete, click the Refresh button next to the Model drop-down in the top left.
- Uncheck the Autoload the model checkbox. This allows you to select the correct settings before attempting to load the model.
- Select the model from the Model drop-down list.
- On the right, set the following settings:
- Wbits = 4
- Groupsize = 128
- Model_Type = LLaMA
- Click Save settings for this model.
- Click Load Model.
<hr> <center>license: cc-by-nc-4.0</center>