TW-Pythia-6.9B-Chat

Taiwanese Mandarin Pythia Language Model, instruction-tuned for dialogue.

Version 0.2

Model Details

The TW-Pythia model is derived from the Apache-2.0-licenced Pythia language model, with 8000 new Traditional Chinese tokens added, embed layers resized and re-trained.

Basics

• Developed by: @zetavg based on EleutherAI's Pythia language model.
• Model type: Transformer-based GPT-NeoX Causal Language Model
• Languages: English, Traditional Chinese
• License: Unknown due to unconfirmed usage license of the training data
• Derived from model: EleutherAI/pythia-6.9b

Model Sources

• Repository: https://github.com/zetavg/twlm
• Demo: See https://hackmd.io/@z/twlm-demo

Uses

Currently, this model has not demonstrated any practical value in Traditional Chinese processing without further training, but it does possess some basic Chinese-English translation capabilities.

Training Details

Training Data

• 200k English ↔ Traditional Chinese Sentences from the COCT Database.
• ~8k English and Traditional Chinese mixed ShareGPT data.

Training Procedure

First, we build a BPE tokenizer based on the original Pythia tokenizer with 8000 new Traditional Chinese tokens added.

Then, we resize the embedding layer of the pythia-6.9b model to accommodate the new vocabulary size, and we train only the input/output embedding layers to allow the model to learn the new Traditional Chinese words and phrases.

At last, LoRA weights are added to the model and fine-tuned for instruction following.

Training Hyperparameters

• Training regime: fp32
• See: https://github.com/zetavg/twlm/blob/main/configs/ta01_p7b.yaml

Hardware

• 1xH100 80GB GPU on Lambda Cloud (with Skypilot), about 20h in total.