Model description

• This model was trained on ZH, JA, KO's Wikipedia (5 epochs).

How to use

from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("conan1024hao/cjkbert-small")
model = AutoModelForMaskedLM.from_pretrained("conan1024hao/cjkbert-small")

• Before you fine-tune downstream tasks, you don't need any text segmentation.
• (Though you may obtain better results if you applied morphological analysis to the data before fine-tuning)

Morphological analysis tools

• ZH: For Chinese, we use LTP.
• JA: For Japanese, we use Juman++.
• KO: For Korean, we use KoNLPy(Kkma class).

Tokenization

• We use character-based tokenization with whole-word-masking strategy.

Model size

• vocab_size: 15015
• num_hidden_layers: 4
• hidden_size: 512
• num_attention_heads: 8
• param_num: 25M