The Hugging Face fast tokenizer for LLM-jp 13B.

The vocab size is 50,570.

Requirements:

• transformers>=4.34.0
• tokenizers>=0.14.0
• torch

Usage:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("llm-jp/hf-fast-tokenizer-v21b3")