Upload the tokenizer for the LAMA model in Meta. It is derived from Hugging Face's LlamaTokenizerFast and the license follows the licenses of previous distributions, including all references and metadata.

Install

pip install tokenizers==0.13.3

Usage

from transformers import LlamaTokenizerFast

tokenizer = LlamaTokenizerFast.from_pretrained("dsdanielpark/llama-tokenizer")
tokenizer.encode("Hi, I'm Minwoo Park from seoul, korea.")

References

[1] https://pypi.org/project/tokenizers/ <br> [2] https://huggingface.co/docs/transformers/model_doc/llama