polish-fast-tokenizer

This is polish fast tokenizer.

Number of documents used to train tokenizer:

• 25 088 398

Sample usge with transformers:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('radlab/polish-fast-tokenizer')
tokenizer.decode(tokenizer("Ala ma kota i psa").input_ids)