A copy of Meta's llama tokenizer, with three tokens added to mask PII:
- |||EMAIL_ADDRESS|||
- |||PHONE_NUMBER|||
- |||IP_ADDRESS|||
A copy of Meta's llama tokenizer, with three tokens added to mask PII:
|||EMAIL_ADDRESS||||||PHONE_NUMBER||||||IP_ADDRESS|||