This is a smaller version of the XLM-RoBERTa model with only Ukrainian and some English embeddings left.
- The original model has 470M parameters, with 384M of them being input and output embeddings.
- After shrinking the
sentencepiecevocabulary from 250K to 31K (top 25K Ukrainian tokens and top English tokens) the number of model parameters reduced to 134M parameters, and model size reduced from 1GB to 400MB.