This is based on SloBERTa (https://huggingface.co/EMBEDDIA/sloberta) but fine-tuned for 5 epochs on the text of all Slovenian-language documents available on the Slovenian Digital Library (https://dlib.si) from the period 1850-1919. This was about 8.2 GB of text. Note that it also contained a lot of OCR errors.