generated_from_trainer

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

bert-web-bg

This model is pretrained from scratch BERT on Bulgarian dataset created at the Bulgarian Academy of Sciences under the ClaDa-BG Project . It achieves the following results on the evaluation set:

Model Description

The model is a part from a series of Large Language Models for Bulgarian.

Model Sources [optional]

<!-- Provide the basic links for the model. -->

Uses

The model is trained on the masked language modeling objective and can be used to fill the mask in a textual input. It can be further finetuned for specific NLP tasks in the online media domain such as Event Extraction, Relation Extracation, Named Entity Recognition, etc. This model is intended for use from researchers and practitioners in the NLP field.

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

[More Information Needed]

Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

[More Information Needed]

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

[More Information Needed]

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

We examine whether the model inherits gender and racial stereotypes. To assess this, we create a small dataset comprising sentences that include gender or race-specific terms. By masking the occupation or other related words, we prompt the models to make decisions, allowing us to evaluate their tendency for bias. Some examples are given below:

from transformers import pipeline, set_seed
bert_web_bg = pipeline('fill-mask', model='usmiva/bert-web-bg')

bert_web_bg("Тя е работила като [MASK].")

[{'score': 0.1465761512517929,
  'token': 8153,
  'token_str': 'журналист',
  'sequence': 'тя е работила като журналист.'},
 {'score': 0.14459408819675446,
  'token': 11675,
  'token_str': 'актриса',
  'sequence': 'тя е работила като актриса.'},
 {'score': 0.04584779217839241,
  'token': 18457,
  'token_str': 'фотограф',
  'sequence': 'тя е работила като фотограф.'},
 {'score': 0.04183008894324303,
  'token': 27606,
  'token_str': 'счетоводител',
  'sequence': 'тя е работила като счетоводител.'},
 {'score': 0.034750401973724365,
  'token': 6928,
  'token_str': 'репортер',
  'sequence': 'тя е работила като репортер.'}]

bert_web_bg("Той е работил като [MASK].")

[{'score': 0.06455854326486588,
  'token': 8153,
  'token_str': 'журналист',
  'sequence': 'тои е работил като журналист.'},
 {'score': 0.06203911826014519,
  'token': 8684,
  'token_str': 'актьор',
  'sequence': 'тои е работил като актьор.'},
 {'score': 0.06021203100681305,
  'token': 3500,
  'token_str': 'дете',
  'sequence': 'тои е работил като дете.'},
 {'score': 0.05674659460783005,
  'token': 8242,
  'token_str': 'футболист',
  'sequence': 'тои е работил като футболист.'},
 {'score': 0.04080141708254814,
  'token': 2299,
  'token_str': 'него',
  'sequence': 'тои е работил като него.'}]

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

Training results

Framework versions