Vi-XLM-RoBERTa base model (uncased)

MODEL IS BEING TRAINED (Pending...)

• Progress: ▓▓░░░░░░░░ 25.00%
• Epochs: 10/40 (2022-12-06)

Logs

<a href="https://wandb.ai/anhdungitvn/test/reports/perplexity-22-10-21-21-03-10---VmlldzoyODMwOTQ3?accessToken=lk98cqcsbmcdl4ftjw8vu2bfoc6ifigal1p2db3zo27cc7xwzedfivk3o4aeei3h">Perplexity</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/Training-loss-22-10-21-21-04-12---VmlldzoyODMwOTU4?accessToken=8kfl35d92xxa1eockvrf08p8r65zhlrm1dxfl7vhu5avc3cfozl897n030t1q1f6">Train Loss</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/eval_loss-22-10-21-21-04-24---VmlldzoyODMwOTYx?accessToken=3efesxgcurawkzl93anh4w4hz28c1fl5oexrnwb0mo0pz118fz6mappkpr434tp9">Eval Loss</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/lr-22-10-21-21-03-57---VmlldzoyODMwOTU1?accessToken=ggggf7fe4c4rvhexuonybf3m6mhrdn8tz7o4qjqwbml6k0h9f0953zpsbrympktq">Learning Rate</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/global_step-22-10-21-21-04-36---VmlldzoyODMwOTYy?accessToken=cjd65vgz6uviswqqi5tbud14g9v4bxhjpuccal4hqmrm1h4vqfxithrf4wg0u7xq">Global Step</a>

Gradients by layers

<a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-embeddings-LayerNorm-weight-22-12-06-20-31-20---VmlldzozMDk0NTkx?accessToken=vepursxxx3s66rvrnwtiep9jhx8vjkoid1oyk3gd83b6ertq6r70qhotjg9ne36s">Embedding</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-0-attention-output-LayerNorm-weight-22-12-06-20-30-40---VmlldzozMDk0NTg4?accessToken=x975ir3hd7smhqm64dahbgtv76uwklmgf87j7izkkjbcgu41ed9zcfkk3rzb1nmm">L0-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-1-attention-output-LayerNorm-weight-22-12-06-20-30-24---VmlldzozMDk0NTg2?accessToken=gwqxn1asm08k2jrdhnpa2oiprley2cmpqgnazjxctorq53yfb0zq5yqq8p46ubzq">L1-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-2-attention-output-LayerNorm-weight-22-12-06-20-30-09---VmlldzozMDk0NTgz?accessToken=b6kr19k4qutc12i1hsfki4jryr8ou5pzbmlkvyq6zwxj25tw1txe5vsd8t4li4i3">L2-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-3-attention-output-LayerNorm-weight-22-12-06-20-29-49---VmlldzozMDk0NTgx?accessToken=12klldmz2xpfjcw0bi6a5ggqe3ebtcdq9yismzaz562xajxb696zqcgvsdxp3ext">L3-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-4-attention-output-LayerNorm-weight-22-12-06-20-29-28---VmlldzozMDk0NTc2?accessToken=qgbz1njt7vwi0aspw7bh4n6ad4i0bctctgedq7qmsd4lds9l758ibbm5g4ndfs9f">L4-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-5-attention-output-LayerNorm-weight-22-12-06-20-29-11---VmlldzozMDk0NTc1?accessToken=wytlftyuujtgpmxxwk6q3kd30zjre3qiloa6w6h1aw6eb2169udu3lwtgbbyhd8x">L5-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-6-attention-output-LayerNorm-weight-22-12-06-20-28-53---VmlldzozMDk0NTcx?accessToken=k2ni3ba1xx50vpeto3xpnn1hc0ypvxd7im5mrk1pq47krl64fc765pjr1zt14eyc">L6-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-7-attention-output-LayerNorm-weight-22-12-06-20-28-37---VmlldzozMDk0NTY3?accessToken=kwsawb1wu60ecq1pnvtiim7sumqqca9uqn7rcv2o0ize794h49cgvchv680nju1z">L7-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-8-attention-output-LayerNorm-weight-22-12-06-20-28-20---VmlldzozMDk0NTY0?accessToken=uaqu2tqx39ki634m6j3fywcs76tzyp3q9i1riw9af52pehym9p3ukyamsyldq6te">L8-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-9-attention-output-LayerNorm-weight-22-12-06-20-27-57---VmlldzozMDk0NTYz?accessToken=8fvpijkrfvaywa909woxz38qa0dj8jsm6wyjd3tpubemoakzf3u84p3yel3g5b4a">L9-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-10-attention-output-LayerNorm-weight-22-12-06-20-26-45---VmlldzozMDk0NTU3?accessToken=qv18nauts75orcu6b3lpt227g3aa88wfnpy2lnckwn771pp5j4fn270lrnrxvaie">L10-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-11-attention-output-LayerNorm-weight-22-12-06-20-25-42---VmlldzozMDk0NTUz?accessToken=6xc2nhzbbctc7o0h8ztaspy8m1ha9r1qoldav2e1rilvwl1yu48frels9134ittl">L11-Attention</a> | <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-lm_head-layer_norm-weight-22-12-06-20-32-36---VmlldzozMDk0NTk4?accessToken=t2zroq4rqqu12sak0v8hohmkkqgjuqfgni6apqtearwxa01d3yppglhtux91y4ga">LM Head</a>

This model is Vietnamese XLM Robert, base, uncased.

Model description

Intended uses & limitations

You can use the raw model for either masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.

How to use

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import AutoTokenizer, XLMRobertaForMaskedLM
tokenizer = AutoTokenizer.from_pretrained('anhdungitvn/vi-xlm-roberta-base')
model = XLMRobertaForMaskedLM.from_pretrained('anhdungitvn/vi-xlm-roberta-base')
text = "Câu bằng tiếng Việt."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

and in TensorFlow:

from transformers import AutoTokenizer, XLMRobertaForMaskedLM
tokenizer = AutoTokenizer.from_pretrained('anhdungitvn/vi-xlm-roberta-base')
model = XLMRobertaForMaskedLM.from_pretrained('anhdungitvn/vi-xlm-roberta-base')
text = "Câu bằng tiếng Việt."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

Limitations and bias

Even if the training data used for this model could be characterized as fairly neutral, this model can have biased predictions.

Training data

• Vietnamese Wiki (2022, 1GB).
• Vietnamese News (2022, 17.2GB).

Training procedure

Model was pretrained:

• Tokenizer: SentencePiece (BPE) with a vocab of 32000.
• Model type: xlmroberta
• Optimizer: AdamW
• Learning rate: 100µ

Evaluation results

Pretraining metrics and results:

When fine-tuned on downstream tasks, this model achieves the following results:

Downstream Task Dataset:

• <a href="https://www.aivivn.com/contests/6">SC: Sentiment Classification</a>

  • <a href="https://huggingface.co/datasets/anhdungitvn/sccr">a copy version</a>

• <a href="https://huggingface.co/datasets/truongpdd/Covid-19-ner-lowercased">CN: Covid-19 NER</a>

• <a href="https://huggingface.co/datasets/truongpdd/new_categorical_dataset">NC: News Classification</a>

• <a href="https://huggingface.co/datasets/truongpdd/VSLP_2016_ASC">VSLP_2016_ASC</a>

Evaluation results

SC: Sentiment Classification (Phân loại sắc thái bình luận)

<img width="1024px" src="https://i.postimg.cc/02fLqVDP/sc-cosine-schedule-with-warmup.png">

Logs

<a href="https://wandb.ai/anhdungitvn/test_cls/reports/-tp-tn-tp-tn-fp-fn-22-10-21-09-50-27---VmlldzoyODI4MDcz?accessToken=axp3jb8pw01a7susb3n6pu0i5mfbay66v21oeae1dkx93km41rcuhvrkkqfqpar5">Perplexity</a>

BibTeX entry and citation info

@article{2022,
  title={x},
  author={x},
  journal={ArXiv},
  year={2022},
  volume={x}
}

<a href="https://huggingface.co/exbert/?model=anhdungitvn/vi-xlm-roberta-base"> <img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png"> </a>