generated_from_trainer

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

Indojave: IndoBERT-base

About

This is a pre-trained masked language model for code-mixed Indonesian-Javanese-English tweets data. This model is trained based on IndoBERT model utilizing Hugging Face's Transformers library.

Pre-training Data

The Twitter data is collected from January 2022 until January 2023. The tweets are collected using 8698 random keyword phrases. To make sure the retrieved data are code-mixed, we use keyword phrases that contain code-mixed Indonesian, Javanese, or English words. The following are few examples of the keyword phrases:

We acquire 40,788,384 raw tweets. We apply first stage pre-processing tasks such as:

After the first stage pre-processing, we obtain 17,385,773 tweets. In the second stage pre-processing, we do the following pre-processing tasks:

Finally, we have 28,121,693 sentences for the training process. This pretraining data will not be opened to public due to Twitter policy.

Model

Model name Base model Size of training data Size of validation data
indojave-codemixed-indobert-base IndoBERT 2.24 GB of text 249 MB of text

Evaluation Results

We train the data with 3 epochs and total steps of 296K for 4 days. The following are the results obtained from the training:

train loss eval loss eval perplexity
2.2431 1.9968 7.3657

How to use

Load model and tokenizer

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("fathan/indojave-codemixed-indobert-base")
model = AutoModel.from_pretrained("fathan/indojave-codemixed-indobert-base")

Masked language model

from transformers import pipeline

pretrained_model = "fathan/indojave-codemixed-indobert-base"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_model,
    tokenizer=pretrained_model
)

Training hyperparameters

The following hyperparameters were used during training:

Framework versions