$ source .etc/Script/activate

BERT

자체 데이터로 Bert Training 하기 RoBERTa/BERT/DistilBERT and masked language modeling pretrainging

아래 경로에 model 생성됨 D:\tmp\test-mlm

python run_mlm.py \
    --model_name_or_path roberta-base \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-mlm

ref

repo