一、 个人在openwebtext数据集上训练得到的electra-small模型
二、 复现结果(dev dataset)
| Model | CoLA | SST | MRPC | STS | QQP | MNLI | QNLI | RTE | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| ELECTRA-Small-OWT(original) | 56.8 | 88.3 | 87.4 | 86.8 | 88.3 | 78.9 | 87.9 | 68.5 | 80.36 |
| ELECTRA-Small-OWT (this) | 55.82 | 89.67 | 87.0 | 86.96 | 89.28 | 80.08 | 87.50 | 66.07 | 80.30 |
三、 训练细节
- 数据集 openwebtext
- 训练batch_size 256
- 学习率lr 5e-4
- 最大句子长度max_seqlen 128
- 训练total step 62.5W
- GPU RTX3090
- 训练时间总共耗费2.5天
四、 使用
from transformers import pipeline
fill_mask = pipeline(
"fill-mask",
model="junnyu/electra_small_generator",
tokenizer="junnyu/electra_small_generator"
)
print(
fill_mask("HuggingFace is creating a [MASK] that the community uses to solve NLP tasks.")
)
