<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->
synpre_delete_1M_t5-small
This model is a fine-tuned version of t5-small on the tyzhu/synpre_delete_1M dataset. It achieves the following results on the evaluation set:
- Loss: 7.1370
- Bleu: 0.6528
- Gen Len: 127.5666
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 128
- eval_batch_size: 128
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant_with_warmup
- lr_scheduler_warmup_steps: 10000
- training_steps: 80000
Training results
| Training Loss | Epoch | Step | Validation Loss | Bleu | Gen Len |
|---|---|---|---|---|---|
| 10.3034 | 0.64 | 5000 | 10.1849 | 0.0 | 97.8953 |
| 9.777 | 1.28 | 10000 | 9.6555 | 0.4844 | 87.4314 |
| 9.5156 | 1.92 | 15000 | 9.3561 | 0.2252 | 216.9687 |
| 9.3497 | 2.56 | 20000 | 9.2006 | 0.245 | 200.9628 |
| 9.2005 | 3.2 | 25000 | 9.1012 | 0.2684 | 184.8855 |
| 9.1503 | 3.84 | 30000 | 9.0289 | 0.261 | 188.3473 |
| 9.0687 | 4.48 | 35000 | 8.9712 | 0.2677 | 184.6629 |
| 9.0207 | 5.12 | 40000 | 8.9276 | 0.2659 | 186.7811 |
| 8.9574 | 5.76 | 45000 | 8.8674 | 0.2761 | 179.3734 |
| 8.9249 | 6.4 | 50000 | 8.8103 | 0.28 | 178.7343 |
| 8.8734 | 7.04 | 55000 | 8.7500 | 0.3151 | 158.8231 |
| 8.833 | 7.68 | 60000 | 8.7039 | 0.3522 | 141.6075 |
| 8.6513 | 8.32 | 65000 | 8.4870 | 0.3225 | 163.0082 |
| 8.2889 | 8.96 | 70000 | 8.1285 | 0.3305 | 171.8776 |
| 7.8159 | 9.6 | 75000 | 7.6354 | 0.3969 | 169.3054 |
| 7.4898 | 10.24 | 80000 | 7.1370 | 0.6528 | 127.5666 |
Framework versions
- Transformers 4.34.0
- Pytorch 2.1.0+cu121
- Datasets 2.14.5
- Tokenizers 0.14.1
