<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->
synpre_union_1M_t5-small
This model is a fine-tuned version of t5-small on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.2342
- Bleu: 74.1557
- Gen Len: 50.3991
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 128
- eval_batch_size: 128
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant_with_warmup
- lr_scheduler_warmup_steps: 10000
- training_steps: 80000
Training results
| Training Loss | Epoch | Step | Validation Loss | Bleu | Gen Len |
|---|---|---|---|---|---|
| 9.2943 | 0.64 | 5000 | 9.1406 | 0.015 | 123.3894 |
| 8.1336 | 1.28 | 10000 | 7.7444 | 0.5478 | 43.7919 |
| 3.473 | 1.92 | 15000 | 1.8592 | 18.4344 | 46.2609 |
| 1.7308 | 2.56 | 20000 | 1.1329 | 32.37 | 47.7555 |
| 1.1924 | 3.2 | 25000 | 0.9239 | 42.6395 | 49.3666 |
| 0.9842 | 3.84 | 30000 | 0.8013 | 45.8171 | 49.5368 |
| 0.84 | 4.48 | 35000 | 0.7371 | 49.2807 | 50.0233 |
| 0.7423 | 5.12 | 40000 | 0.6127 | 57.2094 | 50.392 |
| 0.6675 | 5.76 | 45000 | 0.5601 | 57.6268 | 50.3699 |
| 0.5964 | 6.4 | 50000 | 0.4787 | 60.9757 | 50.4021 |
| 0.5209 | 7.04 | 55000 | 0.4313 | 61.9149 | 50.3289 |
| 0.4824 | 7.68 | 60000 | 0.3971 | 63.0695 | 50.3153 |
| 0.4336 | 8.32 | 65000 | 0.3602 | 68.2965 | 50.4423 |
| 0.3952 | 8.96 | 70000 | 0.3083 | 69.6838 | 50.4396 |
| 0.3543 | 9.6 | 75000 | 0.2679 | 71.0319 | 50.3738 |
| 0.3213 | 10.24 | 80000 | 0.2342 | 74.1557 | 50.3991 |
Framework versions
- Transformers 4.31.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.13.1
- Tokenizers 0.13.3
