<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->
synpre_set_1M_t5-large
This model is a fine-tuned version of t5-large on the tyzhu/synpre_set_1M dataset. It achieves the following results on the evaluation set:
- Loss: 0.0590
- Bleu: 99.0948
- Gen Len: 59.3454
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 128
- eval_batch_size: 128
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant_with_warmup
- lr_scheduler_warmup_steps: 10000
- training_steps: 200000
Training results
| Training Loss | Epoch | Step | Validation Loss | Bleu | Gen Len |
|---|---|---|---|---|---|
| 9.3177 | 0.64 | 5000 | 9.1894 | 0.0191 | 84.117 |
| 8.6899 | 1.28 | 10000 | 8.4473 | 0.2534 | 38.6534 |
| 8.2946 | 1.92 | 15000 | 7.8728 | 0.3046 | 34.5745 |
| 8.0626 | 2.56 | 20000 | 7.6207 | 0.3269 | 31.4656 |
| 7.9075 | 3.2 | 25000 | 7.4554 | 0.3091 | 29.724 |
| 7.8651 | 3.84 | 30000 | 7.3634 | 0.3035 | 27.3112 |
| 6.8274 | 4.48 | 35000 | 6.2226 | 0.6748 | 28.2529 |
| 3.9988 | 5.12 | 40000 | 2.4000 | 23.6952 | 51.826 |
| 0.8097 | 5.76 | 45000 | 0.2258 | 87.1415 | 61.5628 |
| 0.1402 | 6.4 | 50000 | 0.0590 | 99.0948 | 59.3454 |
Framework versions
- Transformers 4.34.0
- Pytorch 2.1.0+cu121
- Datasets 2.14.5
- Tokenizers 0.14.1
