recipe-distilbert-upper-tIs
This model is a fine-tuned version of distilbert-base-uncased on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.8746
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 256
- eval_batch_size: 256
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.67 | 1.0 | 1353 | 1.2945 |
| 1.2965 | 2.0 | 2706 | 1.1547 |
| 1.1904 | 3.0 | 4059 | 1.0846 |
| 1.1272 | 4.0 | 5412 | 1.0407 |
| 1.0857 | 5.0 | 6765 | 1.0039 |
| 1.0549 | 6.0 | 8118 | 0.9802 |
| 1.03 | 7.0 | 9471 | 0.9660 |
| 1.01 | 8.0 | 10824 | 0.9474 |
| 0.9931 | 9.0 | 12177 | 0.9365 |
| 0.9807 | 10.0 | 13530 | 0.9252 |
| 0.9691 | 11.0 | 14883 | 0.9105 |
| 0.9601 | 12.0 | 16236 | 0.9079 |
| 0.9503 | 13.0 | 17589 | 0.8979 |
| 0.9436 | 14.0 | 18942 | 0.8930 |
| 0.9371 | 15.0 | 20295 | 0.8875 |
| 0.9322 | 16.0 | 21648 | 0.8851 |
| 0.9279 | 17.0 | 23001 | 0.8801 |
| 0.9254 | 18.0 | 24354 | 0.8812 |
| 0.9227 | 19.0 | 25707 | 0.8768 |
| 0.9232 | 20.0 | 27060 | 0.8746 |
Framework versions
- Transformers 4.19.0.dev0
- Pytorch 1.11.0+cu102
- Datasets 2.3.2
- Tokenizers 0.12.1