BERiT_2000_custom_architecture
This model is a fine-tuned version of roberta-base on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 6.0153
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 16.6991 | 0.19 | 500 | 8.9825 |
| 8.259 | 0.39 | 1000 | 7.5650 |
| 7.3895 | 0.58 | 1500 | 7.1084 |
| 7.0328 | 0.77 | 2000 | 6.8799 |
| 6.8743 | 0.97 | 2500 | 6.7598 |
| 6.7775 | 1.16 | 3000 | 6.5915 |
| 6.6348 | 1.36 | 3500 | 6.4513 |
| 6.5759 | 1.55 | 4000 | 6.3394 |
| 6.5243 | 1.74 | 4500 | 6.3336 |
| 6.4492 | 1.94 | 5000 | 6.2714 |
| 6.4472 | 2.13 | 5500 | 6.2921 |
| 6.4283 | 2.32 | 6000 | 6.1922 |
| 6.3508 | 2.52 | 6500 | 6.2112 |
| 6.3838 | 2.71 | 7000 | 6.1727 |
| 6.3303 | 2.9 | 7500 | 6.2093 |
| 6.3067 | 3.1 | 8000 | 6.1984 |
| 6.3099 | 3.29 | 8500 | 6.1589 |
| 6.2806 | 3.49 | 9000 | 6.1732 |
| 6.2861 | 3.68 | 9500 | 6.1257 |
| 6.2645 | 3.87 | 10000 | 6.1655 |
| 6.2992 | 4.07 | 10500 | 6.1156 |
| 6.2331 | 4.26 | 11000 | 6.1212 |
| 6.2247 | 4.45 | 11500 | 6.1991 |
| 6.2235 | 4.65 | 12000 | 6.1181 |
| 6.2354 | 4.84 | 12500 | 6.1469 |
| 6.2157 | 5.03 | 13000 | 6.1170 |
| 6.2076 | 5.23 | 13500 | 6.1128 |
| 6.2085 | 5.42 | 14000 | 6.1079 |
| 6.1917 | 5.62 | 14500 | 6.1511 |
| 6.1917 | 5.81 | 15000 | 6.1032 |
| 6.1887 | 6.0 | 15500 | 6.0877 |
| 6.1895 | 6.2 | 16000 | 6.0876 |
| 6.1685 | 6.39 | 16500 | 6.0734 |
| 6.1709 | 6.58 | 17000 | 6.1039 |
| 6.1442 | 6.78 | 17500 | 6.1347 |
| 6.126 | 6.97 | 18000 | 6.0571 |
| 6.1587 | 7.16 | 18500 | 6.0808 |
| 6.1349 | 7.36 | 19000 | 5.9921 |
| 6.1487 | 7.55 | 19500 | 6.0548 |
| 6.1362 | 7.75 | 20000 | 6.0746 |
| 6.1581 | 7.94 | 20500 | 6.0689 |
| 6.1225 | 8.13 | 21000 | 6.0916 |
| 6.1233 | 8.33 | 21500 | 6.0504 |
| 6.1192 | 8.52 | 22000 | 6.0630 |
| 6.0843 | 8.71 | 22500 | 6.0927 |
| 6.1144 | 8.91 | 23000 | 6.0464 |
| 6.1012 | 9.1 | 23500 | 6.0872 |
| 6.1118 | 9.3 | 24000 | 6.0153 |
Framework versions
- Transformers 4.24.0
- Pytorch 1.12.1+cu113
- Datasets 2.6.1
- Tokenizers 0.13.2