<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->
8_roberta-large_train_korquad-1_aihub10_final
This model is a fine-tuned version of klue/roberta-large on the None dataset. It achieves the following results on the evaluation set:
- Exact Match: 82.0646
- F1: 89.8532
- Loss: 0.6855
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 15
- mixed_precision_training: Native AMP
- label_smoothing_factor: 0.1
Training results
Training Loss | Epoch | Step | Exact Match | F1 | Validation Loss |
---|---|---|---|---|---|
5.2079 | 0.09 | 1000 | 16.8859 | 22.6297 | 3.7930 |
1.6052 | 0.18 | 2000 | 69.8685 | 80.7048 | 1.0508 |
0.981 | 0.27 | 3000 | 74.8904 | 85.0342 | 0.8213 |
0.8183 | 0.36 | 4000 | 77.0028 | 86.6159 | 0.6270 |
0.6779 | 0.45 | 5000 | 77.8663 | 87.1009 | 0.6006 |
0.6535 | 0.54 | 6000 | 78.4775 | 87.4922 | 0.5645 |
0.6171 | 0.63 | 7000 | 78.9159 | 87.7604 | 0.5454 |
0.5953 | 0.72 | 8000 | 79.2082 | 87.9666 | 0.5474 |
0.5967 | 0.81 | 9000 | 79.1949 | 88.0747 | 0.5449 |
0.6081 | 0.9 | 10000 | 79.6067 | 88.4190 | 0.5483 |
0.5652 | 0.99 | 11000 | 79.8857 | 88.4750 | 0.5527 |
0.5434 | 1.08 | 12000 | 79.5005 | 88.3638 | 0.5483 |
0.5618 | 1.17 | 13000 | 79.8592 | 88.6312 | 0.5396 |
0.5301 | 1.26 | 14000 | 80.2312 | 88.7555 | 0.5229 |
0.5335 | 1.35 | 15000 | 80.4172 | 88.9265 | 0.5171 |
0.5 | 1.44 | 16000 | 80.4172 | 89.0081 | 0.5107 |
0.5207 | 1.53 | 17000 | 80.6563 | 89.1881 | 0.5117 |
0.4872 | 1.62 | 18000 | 80.5899 | 88.9920 | 0.5259 |
0.5139 | 1.71 | 19000 | 80.4703 | 88.8763 | 0.5186 |
0.5104 | 1.8 | 20000 | 80.2976 | 88.8165 | 0.5151 |
0.4917 | 1.89 | 21000 | 80.6032 | 88.9459 | 0.5312 |
0.5142 | 1.98 | 22000 | 80.6696 | 88.9736 | 0.5181 |
0.4655 | 2.07 | 23000 | 80.9619 | 89.1750 | 0.5112 |
0.4731 | 2.16 | 24000 | 81.0947 | 89.2199 | 0.4971 |
0.4563 | 2.25 | 25000 | 81.0416 | 89.1501 | 0.5015 |
0.4447 | 2.34 | 26000 | 81.1080 | 89.3572 | 0.5049 |
0.4751 | 2.43 | 27000 | 81.1612 | 89.2913 | 0.5020 |
0.4647 | 2.52 | 28000 | 81.2010 | 89.3781 | 0.4971 |
0.4672 | 2.61 | 29000 | 80.9220 | 89.1665 | 0.5073 |
0.4628 | 2.7 | 30000 | 81.2807 | 89.5747 | 0.5156 |
0.4397 | 2.79 | 31000 | 81.4269 | 89.5239 | 0.5098 |
0.4561 | 2.88 | 32000 | 81.5597 | 89.5456 | 0.4966 |
0.4449 | 2.97 | 33000 | 81.5730 | 89.6600 | 0.4973 |
0.435 | 3.06 | 34000 | 81.2143 | 89.4132 | 0.5020 |
0.4271 | 3.15 | 35000 | 81.5464 | 89.6768 | 0.4956 |
0.43 | 3.24 | 36000 | 81.5597 | 89.6178 | 0.5220 |
0.4348 | 3.33 | 37000 | 81.3737 | 89.5019 | 0.5107 |
0.4215 | 3.42 | 38000 | 81.5730 | 89.6766 | 0.4922 |
0.4287 | 3.51 | 39000 | 81.6660 | 89.7750 | 0.5166 |
0.4235 | 3.6 | 40000 | 81.5996 | 89.6392 | 0.4932 |
0.4314 | 3.69 | 41000 | 81.5996 | 89.7113 | 0.4995 |
0.4326 | 3.77 | 42000 | 81.5863 | 89.7004 | 0.4949 |
0.4175 | 3.86 | 43000 | 81.7324 | 89.6672 | 0.4966 |
0.4327 | 3.95 | 44000 | 81.5066 | 89.7201 | 0.4883 |
0.4266 | 4.04 | 45000 | 81.6660 | 89.8173 | 0.5186 |
0.4145 | 4.13 | 46000 | 81.2940 | 89.6006 | 0.4966 |
0.3995 | 4.22 | 47000 | 81.7191 | 89.6950 | 0.4963 |
0.3853 | 4.31 | 48000 | 81.8387 | 89.7038 | 0.5078 |
0.3893 | 4.4 | 49000 | 81.8254 | 89.7305 | 0.4932 |
0.3977 | 4.49 | 50000 | 81.7457 | 89.8541 | 0.5044 |
0.389 | 4.58 | 51000 | 81.7324 | 89.7325 | 0.5322 |
0.3921 | 4.67 | 52000 | 81.8387 | 89.8072 | 0.5122 |
0.4008 | 4.76 | 53000 | 81.6660 | 89.6847 | 0.5137 |
0.4053 | 4.85 | 54000 | 81.7723 | 89.8425 | 0.4932 |
0.3982 | 4.94 | 55000 | 82.0779 | 90.0053 | 0.5020 |
0.4107 | 5.03 | 56000 | 82.0114 | 89.9584 | 0.4983 |
0.3549 | 5.12 | 57000 | 82.1576 | 90.0642 | 0.5098 |
0.3996 | 5.21 | 58000 | 82.1974 | 90.1950 | 0.4907 |
0.3809 | 5.3 | 59000 | 82.0114 | 89.8741 | 0.5078 |
0.3706 | 5.39 | 60000 | 82.0646 | 89.9369 | 0.5249 |
0.3897 | 5.48 | 61000 | 81.9583 | 89.9503 | 0.5063 |
0.376 | 5.57 | 62000 | 81.9981 | 89.8234 | 0.5229 |
0.3666 | 5.66 | 63000 | 81.9317 | 89.9506 | 0.5332 |
0.3568 | 5.75 | 64000 | 81.9981 | 89.9200 | 0.5244 |
0.3677 | 5.84 | 65000 | 82.1443 | 89.9477 | 0.5259 |
0.3734 | 5.93 | 66000 | 82.5030 | 90.1477 | 0.5220 |
0.3622 | 6.02 | 67000 | 82.1177 | 90.0190 | 0.5557 |
0.3671 | 6.11 | 68000 | 81.8387 | 90.0083 | 0.5200 |
0.3563 | 6.2 | 69000 | 81.9849 | 89.9346 | 0.5405 |
0.3773 | 6.29 | 70000 | 81.8387 | 89.7463 | 0.4966 |
0.3662 | 6.38 | 71000 | 81.9716 | 89.9233 | 0.5225 |
0.3461 | 6.47 | 72000 | 82.1177 | 89.9910 | 0.5259 |
0.3412 | 6.56 | 73000 | 82.2240 | 90.1485 | 0.5234 |
0.3695 | 6.65 | 74000 | 82.1177 | 89.9749 | 0.5166 |
0.3462 | 6.74 | 75000 | 81.9184 | 89.9214 | 0.5234 |
0.3696 | 6.83 | 76000 | 82.3568 | 90.1955 | 0.5220 |
0.3651 | 6.92 | 77000 | 81.8520 | 90.0530 | 0.5215 |
0.3595 | 7.01 | 78000 | 82.1443 | 90.0629 | 0.5186 |
0.3284 | 7.1 | 79000 | 82.4100 | 90.0797 | 0.5601 |
0.3288 | 7.19 | 80000 | 82.2506 | 90.1146 | 0.5518 |
0.3268 | 7.28 | 81000 | 82.1841 | 90.1520 | 0.5557 |
0.3291 | 7.37 | 82000 | 82.0779 | 90.0713 | 0.5557 |
0.3355 | 7.46 | 83000 | 82.2373 | 90.1808 | 0.5483 |
0.3382 | 7.55 | 84000 | 82.1576 | 89.9746 | 0.5361 |
0.3377 | 7.64 | 85000 | 82.3303 | 90.0823 | 0.5459 |
0.3614 | 7.73 | 86000 | 82.2373 | 90.0396 | 0.5093 |
0.3522 | 7.82 | 87000 | 82.2107 | 90.0806 | 0.5225 |
0.326 | 7.91 | 88000 | 81.9583 | 89.9022 | 0.5400 |
0.3395 | 8.0 | 89000 | 82.3436 | 90.1668 | 0.5249 |
0.3279 | 8.09 | 90000 | 82.1974 | 90.0841 | 0.5278 |
0.3129 | 8.18 | 91000 | 81.9051 | 89.9340 | 0.5474 |
0.3304 | 8.27 | 92000 | 82.2240 | 90.0328 | 0.5547 |
0.3208 | 8.36 | 93000 | 82.2373 | 90.1196 | 0.5566 |
0.3282 | 8.45 | 94000 | 82.1974 | 89.9974 | 0.5591 |
0.324 | 8.54 | 95000 | 82.2506 | 90.0156 | 0.5737 |
0.3314 | 8.63 | 96000 | 82.1841 | 90.0330 | 0.5503 |
0.3279 | 8.72 | 97000 | 82.4233 | 90.1501 | 0.5625 |
0.3106 | 8.81 | 98000 | 82.1841 | 90.0287 | 0.5889 |
0.3384 | 8.9 | 99000 | 82.1974 | 89.9639 | 0.5513 |
0.3298 | 8.99 | 100000 | 82.2506 | 90.0830 | 0.5752 |
0.3021 | 9.08 | 101000 | 82.1044 | 90.0205 | 0.5850 |
0.3079 | 9.17 | 102000 | 82.2639 | 90.1064 | 0.5840 |
0.2917 | 9.26 | 103000 | 82.2107 | 90.0571 | 0.5859 |
0.3225 | 9.35 | 104000 | 82.4366 | 90.2246 | 0.5654 |
0.2892 | 9.44 | 105000 | 82.2639 | 90.1516 | 0.5898 |
0.3063 | 9.53 | 106000 | 82.4233 | 90.1390 | 0.5879 |
0.3045 | 9.62 | 107000 | 82.2506 | 90.0909 | 0.5938 |
0.3058 | 9.71 | 108000 | 82.3037 | 90.0736 | 0.6152 |
0.2946 | 9.8 | 109000 | 82.1974 | 90.0984 | 0.5742 |
0.3143 | 9.89 | 110000 | 81.8919 | 89.8635 | 0.5840 |
0.3078 | 9.98 | 111000 | 82.1177 | 89.9754 | 0.5859 |
0.2861 | 10.07 | 112000 | 82.1443 | 89.9743 | 0.6001 |
0.2902 | 10.16 | 113000 | 82.1576 | 89.9842 | 0.6035 |
0.2766 | 10.25 | 114000 | 82.2771 | 90.1896 | 0.6001 |
0.3008 | 10.34 | 115000 | 82.2904 | 90.1552 | 0.6001 |
0.2898 | 10.43 | 116000 | 82.0779 | 90.0833 | 0.6162 |
0.2916 | 10.52 | 117000 | 81.9450 | 89.8734 | 0.6318 |
0.2964 | 10.61 | 118000 | 82.1044 | 89.9503 | 0.5957 |
0.2821 | 10.7 | 119000 | 82.2107 | 90.0871 | 0.5908 |
0.2883 | 10.79 | 120000 | 82.3568 | 90.1520 | 0.5947 |
0.2929 | 10.88 | 121000 | 82.2639 | 90.0949 | 0.5786 |
0.2939 | 10.97 | 122000 | 82.0779 | 90.0528 | 0.5542 |
0.2705 | 11.06 | 123000 | 82.1841 | 89.9515 | 0.6396 |
0.2618 | 11.15 | 124000 | 82.1709 | 90.0294 | 0.6494 |
0.2762 | 11.23 | 125000 | 82.1841 | 89.9757 | 0.6289 |
0.2781 | 11.32 | 126000 | 82.2639 | 90.1137 | 0.6274 |
0.276 | 11.41 | 127000 | 82.4897 | 90.2755 | 0.6245 |
0.2828 | 11.5 | 128000 | 82.2107 | 90.0683 | 0.6147 |
0.285 | 11.59 | 129000 | 82.1974 | 89.9957 | 0.6240 |
0.271 | 11.68 | 130000 | 82.3037 | 90.0343 | 0.6279 |
0.2925 | 11.77 | 131000 | 82.0779 | 89.9050 | 0.6260 |
0.2824 | 11.86 | 132000 | 82.0911 | 90.0220 | 0.6416 |
0.3018 | 11.95 | 133000 | 82.3436 | 89.9673 | 0.6294 |
0.2762 | 12.04 | 134000 | 82.2506 | 90.1139 | 0.6479 |
0.246 | 12.13 | 135000 | 82.2240 | 90.0564 | 0.6738 |
0.2612 | 12.22 | 136000 | 82.2107 | 90.0322 | 0.6680 |
0.2491 | 12.31 | 137000 | 82.0513 | 89.8680 | 0.6592 |
0.2648 | 12.4 | 138000 | 82.0513 | 90.0025 | 0.6338 |
0.2596 | 12.49 | 139000 | 82.2373 | 90.1338 | 0.6558 |
0.2696 | 12.58 | 140000 | 82.2107 | 90.1526 | 0.6523 |
0.2778 | 12.67 | 141000 | 82.3303 | 90.1032 | 0.6650 |
0.2784 | 12.76 | 142000 | 82.0114 | 89.9141 | 0.6289 |
0.2639 | 12.85 | 143000 | 82.0114 | 89.8360 | 0.6680 |
0.2584 | 12.94 | 144000 | 82.1841 | 89.9868 | 0.6484 |
0.2271 | 13.03 | 145000 | 82.0114 | 89.9543 | 0.6748 |
0.2373 | 13.12 | 146000 | 82.1443 | 90.0159 | 0.6807 |
0.2576 | 13.21 | 147000 | 82.2771 | 90.1674 | 0.6699 |
0.2483 | 13.3 | 148000 | 82.2107 | 89.9238 | 0.6602 |
0.2728 | 13.39 | 149000 | 81.8387 | 89.8637 | 0.6338 |
0.2632 | 13.48 | 150000 | 81.8520 | 89.9056 | 0.6240 |
0.2484 | 13.57 | 151000 | 82.1310 | 89.9535 | 0.6768 |
0.2428 | 13.66 | 152000 | 82.2240 | 90.1379 | 0.6812 |
0.255 | 13.75 | 153000 | 82.2639 | 90.0505 | 0.6836 |
0.2632 | 13.84 | 154000 | 82.1841 | 90.0297 | 0.6680 |
0.2366 | 13.93 | 155000 | 82.0380 | 89.9797 | 0.6953 |
0.2571 | 14.02 | 156000 | 82.2373 | 90.0496 | 0.6763 |
0.2332 | 14.11 | 157000 | 82.1974 | 89.9976 | 0.7007 |
0.2333 | 14.2 | 158000 | 82.1709 | 90.1054 | 0.7109 |
0.245 | 14.29 | 159000 | 82.0114 | 90.0034 | 0.6748 |
0.2281 | 14.38 | 160000 | 82.1576 | 90.0718 | 0.7031 |
0.2323 | 14.47 | 161000 | 82.0380 | 90.0130 | 0.7012 |
0.2264 | 14.56 | 162000 | 82.1310 | 90.0605 | 0.7090 |
0.2365 | 14.65 | 163000 | 81.9317 | 89.7701 | 0.7334 |
0.2421 | 14.74 | 164000 | 82.1443 | 89.9842 | 0.6992 |
0.2397 | 14.83 | 165000 | 82.1841 | 89.9864 | 0.7256 |
0.2607 | 14.92 | 166000 | 82.0646 | 89.8532 | 0.6855 |
Framework versions
- Transformers 4.25.1
- Pytorch 1.13.0+cu117
- Datasets 2.7.1
- Tokenizers 0.13.2