<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->
enlm-r
This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 1.4837
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0006
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 128
- total_train_batch_size: 8192
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
- lr_scheduler_type: polynomial
- lr_scheduler_warmup_steps: 24000
- num_epochs: 81
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 6.4 | 0.33 | 160 | 10.7903 |
| 6.4 | 0.66 | 320 | 10.1431 |
| 6.4 | 0.99 | 480 | 9.8708 |
| 6.4 | 0.33 | 640 | 9.3884 |
| 6.4 | 0.66 | 800 | 8.7352 |
| 6.4 | 0.99 | 960 | 8.3341 |
| 6.4 | 1.33 | 1120 | 8.0614 |
| 6.4 | 1.66 | 1280 | 7.8582 |
| 4.2719 | 1.99 | 1440 | 7.4879 |
| 3.2 | 3.3 | 1600 | 7.2689 |
| 3.2 | 3.63 | 1760 | 7.1434 |
| 3.2 | 3.96 | 1920 | 7.0576 |
| 3.2 | 4.29 | 2080 | 7.0030 |
| 3.2 | 4.62 | 2240 | 6.9612 |
| 3.2 | 4.95 | 2400 | 6.9394 |
| 3.2 | 5.28 | 2560 | 6.9559 |
| 3.2 | 5.61 | 2720 | 6.8964 |
| 3.2 | 5.94 | 2880 | 6.8939 |
| 3.2 | 6.27 | 3040 | 6.8871 |
| 3.2 | 6.6 | 3200 | 6.8771 |
| 3.2 | 6.93 | 3360 | 6.8617 |
| 3.2 | 7.26 | 3520 | 6.8472 |
| 3.2 | 7.59 | 3680 | 6.8283 |
| 3.2 | 7.92 | 3840 | 6.8082 |
| 3.2 | 8.25 | 4000 | 6.8119 |
| 3.2 | 8.58 | 4160 | 6.7962 |
| 3.2 | 8.91 | 4320 | 6.7751 |
| 3.2 | 9.24 | 4480 | 6.7405 |
| 3.2 | 9.57 | 4640 | 6.7412 |
| 3.2 | 9.9 | 4800 | 6.7279 |
| 3.2 | 10.22 | 4960 | 6.7069 |
| 3.2 | 10.55 | 5120 | 6.6998 |
| 3.2 | 10.88 | 5280 | 6.6875 |
| 3.2 | 11.22 | 5440 | 6.6580 |
| 3.2 | 11.55 | 5600 | 6.6402 |
| 3.2 | 11.88 | 5760 | 6.6281 |
| 3.2 | 12.21 | 5920 | 6.6181 |
| 3.2 | 12.54 | 6080 | 6.5995 |
| 3.2 | 12.87 | 6240 | 6.5970 |
| 3.2 | 13.2 | 6400 | 6.5772 |
| 3.2 | 13.53 | 6560 | 6.5594 |
| 3.2 | 13.85 | 6720 | 6.5400 |
| 3.2 | 14.19 | 6880 | 6.5396 |
| 3.2 | 14.51 | 7040 | 6.5211 |
| 3.2 | 14.84 | 7200 | 6.5140 |
| 3.2 | 15.18 | 7360 | 6.4002 |
| 3.2 | 15.5 | 7520 | 6.3170 |
| 3.2 | 15.83 | 7680 | 6.2621 |
| 3.2 | 16.16 | 7840 | 6.2253 |
| 3.2 | 16.49 | 8000 | 6.1722 |
| 3.2 | 16.82 | 8160 | 6.1106 |
| 3.2 | 17.15 | 8320 | 6.1281 |
| 3.2 | 17.48 | 8480 | 6.0019 |
| 3.2 | 17.81 | 8640 | 5.9069 |
| 3.2 | 18.14 | 8800 | 5.7105 |
| 3.2 | 18.47 | 8960 | 5.2741 |
| 3.2 | 18.8 | 9120 | 5.0369 |
| 5.0352 | 19.13 | 9280 | 4.8148 |
| 4.5102 | 19.26 | 9440 | 4.3175 |
| 4.1247 | 19.59 | 9600 | 3.9518 |
| 3.8443 | 20.12 | 9760 | 3.6712 |
| 3.6334 | 20.45 | 9920 | 3.4654 |
| 3.4698 | 20.78 | 10080 | 3.2994 |
| 3.3267 | 21.11 | 10240 | 3.1638 |
| 3.2173 | 21.44 | 10400 | 3.0672 |
| 3.1255 | 21.77 | 10560 | 2.9687 |
| 3.0344 | 22.1 | 10720 | 2.8865 |
| 2.9645 | 22.43 | 10880 | 2.8104 |
| 2.9046 | 22.76 | 11040 | 2.7497 |
| 2.8707 | 23.09 | 11200 | 2.7040 |
| 2.7903 | 23.42 | 11360 | 2.6416 |
| 2.7339 | 23.75 | 11520 | 2.5891 |
| 2.6894 | 24.08 | 11680 | 2.5370 |
| 2.6461 | 24.41 | 11840 | 2.4960 |
| 2.5976 | 24.74 | 12000 | 2.4508 |
| 2.5592 | 25.07 | 12160 | 2.4194 |
| 2.5305 | 25.4 | 12320 | 2.3790 |
| 2.4993 | 25.73 | 12480 | 2.3509 |
| 2.465 | 26.06 | 12640 | 2.3173 |
| 2.4455 | 26.39 | 12800 | 2.2934 |
| 2.4107 | 26.72 | 12960 | 2.2701 |
| 2.3883 | 27.05 | 13120 | 2.2378 |
| 2.3568 | 27.38 | 13280 | 2.2079 |
| 2.3454 | 27.71 | 13440 | 2.1919 |
| 2.3207 | 28.04 | 13600 | 2.1671 |
| 2.2963 | 28.37 | 13760 | 2.1513 |
| 2.2738 | 28.7 | 13920 | 2.1326 |
| 2.2632 | 29.03 | 14080 | 2.1176 |
| 2.2413 | 29.36 | 14240 | 2.0913 |
| 2.2193 | 29.69 | 14400 | 2.0772 |
| 2.2169 | 30.02 | 14560 | 2.0692 |
| 2.1848 | 30.35 | 14720 | 2.0411 |
| 2.1693 | 30.68 | 14880 | 2.0290 |
| 2.1964 | 31.01 | 15040 | 2.0169 |
| 2.1467 | 31.34 | 15200 | 2.0016 |
| 2.1352 | 31.67 | 15360 | 1.9880 |
| 2.1152 | 32.0 | 15520 | 1.9727 |
| 2.1098 | 32.33 | 15680 | 1.9604 |
| 2.0888 | 32.66 | 15840 | 1.9521 |
| 2.0837 | 32.99 | 16000 | 1.9394 |
| 2.0761 | 33.32 | 16160 | 1.9366 |
| 2.0635 | 33.65 | 16320 | 1.9200 |
| 2.0631 | 33.98 | 16480 | 1.9147 |
| 2.0448 | 34.31 | 16640 | 1.9053 |
| 2.0452 | 34.64 | 16800 | 1.8937 |
| 2.0303 | 34.97 | 16960 | 1.8801 |
| 2.0184 | 35.3 | 17120 | 1.8752 |
| 2.0115 | 35.63 | 17280 | 1.8667 |
| 2.0042 | 35.96 | 17440 | 1.8626 |
| 2.002 | 36.29 | 17600 | 1.8565 |
| 1.9918 | 36.62 | 17760 | 1.8475 |
| 1.9868 | 36.95 | 17920 | 1.8420 |
| 1.9796 | 37.28 | 18080 | 1.8376 |
| 1.976 | 37.61 | 18240 | 1.8318 |
| 1.9647 | 37.94 | 18400 | 1.8225 |
| 1.9561 | 38.27 | 18560 | 1.8202 |
| 1.9544 | 38.6 | 18720 | 1.8084 |
| 1.9454 | 38.93 | 18880 | 1.8057 |
| 1.9333 | 39.26 | 19040 | 1.8030 |
| 1.9411 | 39.59 | 19200 | 1.7966 |
| 1.9289 | 39.92 | 19360 | 1.7865 |
| 1.9261 | 40.25 | 19520 | 1.7815 |
| 1.9207 | 40.58 | 19680 | 1.7881 |
| 1.9164 | 40.91 | 19840 | 1.7747 |
| 1.9152 | 41.24 | 20000 | 1.7786 |
| 1.914 | 41.57 | 20160 | 1.7664 |
| 1.901 | 41.9 | 20320 | 1.7586 |
| 1.8965 | 42.23 | 20480 | 1.7554 |
| 1.8982 | 42.56 | 20640 | 1.7524 |
| 1.8941 | 42.89 | 20800 | 1.7460 |
| 1.8834 | 43.22 | 20960 | 1.7488 |
| 1.8841 | 43.55 | 21120 | 1.7486 |
| 1.8846 | 43.88 | 21280 | 1.7424 |
| 1.8763 | 44.21 | 21440 | 1.7352 |
| 1.8688 | 44.54 | 21600 | 1.7349 |
| 1.8714 | 44.87 | 21760 | 1.7263 |
| 1.8653 | 45.2 | 21920 | 1.7282 |
| 1.8673 | 45.53 | 22080 | 1.7195 |
| 1.8682 | 45.85 | 22240 | 1.7266 |
| 1.8532 | 46.19 | 22400 | 1.7180 |
| 1.8553 | 46.51 | 22560 | 1.7137 |
| 1.8569 | 46.84 | 22720 | 1.7158 |
| 1.8469 | 47.18 | 22880 | 1.7117 |
| 1.845 | 47.5 | 23040 | 1.7031 |
| 1.8475 | 47.83 | 23200 | 1.7089 |
| 1.845 | 48.16 | 23360 | 1.7018 |
| 1.8391 | 48.49 | 23520 | 1.6945 |
| 1.8456 | 48.82 | 23680 | 1.7015 |
| 1.8305 | 49.15 | 23840 | 1.6964 |
| 1.8324 | 49.48 | 24000 | 1.6900 |
| 1.7763 | 49.81 | 24160 | 1.6449 |
| 1.7728 | 50.14 | 24320 | 1.6436 |
| 1.7576 | 50.47 | 24480 | 1.6268 |
| 1.7354 | 50.8 | 24640 | 1.6088 |
| 1.74 | 51.13 | 24800 | 1.6156 |
| 1.7251 | 51.06 | 24960 | 1.6041 |
| 1.719 | 51.39 | 25120 | 1.5938 |
| 1.7257 | 52.12 | 25280 | 1.5983 |
| 1.7184 | 52.45 | 25440 | 1.5919 |
| 1.7093 | 52.78 | 25600 | 1.5848 |
| 1.7114 | 53.11 | 25760 | 1.5922 |
| 1.7076 | 53.44 | 25920 | 1.5843 |
| 1.7 | 53.77 | 26080 | 1.5807 |
| 1.7027 | 54.1 | 26240 | 1.5811 |
| 1.704 | 54.43 | 26400 | 1.5766 |
| 1.6958 | 54.76 | 26560 | 1.5756 |
| 1.6976 | 55.09 | 26720 | 1.5773 |
| 1.6944 | 55.42 | 26880 | 1.5725 |
| 1.6891 | 55.75 | 27040 | 1.5685 |
| 1.6936 | 56.08 | 27200 | 1.5750 |
| 1.6893 | 56.41 | 27360 | 1.5696 |
| 1.6886 | 56.74 | 27520 | 1.5643 |
| 1.6936 | 57.07 | 27680 | 1.5691 |
| 1.6883 | 57.4 | 27840 | 1.5718 |
| 1.6832 | 57.73 | 28000 | 1.5660 |
| 1.9222 | 28.03 | 28160 | 1.7107 |
| 1.7838 | 28.19 | 28320 | 1.6345 |
| 1.7843 | 28.36 | 28480 | 1.6445 |
| 1.7809 | 28.52 | 28640 | 1.6461 |
| 1.783 | 28.69 | 28800 | 1.6505 |
| 1.7869 | 28.85 | 28960 | 1.6364 |
| 1.778 | 29.02 | 29120 | 1.6363 |
| 1.775 | 29.18 | 29280 | 1.6364 |
| 1.7697 | 29.34 | 29440 | 1.6345 |
| 1.7719 | 29.51 | 29600 | 1.6261 |
| 1.7454 | 61.16 | 29760 | 1.6099 |
| 1.741 | 61.49 | 29920 | 1.6006 |
| 1.7314 | 62.02 | 30080 | 1.6041 |
| 1.7314 | 62.35 | 30240 | 1.5914 |
| 1.7246 | 62.68 | 30400 | 1.5917 |
| 1.7642 | 63.01 | 30560 | 1.5923 |
| 1.7221 | 63.34 | 30720 | 1.5857 |
| 1.7185 | 63.67 | 30880 | 1.5836 |
| 1.7022 | 64.0 | 31040 | 1.5836 |
| 1.7107 | 64.33 | 31200 | 1.5739 |
| 1.7082 | 64.66 | 31360 | 1.5724 |
| 1.7055 | 64.99 | 31520 | 1.5734 |
| 1.7019 | 65.32 | 31680 | 1.5707 |
| 1.699 | 65.65 | 31840 | 1.5649 |
| 1.6963 | 65.98 | 32000 | 1.5685 |
| 1.6935 | 66.31 | 32160 | 1.5673 |
| 1.6899 | 66.64 | 32320 | 1.5648 |
| 1.6869 | 66.97 | 32480 | 1.5620 |
| 1.6867 | 67.3 | 32640 | 1.5564 |
| 1.6861 | 67.63 | 32800 | 1.5552 |
| 1.6831 | 67.96 | 32960 | 1.5496 |
| 1.6778 | 68.29 | 33120 | 1.5479 |
| 1.6742 | 68.62 | 33280 | 1.5501 |
| 1.6737 | 68.95 | 33440 | 1.5441 |
| 1.6725 | 69.28 | 33600 | 1.5399 |
| 1.6683 | 69.61 | 33760 | 1.5398 |
| 1.6689 | 69.94 | 33920 | 1.5374 |
| 1.6634 | 70.27 | 34080 | 1.5385 |
| 1.6638 | 70.6 | 34240 | 1.5332 |
| 1.6614 | 70.93 | 34400 | 1.5329 |
| 1.6544 | 71.26 | 34560 | 1.5292 |
| 1.6532 | 71.59 | 34720 | 1.5268 |
| 1.6511 | 71.92 | 34880 | 1.5225 |
| 1.6506 | 72.25 | 35040 | 1.5219 |
| 1.6496 | 72.58 | 35200 | 1.5202 |
| 1.6468 | 72.91 | 35360 | 1.5199 |
| 1.6424 | 73.24 | 35520 | 1.5220 |
| 1.642 | 73.57 | 35680 | 1.5145 |
| 1.6415 | 73.9 | 35840 | 1.5139 |
| 1.6419 | 74.23 | 36000 | 1.5120 |
| 1.633 | 74.56 | 36160 | 1.5113 |
| 1.6354 | 74.89 | 36320 | 1.5139 |
| 1.6312 | 75.22 | 36480 | 1.5068 |
| 1.6298 | 75.55 | 36640 | 1.5056 |
| 1.6268 | 75.88 | 36800 | 1.5000 |
| 1.6277 | 76.21 | 36960 | 1.5033 |
| 1.6198 | 76.54 | 37120 | 1.4988 |
| 1.6246 | 76.87 | 37280 | 1.4978 |
| 1.6184 | 77.2 | 37440 | 1.4966 |
| 1.6187 | 77.53 | 37600 | 1.4954 |
| 1.6192 | 77.85 | 37760 | 1.4951 |
| 1.6134 | 78.19 | 37920 | 1.4936 |
| 1.6176 | 78.51 | 38080 | 1.4908 |
| 1.6103 | 78.84 | 38240 | 1.4904 |
| 1.612 | 79.18 | 38400 | 1.4919 |
| 1.611 | 79.5 | 38560 | 1.4891 |
| 1.6082 | 79.83 | 38720 | 1.4837 |
| 1.6047 | 80.16 | 38880 | 1.4859 |
| 1.6058 | 80.49 | 39040 | 1.4814 |
| 1.602 | 80.82 | 39200 | 1.4837 |
Framework versions
- Transformers 4.20.1
- Pytorch 1.11.0
- Datasets 2.3.2
- Tokenizers 0.12.1
