<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->
codet5p-770m-py-codebleu-128-True-1e-06-0.1
This model is a fine-tuned version of Salesforce/codet5p-770m-py on the mbpp dataset. It achieves the following results on the evaluation set:
- Loss: 0.8171
- Codebleu: 0.0852
- Ngram Match Score: 0.0135
- Weighted Ngram Match Score: 0.0421
- Syntax Match Score: 0.1180
- Dataflow Match Score: 0.0810
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-06
- train_batch_size: 6
- eval_batch_size: 6
- seed: 42
- gradient_accumulation_steps: 128
- total_train_batch_size: 768
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 50
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Codebleu | Ngram Match Score | Weighted Ngram Match Score | Syntax Match Score | Dataflow Match Score |
|---|---|---|---|---|---|---|---|---|
| 0.4807 | 1.0 | 1 | 0.9113 | 0.0039 | 0.0000 | 0.0000 | 0.0048 | 0.0049 |
| 0.4928 | 2.0 | 2 | 0.9113 | 0.0039 | 0.0000 | 0.0000 | 0.0048 | 0.0049 |
| 0.2434 | 3.0 | 4 | 0.9113 | 0.0039 | 0.0000 | 0.0000 | 0.0048 | 0.0049 |
| 0.4813 | 4.0 | 5 | 0.9112 | 0.0055 | 0.0000 | 0.0000 | 0.0067 | 0.0070 |
| 0.2397 | 5.0 | 7 | 0.9111 | 0.0063 | 0.0000 | 0.0002 | 0.0072 | 0.0084 |
| 0.4858 | 6.0 | 8 | 0.9109 | 0.0065 | 0.0000 | 0.0002 | 0.0072 | 0.0091 |
| 0.242 | 7.0 | 10 | 0.9107 | 0.0124 | 0.0000 | 0.0012 | 0.0173 | 0.0133 |
| 0.4909 | 8.0 | 11 | 0.9104 | 0.0124 | 0.0000 | 0.0012 | 0.0173 | 0.0133 |
| 0.2437 | 9.0 | 13 | 0.9094 | 0.0136 | 0.0000 | 0.0028 | 0.0178 | 0.0154 |
| 0.4811 | 10.0 | 14 | 0.9091 | 0.0176 | 0.0000 | 0.0045 | 0.0212 | 0.0216 |
| 0.2439 | 11.0 | 16 | 0.9056 | 0.0208 | 0.0000 | 0.0057 | 0.0260 | 0.0244 |
| 0.4809 | 12.0 | 17 | 0.9051 | 0.0241 | 0.0000 | 0.0107 | 0.0289 | 0.0286 |
| 0.2428 | 13.0 | 19 | 0.9038 | 0.0334 | 0.0003 | 0.0159 | 0.0424 | 0.0370 |
| 0.4813 | 14.0 | 20 | 0.9034 | 0.0395 | 0.0007 | 0.0195 | 0.0477 | 0.0461 |
| 0.2429 | 15.0 | 22 | 0.8926 | 0.0464 | 0.0013 | 0.0214 | 0.0573 | 0.0531 |
| 0.473 | 16.0 | 23 | 0.8908 | 0.0491 | 0.0020 | 0.0221 | 0.0588 | 0.0580 |
| 0.2385 | 17.0 | 25 | 0.8889 | 0.0637 | 0.0060 | 0.0316 | 0.0814 | 0.0684 |
| 0.472 | 18.0 | 26 | 0.8870 | 0.0634 | 0.0053 | 0.0286 | 0.0829 | 0.0670 |
| 0.2385 | 19.0 | 28 | 0.8847 | 0.0744 | 0.0072 | 0.0294 | 0.1021 | 0.0747 |
| 0.4688 | 20.0 | 29 | 0.8838 | 0.0780 | 0.0093 | 0.0346 | 0.1079 | 0.0761 |
| 0.2304 | 21.0 | 31 | 0.8609 | 0.0785 | 0.0100 | 0.0357 | 0.1079 | 0.0768 |
| 0.451 | 22.0 | 32 | 0.8594 | 0.0806 | 0.0105 | 0.0363 | 0.1122 | 0.0775 |
| 0.2238 | 23.0 | 34 | 0.8571 | 0.0804 | 0.0107 | 0.0362 | 0.1132 | 0.0761 |
| 0.4484 | 24.0 | 35 | 0.8561 | 0.0804 | 0.0109 | 0.0363 | 0.1132 | 0.0761 |
| 0.2224 | 25.0 | 37 | 0.8525 | 0.0804 | 0.0109 | 0.0363 | 0.1132 | 0.0761 |
| 0.4454 | 26.0 | 38 | 0.8517 | 0.0794 | 0.0109 | 0.0362 | 0.1113 | 0.0754 |
| 0.2234 | 27.0 | 40 | 0.8466 | 0.0797 | 0.0118 | 0.0385 | 0.1113 | 0.0754 |
| 0.4458 | 28.0 | 41 | 0.8348 | 0.0797 | 0.0117 | 0.0385 | 0.1113 | 0.0754 |
| 0.2151 | 29.0 | 43 | 0.8318 | 0.0800 | 0.0118 | 0.0385 | 0.1113 | 0.0761 |
| 0.4204 | 30.0 | 44 | 0.8333 | 0.0800 | 0.0119 | 0.0385 | 0.1113 | 0.0761 |
| 0.2117 | 31.0 | 46 | 0.8312 | 0.0800 | 0.0118 | 0.0385 | 0.1113 | 0.0761 |
| 0.4165 | 32.0 | 47 | 0.8287 | 0.0808 | 0.0122 | 0.0385 | 0.1118 | 0.0775 |
| 0.2073 | 33.0 | 49 | 0.8216 | 0.0842 | 0.0135 | 0.0421 | 0.1171 | 0.0796 |
| 0.4114 | 34.0 | 50 | 0.8171 | 0.0852 | 0.0135 | 0.0421 | 0.1180 | 0.0810 |
Framework versions
- Transformers 4.30.0.dev0
- Pytorch 2.0.1
- Datasets 2.13.1
- Tokenizers 0.13.3
