NB-ROBERTA Training Code
This is the current training code for the planned nb-roberta models.
We are currently planning to run the following experiments:
| Name | nb-roberta-base-old (C) |
| Corpus | NbAiLab/nb_bert |
| Pod size | v4-64 |
| Batch size | 62*4*8 = 1984 = 2k |
| Learning rate | 3e-4 (RoBERTa article is using 6e-4 and bs=8k) |
| Number of steps | 250k |
| Name | nb-roberta-base-ext (B) |
| Corpus | NbAiLab/nbailab_extended |
| Pod size | v4-64 |
| Batch size | 62*4*8 = 1984 = 2k |
| Learning rate | 3e-4 (RoBERTa article is using 6e-4 and bs=8k) |
| Number of steps | 250k |
| Name | nb-roberta-large-ext |
| Corpus | NbAiLab/nbailab_extended |
| Pod size | v4-64 |
| Batch size | 32*4*8 = 2024 = 1k |
| Learning rate | 2-e4 (RoBERTa article is using 4e-4 and bs=8k) |
| Number of steps | 500k |
| Name | nb-roberta-base-scandi |
| Corpus | NbAiLab/scandinavian |
| Pod size | v4-64 |
| Batch size | 62*4*8 = 1984 = 2k |
| Learning rate | 3e-4 (RoBERTa article is using 6e-4 and bs=8k) |
| Number of steps | 250k |
| Name | nb-roberta-large-scandi |
| Corpus | NbAiLab/scandinavian |
| Pod size | v4-64 |
| Batch size | 32*4*8 = 1024 = 1k |
| Learning rate | 2-e4 (RoBERTa article is using 4e-4 and bs=8k) |
| Number of steps | 500k |
Calculations
Some basic that we used when estimating the number of training steps:
- The Scandinavic Corpus is 85GB
- The Scandinavic Corpus contains 13B words
- With a conversion factor of 2.3, this is estimated to around 30B tokens
- 30B tokens / (512 seq length * 3000 batch size) = 20.000 steps