BL-pythia-31m-simpleRW-lite-2048-scratch

This model is a fine-tuned version of EleutherAI/pythia-31m on the None dataset. It achieves the following results on the evaluation set:

Loss: 4.7136
Accuracy: 0.2662

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

2040 ***** eval metrics *****
2041   epoch                   =        3.0
2042   eval_accuracy           =     0.2668
2043   eval_loss               =     4.7076
2044   eval_runtime            = 0:00:21.04
2045   eval_samples            =        500
2046   eval_samples_per_second =     23.759
2047   eval_steps_per_second   =      11.88
2048   perplexity              =   110.7897

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0005
train_batch_size: 2
eval_batch_size: 2
seed: 80085
gradient_accumulation_steps: 64
total_train_batch_size: 128
optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-07
lr_scheduler_type: inverse_sqrt
lr_scheduler_warmup_ratio: 0.05
num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
7.0159	0.13	100	7.1022	0.1180
6.2257	0.27	200	6.3526	0.1508
5.8611	0.4	300	5.9888	0.1735
5.5514	0.54	400	5.7552	0.1855
5.3824	0.67	500	5.5883	0.1948
5.344	0.81	600	5.4697	0.2017
5.1925	0.94	700	5.3717	0.2073
5.0814	1.08	800	5.2932	0.2121
5.0865	1.21	900	5.2280	0.2162
4.9602	1.35	1000	5.1672	0.2207
4.957	1.48	1100	5.1144	0.2247
4.8489	1.62	1200	5.0617	0.2299
4.79	1.75	1300	5.0122	0.2349
4.8005	1.89	1400	4.9637	0.2400
4.7409	2.02	1500	4.9216	0.2448
4.6674	2.16	1600	4.8815	0.2488
4.6729	2.29	1700	4.8475	0.2526
4.7071	2.43	1800	4.8156	0.2555
4.4937	2.56	1900	4.7841	0.2588
4.5153	2.7	2000	4.7573	0.2615
4.5512	2.83	2100	4.7345	0.2637
4.5153	2.96	2200	4.7136	0.2662

Framework versions

Transformers 4.34.0.dev0
Pytorch 2.2.0.dev20230915+cu118
Datasets 2.14.5
Tokenizers 0.13.3