<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

BEE-spoke-data/verysmol_llama-v8-minipile_x2

This is still a work-in-progress and should be treated as such.

Model description

This is an autogressive smol language model. It generates text.

It achieves the following results on the evaluation set:

• Loss: 2.7521
• Accuracy: 0.4686

Intended uses & limitations

Doing things. Limitations are that it is smol.

Additionally, <insert generic, emotionless, and corporate statement about bias in language models here>.

Data

Most recent training run was on JeanKaddour/minipile for 2 epochs. Otherwise, please refer to the below quote:

UnFoRtUnAtElY We'rE UnAbLe tO ShArE DeTaIlS AbOuT ThE TrAiNiNg aNd tHe dAtAsEtS (eXtRaCtEd fRoM ThE OpEn wEb) DuE To tHe hIgHlY CoMpEtItIvE NaTuRe oF ThE FiElD.

evals

harness

some improvements and some degradations over prev versions. May indicate the last dataset in curricula matters/needs to be chosen specifically

hf-causal-experimental (pretrained=BEE-spoke-data/verysmol_llama-v8-minipile_x2,revision=main,trust_remote_code=True,dtype='float'), limit: None, provide_description: False, num_fewshot: 0, batch_size: 16

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.00015
• train_batch_size: 8
• eval_batch_size: 8
• seed: 5404
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-07
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 2.0

Training results

...