Fuxi
负屃是一个Python代码生成模型，训练过程主要参考了Training a causal language model from scratch
在中国神话中，负屃是龙的第八个儿子。似龙形，平生好文。负屃十分爱好闪耀着艺术光彩的碑文，它甘愿化做图案文龙去衬托这些传世的文学珍品。
Fuxi is a Python code generation model, and the training process mainly refers to Training a causal language model from scratch.
In Chinese mythology, Fuxi is the eighth son of the Dragon. He resembles a dragon, and has a strong affinity for literature. Fuxi is particularly fond of inscriptions shining with artistic brilliance, and is willing to transform into an ornamental dragon to set off these enduring literary treasures.
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 512
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 1000
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 2.3403 | 0.15 | 5000 | 1.5401 |
| 1.5052 | 0.31 | 10000 | 1.3514 |
| 1.3657 | 0.46 | 15000 | 1.2464 |
| 1.2715 | 0.61 | 20000 | 1.1665 |
| 1.1977 | 0.77 | 25000 | 1.1059 |
| 1.1498 | 0.92 | 30000 | 1.0777 |
Framework versions
- Transformers 4.30.2
- Pytorch 2.0.1+cu117
- Datasets 2.13.1
- Tokenizers 0.13.3
