Finetuning Overview:
Model Used: tiiuae/falcon-40b
Dataset: Databricks-dolly-15k
Dataset Insights:
The Databricks-dolly-15k dataset, comprising over 15,000 records, stands as a testament to the dedication of numerous Databricks professionals. Aimed at refining the interactive capabilities of systems like ChatGPT, the dataset offers:
- Prompt/response pairs across eight distinct instruction categories.
- A blend of the seven categories from the InstructGPT paper and an open-ended category.
- Original content, devoid of generative AI influence and primarily offline-sourced, with exceptions for Wikipedia references.
- Interactive sessions where contributors could address and rephrase peer questions.
Note: Some data categories incorporate Wikipedia references, evident from bracketed citation numbers, e.g., [42]. Exclusion is recommended for downstream applications.
Finetuning Details:
Leveraging MonsterAPI's no-code LLM finetuner, our finetuning emphasized:
- Cost-Effectiveness: A complete run at just
$11.8. - Efficiency: Using an A6000 48GB GPU, the session concluded in 5 hours and 40 minutes.
Hyperparameters & Additional Details:
- Epochs: 1
- Learning Rate: 0.0002
- Data Split: Training 90% / Validation 10%
- Gradient Accumulation Steps: 4
Prompt Structure:
### INSTRUCTION:
[instruction]
[context]
### RESPONSE:
[response]
Loss metrics
Training loss:
license: apache-2.0
