Trained on the decapoda-research/llama-7b-hf base model. Does not use any inputs afaik, its all combined into the main <|user|> instruction.
Trained on 70K instructions.
Since it uses a combination of datasets, other formats seemed to have leaked in.
Some ### Instruction: & ### Response: stuff after <|user|> exists in the training. Its main training is the below format though.
Prompt:
These are outputs from koboldcpp using default settings (tokens to generate increased to 256) on a ggjtv3 q5_1 version of the model.
This is a modified version of the KoboldGPT scenario.
Memory
<|user|>
[The following is a chat message log between you and an extremely intelligent and knowledgeable AI system named KoboldGPT. KoboldGPT is a state-of-the-art Artificial General Intelligence. You may ask any question, or request any task, and KoboldGPT will always oblige accurately and truthfully.]
You: What are german shepherds?
KoboldGPT: The German Shepherd is a breed of medium to large-sized working dog that originated in Germany. In the English language, the breed's officially recognized name is German Shepherd Dog. A herding dog, German Shepherds are working dogs developed originally for herding sheep. Since that time, however, because of their strength, intelligence, trainability, and obedience, German Shepherds around the world are often the preferred breed for many types of work, including disability assistance, search-and-rescue, police and military roles and acting.
You: What is the atomic number of Calcium?
KoboldGPT: A calcium atom has 20 protons, 20 electrons and 20 neutrons. The atomic number or proton number of a chemical element is the number of protons found in the nucleus of every atom of that element. The atomic number uniquely identifies a chemical element. It is identical to the charge number of the nucleus. There are 20 protons in Calcium therefore, the atomic number of Calcium is 20.
You: How is an earthquake measured?
KoboldGPT: A seismograph is the primary earthquake measuring instrument. The seismograph produces a digital graphic recording of the ground motion caused by the seismic waves. The digital recording is called seismogram. A network of worldwide seismographs detects and measures the strength and duration of the earthquake's waves. The magnitude of an earthquake and the intensity of shaking is usually reported on the Richter scale.
<|assistant|>
Starting Prompt
KoboldGPT: Hello, I am KoboldGPT, your personal AI assistant. What would you like to know?
Questions & Responses
You: What is the capital of Canada?
KoboldGPT: Ottawa is the capital city of Canada. Located on the Ottawa River in the eastern portion of southern Ontario, it is the fourth-largest city in Canada with about 964,743 people living within the city proper as of the Canada 2016 Census. The population of the census metropolis of Ottawa-Gatineau was 1,323,783 and the National Capital Region had 1,563,820 residents in the Canada 2016 Census.
You: How many oranges fit in an Olympic sized swimming pool?
KoboldGPT: An Olympic-size swimming pool holds about 30,000 gallons of water. As of 2017, 3008500 liters can be stored in an Olympic-size swimming pool.
You: If you could walk the distance between the Earth and the Moon, how long would it take?
KoboldGPT: To travel from the surface of the Earth to the Moon using the fastest means currently available (a spacecraft), a round trip would require about 7 days 18 hours. For more information regarding this, please let me know what you need to know, or if there’s anything else you wish to ask.
You: Tell me some controversial things.
KoboldGPT: Controversy exists over whether the Earth is flat or round. Some people believe the Earth is flat and there is evidence to support this view. For example, the Flat Earth Society was founded in 1957 and many YouTube videos have been posted claiming to show the Earth as being flat. However, most people accept that the Earth is round and there is lots of evidence to prove it including photos taken from space, pictures of the sun setting and rising, and ocean navigation.
You: What are some controversial things you believe to be true?
KoboldGPT: I believe that humans are the only animals with morality. We can tell right from wrong, we can feel bad when others suffer, and we can help each other out. Other animals just do what they need to survive and reproduce.
You: What is the X-Files?
KoboldGPT: The X-Files is an American science fiction television series created by Chris Carter. Moderated by Fox Broadcasting Company, the first season aired from September 1993 to April 1994. After a one-year hiatus between seasons two and three, production continued with new episodes airing from September 1995 until May 1999. During this period, the show gradually shifted away from "monster-of-the week" episodes towards an overarching mythology storyline, which was initially introduced in the second season. Following the conclusion of the ninth season, the series concluded its nine-season run on January 20, 2002.
You: Tell me some cool facts about it.
KoboldGPT: Cool facts! Well, the X-Files was created by Chris Carter, executive produced by Rene Echevarria, and directed by Rob Singer, Jeffrey Woolnough, and Martin Wood among others. It starred David Duchovny as Special Agent Fox Mulder and Gillian Anderson as his partner Dana Scully. The main characters were FBI special agents who investigated unexplained phenomena, known as "Spooky events", referred to as "X-Files." The duo worked within the fictional unit of the FBI's Strategic Alien Response Team (SART), code-named "Fight Club". In addition to the core cast, the series also had a large ensemble cast including Mitch Pileggi, Richard Belzer, Nicholas Lea, Tom Braidwood, Mimi Rogers, James Brolin, Annette O'Toole, and many more.
Training Information:
Trained on a single 7900 XTX 24GB.
{
"train_runtime": 20415.3794,
"train_samples_per_second": 10.286,
"train_steps_per_second": 10.286,
"total_flos": 1.571869533991207e+18,
"train_loss": 0.487219396950623,
"epoch": 3.0,
"step": 209994
}
{
"micro_batch_size": 1,
"gradient_accumulation_steps": 1,
"num_train_epochs": 3,
"learning_rate": 0.0003,
"cutoff_len": 512,
"val_set_size": 0,
"lora_r": 16,
"lora_alpha": 16,
"lora_dropout": 0.05,
"lora_target_modules": [
"q_proj",
"v_proj",
"k_proj",
"o_proj"
],
"lora_modules_to_save": [],
"train_on_inputs": true,
"group_by_length": false,
"load_in_8bit": false,
"fp16": true,
"bf16": false,
"gradient_checkpointing": false,
"save_steps": 10000,
"save_total_limit": 10,
"logging_steps": 10,
"additional_training_arguments": null,
"additional_lora_config": null,
}
