A 19th Century Realism Tuning of GPT-2

This model is a fine-tuning of the GPT-2 model found here: https://huggingface.co/gpt2. This paper introduces the model.

Model Description

To develop this model, the standard GPT-2 model was fine-tuned on four 19th century realism novels. The GPT-2 pre-training process is self-supervised and involves the model predicting the next word in a sequence. You can learn more about the GPT-2 model at the link in the previous section.

For this model, the training process was extended on the more confined domain of 19th century realism to get a more specialized model. The training process was the same as the pre-training process.

Intended Uses and Limitations

This model is intended to be used for text generation.

How to Use

The model can be instantiated and used as follows.

from transformers import AutoModelForCausalLM, GPT2Tokenizer


model = AutoModelForCausalLM.from_pretrained('CharlieKincs/19th_century_gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.pad_token = tokenizer.eos_token

inputs = tokenizer(['I am going to lunch and', 'I was watching television the other day and saw'], padding=True, return_tensors='pt').input_ids
outputs = model.generate(inputs, max_new_tokens=110, do_sample=True, top_k=50, top_p=0.95)
decoded_outputs = tokenizer.batch_decode(outputs, skip_special_tokens=True)

Limitations

This model was trained on unfiltered data from 19th century novels, so some of the generation may not be appropriate for modern times.

Training Data

This model was fine-tuned on the text from the following novels: The Idiot by Fyodor Dostoevsky (244,592 tokens), The Brothers Karamazov by Fyodor Dostoevsky (353,886 tokens), Anna Karenina by Leo Tolstoy (352,812 tokens), and David Copperfield by Charles Dickens (357,861 tokens). The total number of tokens in this corpus is 1,309,151. You can find the corpus on https://github.com/charliekincs/tuned_gpt2_corpus.

The corpus was collected on February 21, 2023 from Project Gutenberg. It was collected using the curl command and the goal was to assemble a corpus that reflected the style of 19th century realism.

The text was pre-processed as shown on the GPT-2 page (link in first section). The particular tokenizer object that was used was GPT2Tokenizer.

Evaluation

The following table shows the loss reported during training: