question generation

Model Card of lmqg/bart-base-squad-qg

This model is fine-tuned version of facebook/bart-base for question generation task on the lmqg/qg_squad (dataset_name: default) via lmqg.

Overview

Usage

from lmqg import TransformersQG

# initialize model
model = TransformersQG(language="en", model="lmqg/bart-base-squad-qg")

# model prediction
questions = model.generate_q(list_context="William Turner was an English painter who specialised in watercolour landscapes", list_answer="William Turner")

from transformers import pipeline

pipe = pipeline("text2text-generation", "lmqg/bart-base-squad-qg")
output = pipe("<hl> Beyonce <hl> further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records.")

Evaluation

Score Type Dataset
BERTScore 90.87 default lmqg/qg_squad
Bleu_1 56.92 default lmqg/qg_squad
Bleu_2 40.98 default lmqg/qg_squad
Bleu_3 31.44 default lmqg/qg_squad
Bleu_4 24.68 default lmqg/qg_squad
METEOR 26.05 default lmqg/qg_squad
MoverScore 64.47 default lmqg/qg_squad
ROUGE_L 52.66 default lmqg/qg_squad
Score Type Dataset
QAAlignedF1Score (BERTScore) 95.49 default lmqg/qg_squad
QAAlignedF1Score (MoverScore) 70.38 default lmqg/qg_squad
QAAlignedPrecision (BERTScore) 95.55 default lmqg/qg_squad
QAAlignedPrecision (MoverScore) 70.67 default lmqg/qg_squad
QAAlignedRecall (BERTScore) 95.44 default lmqg/qg_squad
QAAlignedRecall (MoverScore) 70.1 default lmqg/qg_squad
Score Type Dataset
QAAlignedF1Score (BERTScore) 92.84 default lmqg/qg_squad
QAAlignedF1Score (MoverScore) 64.24 default lmqg/qg_squad
QAAlignedPrecision (BERTScore) 92.75 default lmqg/qg_squad
QAAlignedPrecision (MoverScore) 64.46 default lmqg/qg_squad
QAAlignedRecall (BERTScore) 92.95 default lmqg/qg_squad
QAAlignedRecall (MoverScore) 64.11 default lmqg/qg_squad
Dataset Type BERTScore Bleu_4 METEOR MoverScore ROUGE_L Link
lmqg/qg_squadshifts amazon 90.49 5.82 21.27 60.27 23.82 link
lmqg/qg_squadshifts new_wiki 93.07 10.73 26.23 65.67 28.44 link
lmqg/qg_squadshifts nyt 92.36 7.65 24.43 63.69 23.9 link
lmqg/qg_squadshifts reddit 90.57 5.38 20.4 60.14 21.41 link
lmqg/qg_subjqa books 87.75 0.0 11.52 55.21 10.77 link
lmqg/qg_subjqa electronics 87.6 0.0 14.87 56.07 14.29 link
lmqg/qg_subjqa grocery 87.38 0.6 15.53 56.63 12.49 link
lmqg/qg_subjqa movies 87.73 1.08 12.86 55.55 13.9 link
lmqg/qg_subjqa restaurants 87.71 0.0 11.47 54.91 12.16 link
lmqg/qg_subjqa tripadvisor 88.78 1.02 13.92 55.91 13.41 link

Training hyperparameters

The following hyperparameters were used during fine-tuning:

The full configuration can be found at fine-tuning config file.

Citation

@inproceedings{ushio-etal-2022-generative,
    title = "{G}enerative {L}anguage {M}odels for {P}aragraph-{L}evel {Q}uestion {G}eneration",
    author = "Ushio, Asahi  and
        Alva-Manchego, Fernando  and
        Camacho-Collados, Jose",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, U.A.E.",
    publisher = "Association for Computational Linguistics",
}