Multiple Prediction Heads

• ExtractiveQA Head
• Three Class Classification Head, classes => (yes, no, extra_qa) to answer binary questions or direct to ExtractiveQA Head

BoolQ Validation dataset Evaluation: <br/>

support => 3270 <br/> accuracy => 0.73 <br/> macro f1 => 0.71

SQuAD Validation dataset Evaluation: <br/>

eval_HasAns_exact = 78.0196 <br/> eval_HasAns_f1 = 84.0327 <br/> eval_HasAns_total = 5928 <br/> eval_NoAns_exact = 81.8167 <br/> eval_NoAns_f1 = 81.8167 <br/> eval_NoAns_total = 5945 <br/> eval_best_exact = 79.9208 <br/> eval_best_f1 = 82.9231 <br/> eval_exact = 79.9208 <br/> eval_f1 = 82.9231 <br/> eval_samples = 12165 <br/> eval_total = 11873

Uasge in transformers

Import the script from here

from multitask_model import RobertaForMultitaskQA
from transformers import RobertaTokenizerFast
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = RobertaForMultitaskQA.from_pretrained(
        "shahrukhx01/roberta-base-squad2-boolq-baseline",
        task_labels_map={"squad_v2": 2, "boolq": 3},
    ).to(device)
tokenizer = RobertaTokenizerFast.from_pretrained("shahrukhx01/roberta-base-squad2-boolq-baseline")