microsoft/xtremedistil-l6-h256-uncased fined-tuned on SQuAD (https://huggingface.co/datasets/squad)
Hyperparameters:
- epochs: 1
- lr: 1e-5
- train batch sie: 16
- optimizer: adamW
- lr_scheduler: linear
- num warming steps: 0
- max_length: 512
Results on the dev set:
- 'exact_match': 62.66792809839168
- 'f1': 74.99490608582015