SpanMarker with bert-base-uncased on BioNLP2004
This is a SpanMarker model trained on the BioNLP2004 dataset that can be used for Named Entity Recognition. This SpanMarker model uses bert-base-uncased as the underlying encoder. See train.py for the training script.
Model Details
Model Description
- Model Type: SpanMarker
- Encoder: bert-base-uncased
- Maximum Sequence Length: 256 tokens
- Maximum Entity Length: 8 words
- Training Dataset: BioNLP2004
- Language: en
- License: other
Model Sources
Model Labels
| Label |
Examples |
| DNA |
"immunoglobulin heavy-chain enhancer", "enhancer", "immunoglobulin heavy-chain ( IgH ) enhancer" |
| RNA |
"GATA-1 mRNA", "c-myb mRNA", "antisense myb RNA" |
| cell_line |
"monocytic U937 cells", "TNF-treated HUVECs", "HUVECs" |
| cell_type |
"B cells", "non-B cells", "human red blood cells" |
| protein |
"ICAM-1", "VCAM-1", "NADPH oxidase" |
Evaluation
Metrics
| Label |
Precision |
Recall |
F1 |
| all |
0.7290 |
0.7983 |
0.7621 |
| DNA |
0.7174 |
0.7505 |
0.7336 |
| RNA |
0.6977 |
0.7692 |
0.7317 |
| cell_line |
0.5831 |
0.7020 |
0.6370 |
| cell_type |
0.8222 |
0.7381 |
0.7779 |
| protein |
0.7196 |
0.8407 |
0.7755 |
Uses
Direct Use
from span_marker import SpanMarkerModel
# Download from the 🤗 Hub
model = SpanMarkerModel.from_pretrained("tomaarsen/span-marker-bert-base-uncased-bionlp")
# Run inference
entities = model.predict("In erythroid cells most of the transcription activity was contained in a 150 bp promoter fragment with binding sites for transcription factors AP2, Sp1 and the erythroid-specific GATA-1.")
Downstream Use
You can finetune this model on your own dataset.
Click to expand
from span_marker import SpanMarkerModel, Trainer
# Download from the 🤗 Hub
model = SpanMarkerModel.from_pretrained("tomaarsen/span-marker-bert-base-uncased-bionlp")
# Specify a Dataset with "tokens" and "ner_tag" columns
dataset = load_dataset("conll2003") # For example CoNLL2003
# Initialize a Trainer using the pretrained model & dataset
trainer = Trainer(
model=model,
train_dataset=dataset["train"],
eval_dataset=dataset["validation"],
)
trainer.train()
trainer.save_model("tomaarsen/span-marker-bert-base-uncased-bionlp-finetuned")
Training Details
Training Set Metrics
| Training set |
Min |
Median |
Max |
| Sentence length |
2 |
26.5790 |
166 |
| Entities per sentence |
0 |
2.7528 |
23 |
Training Hyperparameters
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3
Training Results
| Epoch |
Step |
Validation Loss |
Validation Precision |
Validation Recall |
Validation F1 |
Validation Accuracy |
| 0.4505 |
300 |
0.0210 |
0.7497 |
0.7659 |
0.7577 |
0.9254 |
| 0.9009 |
600 |
0.0162 |
0.8048 |
0.8217 |
0.8131 |
0.9432 |
| 1.3514 |
900 |
0.0154 |
0.8126 |
0.8249 |
0.8187 |
0.9434 |
| 1.8018 |
1200 |
0.0149 |
0.8148 |
0.8451 |
0.8296 |
0.9481 |
| 2.2523 |
1500 |
0.0150 |
0.8297 |
0.8438 |
0.8367 |
0.9501 |
| 2.7027 |
1800 |
0.0145 |
0.8280 |
0.8443 |
0.8361 |
0.9501 |
Environmental Impact
Carbon emissions were measured using CodeCarbon.
- Carbon Emitted: 0.045 kg of CO2
- Hours Used: 0.296 hours
Training Hardware
- On Cloud: No
- GPU Model: 1 x NVIDIA GeForce RTX 3090
- CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
- RAM Size: 31.78 GB
Framework Versions
- Python: 3.9.16
- SpanMarker: 1.3.1.dev
- Transformers : 4.29.2
- PyTorch: 2.0.1+cu118
- Datasets: 2.14.3
- Tokenizers: 0.13.2