ZINC-deberta-base-output

This model is a fine-tuned version of microsoft/deberta-base on the sagawa/ZINC-canonicalized dataset. It achieves the following results on the evaluation set:

Loss: 0.0237
Accuracy: 0.9900

Model description

We trained deberta-base on SMILES from ZINC using the task of masked-language modeling (MLM). Its tokenizer is a character-level tokenizer trained on ZINC.

Intended uses & limitations

This model can be used for the prediction of molecules' properties, reactions, or interactions with proteins by changing the way of finetuning.

Training and evaluation data

We downloaded ZINC data and canonicalized them using RDKit. Then, we droped duplicates. The total number of data is 22992522, and they were randomly split into train:validation=10:1.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 20
eval_batch_size: 32
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 10.0

Training results

Training Loss	Epoch	Step	Accuracy	Validation Loss
0.045	1.06	100000	0.9842	0.0409
0.0372	2.13	200000	0.9864	0.0346
0.0337	3.19	300000	0.9874	0.0314
0.0318	4.25	400000	0.9882	0.0293
0.0296	5.31	500000	0.0277	0.9887
0.0289	6.38	600000	0.0264	0.9891
0.0267	7.44	700000	0.0253	0.9894
0.0261	8.5	800000	0.0243	0.9898
0.025	9.57	900000	0.0238	0.9900

Framework versions

Transformers 4.22.0.dev0
Pytorch 1.12.0
Datasets 2.4.1.dev0
Tokenizers 0.11.6