CrisisTransformers

CrisisTransformers is a family of pre-trained language models and sentence encoders introduced in the paper "CrisisTransformers: Pre-trained language models and sentence encoders for crisis-related social media texts". The models were trained based on the RoBERTa pre-training procedure on a massive corpus of over 15 billion word tokens sourced from tweets associated with 30+ crisis events such as disease outbreaks, natural disasters, conflicts, etc. Please refer to the associated paper for more details.

CrisisTransformers were evaluated on 18 public crisis-specific datasets against strong baselines such as BERT, RoBERTa, BERTweet, etc. Our pre-trained models outperform the baselines across all 18 datasets in classification tasks, and our best-performing sentence-encoder outperforms the state-of-the-art by more than 17% in sentence encoding tasks.

Uses

CrisisTransformers has 8 pre-trained models and a sentence encoder. The pre-trained models should be finetuned for downstream tasks just like BERT and RoBERTa. The sentence encoder can be used out-of-the-box just like Sentence-Transformers for sentence encoding to facilitate tasks such as semantic search, clustering, topic modelling.

Models and naming conventions

CT-M1 models were trained from scratch up to 40 epochs, while CT-M2 models were initialized with pre-trained RoBERTa's weights and CT-M3 models were initialized with pre-trained BERTweet's weights and both trained for up to 20 epochs. OneLook represents the checkpoint after 1 epoch, BestLoss represents the checkpoint with the lowest loss during training, and Complete represents the checkpoint after completing all epochs. SE represents sentence encoder.

Results

Here are the main results from the associated paper.

<p float="left"> <a href="https://raw.githubusercontent.com/rabindralamsal/images/main/cls.png"><img width="100%" alt="classification" src="https://raw.githubusercontent.com/rabindralamsal/images/main/cls.png"></a> <a href="https://raw.githubusercontent.com/rabindralamsal/images/main/se.png"><img width="50%" alt="sentence encoding" src="https://raw.githubusercontent.com/rabindralamsal/images/main/se.png"></a> </p>

Citation

If you use CrisisTransformers, please cite the following paper:

@misc{lamsal2023crisistransformers,
      title={CrisisTransformers: Pre-trained language models and sentence encoders for crisis-related social media texts}, 
      author={Rabindra Lamsal and
		      Maria Rodriguez Read and
		      Shanika Karunasekera},
      year={2023},
      eprint={2309.05494},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}