This model represents an enhanced iteration of the SecureBERT model, trained on a corpus eight times larger than its predecessor, over the course of 400 hours, leveraging the computational power of 8xA100 GPUs. The innovation, known as SecureBERT_Plus, brings forth an average improvment of 9% in the performance of the Masked Language Model (MLM) task. This advancement signifies a substantial stride towards achieving heightened proficiency in language understanding and representation learning within the cybersecurity knowledge.
SecureBERT is a domain-specific language model based on RoBERTa which is trained on a huge amount of cybersecurity data and fine-tuned/tweaked to understand/represent cybersecurity textual data.
Related cybersecurity language models:
