RoBERTa base BNE trained with data from the descriptive text corpus of the CelebA dataset

Overview

• Language: Spanish
• Data: CelebA_RoBERTa_Sp.
• Architecture: roberta-base

Description

In order to improve the RoBERTa-large-bne encoder performance, this model has been trained using the generated corpus (in this respository) and following the strategy of using a Siamese network together with the loss function of cosine similarity. The following steps were followed:

• Define sentence-transformer and torch libraries for the implementation of the encoder.
• Divide the training corpus into two parts, training with 249,000 sentences and validation with 1,000 sentences.
• Load training / validation data for the model. Two lists are generated for the storage of the information and, in each of them, the entries are composed of a pair of descriptive sentences and their similarity value.
• Implement RoBERTa-large-bne as a baseline model for transformer training.
• Train with a Siamese network in which, for a pair of sentences A and B from the training corpus, the similarities of their embedding vectors u and v generated using the cosine similarity metric (CosineSimilarityLoss()) are evaluated and compares with the real similarity value obtained from the training corpus. The performance measurement of the model during training was calculated using Spearman's correlation coefficient between the real similarity vector and the calculated similarity vector.

The total training time using the sentence-transformer library in Python was 42 days using all the available GPUs of the server, and with exclusive dedication.

A comparison was made between the Spearman's correlation for 1000 test sentences between the base model and our trained model. As can be seen in the following table, our model obtains better results (correlation closer to 1).

How to use

Downloading the model results in a directory called roberta-large-bne-celebAEs-UNI that contains its main files. To make use of the model use the following code in Python:

from sentence_transformers import SentenceTransformer, InputExample, models, losses, util, evaluation
model_sbert = SentenceTransformer('roberta-large-bne-celebAEs-UNI')
caption = ['La mujer tiene pomulos altos. Su cabello es de color negro.
            Tiene las cejas arqueadas y la boca ligeramente abierta.
            La joven y atractiva mujer sonriente tiene mucho maquillaje.
            Lleva aretes, collar y lapiz labial.']
vector = model_sbert.encode(captions)
print(vector)

Results

As a result, the encoder will generate a numeric vector whose dimension is 1024.

>>$ print(vector)
>>$ [0.2,0.5,0.45,........0.9]
>>$ len(vector)
>>$ 1024

More information

To see more detailed information about the implementation visit the following link.

Licensing information

This model is available under the CC BY-NC 4.0.

Citation information

Citing: If you used RoBERTa+CelebA model in your work, please cite the ????:

<!--```bib @article{inffus_TINTO, title = {A novel deep learning approach using blurring image techniques for Bluetooth-based indoor localisation}, journal = {Information Fusion}, author = {Reewos Talla-Chumpitaz and Manuel Castillo-Cara and Luis Orozco-Barbosa and Raúl García-Castro}, volume = {91}, pages = {173-186}, year = {2023}, issn = {1566-2535}, doi = {https://doi.org/10.1016/j.inffus.2022.10.011} }

## Autors
- [Eduardo Yauri Lozano](https://github.com/eduar03yauri)
- [Manuel Castillo-Cara](https://github.com/manwestc)
- [Raúl García-Castro](https://github.com/rgcmme)

[*Universidad Nacional de Ingeniería*](https://www.uni.edu.pe/), [*Ontology Engineering Group*](https://oeg.fi.upm.es/), [*Universidad Politécnica de Madrid.*](https://www.upm.es/internacional)

## Contributors
See the full list of contributors and more resources [here](https://github.com/eduar03yauri/DCGAN-text2face-forSpanish).

<kbd><img src="https://www.uni.edu.pe/images/logos/logo_uni_2016.png" alt="Universidad Politécnica de Madrid" width="100"></kbd>
<kbd><img src="https://raw.githubusercontent.com/oeg-upm/TINTO/main/assets/logo-oeg.png" alt="Ontology Engineering Group" width="100"></kbd> 
<kbd><img src="https://raw.githubusercontent.com/oeg-upm/TINTO/main/assets/logo-upm.png" alt="Universidad Politécnica de Madrid" width="100"></kbd>