e5-large-mnli

This model is a fine-tuned version of intfloat/e5-large on the glue dataset.

Model description

Text Embeddings by Weakly-Supervised Contrastive Pre-training. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei, arXiv 2022

How to use the model

With the zero-shot classification pipeline

The model can be loaded with the zero-shot-classification pipeline like so:

from transformers import pipeline
classifier = pipeline("zero-shot-classification",
                      model="mjwong/e5-large-mnli")

You can then use this pipeline to classify sequences into any of the class names you specify.

sequence_to_classify = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(sequence_to_classify, candidate_labels)

If more than one candidate label can be correct, pass multi_class=True to calculate each class independently:

candidate_labels = ['travel', 'cooking', 'dancing', 'exploration']
classifier(sequence_to_classify, candidate_labels, multi_class=True)

With manual PyTorch

The model can also be applied on NLI tasks like so:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# device = "cuda:0" or "cpu"
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

model_name = "mjwong/e5-large-mnli"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

premise = "But I thought you'd sworn off coffee."
hypothesis = "I thought that you vowed to drink more coffee."

input = tokenizer(premise, hypothesis, truncation=True, return_tensors="pt")
output = model(input["input_ids"].to(device))
prediction = torch.softmax(output["logits"][0], -1).tolist()
label_names = ["entailment", "neutral", "contradiction"]
prediction = {name: round(float(pred) * 100, 2) for pred, name in zip(prediction, label_names)}
print(prediction)

Eval results

The model was evaluated using the dev sets for MultiNLI and test sets for ANLI. The metric used is accuracy.

Datasets mnli_dev_m mnli_dev_mm anli_test_r1 anli_test_r2 anli_test_r3
e5-base-v2-mnli-anli 0.812 0.809 0.557 0.460 0.448
e5-large-mnli 0.868 0.869 0.301 0.296 0.294
e5-large-mnli-anli 0.843 0.848 0.646 0.484 0.458
e5-large-v2-mnli 0.875 0.876 0.354 0.298 0.313
e5-large-v2-mnli-anli 0.846 0.848 0.638 0.474 0.479

Training hyperparameters

The following hyperparameters were used during training:

Framework versions