Model Card for Vera

<!-- Provide a quick summary of what the model is/does. -->

Vera is a commonsense statement verification model. See our paper at: https://arxiv.org/abs/2305.03695.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

Given a commonsense statement as input, Vera predicts the plausibility of this statement. Vera outputs a real-valued score in the range [0, 1]. A score of 1 means the statement is correct according to commonsense, and a score of 0 means the statement is incorrect. This score is calibrated, so a score between 0 and 1 can be interpreted as Vera's confidence that the statement is correct.

• Developed by: Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi
• Shared by: Jiacheng (Gary) Liu
• Model type: Transformers
• Language(s) (NLP): English
• License: MIT
• Finetuned from model: T5-v1.1-XXL

Model Sources

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/liujch1998/vera
• Paper: https://arxiv.org/abs/2305.03695
• Demo: https://huggingface.co/spaces/liujch1998/vera

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

Vera is intended to predict the correctness of commonsense statements.

Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

Vera can be used to detect commonsense errors made by generative LMs (e.g., ChatGPT), or filter noisy commonsense knowledge generated by other LMs (e.g., Rainier).

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

Vera is a research prototype and may make mistakes. Do not use for making critical decisions. It is intended to predict the correctness of commonsense statements, and may be unreliable when taking input out of this scope.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

See the Limitations and Ethics Statement section of our paper.

How to Get Started with the Model

Use the code below to get started with the model.

tokenizer = transformers.AutoTokenizer.from_pretrained('liujch1998/vera')
model = transformers.T5EncoderModel.from_pretrained('liujch1998/vera')
model.D = model.shared.embedding_dim
linear = torch.nn.Linear(model.D, 1, dtype=model.dtype)
linear.weight = torch.nn.Parameter(model.shared.weight[32099, :].unsqueeze(0))
linear.bias = torch.nn.Parameter(model.shared.weight[32098, 0].unsqueeze(0))
model.eval()
t = model.shared.weight[32097, 0].item() # temperature for calibration

statement = 'Please enter a commonsense statement here.'
input_ids = tokenizer.batch_encode_plus([statement], return_tensors='pt', padding='longest', truncation='longest_first', max_length=128).input_ids
with torch.no_grad():
    output = model(input_ids)
    last_hidden_state = output.last_hidden_state
    hidden = last_hidden_state[0, -1, :]
    logit = linear(hidden).squeeze(-1)
    logit_calibrated = logit / t
    score_calibrated = logit_calibrated.sigmoid()
    # score_calibrated is Vera's final output plausibility score

You may also refer to https://huggingface.co/spaces/liujch1998/vera/blob/main/app.py#L27-L98 for implementation.

Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

@article{Liu2023VeraAG,
  title={Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements},
  author={Jiacheng Liu and Wenya Wang and Dianzhuo Wang and Noah A. Smith and Yejin Choi and Hanna Hajishirzi},
  journal={ArXiv},
  year={2023},
  volume={abs/2305.03695}
}

Model Card Contact

Jiacheng (Gary) Liu