Overview

This model is based on CLIP model and test on four kinds of animal datasets and ten kinds of animal datasets. CLIP model is a zero-shot pre-trained model so we don't need train model. We just input possible classes and image dataset to use model. Possible classes can be defined by yourself, it can be dataset labels or other description.

Text-image matching

Model Input

Class = ["dog", "cat", "rabbit","squirrel"]
image = preprocess(Image.open("/content/drive/MyDrive/Transformer_CLIP/Golden_Retriever.jpeg")).unsqueeze(0).to(device)
text = clip.tokenize(Class).to(device)

Model Process

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    
    logits_per_image, logits_per_text = model(image, text)
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()

res = 0
pos = -1
for j in range(len(probs[0])):
  if probs[0][j] > res:
    res = probs[0][j]
    pos = j


print("The options available are: " + str(Class))
for i in range(len(probs[0])):
  print("The probability of " + str(Class[i]) + " is " + str(probs[0][i]))

print("Model thinks this photo is most likely a " + Class[pos])

Animal Classification

Demo code (Python Notebook)

https://huggingface.co/Haofeng/CLIP_animal_classification/blob/main/CLIP_classfication.ipynb https://huggingface.co/Haofeng/CLIP_animal_classification/blob/main/CLIP_classfication10.ipynb

Dataset reference

https://www.kaggle.com/datasets/alessiocorrado99/animals10 https://www.kaggle.com/datasets/ayushv322/animal-classification