SEED v1 tokenizer and detokenizer - Work in Progress

NOTE: We are not assocaited with TencentAI Lab. Check out their recent seed2 models: https://huggingface.co/AILab-CVC/SEED

• This repo makes it easier to use AILab's SEEDv1 models.
• SEEDv1 tokens can be used to teach an LLM how to see and draw images! See this paper: https://arxiv.org/pdf/2307.08041.pdf
• The seed tokenizer creates tokens from an image
• The tokenizer can also detokenize the seed tokens into an vector (output_proj) which can then be used by an stable diffusion model to draw the image
• This repo has the models and some modified versions of the code to make it easier to use the seed tokenizer.

Models from https://github.com/AILab-CVC/SEED and which is under Apache 2.0

• causal_qformer.pth
• seed_tokenizer.pth

And model from https://github.com/salesforce/LAVIS which is under BSD-3

• eva_vit_g.pth

Note:

• This repo does not include the SEED author's version of stable diffusion (SD) model, which can be found here: https://drive.google.com/drive/folders/1xmVXuttQfBPBOe4ZR96Wu1X34uzPkxsS?usp=drive_link
• If you wish to use the SD model, you need to clone the original SEED github which has the code to run the SD model.
• Question: Is this just an off the shelf SD-Unet v1-4 (https://huggingface.co/CompVis/stable-diffusion-v1-4)?

Usage:

git clone https://huggingface.co/ontocord/seed
pip install -q salesforce-lavis
pip install -q kornia

import torch
from PIL import Image
from torchvision import transforms
from torchvision.transforms.functional import InterpolationMode
from seed.visual_tokenizer import SEEDVisualTokenizer

# setup device to use
device = torch.device("cuda") if torch.cuda.is_available() else "cpu"

def get_transform(image_size):
    transform = transforms.Compose([
        transforms.Resize((image_size, image_size), interpolation=InterpolationMode.BICUBIC),
        transforms.ToTensor(),
        transforms.Normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711))
    ])
    return transform

# pass in the directory where the models are located
SEED_tokenizer = SEEDVisualTokenizer("./seed/").to(device)

raw_image = Image.open('<your image>').convert("RGB")
image = get_transform(image_size=224)(raw_image).unsqueeze(0).to(device)


with torch.no_grad():
        #tokenize to get the seed tokens and the eva_g embedding. the eva_g embedding can be used for semantic similarity or fed into LLM similar to Llava or EMU.
        visual_codes, eva_g_embed = SEED_tokenizer.predict_seed_tokens(image)

        #detokenize the visual_codes into an output_proj usable by SD to create image
        sd_output_proj = SEED_tokenizer.predict_reverse_output_proj_from_seed_tokens(visual_codes)

        # get the eva_g_embed only
        image_embeds = SEED_tokenizer.visual_encoder(image)
     

Citations:

If you find this package useful, please cite Ontocord.AI and this repo as well as AILab's:

      title={Planting a SEED of Vision in Large Language Model},
      author={Yuying Ge and Yixiao Ge and Ziyun Zeng and Xintao Wang and Ying Shan},
      year={2023},
      eprint={2307.08041},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}