ORIGINAL MODEL BY timbrooks

This is just a clone of https://huggingface.co/timbrooks/instruct-pix2pix.

However, I replaced the 7.7 GB ckpt & satetensor files with more convenient 2.13 GB pruned fp16 versions for convenience sake.

Conversion from diffusers format to ckpt & satetensor formats was done with https://github.com/ShivamShrirao/diffusers/blob/main/scripts/convert_diffusers_to_original_stable_diffusion.py.

InstructPix2Pix: Learning to Follow Image Editing Instructions

GitHub: https://github.com/timothybrooks/instruct-pix2pix <img src='https://instruct-pix2pix.timothybrooks.com/teaser.jpg'/>

Example

To use InstructPix2Pix, install diffusers using main for now. The pipeline will be available in the next release

pip install diffusers accelerate safetensors transformers

import PIL
import requests
import torch
from diffusers import StableDiffusionInstructPix2PixPipeline, EulerAncestralDiscreteScheduler

model_id = "timbrooks/instruct-pix2pix"
pipe = StableDiffusionInstructPix2PixPipeline.from_pretrained(model_id, torch_dtype=torch.float16, safety_checker=None)
pipe.to("cuda")
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)

url = "https://raw.githubusercontent.com/timothybrooks/instruct-pix2pix/main/imgs/example.jpg"
def download_image(url):
    image = PIL.Image.open(requests.get(url, stream=True).raw)
    image = PIL.ImageOps.exif_transpose(image)
    image = image.convert("RGB")
    return image
image = download_image(URL)

prompt = "turn him into cyborg"
images = pipe(prompt, image=image, num_inference_steps=10, image_guidance_scale=1).images
images[0]