t2iadapter_depth_sd15v2

作者 TencentARC · 下载 799 · 点赞 0 · 更新 2023-11-03

art t2i-adapter controlnet stable-diffusion image-to-image

T2I Adapter - Depth

T2I Adapter is a network providing additional conditioning to stable diffusion. Each t2i checkpoint takes a different type of conditioning as input and is used with a specific base stable diffusion checkpoint.

This checkpoint provides conditioning on depth for the stable diffusion 1.5 checkpoint.

Model Details

Checkpoints

Model Name Control Image Overview Control Image Example Generated Image Example
TencentARC/t2iadapter_color_sd14v1
Trained with spatial color palette
A image with 8x8 color palette. t2iadapter_depth_sd15v2 预览 1 t2iadapter_depth_sd15v2 预览 2
TencentARC/t2iadapter_canny_sd14v1
Trained with canny edge detection
A monochrome image with white edges on a black background. t2iadapter_depth_sd15v2 预览 3 t2iadapter_depth_sd15v2 预览 4
TencentARC/t2iadapter_sketch_sd14v1
Trained with PidiNet edge detection
A hand-drawn monochrome image with white outlines on a black background. t2iadapter_depth_sd15v2 预览 5 t2iadapter_depth_sd15v2 预览 6
TencentARC/t2iadapter_depth_sd14v1
Trained with Midas depth estimation
A grayscale image with black representing deep areas and white representing shallow areas. t2iadapter_depth_sd15v2 预览 7 t2iadapter_depth_sd15v2 预览 8
TencentARC/t2iadapter_openpose_sd14v1
Trained with OpenPose bone image
A OpenPose bone image. t2iadapter_depth_sd15v2 预览 9 t2iadapter_depth_sd15v2 预览 10
TencentARC/t2iadapter_keypose_sd14v1
Trained with mmpose skeleton image
A mmpose skeleton image. t2iadapter_depth_sd15v2 预览 11 t2iadapter_depth_sd15v2 预览 12
TencentARC/t2iadapter_seg_sd14v1
Trained with semantic segmentation
An custom segmentation protocol image. t2iadapter_depth_sd15v2 预览 13 t2iadapter_depth_sd15v2 预览 14
TencentARC/t2iadapter_canny_sd15v2
TencentARC/t2iadapter_depth_sd15v2
TencentARC/t2iadapter_sketch_sd15v2
TencentARC/t2iadapter_zoedepth_sd15v1

Example

  1. Dependencies
pip install diffusers transformers controlnet_aux
  1. Run code:
from controlnet_aux import MidasDetector
from PIL import Image
from diffusers import T2IAdapter, StableDiffusionAdapterPipeline
import torch

midas = MidasDetector.from_pretrained("lllyasviel/Annotators")

image = Image.open('./images/depth_input.png')

image = midas(image)

image.save('./images/depth.png')

adapter = T2IAdapter.from_pretrained("TencentARC/t2iadapter_depth_sd15v2", torch_dtype=torch.float16)
pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", adapter=adapter, safety_checker=None, torch_dtype=torch.float16, variant="fp16"
)

pipe.to('cuda')

generator = torch.Generator().manual_seed(1)

depth_out = pipe(prompt="storm trooper giving a speech", image=image, generator=generator).images[0]

depth_out.save('./images/depth_output.png')

depth_input depth depth_output