Restore missing RGB channels

Restore a missing channel of a RGB image by using ControlNet to guide image generation of Stable Diffusion to infer missing channel from the other two channels.

• See accompanying discussion at github.com - Channels RGB with detailed report and evaluations.
• To restore images with missing channels you can use this space.
• For evaluation images see the corresponding .zip's at "files".
• To run your own evaluations you can use this script at gitlab.com.

Training

accelerate launch train_controlnet.py \
  --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
  --train_batch_size=4 \
  --gradient_accumulation_steps=8 \
  --proportion_empty_prompts=0.5
  --mixed_precision="fp16" \
  --learning_rate=1e-5 \
  --enable_xformers_memory_efficient_attention \
  --use_8bit_adam \
  --set_grads_to_none \
  --seed=0 \
  --num_train_epochs=2

Image dataset

• laion2B-en aesthetics>=6.5 dataset
• --min_image_size 512 --max_aspect_ratio 2 --resize_mode="center_crop" --image_size 512
• Cleaned with fastdup default settings
• Data augmented with right-left flipped images
• Resulting in 214244 images
• Set whole channel to 0 by alternating between R-G-B channels