Riffusion

Riffusion is an app for real-time music generation with stable diffusion.

Read about it at https://www.riffusion.com/about and try it at https://www.riffusion.com/.

• Web app: https://github.com/hmartiro/riffusion-app
• Inference server: https://github.com/hmartiro/riffusion-inference
• Model checkpoint: https://huggingface.co/riffusion/riffusion-model-v1

This repository contains the model files, including:

• a diffusers formated library
• a compiled checkpoint file
• a traced unet for improved inference speed
• a seed image library for use with riffusion-app

Riffusion v1 Model

Riffusion is a latent text-to-image diffusion model capable of generating spectrogram images given any text input. These spectrograms can be converted into audio clips.

The model was created by Seth Forsgren and Hayk Martiros as a hobby project.

You can use the Riffusion model directly, or try the Riffusion web app.

The Riffusion model was created by fine-tuning the Stable-Diffusion-v1-5 checkpoint. Read about Stable Diffusion here 🤗's Stable Diffusion blog.

Model Details

• Developed by: Seth Forsgren, Hayk Martiros
• Model type: Diffusion-based text-to-image generation model
• Language(s): English
• License: The CreativeML OpenRAIL M license is an Open RAIL M license, adapted from the work that BigScience and the RAIL Initiative are jointly carrying in the area of responsible AI licensing. See also the article about the BLOOM Open RAIL license on which our license is based.
• Model Description: This is a model that can be used to generate and modify images based on text prompts. It is a Latent Diffusion Model that uses a fixed, pretrained text encoder (CLIP ViT-L/14) as suggested in the Imagen paper.

Direct Use

The model is intended for research purposes only. Possible research areas and tasks include

• Generation of artworks, audio, and use in creative processes.
• Applications in educational or creative tools.
• Research on generative models.

Citation

If you build on this work, please cite it as follows:

@software{Forsgren_Martiros_2022,
  author = {Forsgren, Seth* and Martiros, Hayk*},
  title = {{Riffusion - Stable diffusion for real-time music generation}},
  url = {https://riffusion.com/about},
  year = {2022}
}