README.md 14.1 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<!---
Copyright 2022 - The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

Patrick von Platen's avatar
Patrick von Platen committed
17
18
<p align="center">
    <br>
19
    <img src="https://raw.githubusercontent.com/huggingface/diffusers/main/docs/source/en/imgs/diffusers_library.jpg" width="400"/>
Patrick von Platen's avatar
Patrick von Platen committed
20
21
22
    <br>
<p>
<p align="center">
Anton Lozhkov's avatar
Anton Lozhkov committed
23
    <a href="https://github.com/huggingface/diffusers/blob/main/LICENSE">
Patrick von Platen's avatar
Patrick von Platen committed
24
25
26
        <img alt="GitHub" src="https://img.shields.io/github/license/huggingface/datasets.svg?color=blue">
    </a>
    <a href="https://github.com/huggingface/diffusers/releases">
Anton Lozhkov's avatar
Anton Lozhkov committed
27
        <img alt="GitHub release" src="https://img.shields.io/github/release/huggingface/diffusers.svg">
Patrick von Platen's avatar
Patrick von Platen committed
28
    </a>
Patrick von Platen's avatar
Patrick von Platen committed
29
30
31
    <a href="https://pepy.tech/project/diffusers">
        <img alt="GitHub release" src="https://static.pepy.tech/badge/diffusers/month">
    </a>
Patrick von Platen's avatar
Patrick von Platen committed
32
    <a href="CODE_OF_CONDUCT.md">
33
34
35
36
        <img alt="Contributor Covenant" src="https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg">
    </a>
    <a href="https://twitter.com/diffuserslib">
        <img alt="X account" src="https://img.shields.io/twitter/url/https/twitter.com/diffuserslib.svg?style=social&label=Follow%20%40diffuserslib">
Patrick von Platen's avatar
Patrick von Platen committed
37
38
39
    </a>
</p>

Steven Liu's avatar
Steven Liu committed
40
🤗 Diffusers is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. Whether you're looking for a simple inference solution or training your own diffusion models, 🤗 Diffusers is a modular toolbox that supports both. Our library is designed with a focus on [usability over performance](https://huggingface.co/docs/diffusers/conceptual/philosophy#usability-over-performance), [simple over easy](https://huggingface.co/docs/diffusers/conceptual/philosophy#simple-over-easy), and [customizability over abstractions](https://huggingface.co/docs/diffusers/conceptual/philosophy#tweakable-contributorfriendly-over-abstraction).
Patrick von Platen's avatar
Patrick von Platen committed
41

Steven Liu's avatar
Steven Liu committed
42
🤗 Diffusers offers three core components:
Patrick von Platen's avatar
Patrick von Platen committed
43

Steven Liu's avatar
Steven Liu committed
44
45
- State-of-the-art [diffusion pipelines](https://huggingface.co/docs/diffusers/api/pipelines/overview) that can be run in inference with just a few lines of code.
- Interchangeable noise [schedulers](https://huggingface.co/docs/diffusers/api/schedulers/overview) for different diffusion speeds and output quality.
46
- Pretrained [models](https://huggingface.co/docs/diffusers/api/models/overview) that can be used as building blocks, and combined with schedulers, for creating your own end-to-end diffusion systems.
47

48
49
## Installation

50
We recommend installing 🤗 Diffusers in a virtual environment from PyPI or Conda. For more details about installing [PyTorch](https://pytorch.org/get-started/locally/) and [Flax](https://flax.readthedocs.io/en/latest/#installation), please refer to their official documentation.
51

Steven Liu's avatar
Steven Liu committed
52
53
54
### PyTorch

With `pip` (official package):
55

56
```bash
57
pip install --upgrade diffusers[torch]
58
59
```

Steven Liu's avatar
Steven Liu committed
60
With `conda` (maintained by the community):
61
62
63
64
65

```sh
conda install -c conda-forge diffusers
```

Steven Liu's avatar
Steven Liu committed
66
### Flax
67

Steven Liu's avatar
Steven Liu committed
68
With `pip` (official package):
69
70
71
72
73

```bash
pip install --upgrade diffusers[flax]
```

Steven Liu's avatar
Steven Liu committed
74
### Apple Silicon (M1/M2) support
75

Steven Liu's avatar
Steven Liu committed
76
Please refer to the [How to use Stable Diffusion in Apple Silicon](https://huggingface.co/docs/diffusers/optimization/mps) guide.
77

Patrick von Platen's avatar
Patrick von Platen committed
78
79
## Quickstart

M. Tolga Cangöz's avatar
M. Tolga Cangöz committed
80
Generating outputs is super easy with 🤗 Diffusers. To generate an image from text, use the `from_pretrained` method to load any pretrained diffusion model (browse the [Hub](https://huggingface.co/models?library=diffusers&sort=downloads) for 19000+ checkpoints):
81

82
```python
Steven Liu's avatar
Steven Liu committed
83
from diffusers import DiffusionPipeline
Patrick von Platen's avatar
Patrick von Platen committed
84
import torch
85

Patrick von Platen's avatar
Patrick von Platen committed
86
pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
Steven Liu's avatar
Steven Liu committed
87
88
pipeline.to("cuda")
pipeline("An image of a squirrel in Picasso style").images[0]
89
90
```

Steven Liu's avatar
Steven Liu committed
91
You can also dig into the models and schedulers toolbox to build your own diffusion system:
92
93

```python
Steven Liu's avatar
Steven Liu committed
94
from diffusers import DDPMScheduler, UNet2DModel
95
from PIL import Image
96
import torch
97

Steven Liu's avatar
Steven Liu committed
98
99
100
101
102
scheduler = DDPMScheduler.from_pretrained("google/ddpm-cat-256")
model = UNet2DModel.from_pretrained("google/ddpm-cat-256").to("cuda")
scheduler.set_timesteps(50)

sample_size = model.config.sample_size
103
noise = torch.randn((1, 3, sample_size, sample_size), device="cuda")
Steven Liu's avatar
Steven Liu committed
104
105
106
107
108
input = noise

for t in scheduler.timesteps:
    with torch.no_grad():
        noisy_residual = model(input, t).sample
qwjaskzxl's avatar
qwjaskzxl committed
109
110
        prev_noisy_sample = scheduler.step(noisy_residual, t, input).prev_sample
        input = prev_noisy_sample
Steven Liu's avatar
Steven Liu committed
111
112
113

image = (input / 2 + 0.5).clamp(0, 1)
image = image.cpu().permute(0, 2, 3, 1).numpy()[0]
qwjaskzxl's avatar
qwjaskzxl committed
114
image = Image.fromarray((image * 255).round().astype("uint8"))
Steven Liu's avatar
Steven Liu committed
115
116
117
118
119
120
121
122
123
image
```

Check out the [Quickstart](https://huggingface.co/docs/diffusers/quicktour) to launch your diffusion journey today!

## How to navigate the documentation

| **Documentation**                                                   | **What can I learn?**                                                                                                                                                                           |
|---------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
Patrick von Platen's avatar
Patrick von Platen committed
124
125
126
127
| [Tutorial](https://huggingface.co/docs/diffusers/tutorials/tutorial_overview)                                                            | A basic crash course for learning how to use the library's most important features like using models and schedulers to build your own diffusion system, and training your own diffusion model.  |
| [Loading](https://huggingface.co/docs/diffusers/using-diffusers/loading_overview)                                                             | Guides for how to load and configure all the components (pipelines, models, and schedulers) of the library, as well as how to use different schedulers.                                         |
| [Pipelines for inference](https://huggingface.co/docs/diffusers/using-diffusers/pipeline_overview)                                             | Guides for how to use pipelines for different inference tasks, batched generation, controlling generated outputs and randomness, and how to contribute a pipeline to the library.               |
| [Optimization](https://huggingface.co/docs/diffusers/optimization/opt_overview)                                                        | Guides for how to optimize your diffusion model to run faster and consume less memory.                                                                                                          |
Steven Liu's avatar
Steven Liu committed
128
| [Training](https://huggingface.co/docs/diffusers/training/overview) | Guides for how to train a diffusion model for different tasks with different training techniques.                                                                                               |
Patrick von Platen's avatar
Patrick von Platen committed
129
130
## Contribution

131
We ❤️  contributions from the open-source community!
Patrick von Platen's avatar
Patrick von Platen committed
132
133
134
135
136
137
If you want to contribute to this library, please check out our [Contribution guide](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md).
You can look out for [issues](https://github.com/huggingface/diffusers/issues) you'd like to tackle to contribute to the library.
- See [Good first issues](https://github.com/huggingface/diffusers/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) for general opportunities to contribute
- See [New model/pipeline](https://github.com/huggingface/diffusers/issues?q=is%3Aopen+is%3Aissue+label%3A%22New+pipeline%2Fmodel%22) to contribute exciting new diffusion models / diffusion pipelines
- See [New scheduler](https://github.com/huggingface/diffusers/issues?q=is%3Aopen+is%3Aissue+label%3A%22New+scheduler%22)

138
Also, say 👋 in our public Discord channel <a href="https://discord.gg/G7tWnz98XR"><img alt="Join us on Discord" src="https://img.shields.io/discord/823813159592001537?color=5865F2&logo=discord&logoColor=white"></a>. We discuss the hottest trends about diffusion models, help each other with contributions, personal projects or just hang out ☕.
Patrick von Platen's avatar
Patrick von Platen committed
139

Patrick von Platen's avatar
Patrick von Platen committed
140
141
142
143
144
145
146
147
148
149
150

## Popular Tasks & Pipelines

<table>
  <tr>
    <th>Task</th>
    <th>Pipeline</th>
    <th>🤗 Hub</th>
  </tr>
  <tr style="border-top: 2px solid black">
    <td>Unconditional Image Generation</td>
151
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/ddpm"> DDPM </a></td>
Patrick von Platen's avatar
Patrick von Platen committed
152
153
154
155
    <td><a href="https://huggingface.co/google/ddpm-ema-church-256"> google/ddpm-ema-church-256 </a></td>
  </tr>
  <tr style="border-top: 2px solid black">
    <td>Text-to-Image</td>
156
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img">Stable Diffusion Text-to-Image</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
157
158
159
160
      <td><a href="https://huggingface.co/runwayml/stable-diffusion-v1-5"> runwayml/stable-diffusion-v1-5 </a></td>
  </tr>
  <tr>
    <td>Text-to-Image</td>
161
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/unclip">unCLIP</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
162
163
164
165
      <td><a href="https://huggingface.co/kakaobrain/karlo-v1-alpha"> kakaobrain/karlo-v1-alpha </a></td>
  </tr>
  <tr>
    <td>Text-to-Image</td>
166
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/deepfloyd_if">DeepFloyd IF</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
167
168
      <td><a href="https://huggingface.co/DeepFloyd/IF-I-XL-v1.0"> DeepFloyd/IF-I-XL-v1.0 </a></td>
  </tr>
169
170
171
172
173
  <tr>
    <td>Text-to-Image</td>
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/kandinsky">Kandinsky</a></td>
      <td><a href="https://huggingface.co/kandinsky-community/kandinsky-2-2-decoder"> kandinsky-community/kandinsky-2-2-decoder </a></td>
  </tr>
Patrick von Platen's avatar
Patrick von Platen committed
174
175
  <tr style="border-top: 2px solid black">
    <td>Text-guided Image-to-Image</td>
176
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/controlnet">ControlNet</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
177
178
179
180
      <td><a href="https://huggingface.co/lllyasviel/sd-controlnet-canny"> lllyasviel/sd-controlnet-canny </a></td>
  </tr>
  <tr>
    <td>Text-guided Image-to-Image</td>
181
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/pix2pix">InstructPix2Pix</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
182
183
184
185
      <td><a href="https://huggingface.co/timbrooks/instruct-pix2pix"> timbrooks/instruct-pix2pix </a></td>
  </tr>
  <tr>
    <td>Text-guided Image-to-Image</td>
186
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/img2img">Stable Diffusion Image-to-Image</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
187
188
189
190
      <td><a href="https://huggingface.co/runwayml/stable-diffusion-v1-5"> runwayml/stable-diffusion-v1-5 </a></td>
  </tr>
  <tr style="border-top: 2px solid black">
    <td>Text-guided Image Inpainting</td>
191
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/inpaint">Stable Diffusion Inpainting</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
192
193
194
195
      <td><a href="https://huggingface.co/runwayml/stable-diffusion-inpainting"> runwayml/stable-diffusion-inpainting </a></td>
  </tr>
  <tr style="border-top: 2px solid black">
    <td>Image Variation</td>
196
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/image_variation">Stable Diffusion Image Variation</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
197
198
199
200
      <td><a href="https://huggingface.co/lambdalabs/sd-image-variations-diffusers"> lambdalabs/sd-image-variations-diffusers </a></td>
  </tr>
  <tr style="border-top: 2px solid black">
    <td>Super Resolution</td>
201
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/upscale">Stable Diffusion Upscale</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
202
203
204
205
      <td><a href="https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler"> stabilityai/stable-diffusion-x4-upscaler </a></td>
  </tr>
  <tr>
    <td>Super Resolution</td>
206
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/latent_upscale">Stable Diffusion Latent Upscale</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
207
208
209
210
      <td><a href="https://huggingface.co/stabilityai/sd-x2-latent-upscaler"> stabilityai/sd-x2-latent-upscaler </a></td>
  </tr>
</table>

Patrick von Platen's avatar
Patrick von Platen committed
211
## Popular libraries using 🧨 Diffusers
Patrick von Platen's avatar
Patrick von Platen committed
212

213
214
215
216
- https://github.com/microsoft/TaskMatrix
- https://github.com/invoke-ai/InvokeAI
- https://github.com/apple/ml-stable-diffusion
- https://github.com/Sanster/lama-cleaner
Patrick von Platen's avatar
Patrick von Platen committed
217
- https://github.com/IDEA-Research/Grounded-Segment-Anything
218
219
- https://github.com/ashawkey/stable-dreamfusion
- https://github.com/deep-floyd/IF
Patrick von Platen's avatar
Patrick von Platen committed
220
221
- https://github.com/bentoml/BentoML
- https://github.com/bmaltais/kohya_ss
M. Tolga Cangöz's avatar
M. Tolga Cangöz committed
222
- +8000 other amazing GitHub repositories 💪
Patrick von Platen's avatar
Patrick von Platen committed
223

224
Thank you for using us ❤️.
Patrick von Platen's avatar
Patrick von Platen committed
225

226
227
228
229
230
231
## Credits

This library concretizes previous work by many different authors and would not have been possible without their great research and implementations. We'd like to thank, in particular, the following implementations which have helped us in our development and without which the API could not have been as polished today:

- @CompVis' latent diffusion models library, available [here](https://github.com/CompVis/latent-diffusion)
- @hojonathanho original DDPM implementation, available [here](https://github.com/hojonathanho/diffusion) as well as the extremely useful translation into PyTorch by @pesser, available [here](https://github.com/pesser/pytorch_diffusion)
Steven Liu's avatar
Steven Liu committed
232
- @ermongroup's DDIM implementation, available [here](https://github.com/ermongroup/ddim)
233
234
- @yang-song's Score-VE and Score-VP implementations, available [here](https://github.com/yang-song/score_sde_pytorch)

Patrick von Platen's avatar
Patrick von Platen committed
235
We also want to thank @heejkoo for the very helpful overview of papers, code and resources on diffusion models, available [here](https://github.com/heejkoo/Awesome-Diffusion-Models) as well as @crowsonkb and @rromb for useful discussions and insights.
Patrick von Platen's avatar
Patrick von Platen committed
236
237
238

## Citation

Patrick von Platen's avatar
Patrick von Platen committed
239
```bibtex
Patrick von Platen's avatar
Patrick von Platen committed
240
@misc{von-platen-etal-2022-diffusers,
Patrick von Platen's avatar
Patrick von Platen committed
241
  author = {Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Thomas Wolf},
Patrick von Platen's avatar
Patrick von Platen committed
242
243
244
245
246
247
  title = {Diffusers: State-of-the-art diffusion models},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/huggingface/diffusers}}
}
Patrick von Platen's avatar
Patrick von Platen committed
248
```