You need to sign in or sign up before continuing.
README.md 14.1 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<!---
Copyright 2022 - The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

Patrick von Platen's avatar
Patrick von Platen committed
17
18
<p align="center">
    <br>
19
    <img src="https://raw.githubusercontent.com/huggingface/diffusers/main/docs/source/en/imgs/diffusers_library.jpg" width="400"/>
Patrick von Platen's avatar
Patrick von Platen committed
20
21
22
    <br>
<p>
<p align="center">
23
24
25
26
27
    <a href="https://github.com/huggingface/diffusers/blob/main/LICENSE"><img alt="GitHub" src="https://img.shields.io/github/license/huggingface/datasets.svg?color=blue"></a>
    <a href="https://github.com/huggingface/diffusers/releases"><img alt="GitHub release" src="https://img.shields.io/github/release/huggingface/diffusers.svg"></a>
    <a href="https://pepy.tech/project/diffusers"><img alt="GitHub release" src="https://static.pepy.tech/badge/diffusers/month"></a>
    <a href="CODE_OF_CONDUCT.md"><img alt="Contributor Covenant" src="https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg"></a>
    <a href="https://twitter.com/diffuserslib"><img alt="X account" src="https://img.shields.io/twitter/url/https/twitter.com/diffuserslib.svg?style=social&label=Follow%20%40diffuserslib"></a>
Patrick von Platen's avatar
Patrick von Platen committed
28
29
</p>

Steven Liu's avatar
Steven Liu committed
30
🤗 Diffusers is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. Whether you're looking for a simple inference solution or training your own diffusion models, 🤗 Diffusers is a modular toolbox that supports both. Our library is designed with a focus on [usability over performance](https://huggingface.co/docs/diffusers/conceptual/philosophy#usability-over-performance), [simple over easy](https://huggingface.co/docs/diffusers/conceptual/philosophy#simple-over-easy), and [customizability over abstractions](https://huggingface.co/docs/diffusers/conceptual/philosophy#tweakable-contributorfriendly-over-abstraction).
Patrick von Platen's avatar
Patrick von Platen committed
31

Steven Liu's avatar
Steven Liu committed
32
🤗 Diffusers offers three core components:
Patrick von Platen's avatar
Patrick von Platen committed
33

Steven Liu's avatar
Steven Liu committed
34
35
- State-of-the-art [diffusion pipelines](https://huggingface.co/docs/diffusers/api/pipelines/overview) that can be run in inference with just a few lines of code.
- Interchangeable noise [schedulers](https://huggingface.co/docs/diffusers/api/schedulers/overview) for different diffusion speeds and output quality.
36
- Pretrained [models](https://huggingface.co/docs/diffusers/api/models/overview) that can be used as building blocks, and combined with schedulers, for creating your own end-to-end diffusion systems.
37

38
39
## Installation

Steven Liu's avatar
Steven Liu committed
40
We recommend installing 🤗 Diffusers in a virtual environment from PyPI or Conda. For more details about installing [PyTorch](https://pytorch.org/get-started/locally/), please refer to their official documentation.
41

Steven Liu's avatar
Steven Liu committed
42
43
44
### PyTorch

With `pip` (official package):
45

46
```bash
47
pip install --upgrade diffusers[torch]
48
49
```

Steven Liu's avatar
Steven Liu committed
50
With `conda` (maintained by the community):
51
52
53
54
55

```sh
conda install -c conda-forge diffusers
```

Steven Liu's avatar
Steven Liu committed
56
### Apple Silicon (M1/M2) support
57

Steven Liu's avatar
Steven Liu committed
58
Please refer to the [How to use Stable Diffusion in Apple Silicon](https://huggingface.co/docs/diffusers/optimization/mps) guide.
59

Patrick von Platen's avatar
Patrick von Platen committed
60
61
## Quickstart

62
Generating outputs is super easy with 🤗 Diffusers. To generate an image from text, use the `from_pretrained` method to load any pretrained diffusion model (browse the [Hub](https://huggingface.co/models?library=diffusers&sort=downloads) for 30,000+ checkpoints):
63

64
```python
Steven Liu's avatar
Steven Liu committed
65
from diffusers import DiffusionPipeline
Patrick von Platen's avatar
Patrick von Platen committed
66
import torch
67

68
pipeline = DiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16)
Steven Liu's avatar
Steven Liu committed
69
70
pipeline.to("cuda")
pipeline("An image of a squirrel in Picasso style").images[0]
71
72
```

Steven Liu's avatar
Steven Liu committed
73
You can also dig into the models and schedulers toolbox to build your own diffusion system:
74
75

```python
Steven Liu's avatar
Steven Liu committed
76
from diffusers import DDPMScheduler, UNet2DModel
77
from PIL import Image
78
import torch
79

Steven Liu's avatar
Steven Liu committed
80
81
82
83
84
scheduler = DDPMScheduler.from_pretrained("google/ddpm-cat-256")
model = UNet2DModel.from_pretrained("google/ddpm-cat-256").to("cuda")
scheduler.set_timesteps(50)

sample_size = model.config.sample_size
85
noise = torch.randn((1, 3, sample_size, sample_size), device="cuda")
Steven Liu's avatar
Steven Liu committed
86
87
88
89
90
input = noise

for t in scheduler.timesteps:
    with torch.no_grad():
        noisy_residual = model(input, t).sample
qwjaskzxl's avatar
qwjaskzxl committed
91
92
        prev_noisy_sample = scheduler.step(noisy_residual, t, input).prev_sample
        input = prev_noisy_sample
Steven Liu's avatar
Steven Liu committed
93
94
95

image = (input / 2 + 0.5).clamp(0, 1)
image = image.cpu().permute(0, 2, 3, 1).numpy()[0]
qwjaskzxl's avatar
qwjaskzxl committed
96
image = Image.fromarray((image * 255).round().astype("uint8"))
Steven Liu's avatar
Steven Liu committed
97
98
99
100
101
102
103
104
105
image
```

Check out the [Quickstart](https://huggingface.co/docs/diffusers/quicktour) to launch your diffusion journey today!

## How to navigate the documentation

| **Documentation**                                                   | **What can I learn?**                                                                                                                                                                           |
|---------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
Patrick von Platen's avatar
Patrick von Platen committed
106
| [Tutorial](https://huggingface.co/docs/diffusers/tutorials/tutorial_overview)                                                            | A basic crash course for learning how to use the library's most important features like using models and schedulers to build your own diffusion system, and training your own diffusion model.  |
107
108
| [Loading](https://huggingface.co/docs/diffusers/using-diffusers/loading)                                                             | Guides for how to load and configure all the components (pipelines, models, and schedulers) of the library, as well as how to use different schedulers.                                         |
| [Pipelines for inference](https://huggingface.co/docs/diffusers/using-diffusers/overview_techniques)                                             | Guides for how to use pipelines for different inference tasks, batched generation, controlling generated outputs and randomness, and how to contribute a pipeline to the library.               |
109
| [Optimization](https://huggingface.co/docs/diffusers/optimization/fp16)                                                        | Guides for how to optimize your diffusion model to run faster and consume less memory.                                                                                                          |
Steven Liu's avatar
Steven Liu committed
110
| [Training](https://huggingface.co/docs/diffusers/training/overview) | Guides for how to train a diffusion model for different tasks with different training techniques.                                                                                               |
Patrick von Platen's avatar
Patrick von Platen committed
111
112
## Contribution

113
We ❤️  contributions from the open-source community!
Patrick von Platen's avatar
Patrick von Platen committed
114
115
116
117
118
119
If you want to contribute to this library, please check out our [Contribution guide](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md).
You can look out for [issues](https://github.com/huggingface/diffusers/issues) you'd like to tackle to contribute to the library.
- See [Good first issues](https://github.com/huggingface/diffusers/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) for general opportunities to contribute
- See [New model/pipeline](https://github.com/huggingface/diffusers/issues?q=is%3Aopen+is%3Aissue+label%3A%22New+pipeline%2Fmodel%22) to contribute exciting new diffusion models / diffusion pipelines
- See [New scheduler](https://github.com/huggingface/diffusers/issues?q=is%3Aopen+is%3Aissue+label%3A%22New+scheduler%22)

120
Also, say 👋 in our public Discord channel <a href="https://discord.gg/G7tWnz98XR"><img alt="Join us on Discord" src="https://img.shields.io/discord/823813159592001537?color=5865F2&logo=discord&logoColor=white"></a>. We discuss the hottest trends about diffusion models, help each other with contributions, personal projects or just hang out ☕.
Patrick von Platen's avatar
Patrick von Platen committed
121

Patrick von Platen's avatar
Patrick von Platen committed
122
123
124
125
126
127
128
129
130
131
132

## Popular Tasks & Pipelines

<table>
  <tr>
    <th>Task</th>
    <th>Pipeline</th>
    <th>🤗 Hub</th>
  </tr>
  <tr style="border-top: 2px solid black">
    <td>Unconditional Image Generation</td>
133
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/ddpm"> DDPM </a></td>
Patrick von Platen's avatar
Patrick von Platen committed
134
135
136
137
    <td><a href="https://huggingface.co/google/ddpm-ema-church-256"> google/ddpm-ema-church-256 </a></td>
  </tr>
  <tr style="border-top: 2px solid black">
    <td>Text-to-Image</td>
138
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img">Stable Diffusion Text-to-Image</a></td>
139
      <td><a href="https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5"> stable-diffusion-v1-5/stable-diffusion-v1-5 </a></td>
Patrick von Platen's avatar
Patrick von Platen committed
140
141
142
  </tr>
  <tr>
    <td>Text-to-Image</td>
143
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/unclip">unCLIP</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
144
145
146
147
      <td><a href="https://huggingface.co/kakaobrain/karlo-v1-alpha"> kakaobrain/karlo-v1-alpha </a></td>
  </tr>
  <tr>
    <td>Text-to-Image</td>
148
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/deepfloyd_if">DeepFloyd IF</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
149
150
      <td><a href="https://huggingface.co/DeepFloyd/IF-I-XL-v1.0"> DeepFloyd/IF-I-XL-v1.0 </a></td>
  </tr>
151
152
153
154
155
  <tr>
    <td>Text-to-Image</td>
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/kandinsky">Kandinsky</a></td>
      <td><a href="https://huggingface.co/kandinsky-community/kandinsky-2-2-decoder"> kandinsky-community/kandinsky-2-2-decoder </a></td>
  </tr>
Patrick von Platen's avatar
Patrick von Platen committed
156
157
  <tr style="border-top: 2px solid black">
    <td>Text-guided Image-to-Image</td>
158
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/controlnet">ControlNet</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
159
160
161
162
      <td><a href="https://huggingface.co/lllyasviel/sd-controlnet-canny"> lllyasviel/sd-controlnet-canny </a></td>
  </tr>
  <tr>
    <td>Text-guided Image-to-Image</td>
163
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/pix2pix">InstructPix2Pix</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
164
165
166
167
      <td><a href="https://huggingface.co/timbrooks/instruct-pix2pix"> timbrooks/instruct-pix2pix </a></td>
  </tr>
  <tr>
    <td>Text-guided Image-to-Image</td>
168
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/img2img">Stable Diffusion Image-to-Image</a></td>
169
      <td><a href="https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5"> stable-diffusion-v1-5/stable-diffusion-v1-5 </a></td>
Patrick von Platen's avatar
Patrick von Platen committed
170
171
172
  </tr>
  <tr style="border-top: 2px solid black">
    <td>Text-guided Image Inpainting</td>
173
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/inpaint">Stable Diffusion Inpainting</a></td>
174
      <td><a href="https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting"> stable-diffusion-v1-5/stable-diffusion-inpainting </a></td>
Patrick von Platen's avatar
Patrick von Platen committed
175
176
177
  </tr>
  <tr style="border-top: 2px solid black">
    <td>Image Variation</td>
178
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/image_variation">Stable Diffusion Image Variation</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
179
180
181
182
      <td><a href="https://huggingface.co/lambdalabs/sd-image-variations-diffusers"> lambdalabs/sd-image-variations-diffusers </a></td>
  </tr>
  <tr style="border-top: 2px solid black">
    <td>Super Resolution</td>
183
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/upscale">Stable Diffusion Upscale</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
184
185
186
187
      <td><a href="https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler"> stabilityai/stable-diffusion-x4-upscaler </a></td>
  </tr>
  <tr>
    <td>Super Resolution</td>
188
    <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/latent_upscale">Stable Diffusion Latent Upscale</a></td>
Patrick von Platen's avatar
Patrick von Platen committed
189
190
191
192
      <td><a href="https://huggingface.co/stabilityai/sd-x2-latent-upscaler"> stabilityai/sd-x2-latent-upscaler </a></td>
  </tr>
</table>

Patrick von Platen's avatar
Patrick von Platen committed
193
## Popular libraries using 🧨 Diffusers
Patrick von Platen's avatar
Patrick von Platen committed
194

195
196
- https://github.com/microsoft/TaskMatrix
- https://github.com/invoke-ai/InvokeAI
197
- https://github.com/InstantID/InstantID
198
199
- https://github.com/apple/ml-stable-diffusion
- https://github.com/Sanster/lama-cleaner
Patrick von Platen's avatar
Patrick von Platen committed
200
- https://github.com/IDEA-Research/Grounded-Segment-Anything
201
202
- https://github.com/ashawkey/stable-dreamfusion
- https://github.com/deep-floyd/IF
Patrick von Platen's avatar
Patrick von Platen committed
203
204
- https://github.com/bentoml/BentoML
- https://github.com/bmaltais/kohya_ss
205
- +14,000 other amazing GitHub repositories 💪
Patrick von Platen's avatar
Patrick von Platen committed
206

207
Thank you for using us ❤️.
Patrick von Platen's avatar
Patrick von Platen committed
208

209
210
211
212
213
214
## Credits

This library concretizes previous work by many different authors and would not have been possible without their great research and implementations. We'd like to thank, in particular, the following implementations which have helped us in our development and without which the API could not have been as polished today:

- @CompVis' latent diffusion models library, available [here](https://github.com/CompVis/latent-diffusion)
- @hojonathanho original DDPM implementation, available [here](https://github.com/hojonathanho/diffusion) as well as the extremely useful translation into PyTorch by @pesser, available [here](https://github.com/pesser/pytorch_diffusion)
Steven Liu's avatar
Steven Liu committed
215
- @ermongroup's DDIM implementation, available [here](https://github.com/ermongroup/ddim)
216
217
- @yang-song's Score-VE and Score-VP implementations, available [here](https://github.com/yang-song/score_sde_pytorch)

Patrick von Platen's avatar
Patrick von Platen committed
218
We also want to thank @heejkoo for the very helpful overview of papers, code and resources on diffusion models, available [here](https://github.com/heejkoo/Awesome-Diffusion-Models) as well as @crowsonkb and @rromb for useful discussions and insights.
Patrick von Platen's avatar
Patrick von Platen committed
219
220
221

## Citation

Patrick von Platen's avatar
Patrick von Platen committed
222
```bibtex
Patrick von Platen's avatar
Patrick von Platen committed
223
@misc{von-platen-etal-2022-diffusers,
224
  author = {Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Dhruv Nair and Sayak Paul and William Berman and Yiyi Xu and Steven Liu and Thomas Wolf},
Patrick von Platen's avatar
Patrick von Platen committed
225
226
227
228
229
230
  title = {Diffusers: State-of-the-art diffusion models},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/huggingface/diffusers}}
}
Patrick von Platen's avatar
Patrick von Platen committed
231
```