Unverified Commit 1b456bd5 authored by Ali Imran's avatar Ali Imran Committed by GitHub
Browse files

docs: cleanup of runway model (#12503)

* cleanup of runway model

* quality fixes
parent af769881
...@@ -171,7 +171,7 @@ Also, say 👋 in our public Discord channel <a href="https://discord.gg/G7tWnz9 ...@@ -171,7 +171,7 @@ Also, say 👋 in our public Discord channel <a href="https://discord.gg/G7tWnz9
<tr style="border-top: 2px solid black"> <tr style="border-top: 2px solid black">
<td>Text-guided Image Inpainting</td> <td>Text-guided Image Inpainting</td>
<td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/inpaint">Stable Diffusion Inpainting</a></td> <td><a href="https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/inpaint">Stable Diffusion Inpainting</a></td>
<td><a href="https://huggingface.co/runwayml/stable-diffusion-inpainting"> runwayml/stable-diffusion-inpainting </a></td> <td><a href="https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting"> stable-diffusion-v1-5/stable-diffusion-inpainting </a></td>
</tr> </tr>
<tr style="border-top: 2px solid black"> <tr style="border-top: 2px solid black">
<td>Image Variation</td> <td>Image Variation</td>
......
...@@ -39,7 +39,7 @@ mask_url = "https://huggingface.co/datasets/hf-internal-testing/diffusers-images ...@@ -39,7 +39,7 @@ mask_url = "https://huggingface.co/datasets/hf-internal-testing/diffusers-images
original_image = load_image(img_url).resize((512, 512)) original_image = load_image(img_url).resize((512, 512))
mask_image = load_image(mask_url).resize((512, 512)) mask_image = load_image(mask_url).resize((512, 512))
pipe = StableDiffusionInpaintPipeline.from_pretrained("runwayml/stable-diffusion-inpainting") pipe = StableDiffusionInpaintPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-inpainting")
pipe.vae = AsymmetricAutoencoderKL.from_pretrained("cross-attention/asymmetric-autoencoder-kl-x-1-5") pipe.vae = AsymmetricAutoencoderKL.from_pretrained("cross-attention/asymmetric-autoencoder-kl-x-1-5")
pipe.to("cuda") pipe.to("cuda")
......
...@@ -21,7 +21,7 @@ The Stable Diffusion model can also infer depth based on an image using [MiDaS]( ...@@ -21,7 +21,7 @@ The Stable Diffusion model can also infer depth based on an image using [MiDaS](
> [!TIP] > [!TIP]
> Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently! > Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently!
> >
> If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations! > If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis) and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!
## StableDiffusionDepth2ImgPipeline ## StableDiffusionDepth2ImgPipeline
......
...@@ -21,14 +21,14 @@ The Stable Diffusion model can also be applied to inpainting which lets you edit ...@@ -21,14 +21,14 @@ The Stable Diffusion model can also be applied to inpainting which lets you edit
## Tips ## Tips
It is recommended to use this pipeline with checkpoints that have been specifically fine-tuned for inpainting, such It is recommended to use this pipeline with checkpoints that have been specifically fine-tuned for inpainting, such
as [runwayml/stable-diffusion-inpainting](https://huggingface.co/runwayml/stable-diffusion-inpainting). Default as [stable-diffusion-v1-5/stable-diffusion-inpainting](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting). Default
text-to-image Stable Diffusion checkpoints, such as text-to-image Stable Diffusion checkpoints, such as
[stable-diffusion-v1-5/stable-diffusion-v1-5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) are also compatible but they might be less performant. [stable-diffusion-v1-5/stable-diffusion-v1-5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) are also compatible but they might be less performant.
> [!TIP] > [!TIP]
> Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently! > Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently!
> >
> If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations! > If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis) and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!
## StableDiffusionInpaintPipeline ## StableDiffusionInpaintPipeline
......
...@@ -17,7 +17,7 @@ The Stable Diffusion latent upscaler model was created by [Katherine Crowson](ht ...@@ -17,7 +17,7 @@ The Stable Diffusion latent upscaler model was created by [Katherine Crowson](ht
> [!TIP] > [!TIP]
> Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently! > Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently!
> >
> If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations! > If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis) and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!
## StableDiffusionLatentUpscalePipeline ## StableDiffusionLatentUpscalePipeline
......
...@@ -22,7 +22,7 @@ Stable Diffusion is trained on 512x512 images from a subset of the LAION-5B data ...@@ -22,7 +22,7 @@ Stable Diffusion is trained on 512x512 images from a subset of the LAION-5B data
For more details about how Stable Diffusion works and how it differs from the base latent diffusion model, take a look at the Stability AI [announcement](https://stability.ai/blog/stable-diffusion-announcement) and our own [blog post](https://huggingface.co/blog/stable_diffusion#how-does-stable-diffusion-work) for more technical details. For more details about how Stable Diffusion works and how it differs from the base latent diffusion model, take a look at the Stability AI [announcement](https://stability.ai/blog/stable-diffusion-announcement) and our own [blog post](https://huggingface.co/blog/stable_diffusion#how-does-stable-diffusion-work) for more technical details.
You can find the original codebase for Stable Diffusion v1.0 at [CompVis/stable-diffusion](https://github.com/CompVis/stable-diffusion) and Stable Diffusion v2.0 at [Stability-AI/stablediffusion](https://github.com/Stability-AI/stablediffusion) as well as their original scripts for various tasks. Additional official checkpoints for the different Stable Diffusion versions and tasks can be found on the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations. Explore these organizations to find the best checkpoint for your use-case! You can find the original codebase for Stable Diffusion v1.0 at [CompVis/stable-diffusion](https://github.com/CompVis/stable-diffusion) and Stable Diffusion v2.0 at [Stability-AI/stablediffusion](https://github.com/Stability-AI/stablediffusion) as well as their original scripts for various tasks. Additional official checkpoints for the different Stable Diffusion versions and tasks can be found on the [CompVis](https://huggingface.co/CompVis) and [Stability AI](https://huggingface.co/stabilityai) Hub organizations. Explore these organizations to find the best checkpoint for your use-case!
The table below summarizes the available Stable Diffusion pipelines, their supported tasks, and an interactive demo: The table below summarizes the available Stable Diffusion pipelines, their supported tasks, and an interactive demo:
...@@ -64,7 +64,7 @@ The table below summarizes the available Stable Diffusion pipelines, their suppo ...@@ -64,7 +64,7 @@ The table below summarizes the available Stable Diffusion pipelines, their suppo
<a href="./inpaint">StableDiffusionInpaint</a> <a href="./inpaint">StableDiffusionInpaint</a>
</td> </td>
<td class="px-4 py-2 text-gray-700">inpainting</td> <td class="px-4 py-2 text-gray-700">inpainting</td>
<td class="px-4 py-2"><a href="https://huggingface.co/spaces/runwayml/stable-diffusion-inpainting"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue"/></a> <td class="px-4 py-2"><a href="https://huggingface.co/spaces/stable-diffusion-v1-5/stable-diffusion-inpainting"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue"/></a>
</td> </td>
</tr> </tr>
<tr> <tr>
......
...@@ -36,7 +36,7 @@ Here are some examples for how to use Stable Diffusion 2 for each task: ...@@ -36,7 +36,7 @@ Here are some examples for how to use Stable Diffusion 2 for each task:
> [!TIP] > [!TIP]
> Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently! > Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently!
> >
> If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations! > If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis) and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!
## Text-to-image ## Text-to-image
......
...@@ -25,7 +25,7 @@ The abstract from the paper is: ...@@ -25,7 +25,7 @@ The abstract from the paper is:
> [!TIP] > [!TIP]
> Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently! > Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently!
> >
> If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations! > If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis) and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!
## StableDiffusionPipeline ## StableDiffusionPipeline
......
...@@ -21,7 +21,7 @@ The Stable Diffusion upscaler diffusion model was created by the researchers and ...@@ -21,7 +21,7 @@ The Stable Diffusion upscaler diffusion model was created by the researchers and
> [!TIP] > [!TIP]
> Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently! > Make sure to check out the Stable Diffusion [Tips](overview#tips) section to learn how to explore the tradeoff between scheduler speed and quality, and how to reuse pipeline components efficiently!
> >
> If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis), [Runway](https://huggingface.co/runwayml), and [Stability AI](https://huggingface.co/stabilityai) Hub organizations! > If you're interested in using one of the official checkpoints for a task, explore the [CompVis](https://huggingface.co/CompVis) and [Stability AI](https://huggingface.co/stabilityai) Hub organizations!
## StableDiffusionUpscalePipeline ## StableDiffusionUpscalePipeline
......
...@@ -16,12 +16,12 @@ pipeline.unet.config["in_channels"] ...@@ -16,12 +16,12 @@ pipeline.unet.config["in_channels"]
4 4
``` ```
Inpainting requires 9 channels in the input sample. You can check this value in a pretrained inpainting model like [`runwayml/stable-diffusion-inpainting`](https://huggingface.co/runwayml/stable-diffusion-inpainting): Inpainting requires 9 channels in the input sample. You can check this value in a pretrained inpainting model like [`stable-diffusion-v1-5/stable-diffusion-inpainting`](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting):
```py ```py
from diffusers import StableDiffusionPipeline from diffusers import StableDiffusionPipeline
pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-inpainting", use_safetensors=True) pipeline = StableDiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-inpainting", use_safetensors=True)
pipeline.unet.config["in_channels"] pipeline.unet.config["in_channels"]
9 9
``` ```
......
...@@ -215,7 +215,7 @@ from diffusers import AutoPipelineForInpainting, LCMScheduler ...@@ -215,7 +215,7 @@ from diffusers import AutoPipelineForInpainting, LCMScheduler
from diffusers.utils import load_image, make_image_grid from diffusers.utils import load_image, make_image_grid
pipe = AutoPipelineForInpainting.from_pretrained( pipe = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting", "stable-diffusion-v1-5/stable-diffusion-inpainting",
torch_dtype=torch.float16, torch_dtype=torch.float16,
variant="fp16", variant="fp16",
).to("cuda") ).to("cuda")
......
...@@ -112,7 +112,7 @@ blurred_mask ...@@ -112,7 +112,7 @@ blurred_mask
## Popular models ## Popular models
[Stable Diffusion Inpainting](https://huggingface.co/runwayml/stable-diffusion-inpainting), [Stable Diffusion XL (SDXL) Inpainting](https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1), and [Kandinsky 2.2 Inpainting](https://huggingface.co/kandinsky-community/kandinsky-2-2-decoder-inpaint) are among the most popular models for inpainting. SDXL typically produces higher resolution images than Stable Diffusion v1.5, and Kandinsky 2.2 is also capable of generating high-quality images. [Stable Diffusion Inpainting](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting), [Stable Diffusion XL (SDXL) Inpainting](https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1), and [Kandinsky 2.2 Inpainting](https://huggingface.co/kandinsky-community/kandinsky-2-2-decoder-inpaint) are among the most popular models for inpainting. SDXL typically produces higher resolution images than Stable Diffusion v1.5, and Kandinsky 2.2 is also capable of generating high-quality images.
### Stable Diffusion Inpainting ### Stable Diffusion Inpainting
...@@ -124,7 +124,7 @@ from diffusers import AutoPipelineForInpainting ...@@ -124,7 +124,7 @@ from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained( pipeline = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16" "stable-diffusion-v1-5/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16"
) )
pipeline.enable_model_cpu_offload() pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed # remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
...@@ -244,7 +244,7 @@ make_image_grid([init_image, image], rows=1, cols=2) ...@@ -244,7 +244,7 @@ make_image_grid([init_image, image], rows=1, cols=2)
``` ```
</hfoption> </hfoption>
<hfoption id="runwayml/stable-diffusion-inpainting"> <hfoption id="stable-diffusion-v1-5/stable-diffusion-inpainting">
```py ```py
import torch import torch
...@@ -252,7 +252,7 @@ from diffusers import AutoPipelineForInpainting ...@@ -252,7 +252,7 @@ from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained( pipeline = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16" "stable-diffusion-v1-5/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16"
) )
pipeline.enable_model_cpu_offload() pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed # remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
...@@ -278,7 +278,7 @@ make_image_grid([init_image, image], rows=1, cols=2) ...@@ -278,7 +278,7 @@ make_image_grid([init_image, image], rows=1, cols=2)
</div> </div>
<div> <div>
<img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint-specific.png"/> <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint-specific.png"/>
<figcaption class="mt-2 text-center text-sm text-gray-500">runwayml/stable-diffusion-inpainting</figcaption> <figcaption class="mt-2 text-center text-sm text-gray-500">stable-diffusion-v1-5/stable-diffusion-inpainting</figcaption>
</div> </div>
</div> </div>
...@@ -308,7 +308,7 @@ make_image_grid([init_image, image], rows=1, cols=2) ...@@ -308,7 +308,7 @@ make_image_grid([init_image, image], rows=1, cols=2)
``` ```
</hfoption> </hfoption>
<hfoption id="runwayml/stable-diffusion-inpaint"> <hfoption id="stable-diffusion-v1-5/stable-diffusion-inpaint">
```py ```py
import torch import torch
...@@ -316,7 +316,7 @@ from diffusers import AutoPipelineForInpainting ...@@ -316,7 +316,7 @@ from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained( pipeline = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16" "stable-diffusion-v1-5/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16"
) )
pipeline.enable_model_cpu_offload() pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed # remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
...@@ -340,7 +340,7 @@ make_image_grid([init_image, image], rows=1, cols=2) ...@@ -340,7 +340,7 @@ make_image_grid([init_image, image], rows=1, cols=2)
</div> </div>
<div> <div>
<img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/specific-inpaint-basic.png"/> <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/specific-inpaint-basic.png"/>
<figcaption class="mt-2 text-center text-sm text-gray-500">runwayml/stable-diffusion-inpainting</figcaption> <figcaption class="mt-2 text-center text-sm text-gray-500">stable-diffusion-v1-5/stable-diffusion-inpainting</figcaption>
</div> </div>
</div> </div>
...@@ -358,7 +358,7 @@ from diffusers.utils import load_image, make_image_grid ...@@ -358,7 +358,7 @@ from diffusers.utils import load_image, make_image_grid
device = "cuda" device = "cuda"
pipeline = AutoPipelineForInpainting.from_pretrained( pipeline = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting", "stable-diffusion-v1-5/stable-diffusion-inpainting",
torch_dtype=torch.float16, torch_dtype=torch.float16,
variant="fp16" variant="fp16"
) )
...@@ -396,7 +396,7 @@ from diffusers import AutoPipelineForInpainting ...@@ -396,7 +396,7 @@ from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained( pipeline = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16" "stable-diffusion-v1-5/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16"
) )
pipeline.enable_model_cpu_offload() pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed # remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
...@@ -441,7 +441,7 @@ from diffusers import AutoPipelineForInpainting ...@@ -441,7 +441,7 @@ from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained( pipeline = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16" "stable-diffusion-v1-5/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16"
) )
pipeline.enable_model_cpu_offload() pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed # remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
...@@ -481,7 +481,7 @@ from diffusers import AutoPipelineForInpainting ...@@ -481,7 +481,7 @@ from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained( pipeline = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16" "stable-diffusion-v1-5/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16"
) )
pipeline.enable_model_cpu_offload() pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed # remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
...@@ -606,7 +606,7 @@ from diffusers import AutoPipelineForInpainting, AutoPipelineForImage2Image ...@@ -606,7 +606,7 @@ from diffusers import AutoPipelineForInpainting, AutoPipelineForImage2Image
from diffusers.utils import load_image, make_image_grid from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained( pipeline = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16" "stable-diffusion-v1-5/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16"
) )
pipeline.enable_model_cpu_offload() pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed # remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
...@@ -683,7 +683,7 @@ from diffusers import AutoPipelineForInpainting ...@@ -683,7 +683,7 @@ from diffusers import AutoPipelineForInpainting
from diffusers.utils import make_image_grid from diffusers.utils import make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained( pipeline = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16, "stable-diffusion-v1-5/stable-diffusion-inpainting", torch_dtype=torch.float16,
) )
pipeline.enable_model_cpu_offload() pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed # remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
...@@ -714,7 +714,7 @@ controlnet = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_inpai ...@@ -714,7 +714,7 @@ controlnet = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_inpai
# pass ControlNet to the pipeline # pass ControlNet to the pipeline
pipeline = StableDiffusionControlNetInpaintPipeline.from_pretrained( pipeline = StableDiffusionControlNetInpaintPipeline.from_pretrained(
"runwayml/stable-diffusion-inpainting", controlnet=controlnet, torch_dtype=torch.float16, variant="fp16" "stable-diffusion-v1-5/stable-diffusion-inpainting", controlnet=controlnet, torch_dtype=torch.float16, variant="fp16"
) )
pipeline.enable_model_cpu_offload() pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed # remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
......
...@@ -173,7 +173,7 @@ mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data ...@@ -173,7 +173,7 @@ mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data
init_image = download_image(img_url).resize((512, 512)) init_image = download_image(img_url).resize((512, 512))
mask_image = download_image(mask_url).resize((512, 512)) mask_image = download_image(mask_url).resize((512, 512))
path = "runwayml/stable-diffusion-inpainting" path = "stable-diffusion-v1-5/stable-diffusion-inpainting"
run_compile = True # Set True / False run_compile = True # Set True / False
......
...@@ -28,12 +28,12 @@ pipeline.unet.config["in_channels"] ...@@ -28,12 +28,12 @@ pipeline.unet.config["in_channels"]
4 4
``` ```
인페인팅은 입력 샘플에 9개의 채널이 필요합니다. [`runwayml/stable-diffusion-inpainting`](https://huggingface.co/runwayml/stable-diffusion-inpainting)와 같은 사전학습된 인페인팅 모델에서 이 값을 확인할 수 있습니다: 인페인팅은 입력 샘플에 9개의 채널이 필요합니다. [`stable-diffusion-v1-5/stable-diffusion-inpainting`](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting)와 같은 사전학습된 인페인팅 모델에서 이 값을 확인할 수 있습니다:
```py ```py
from diffusers import StableDiffusionPipeline from diffusers import StableDiffusionPipeline
pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-inpainting") pipeline = StableDiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-inpainting")
pipeline.unet.config["in_channels"] pipeline.unet.config["in_channels"]
9 9
``` ```
......
...@@ -14,7 +14,7 @@ specific language governing permissions and limitations under the License. ...@@ -14,7 +14,7 @@ specific language governing permissions and limitations under the License.
[[open-in-colab]] [[open-in-colab]]
[`StableDiffusionInpaintPipeline`]은 마스크와 텍스트 프롬프트를 제공하여 이미지의 특정 부분을 편집할 수 있도록 합니다. 이 기능은 인페인팅 작업을 위해 특별히 훈련된 [`runwayml/stable-diffusion-inpainting`](https://huggingface.co/runwayml/stable-diffusion-inpainting)과 같은 Stable Diffusion 버전을 사용합니다. [`StableDiffusionInpaintPipeline`]은 마스크와 텍스트 프롬프트를 제공하여 이미지의 특정 부분을 편집할 수 있도록 합니다. 이 기능은 인페인팅 작업을 위해 특별히 훈련된 [`stable-diffusion-v1-5/stable-diffusion-inpainting`](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting)과 같은 Stable Diffusion 버전을 사용합니다.
먼저 [`StableDiffusionInpaintPipeline`] 인스턴스를 불러옵니다: 먼저 [`StableDiffusionInpaintPipeline`] 인스턴스를 불러옵니다:
...@@ -27,7 +27,7 @@ from io import BytesIO ...@@ -27,7 +27,7 @@ from io import BytesIO
from diffusers import StableDiffusionInpaintPipeline from diffusers import StableDiffusionInpaintPipeline
pipeline = StableDiffusionInpaintPipeline.from_pretrained( pipeline = StableDiffusionInpaintPipeline.from_pretrained(
"runwayml/stable-diffusion-inpainting", "stable-diffusion-v1-5/stable-diffusion-inpainting",
torch_dtype=torch.float16, torch_dtype=torch.float16,
) )
pipeline = pipeline.to("cuda") pipeline = pipeline.to("cuda")
...@@ -61,12 +61,3 @@ image = pipe(prompt=prompt, image=init_image, mask_image=mask_image).images[0] ...@@ -61,12 +61,3 @@ image = pipe(prompt=prompt, image=init_image, mask_image=mask_image).images[0]
> [!WARNING] > [!WARNING]
> 이전의 실험적인 인페인팅 구현에서는 품질이 낮은 다른 프로세스를 사용했습니다. 이전 버전과의 호환성을 보장하기 위해 새 모델이 포함되지 않은 사전학습된 파이프라인을 불러오면 이전 인페인팅 방법이 계속 적용됩니다. > 이전의 실험적인 인페인팅 구현에서는 품질이 낮은 다른 프로세스를 사용했습니다. 이전 버전과의 호환성을 보장하기 위해 새 모델이 포함되지 않은 사전학습된 파이프라인을 불러오면 이전 인페인팅 방법이 계속 적용됩니다.
아래 Space에서 이미지 인페인팅을 직접 해보세요!
<iframe
src="https://runwayml-stable-diffusion-inpainting.hf.space"
frameborder="0"
width="850"
height="500"
></iframe>
...@@ -16,12 +16,12 @@ pipeline.unet.config["in_channels"] ...@@ -16,12 +16,12 @@ pipeline.unet.config["in_channels"]
4 4
``` ```
而图像修复任务需要输入样本具有9个通道。您可以在 [`runwayml/stable-diffusion-inpainting`](https://huggingface.co/runwayml/stable-diffusion-inpainting) 这样的预训练修复模型中验证此参数: 而图像修复任务需要输入样本具有9个通道。您可以在 [`stable-diffusion-v1-5/stable-diffusion-inpainting`](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting) 这样的预训练修复模型中验证此参数:
```python ```python
from diffusers import StableDiffusionPipeline from diffusers import StableDiffusionPipeline
pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-inpainting", use_safetensors=True) pipeline = StableDiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-inpainting", use_safetensors=True)
pipeline.unet.config["in_channels"] pipeline.unet.config["in_channels"]
9 9
``` ```
......
...@@ -1328,7 +1328,7 @@ model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined" ...@@ -1328,7 +1328,7 @@ model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined"
# Load Stable Diffusion Inpainting Pipeline with custom pipeline # Load Stable Diffusion Inpainting Pipeline with custom pipeline
pipe = DiffusionPipeline.from_pretrained( pipe = DiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-inpainting", "stable-diffusion-v1-5/stable-diffusion-inpainting",
custom_pipeline="text_inpainting", custom_pipeline="text_inpainting",
segmentation_model=model, segmentation_model=model,
segmentation_processor=processor segmentation_processor=processor
......
...@@ -126,7 +126,7 @@ EXAMPLE_DOC_STRING = """ ...@@ -126,7 +126,7 @@ EXAMPLE_DOC_STRING = """
... "lllyasviel/control_v11p_sd15_inpaint", torch_dtype=torch.float16 ... "lllyasviel/control_v11p_sd15_inpaint", torch_dtype=torch.float16
... ) ... )
>>> pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained( >>> pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(
... "runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16 ... "stable-diffusion-v1-5/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
... ) ... )
>>> pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config) >>> pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
...@@ -347,7 +347,7 @@ class AdaptiveMaskInpaintPipeline( ...@@ -347,7 +347,7 @@ class AdaptiveMaskInpaintPipeline(
[`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`].
safety_checker ([`StableDiffusionSafetyChecker`]): safety_checker ([`StableDiffusionSafetyChecker`]):
Classification module that estimates whether generated images could be considered offensive or harmful. Classification module that estimates whether generated images could be considered offensive or harmful.
Please refer to the [model card](https://huggingface.co/runwayml/stable-diffusion-v1-5) for more details Please refer to the [model card](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) for more details
about a model's potential harms. about a model's potential harms.
feature_extractor ([`~transformers.CLIPImageProcessor`]): feature_extractor ([`~transformers.CLIPImageProcessor`]):
A `CLIPImageProcessor` to extract features from generated images; used as inputs to the `safety_checker`. A `CLIPImageProcessor` to extract features from generated images; used as inputs to the `safety_checker`.
...@@ -429,8 +429,8 @@ class AdaptiveMaskInpaintPipeline( ...@@ -429,8 +429,8 @@ class AdaptiveMaskInpaintPipeline(
"The configuration file of the unet has set the default `sample_size` to smaller than" "The configuration file of the unet has set the default `sample_size` to smaller than"
" 64 which seems highly unlikely .If you're checkpoint is a fine-tuned version of any of the" " 64 which seems highly unlikely .If you're checkpoint is a fine-tuned version of any of the"
" following: \n- CompVis/stable-diffusion-v1-4 \n- CompVis/stable-diffusion-v1-3 \n-" " following: \n- CompVis/stable-diffusion-v1-4 \n- CompVis/stable-diffusion-v1-3 \n-"
" CompVis/stable-diffusion-v1-2 \n- CompVis/stable-diffusion-v1-1 \n- runwayml/stable-diffusion-v1-5" " CompVis/stable-diffusion-v1-2 \n- CompVis/stable-diffusion-v1-1 \n- stable-diffusion-v1-5/stable-diffusion-v1-5"
" \n- runwayml/stable-diffusion-inpainting \n you should change 'sample_size' to 64 in the" " \n- stable-diffusion-v1-5/stable-diffusion-inpainting \n you should change 'sample_size' to 64 in the"
" configuration file. Please make sure to update the config accordingly as leaving `sample_size=32`" " configuration file. Please make sure to update the config accordingly as leaving `sample_size=32`"
" in the config might lead to incorrect results in future versions. If you have downloaded this" " in the config might lead to incorrect results in future versions. If you have downloaded this"
" checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for" " checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for"
...@@ -970,7 +970,7 @@ class AdaptiveMaskInpaintPipeline( ...@@ -970,7 +970,7 @@ class AdaptiveMaskInpaintPipeline(
>>> default_mask_image = download_image(mask_url).resize((512, 512)) >>> default_mask_image = download_image(mask_url).resize((512, 512))
>>> pipe = AdaptiveMaskInpaintPipeline.from_pretrained( >>> pipe = AdaptiveMaskInpaintPipeline.from_pretrained(
... "runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16 ... "stable-diffusion-v1-5/stable-diffusion-inpainting", torch_dtype=torch.float16
... ) ... )
>>> pipe = pipe.to("cuda") >>> pipe = pipe.to("cuda")
...@@ -1095,7 +1095,7 @@ class AdaptiveMaskInpaintPipeline( ...@@ -1095,7 +1095,7 @@ class AdaptiveMaskInpaintPipeline(
# 8. Check that sizes of mask, masked image and latents match # 8. Check that sizes of mask, masked image and latents match
if num_channels_unet == 9: if num_channels_unet == 9:
# default case for runwayml/stable-diffusion-inpainting # default case for stable-diffusion-v1-5/stable-diffusion-inpainting
num_channels_mask = mask.shape[1] num_channels_mask = mask.shape[1]
num_channels_masked_image = masked_image_latents.shape[1] num_channels_masked_image = masked_image_latents.shape[1]
if num_channels_latents + num_channels_mask + num_channels_masked_image != self.unet.config.in_channels: if num_channels_latents + num_channels_mask + num_channels_masked_image != self.unet.config.in_channels:
......
...@@ -62,7 +62,7 @@ class ComposableStableDiffusionPipeline(DiffusionPipeline, StableDiffusionMixin) ...@@ -62,7 +62,7 @@ class ComposableStableDiffusionPipeline(DiffusionPipeline, StableDiffusionMixin)
[`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`].
safety_checker ([`StableDiffusionSafetyChecker`]): safety_checker ([`StableDiffusionSafetyChecker`]):
Classification module that estimates whether generated images could be considered offensive or harmful. Classification module that estimates whether generated images could be considered offensive or harmful.
Please, refer to the [model card](https://huggingface.co/runwayml/stable-diffusion-v1-5) for details. Please, refer to the [model card](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) for details.
feature_extractor ([`CLIPImageProcessor`]): feature_extractor ([`CLIPImageProcessor`]):
Model that extracts features from generated images to be used as inputs for the `safety_checker`. Model that extracts features from generated images to be used as inputs for the `safety_checker`.
""" """
...@@ -145,8 +145,8 @@ class ComposableStableDiffusionPipeline(DiffusionPipeline, StableDiffusionMixin) ...@@ -145,8 +145,8 @@ class ComposableStableDiffusionPipeline(DiffusionPipeline, StableDiffusionMixin)
"The configuration file of the unet has set the default `sample_size` to smaller than" "The configuration file of the unet has set the default `sample_size` to smaller than"
" 64 which seems highly unlikely. If your checkpoint is a fine-tuned version of any of the" " 64 which seems highly unlikely. If your checkpoint is a fine-tuned version of any of the"
" following: \n- CompVis/stable-diffusion-v1-4 \n- CompVis/stable-diffusion-v1-3 \n-" " following: \n- CompVis/stable-diffusion-v1-4 \n- CompVis/stable-diffusion-v1-3 \n-"
" CompVis/stable-diffusion-v1-2 \n- CompVis/stable-diffusion-v1-1 \n- runwayml/stable-diffusion-v1-5" " CompVis/stable-diffusion-v1-2 \n- CompVis/stable-diffusion-v1-1 \n- stable-diffusion-v1-5/stable-diffusion-v1-5"
" \n- runwayml/stable-diffusion-inpainting \n you should change 'sample_size' to 64 in the" " \n- stable-diffusion-v1-5/stable-diffusion-inpainting \n you should change 'sample_size' to 64 in the"
" configuration file. Please make sure to update the config accordingly as leaving `sample_size=32`" " configuration file. Please make sure to update the config accordingly as leaving `sample_size=32`"
" in the config might lead to incorrect results in future versions. If you have downloaded this" " in the config might lead to incorrect results in future versions. If you have downloaded this"
" checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for" " checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for"
......
...@@ -1276,7 +1276,7 @@ class FrescoV2VPipeline(StableDiffusionControlNetImg2ImgPipeline): ...@@ -1276,7 +1276,7 @@ class FrescoV2VPipeline(StableDiffusionControlNetImg2ImgPipeline):
[`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`].
safety_checker ([`StableDiffusionSafetyChecker`]): safety_checker ([`StableDiffusionSafetyChecker`]):
Classification module that estimates whether generated images could be considered offensive or harmful. Classification module that estimates whether generated images could be considered offensive or harmful.
Please refer to the [model card](https://huggingface.co/runwayml/stable-diffusion-v1-5) for more details Please refer to the [model card](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) for more details
about a model's potential harms. about a model's potential harms.
feature_extractor ([`~transformers.CLIPImageProcessor`]): feature_extractor ([`~transformers.CLIPImageProcessor`]):
A `CLIPImageProcessor` to extract features from generated images; used as inputs to the `safety_checker`. A `CLIPImageProcessor` to extract features from generated images; used as inputs to the `safety_checker`.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment