Skip to content

GitLab

Unverified Commit 3eb40786 authored by Steven Liu's avatar Steven Liu Committed by GitHub
Browse files

[docs] Prompting (#12312) 

* init

* fix

* batch inf

* feedback

* update
parent a4bc8454
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 90 additions and 505 deletions
.gitignore
View file @ 3eb40786
...@@ -125,6 +125,9 @@ dmypy.json ...@@ -125,6 +125,9 @@ dmypy.json
.vs .vs
.vscode .vscode
# Cursor
.cursor
# Pycharm # Pycharm
.idea .idea
......
docs/source/en/_toctree.yml
View file @ 3eb40786
...@@ -49,7 +49,7 @@ ...@@ -49,7 +49,7 @@
isExpanded: false isExpanded: false
sections: sections:
- local: using-diffusers/weighted_prompts - local: using-diffusers/weighted_prompts
title: Prompt techniques title: Prompting
- local: using-diffusers/create_a_server - local: using-diffusers/create_a_server
title: Create a server title: Create a server
- local: using-diffusers/batched_inference - local: using-diffusers/batched_inference
......
docs/source/en/using-diffusers/batched_inference.md
View file @ 3eb40786
...@@ -16,43 +16,7 @@ Batch inference processes multiple prompts at a time to increase throughput. It ...@@ -16,43 +16,7 @@ Batch inference processes multiple prompts at a time to increase throughput. It
The downside is increased latency because you must wait for the entire batch to complete, and more GPU memory is required for large batches. The downside is increased latency because you must wait for the entire batch to complete, and more GPU memory is required for large batches.
<hfoptions id="usage"> For text-to-image, pass a list of prompts to the pipeline and for image-to-image, pass a list of images and prompts to the pipeline. The example below demonstrates batched text-to-image inference.
<hfoption id="text-to-image">
For text-to-image, pass a list of prompts to the pipeline.
```py
import torch
from diffusers import DiffusionPipeline
pipeline = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16
).to("cuda")
prompts = [
"cinematic photo of A beautiful sunset over mountains, 35mm photograph, film, professional, 4k, highly detailed",
"cinematic film still of a cat basking in the sun on a roof in Turkey, highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain",
"pixel-art a cozy coffee shop interior, low-res, blocky, pixel art style, 8-bit graphics"
]
images = pipeline(
prompt=prompts,
).images
fig, axes = plt.subplots(2, 2, figsize=(12, 12))
axes = axes.flatten()
for i, image in enumerate(images):
axes[i].imshow(image)
axes[i].set_title(f"Image {i+1}")
axes[i].axis('off')
plt.tight_layout()
plt.show()
```
To generate multiple variations of one prompt, use the `num_images_per_prompt` argument.
```py ```py
import torch import torch
...@@ -61,78 +25,19 @@ from diffusers import DiffusionPipeline ...@@ -61,78 +25,19 @@ from diffusers import DiffusionPipeline
pipeline = DiffusionPipeline.from_pretrained( pipeline = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0", "stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16 torch_dtype=torch.float16,
).to("cuda") device_map="cuda"
)
images = pipeline(
prompt="pixel-art a cozy coffee shop interior, low-res, blocky, pixel art style, 8-bit graphics",
num_images_per_prompt=4
).images
fig, axes = plt.subplots(2, 2, figsize=(12, 12))
axes = axes.flatten()
for i, image in enumerate(images):
axes[i].imshow(image)
axes[i].set_title(f"Image {i+1}")
axes[i].axis('off')
plt.tight_layout()
plt.show()
```
Combine both approaches to generate different variations of different prompts.
```py
images = pipeline(
prompt=prompts,
num_images_per_prompt=2,
).images
fig, axes = plt.subplots(2, 2, figsize=(12, 12))
axes = axes.flatten()
for i, image in enumerate(images):
axes[i].imshow(image)
axes[i].set_title(f"Image {i+1}")
axes[i].axis('off')
plt.tight_layout()
plt.show()
```
</hfoption>
<hfoption id="image-to-image">
For image-to-image, pass a list of input images and prompts to the pipeline.
```py
import torch
from diffusers.utils import load_image
from diffusers import DiffusionPipeline
pipeline = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16
).to("cuda")
input_images = [
load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"),
load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"),
load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/detail-prompt.png")
]
prompts = [ prompts = [
"cinematic photo of a beautiful sunset over mountains, 35mm photograph, film, professional, 4k, highly detailed", "Cinematic shot of a cozy coffee shop interior, warm pastel light streaming through a window where a cat rests. Shallow depth of field, glowing cups in soft focus, dreamy lofi-inspired mood, nostalgic tones, framed like a quiet film scene.",
"cinematic film still of a cat basking in the sun on a roof in Turkey, highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain", "Polaroid-style photograph of a cozy coffee shop interior, bathed in warm pastel light. A cat sits on the windowsill near steaming mugs. Soft, slightly faded tones and dreamy blur evoke nostalgia, a lofi mood, and the intimate, imperfect charm of instant film.",
"pixel-art a cozy coffee shop interior, low-res, blocky, pixel art style, 8-bit graphics" "Soft watercolor illustration of a cozy coffee shop interior, pastel washes of color filling the space. A cat rests peacefully on the windowsill as warm light glows through. Gentle brushstrokes create a dreamy, lofi-inspired atmosphere with whimsical textures and nostalgic calm.",
"Isometric pixel-art illustration of a cozy coffee shop interior in detailed 8-bit style. Warm pastel light fills the space as a cat rests on the windowsill. Blocky furniture and tiny mugs add charm, low-res retro graphics enhance the nostalgic, lofi-inspired game aesthetic."
] ]
images = pipeline( images = pipeline(
prompt=prompts, prompt=prompts,
image=input_images,
guidance_scale=8.0,
strength=0.5
).images ).images
fig, axes = plt.subplots(2, 2, figsize=(12, 12)) fig, axes = plt.subplots(2, 2, figsize=(12, 12))
...@@ -147,24 +52,31 @@ plt.tight_layout() ...@@ -147,24 +52,31 @@ plt.tight_layout()
plt.show() plt.show()
``` ```
<div class="flex justify-center">
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/batch-inference.png"/>
</div>
To generate multiple variations of one prompt, use the `num_images_per_prompt` argument. To generate multiple variations of one prompt, use the `num_images_per_prompt` argument.
```py ```py
import torch import torch
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
from diffusers.utils import load_image
from diffusers import DiffusionPipeline from diffusers import DiffusionPipeline
pipeline = DiffusionPipeline.from_pretrained( pipeline = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0", "stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16 torch_dtype=torch.float16,
).to("cuda") device_map="cuda"
)
input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/detail-prompt.png") prompt="""
Isometric pixel-art illustration of a cozy coffee shop interior in detailed 8-bit style. Warm pastel light fills the
space as a cat rests on the windowsill. Blocky furniture and tiny mugs add charm, low-res retro graphics enhance the
nostalgic, lofi-inspired game aesthetic.
"""
images = pipeline( images = pipeline(
prompt="pixel-art a cozy coffee shop interior, low-res, blocky, pixel art style, 8-bit graphics", prompt=prompt,
image=input_image,
num_images_per_prompt=4 num_images_per_prompt=4
).images ).images
...@@ -180,26 +92,19 @@ plt.tight_layout() ...@@ -180,26 +92,19 @@ plt.tight_layout()
plt.show() plt.show()
``` ```
<div class="flex justify-center">
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/batch-inference-2.png"/>
</div>
Combine both approaches to generate different variations of different prompts. Combine both approaches to generate different variations of different prompts.
```py ```py
input_images = [
load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"),
load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/detail-prompt.png")
]
prompts = [
"cinematic film still of a cat basking in the sun on a roof in Turkey, highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain",
"pixel-art a cozy coffee shop interior, low-res, blocky, pixel art style, 8-bit graphics"
]
images = pipeline( images = pipeline(
prompt=prompts, prompt=prompts,
image=input_images,
num_images_per_prompt=2, num_images_per_prompt=2,
).images ).images
fig, axes = plt.subplots(2, 2, figsize=(12, 12)) fig, axes = plt.subplots(2, 4, figsize=(12, 12))
axes = axes.flatten() axes = axes.flatten()
for i, image in enumerate(images): for i, image in enumerate(images):
...@@ -211,16 +116,18 @@ plt.tight_layout() ...@@ -211,16 +116,18 @@ plt.tight_layout()
plt.show() plt.show()
``` ```
</hfoption> <div class="flex justify-center">
</hfoptions> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/batch-inference-3.png"/>
</div>
## Deterministic generation ## Deterministic generation
Enable reproducible batch generation by passing a list of [Generator’s](https://pytorch.org/docs/stable/generated/torch.Generator.html) to the pipeline and tie each `Generator` to a seed to reuse it. Enable reproducible batch generation by passing a list of [Generator’s](https://pytorch.org/docs/stable/generated/torch.Generator.html) to the pipeline and tie each `Generator` to a seed to reuse it.
Use a list comprehension to iterate over the batch size specified in `range()` to create a unique `Generator` object for each image in the batch. > [!TIP]
> Refer to the [Reproducibility](./reusing_seeds) docs to learn more about deterministic algorithms and the `Generator` object.
Don't multiply the `Generator` by the batch size because that only creates one `Generator` object that is used sequentially for each image in the batch. Use a list comprehension to iterate over the batch size specified in `range()` to create a unique `Generator` object for each image in the batch. Don't multiply the `Generator` by the batch size because that only creates one `Generator` object that is used sequentially for each image in the batch.
```py ```py
generator = [torch.Generator(device="cuda").manual_seed(0)] * 3 generator = [torch.Generator(device="cuda").manual_seed(0)] * 3
...@@ -234,14 +141,16 @@ from diffusers import DiffusionPipeline ...@@ -234,14 +141,16 @@ from diffusers import DiffusionPipeline
pipeline = DiffusionPipeline.from_pretrained( pipeline = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0", "stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16 torch_dtype=torch.float16,
).to("cuda") device_map="cuda"
)
generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(3)] generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(3)]
prompts = [ prompts = [
"cinematic photo of A beautiful sunset over mountains, 35mm photograph, film, professional, 4k, highly detailed", "Cinematic shot of a cozy coffee shop interior, warm pastel light streaming through a window where a cat rests. Shallow depth of field, glowing cups in soft focus, dreamy lofi-inspired mood, nostalgic tones, framed like a quiet film scene.",
"cinematic film still of a cat basking in the sun on a roof in Turkey, highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain", "Polaroid-style photograph of a cozy coffee shop interior, bathed in warm pastel light. A cat sits on the windowsill near steaming mugs. Soft, slightly faded tones and dreamy blur evoke nostalgia, a lofi mood, and the intimate, imperfect charm of instant film.",
"pixel-art a cozy coffee shop interior, low-res, blocky, pixel art style, 8-bit graphics" "Soft watercolor illustration of a cozy coffee shop interior, pastel washes of color filling the space. A cat rests peacefully on the windowsill as warm light glows through. Gentle brushstrokes create a dreamy, lofi-inspired atmosphere with whimsical textures and nostalgic calm.",
"Isometric pixel-art illustration of a cozy coffee shop interior in detailed 8-bit style. Warm pastel light fills the space as a cat rests on the windowsill. Blocky furniture and tiny mugs add charm, low-res retro graphics enhance the nostalgic, lofi-inspired game aesthetic."
] ]
images = pipeline( images = pipeline(
...@@ -261,4 +170,4 @@ plt.tight_layout() ...@@ -261,4 +170,4 @@ plt.tight_layout()
plt.show() plt.show()
``` ```
You can use this to iteratively select an image associated with a seed and then improve on it by crafting a more detailed prompt. You can use this to select an image associated with a seed and iteratively improve on it by crafting a more detailed prompt.
\ No newline at end of file \ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment