pipeline_block.md 5.22 KB
Newer Older
YiYi Xu's avatar
YiYi Xu committed
1
2
3
4
5
6
7
8
9
10
11
12
<!--Copyright 2025 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

Steven Liu's avatar
Steven Liu committed
13
# ModularPipelineBlocks
YiYi Xu's avatar
YiYi Xu committed
14

Steven Liu's avatar
Steven Liu committed
15
[`~modular_pipelines.ModularPipelineBlocks`] is the basic block for building a [`ModularPipeline`]. It defines what components, inputs/outputs, and computation a block should perform for a specific step in a pipeline. A [`~modular_pipelines.ModularPipelineBlocks`] connects with other blocks, using [state](./modular_diffusers_states), to enable the modular construction of workflows.
YiYi Xu's avatar
YiYi Xu committed
16

Steven Liu's avatar
Steven Liu committed
17
A [`~modular_pipelines.ModularPipelineBlocks`] on it's own can't be executed. It is a blueprint for what a step should do in a pipeline. To actually run and execute a pipeline, the [`~modular_pipelines.ModularPipelineBlocks`] needs to be converted into a [`ModularPipeline`].
YiYi Xu's avatar
YiYi Xu committed
18

Steven Liu's avatar
Steven Liu committed
19
This guide will show you how to create a [`~modular_pipelines.ModularPipelineBlocks`].
YiYi Xu's avatar
YiYi Xu committed
20

Steven Liu's avatar
Steven Liu committed
21
## Inputs and outputs
YiYi Xu's avatar
YiYi Xu committed
22

Steven Liu's avatar
Steven Liu committed
23
24
> [!TIP]
> Refer to the [States](./modular_diffusers_states) guide if you aren't familiar with how state works in Modular Diffusers.
YiYi Xu's avatar
YiYi Xu committed
25

Steven Liu's avatar
Steven Liu committed
26
A [`~modular_pipelines.ModularPipelineBlocks`] requires `inputs`, and `intermediate_outputs`.
YiYi Xu's avatar
YiYi Xu committed
27

Steven Liu's avatar
Steven Liu committed
28
- `inputs` are values provided by a user and retrieved from the [`~modular_pipelines.PipelineState`]. This is useful because some workflows resize an image, but the original image is still required. The [`~modular_pipelines.PipelineState`] maintains the original image.
YiYi Xu's avatar
YiYi Xu committed
29

Steven Liu's avatar
Steven Liu committed
30
    Use `InputParam` to define `inputs`.
YiYi Xu's avatar
YiYi Xu committed
31

Steven Liu's avatar
Steven Liu committed
32
33
    ```py
    from diffusers.modular_pipelines import InputParam
YiYi Xu's avatar
YiYi Xu committed
34

Steven Liu's avatar
Steven Liu committed
35
36
37
38
    user_inputs = [
        InputParam(name="image", type_hint="PIL.Image", description="raw input image to process")
    ]
    ```
YiYi Xu's avatar
YiYi Xu committed
39

Steven Liu's avatar
Steven Liu committed
40
- `intermediate_inputs` are values typically created from a previous block but it can also be directly provided if no preceding block generates them. Unlike `inputs`, `intermediate_inputs` can be modified.
YiYi Xu's avatar
YiYi Xu committed
41

Steven Liu's avatar
Steven Liu committed
42
    Use `InputParam` to define `intermediate_inputs`.
YiYi Xu's avatar
YiYi Xu committed
43

Steven Liu's avatar
Steven Liu committed
44
45
46
47
48
    ```py
    user_intermediate_inputs = [
        InputParam(name="processed_image", type_hint="torch.Tensor", description="image that has been preprocessed and normalized"),
    ]
    ```
YiYi Xu's avatar
YiYi Xu committed
49

Steven Liu's avatar
Steven Liu committed
50
- `intermediate_outputs` are new values created by a block and added to the [`~modular_pipelines.PipelineState`]. The `intermediate_outputs` are available as `intermediate_inputs` for subsequent blocks or available as the final output from running the pipeline.
YiYi Xu's avatar
YiYi Xu committed
51

Steven Liu's avatar
Steven Liu committed
52
    Use `OutputParam` to define `intermediate_outputs`.
YiYi Xu's avatar
YiYi Xu committed
53

Steven Liu's avatar
Steven Liu committed
54
55
    ```py
    from diffusers.modular_pipelines import OutputParam
YiYi Xu's avatar
YiYi Xu committed
56

Steven Liu's avatar
Steven Liu committed
57
58
59
60
        user_intermediate_outputs = [
        OutputParam(name="image_latents", description="latents representing the image")
    ]
    ```
YiYi Xu's avatar
YiYi Xu committed
61

Steven Liu's avatar
Steven Liu committed
62
The intermediate inputs and outputs share data to connect blocks. They are accessible at any point, allowing you to track the workflow's progress.
YiYi Xu's avatar
YiYi Xu committed
63

Steven Liu's avatar
Steven Liu committed
64
## Computation logic
YiYi Xu's avatar
YiYi Xu committed
65

Steven Liu's avatar
Steven Liu committed
66
The computation a block performs is defined in the `__call__` method and it follows a specific structure.
YiYi Xu's avatar
YiYi Xu committed
67

Steven Liu's avatar
Steven Liu committed
68
69
70
71
1. Retrieve the [`~modular_pipelines.BlockState`] to get a local view of the `inputs` and `intermediate_inputs`.
2. Implement the computation logic on the `inputs` and `intermediate_inputs`.
3. Update [`~modular_pipelines.PipelineState`] to push changes from the local [`~modular_pipelines.BlockState`] back to the global [`~modular_pipelines.PipelineState`].
4. Return the components and state which becomes available to the next block.
YiYi Xu's avatar
YiYi Xu committed
72
73
74
75
76

```py
def __call__(self, components, state):
    # Get a local view of the state variables this block needs
    block_state = self.get_block_state(state)
Steven Liu's avatar
Steven Liu committed
77

YiYi Xu's avatar
YiYi Xu committed
78
79
    # Your computation logic here
    # block_state contains all your inputs and intermediate_inputs
Steven Liu's avatar
Steven Liu committed
80
81
    # Access them like: block_state.image, block_state.processed_image

YiYi Xu's avatar
YiYi Xu committed
82
83
84
85
86
    # Update the pipeline state with your updated block_states
    self.set_block_state(state, block_state)
    return components, state
```

Steven Liu's avatar
Steven Liu committed
87
### Components and configs
YiYi Xu's avatar
YiYi Xu committed
88

Steven Liu's avatar
Steven Liu committed
89
The components and pipeline-level configs a block needs are specified in [`ComponentSpec`] and [`~modular_pipelines.ConfigSpec`].
YiYi Xu's avatar
YiYi Xu committed
90

Steven Liu's avatar
Steven Liu committed
91
92
- [`ComponentSpec`] contains the expected components used by a block. You need the `name` of the component and ideally a `type_hint` that specifies exactly what the component is.
- [`~modular_pipelines.ConfigSpec`] contains pipeline-level settings that control behavior across all blocks.
YiYi Xu's avatar
YiYi Xu committed
93
94
95
96
97
98
99
100
101
102
103
104
105
106

```py
from diffusers import ComponentSpec, ConfigSpec

expected_components = [
    ComponentSpec(name="unet", type_hint=UNet2DConditionModel),
    ComponentSpec(name="scheduler", type_hint=EulerDiscreteScheduler)
]

expected_config = [
    ConfigSpec("force_zeros_for_empty_prompt", True)
]
```

Steven Liu's avatar
Steven Liu committed
107
When the blocks are converted into a pipeline, the components become available to the block as the first argument in `__call__`.
YiYi Xu's avatar
YiYi Xu committed
108
109
110
111
112
113
114

```py
def __call__(self, components, state):
    # Access components using dot notation
    unet = components.unet
    vae = components.vae
    scheduler = components.scheduler
Steven Liu's avatar
Steven Liu committed
115
```