marigold_usage.md 30.3 KB
Newer Older
1
2
3
<!--
Copyright 2023-2025 Marigold Team, ETH Zürich. All rights reserved.
Copyright 2024-2025 The HuggingFace Team. All rights reserved.
4
5
6
7
8
9
10
11
12
13
14

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

15
# Marigold Computer Vision
16

17
18
19
**Marigold** is a diffusion-based [method](https://huggingface.co/papers/2312.02145) and a collection of [pipelines](../api/pipelines/marigold) designed for 
dense computer vision tasks, including **monocular depth prediction**, **surface normals estimation**, and **intrinsic 
image decomposition**.
20

21
This guide will walk you through using Marigold to generate fast and high-quality predictions for images and videos.
22

23
24
25
Each pipeline is tailored for a specific computer vision task, processing an input RGB image and generating a 
corresponding prediction.
Currently, the following computer vision tasks are implemented:
26

27
28
29
30
31
| Pipeline                                                                                                                                          | Recommended Model Checkpoints                                                                                                                                                                           |                              Spaces (Interactive Apps)                               | Predicted Modalities                                                                                                                                                               |
|---------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------:|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [MarigoldDepthPipeline](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_depth.py)           | [prs-eth/marigold-depth-v1-1](https://huggingface.co/prs-eth/marigold-depth-v1-1)                                                                                                                       |          [Depth Estimation](https://huggingface.co/spaces/prs-eth/marigold)          | [Depth](https://en.wikipedia.org/wiki/Depth_map), [Disparity](https://en.wikipedia.org/wiki/Binocular_disparity)                                                                   |
| [MarigoldNormalsPipeline](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_normals.py)       | [prs-eth/marigold-normals-v1-1](https://huggingface.co/prs-eth/marigold-normals-v1-1)                                                                                                                   | [Surface Normals Estimation](https://huggingface.co/spaces/prs-eth/marigold-normals) | [Surface normals](https://en.wikipedia.org/wiki/Normal_mapping)                                                                                                                    |
| [MarigoldIntrinsicsPipeline](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.py) | [prs-eth/marigold-iid-appearance-v1-1](https://huggingface.co/prs-eth/marigold-iid-appearance-v1-1),<br>[prs-eth/marigold-iid-lighting-v1-1](https://huggingface.co/prs-eth/marigold-iid-lighting-v1-1) | [Intrinsic Image Decomposition](https://huggingface.co/spaces/prs-eth/marigold-iid)  | [Albedo](https://en.wikipedia.org/wiki/Albedo), [Materials](https://www.n.aiq3d.com/wiki/roughnessmetalnessao-map), [Lighting](https://en.wikipedia.org/wiki/Diffuse_reflection)   |
32

33
34
35
36
All original checkpoints are available under the [PRS-ETH](https://huggingface.co/prs-eth/) organization on Hugging Face.
They are designed for use with diffusers pipelines and the [original codebase](https://github.com/prs-eth/marigold), which can also be used to train 
new model checkpoints. 
The following is a summary of the recommended checkpoints, all of which produce reliable results with 1 to 4 steps. 
37

38
39
40
41
42
43
44
45
46
| Checkpoint                                                                                          | Modality     | Comment                                                                                                                                                           |
|-----------------------------------------------------------------------------------------------------|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [prs-eth/marigold-depth-v1-1](https://huggingface.co/prs-eth/marigold-depth-v1-1)                   | Depth        | Affine-invariant depth prediction assigns each pixel a value between 0 (near plane) and 1 (far plane), with both planes determined by the model during inference. |
| [prs-eth/marigold-normals-v0-1](https://huggingface.co/prs-eth/marigold-normals-v0-1)               | Normals      | The surface normals predictions are unit-length 3D vectors in the screen space camera, with values in the range from -1 to 1.                                     |
| [prs-eth/marigold-iid-appearance-v1-1](https://huggingface.co/prs-eth/marigold-iid-appearance-v1-1) | Intrinsics   | InteriorVerse decomposition is comprised of Albedo and two BRDF material properties: Roughness and Metallicity.                                                   | 
| [prs-eth/marigold-iid-lighting-v1-1](https://huggingface.co/prs-eth/marigold-iid-lighting-v1-1)     | Intrinsics   | HyperSim decomposition of an image \\(I\\) is comprised of Albedo \\(A\\), Diffuse shading \\(S\\), and Non-diffuse residual \\(R\\): \\(I = A*S+R\\).            | 

The examples below are mostly given for depth prediction, but they can be universally applied to other supported 
modalities.
47
We showcase the predictions using the same input image of Albert Einstein generated by Midjourney.
48
49
50
51
52
53
This makes it easier to compare visualizations of the predictions across various modalities and checkpoints.

<div class="flex gap-4" style="justify-content: center; width: 100%;">
  <div style="flex: 1 1 50%; max-width: 50%;">
    <img class="rounded-xl" src="https://marigoldmonodepth.github.io/images/einstein.jpg"/>
    <figcaption class="mt-1 text-center text-sm text-gray-500">
54
55
      Example input image for all Marigold pipelines
    </figcaption>
56
57
58
  </div>
</div>

59
## Depth Prediction
60

61
62
To get a depth prediction, load the `prs-eth/marigold-depth-v1-1` checkpoint into [`MarigoldDepthPipeline`], 
put the image through the pipeline, and save the predictions:
63
64
65
66
67
68

```python
import diffusers
import torch

pipe = diffusers.MarigoldDepthPipeline.from_pretrained(
69
    "prs-eth/marigold-depth-v1-1", variant="fp16", torch_dtype=torch.float16
70
71
72
).to("cuda")

image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")
73

74
75
76
77
78
79
80
81
82
depth = pipe(image)

vis = pipe.image_processor.visualize_depth(depth.prediction)
vis[0].save("einstein_depth.png")

depth_16bit = pipe.image_processor.export_depth_to_16bit_png(depth.prediction)
depth_16bit[0].save("einstein_depth_16bit.png")
```

83
84
85
86
The [`~pipelines.marigold.marigold_image_processing.MarigoldImageProcessor.visualize_depth`] function applies one of 
[matplotlib's colormaps](https://matplotlib.org/stable/users/explain/colors/colormaps.html) (`Spectral` by default) to map the predicted pixel values from a single-channel `[0, 1]` 
depth range into an RGB image.
With the `Spectral` colormap, pixels with near depth are painted red, and far pixels are blue.
87
The 16-bit PNG file stores the single channel values mapped linearly from the `[0, 1]` range into `[0, 65535]`.
88
89
Below are the raw and the visualized predictions. The darker and closer areas (mustache) are easier to distinguish in 
the visualization.
90
91
92

<div class="flex gap-4">
  <div style="flex: 1 1 50%; max-width: 50%;">
93
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_einstein_lcm_depth_16bit.png"/>
94
    <figcaption class="mt-1 text-center text-sm text-gray-500">
95
96
      Predicted depth (16-bit PNG)
    </figcaption>
97
98
  </div>
  <div style="flex: 1 1 50%; max-width: 50%;">
99
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_einstein_lcm_depth.png"/>
100
    <figcaption class="mt-1 text-center text-sm text-gray-500">
101
102
      Predicted depth visualization (Spectral)
    </figcaption>
103
104
105
  </div>
</div>

106
## Surface Normals Estimation
107

108
109
Load the `prs-eth/marigold-normals-v1-1` checkpoint into [`MarigoldNormalsPipeline`], put the image through the 
pipeline, and save the predictions:
110
111
112
113
114
115

```python
import diffusers
import torch

pipe = diffusers.MarigoldNormalsPipeline.from_pretrained(
116
    "prs-eth/marigold-normals-v1-1", variant="fp16", torch_dtype=torch.float16
117
118
119
).to("cuda")

image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")
120

121
122
123
124
125
126
normals = pipe(image)

vis = pipe.image_processor.visualize_normals(normals.prediction)
vis[0].save("einstein_normals.png")
```

127
128
129
130
131
132
The [`~pipelines.marigold.marigold_image_processing.MarigoldImageProcessor.visualize_normals`] maps the three-dimensional 
prediction with pixel values in the range `[-1, 1]` into an RGB image.
The visualization function supports flipping surface normals axes to make the visualization compatible with other 
choices of the frame of reference.
Conceptually, each pixel is painted according to the surface normal vector in the frame of reference, where `X` axis 
points right, `Y` axis points up, and `Z` axis points at the viewer.
133
134
135
136
Below is the visualized prediction:

<div class="flex gap-4" style="justify-content: center; width: 100%;">
  <div style="flex: 1 1 50%; max-width: 50%;">
137
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_einstein_lcm_normals.png"/>
138
    <figcaption class="mt-1 text-center text-sm text-gray-500">
139
140
      Predicted surface normals visualization
    </figcaption>
141
142
143
  </div>
</div>

144
145
In this example, the nose tip almost certainly has a point on the surface, in which the surface normal vector points 
straight at the viewer, meaning that its coordinates are `[0, 0, 1]`.
146
This vector maps to the RGB `[128, 128, 255]`, which corresponds to the violet-blue color.
147
148
Similarly, a surface normal on the cheek in the right part of the image has a large `X` component, which increases the 
red hue.
149
Points on the shoulders pointing up with a large `Y` promote green color.
150

151
## Intrinsic Image Decomposition
152

153
154
Marigold provides two models for Intrinsic Image Decomposition (IID): "Appearance" and "Lighting". 
Each model produces Albedo maps, derived from InteriorVerse and Hypersim annotations, respectively.
155

156
157
- The "Appearance" model also estimates Material properties: Roughness and Metallicity.
- The "Lighting" model generates Diffuse Shading and Non-diffuse Residual.
Tolga Cangöz's avatar
Tolga Cangöz committed
158

159
Here is the sample code saving predictions made by the "Appearance" model:
Tolga Cangöz's avatar
Tolga Cangöz committed
160

161
162
163
```python
import diffusers
import torch
Tolga Cangöz's avatar
Tolga Cangöz committed
164

165
166
167
168
169
170
171
172
173
174
175
176
pipe = diffusers.MarigoldIntrinsicsPipeline.from_pretrained(
    "prs-eth/marigold-iid-appearance-v1-1", variant="fp16", torch_dtype=torch.float16
).to("cuda")

image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")

intrinsics = pipe(image)

vis = pipe.image_processor.visualize_intrinsics(intrinsics.prediction, pipe.target_properties)
vis[0]["albedo"].save("einstein_albedo.png")
vis[0]["roughness"].save("einstein_roughness.png")
vis[0]["metallicity"].save("einstein_metallicity.png")
177
178
```

179
Another example demonstrating the predictions made by the "Lighting" model:
180

181
182
183
```python
import diffusers
import torch
Tolga Cangöz's avatar
Tolga Cangöz committed
184

185
186
187
pipe = diffusers.MarigoldIntrinsicsPipeline.from_pretrained(
    "prs-eth/marigold-iid-lighting-v1-1", variant="fp16", torch_dtype=torch.float16
).to("cuda")
Tolga Cangöz's avatar
Tolga Cangöz committed
188

189
image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")
Tolga Cangöz's avatar
Tolga Cangöz committed
190

191
192
193
194
195
196
intrinsics = pipe(image)

vis = pipe.image_processor.visualize_intrinsics(intrinsics.prediction, pipe.target_properties)
vis[0]["albedo"].save("einstein_albedo.png")
vis[0]["shading"].save("einstein_shading.png")
vis[0]["residual"].save("einstein_residual.png")
197
198
```

199
200
201
202
Both models share the same pipeline while supporting different decomposition types.
The exact decomposition parameterization (e.g., sRGB vs. linear space) is stored in the 
`pipe.target_properties` dictionary, which is passed into the 
[`~pipelines.marigold.marigold_image_processing.MarigoldImageProcessor.visualize_intrinsics`] function.
203

204
205
206
Below are some examples showcasing the predicted decomposition outputs. 
All modalities can be inspected in the 
[Intrinsic Image Decomposition](https://huggingface.co/spaces/prs-eth/marigold-iid) Space.
207
208
209

<div class="flex gap-4">
  <div style="flex: 1 1 50%; max-width: 50%;">
210
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/8c7986eaaab5eb9604eb88336311f46a7b0ff5ab/marigold/marigold_einstein_albedo.png"/>
211
    <figcaption class="mt-1 text-center text-sm text-gray-500">
212
      Predicted albedo ("Appearance" model)
213
    </figcaption>
214
215
  </div>
  <div style="flex: 1 1 50%; max-width: 50%;">
216
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/8c7986eaaab5eb9604eb88336311f46a7b0ff5ab/marigold/marigold_einstein_diffuse.png"/>
217
    <figcaption class="mt-1 text-center text-sm text-gray-500">
218
      Predicted diffuse shading ("Lighting" model)
219
    </figcaption>
220
221
222
  </div>
</div>

223
## Speeding up inference
224

225
226
227
228
The above quick start snippets are already optimized for quality and speed, loading the checkpoint, utilizing the 
`fp16` variant of weights and computation, and performing the default number (4) of denoising diffusion steps.
The first step to accelerate inference, at the expense of prediction quality, is to reduce the denoising diffusion 
steps to the minimum:
229

230
231
232
```diff
  import diffusers
  import torch
233

234
235
236
  pipe = diffusers.MarigoldDepthPipeline.from_pretrained(
      "prs-eth/marigold-depth-v1-1", variant="fp16", torch_dtype=torch.float16
  ).to("cuda")
237

238
239
240
241
242
  image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")
  
- depth = pipe(image)
+ depth = pipe(image, num_inference_steps=1)
```
243

244
245
246
247
248
249
250
251
With this change, the `pipe` call completes in 280ms on RTX 3090 GPU.
Internally, the input image is first encoded using the Stable Diffusion VAE encoder, followed by a single denoising 
step performed by the U-Net. 
Finally, the prediction latent is decoded with the VAE decoder into pixel space.
In this setup, two out of three module calls are dedicated to converting between the pixel and latent spaces of the LDM.
Since Marigold's latent space is compatible with Stable Diffusion 2.0, inference can be accelerated by more than 3x, 
reducing the call time to 85ms on an RTX 3090, by using a [lightweight replacement of the SD VAE](../api/models/autoencoder_tiny). 
Note that using a lightweight VAE may slightly reduce the visual quality of the predictions.
252

253
254
255
```diff
  import diffusers
  import torch
256

257
258
259
  pipe = diffusers.MarigoldDepthPipeline.from_pretrained(
      "prs-eth/marigold-depth-v1-1", variant="fp16", torch_dtype=torch.float16
  ).to("cuda")
260

261
262
263
264
265
266
267
+ pipe.vae = diffusers.AutoencoderTiny.from_pretrained(
+     "madebyollin/taesd", torch_dtype=torch.float16
+ ).cuda()

  image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")

  depth = pipe(image, num_inference_steps=1)
268
269
```

270
271
272
So far, we have optimized the number of diffusion steps and model components. Self-attention operations account for a 
significant portion of computations. 
Speeding them up can be achieved by using a more efficient attention processor:
273

274
275
276
277
```diff
  import diffusers
  import torch
+ from diffusers.models.attention_processor import AttnProcessor2_0
278

279
280
281
  pipe = diffusers.MarigoldDepthPipeline.from_pretrained(
      "prs-eth/marigold-depth-v1-1", variant="fp16", torch_dtype=torch.float16
  ).to("cuda")
282

283
284
+ pipe.vae.set_attn_processor(AttnProcessor2_0()) 
+ pipe.unet.set_attn_processor(AttnProcessor2_0())
285

286
  image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")
287

288
289
  depth = pipe(image, num_inference_steps=1)
```
290

Steven Liu's avatar
Steven Liu committed
291
Finally, as suggested in [Optimizations](../optimization/fp16#torchcompile), enabling `torch.compile` can further enhance performance depending on 
292
293
294
the target hardware.
However, compilation incurs a significant overhead during the first pipeline invocation, making it beneficial only when 
the same pipeline instance is called repeatedly, such as within a loop.
295

296
297
298
299
300
301
302
303
```diff
  import diffusers
  import torch
  from diffusers.models.attention_processor import AttnProcessor2_0

  pipe = diffusers.MarigoldDepthPipeline.from_pretrained(
      "prs-eth/marigold-depth-v1-1", variant="fp16", torch_dtype=torch.float16
  ).to("cuda")
304

305
306
  pipe.vae.set_attn_processor(AttnProcessor2_0()) 
  pipe.unet.set_attn_processor(AttnProcessor2_0())
307

308
309
310
311
312
313
+ pipe.vae = torch.compile(pipe.vae, mode="reduce-overhead", fullgraph=True)
+ pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)

  image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")

  depth = pipe(image, num_inference_steps=1)
314
315
```

316
## Maximizing Precision and Ensembling
317

318
319
320
321
322
323
Marigold pipelines have a built-in ensembling mechanism combining multiple predictions from different random latents.
This is a brute-force way of improving the precision of predictions, capitalizing on the generative nature of diffusion.
The ensembling path is activated automatically when the `ensemble_size` argument is set greater or equal than `3`.
When aiming for maximum precision, it makes sense to adjust `num_inference_steps` simultaneously with `ensemble_size`.
The recommended values vary across checkpoints but primarily depend on the scheduler type.
The effect of ensembling is particularly well-seen with surface normals:
324

325
326
```diff
  import diffusers
327

328
  pipe = diffusers.MarigoldNormalsPipeline.from_pretrained("prs-eth/marigold-normals-v1-1").to("cuda")
329

330
  image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")
331

332
333
334
335
336
- depth = pipe(image)
+ depth = pipe(image, num_inference_steps=10, ensemble_size=5)

  vis = pipe.image_processor.visualize_normals(depth.prediction)
  vis[0].save("einstein_normals.png")
337
338
339
340
```

<div class="flex gap-4">
  <div style="flex: 1 1 50%; max-width: 50%;">
341
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_einstein_lcm_normals.png"/>
342
    <figcaption class="mt-1 text-center text-sm text-gray-500">
343
      Surface normals, no ensembling
344
    </figcaption>
345
346
  </div>
  <div style="flex: 1 1 50%; max-width: 50%;">
347
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_einstein_normals.png"/>
348
    <figcaption class="mt-1 text-center text-sm text-gray-500">
349
      Surface normals, with ensembling
350
    </figcaption>
351
352
353
  </div>
</div>

354
355
356
As can be seen, all areas with fine-grained structurers, such as hair, got more conservative and on average more 
correct predictions.
Such a result is more suitable for precision-sensitive downstream tasks, such as 3D reconstruction.
357
358
359

## Frame-by-frame Video Processing with Temporal Consistency

360
361
362
363
Due to Marigold's generative nature, each prediction is unique and defined by the random noise sampled for the latent 
initialization.
This becomes an obvious drawback compared to traditional end-to-end dense regression networks, as exemplified in the 
following videos:
364
365
366

<div class="flex gap-4">
  <div style="flex: 1 1 50%; max-width: 50%;">
367
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_obama.gif"/>
368
369
370
    <figcaption class="mt-1 text-center text-sm text-gray-500">Input video</figcaption>
  </div>
  <div style="flex: 1 1 50%; max-width: 50%;">
371
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_obama_depth_independent.gif"/>
372
373
374
375
    <figcaption class="mt-1 text-center text-sm text-gray-500">Marigold Depth applied to input video frames independently</figcaption>
  </div>
</div>

376
377
378
379
To address this issue, it is possible to pass `latents` argument to the pipelines, which defines the starting point of 
diffusion.
Empirically, we found that a convex combination of the very same starting point noise latent and the latent 
corresponding to the previous frame prediction give sufficiently smooth results, as implemented in the snippet below:
380
381
382
383
384

```python
import imageio
import diffusers
import torch
385
386
387
from diffusers.models.attention_processor import AttnProcessor2_0
from PIL import Image
from tqdm import tqdm
388
389

device = "cuda"
390
path_in = "https://huggingface.co/spaces/prs-eth/marigold-lcm/resolve/c7adb5427947d2680944f898cd91d386bf0d4924/files/video/obama.mp4"
391
392
393
path_out = "obama_depth.gif"

pipe = diffusers.MarigoldDepthPipeline.from_pretrained(
394
    "prs-eth/marigold-depth-v1-1", variant="fp16", torch_dtype=torch.float16
395
396
397
398
).to(device)
pipe.vae = diffusers.AutoencoderTiny.from_pretrained(
    "madebyollin/taesd", torch_dtype=torch.float16
).to(device)
399
400
401
pipe.unet.set_attn_processor(AttnProcessor2_0())
pipe.vae = torch.compile(pipe.vae, mode="reduce-overhead", fullgraph=True)
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
pipe.set_progress_bar_config(disable=True)

with imageio.get_reader(path_in) as reader:
    size = reader.get_meta_data()['size']
    last_frame_latent = None
    latent_common = torch.randn(
        (1, 4, 768 * size[1] // (8 * max(size)), 768 * size[0] // (8 * max(size)))
    ).to(device=device, dtype=torch.float16)

    out = []
    for frame_id, frame in tqdm(enumerate(reader), desc="Processing Video"):
        frame = Image.fromarray(frame)
        latents = latent_common
        if last_frame_latent is not None:
            latents = 0.9 * latents + 0.1 * last_frame_latent

        depth = pipe(
419
420
421
422
423
            frame,
            num_inference_steps=1,
            match_input_resolution=False, 
            latents=latents, 
            output_latent=True,
424
425
426
427
428
429
430
        )
        last_frame_latent = depth.latent
        out.append(pipe.image_processor.visualize_depth(depth.prediction)[0])

    diffusers.utils.export_to_gif(out, path_out, fps=reader.get_meta_data()['fps'])
```

431
Here, the diffusion process starts from the given computed latent.
432
433
The pipeline sets `output_latent=True` to access `out.latent` and computes its contribution to the next frame's latent 
initialization.
434
435
436
437
The result is much more stable now:

<div class="flex gap-4">
  <div style="flex: 1 1 50%; max-width: 50%;">
438
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_obama_depth_independent.gif"/>
439
440
441
    <figcaption class="mt-1 text-center text-sm text-gray-500">Marigold Depth applied to input video frames independently</figcaption>
  </div>
  <div style="flex: 1 1 50%; max-width: 50%;">
442
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_obama_depth_consistent.gif"/>
443
444
445
446
    <figcaption class="mt-1 text-center text-sm text-gray-500">Marigold Depth with forced latents initialization</figcaption>
  </div>
</div>

447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
## Marigold for ControlNet

A very common application for depth prediction with diffusion models comes in conjunction with ControlNet.
Depth crispness plays a crucial role in obtaining high-quality results from ControlNet.
As seen in comparisons with other methods above, Marigold excels at that task.
The snippet below demonstrates how to load an image, compute depth, and pass it into ControlNet in a compatible format:

```python
import torch
import diffusers

device = "cuda"
generator = torch.Generator(device=device).manual_seed(2024)
image = diffusers.utils.load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/controlnet_depth_source.png"
)

pipe = diffusers.MarigoldDepthPipeline.from_pretrained(
465
    "prs-eth/marigold-depth-v1-1", torch_dtype=torch.float16, variant="fp16"
Tolga Cangöz's avatar
Tolga Cangöz committed
466
).to(device)
467
468
469
470
471
472
473

depth_image = pipe(image, generator=generator).prediction
depth_image = pipe.image_processor.visualize_depth(depth_image, color_map="binary")
depth_image[0].save("motorcycle_controlnet_depth.png")

controlnet = diffusers.ControlNetModel.from_pretrained(
    "diffusers/controlnet-depth-sdxl-1.0", torch_dtype=torch.float16, variant="fp16"
Tolga Cangöz's avatar
Tolga Cangöz committed
474
).to(device)
475
476
pipe = diffusers.StableDiffusionXLControlNetPipeline.from_pretrained(
    "SG161222/RealVisXL_V4.0", torch_dtype=torch.float16, variant="fp16", controlnet=controlnet
Tolga Cangöz's avatar
Tolga Cangöz committed
477
).to(device)
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
pipe.scheduler = diffusers.DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True)

controlnet_out = pipe(
    prompt="high quality photo of a sports bike, city",
    negative_prompt="",
    guidance_scale=6.5,
    num_inference_steps=25,
    image=depth_image,
    controlnet_conditioning_scale=0.7,
    control_guidance_end=0.7,
    generator=generator,
).images
controlnet_out[0].save("motorcycle_controlnet_out.png")
```

<div class="flex gap-4">
  <div style="flex: 1 1 33%; max-width: 33%;">
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/controlnet_depth_source.png"/>
    <figcaption class="mt-1 text-center text-sm text-gray-500">
      Input image
    </figcaption>
  </div>
  <div style="flex: 1 1 33%; max-width: 33%;">
501
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/motorcycle_controlnet_depth.png"/>
502
503
504
505
506
    <figcaption class="mt-1 text-center text-sm text-gray-500">
      Depth in the format compatible with ControlNet
    </figcaption>
  </div>
  <div style="flex: 1 1 33%; max-width: 33%;">
507
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/motorcycle_controlnet_out.png"/>
508
509
510
511
512
513
    <figcaption class="mt-1 text-center text-sm text-gray-500">
      ControlNet generation, conditioned on depth and prompt: "high quality photo of a sports bike, city"
    </figcaption>
  </div>
</div>

514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
## Quantitative Evaluation

To evaluate Marigold quantitatively in standard leaderboards and benchmarks (such as NYU, KITTI, and other datasets), 
follow the evaluation protocol outlined in the paper: load the full precision fp32 model and use appropriate values 
for `num_inference_steps` and `ensemble_size`.
Optionally seed randomness to ensure reproducibility. 
Maximizing `batch_size` will deliver maximum device utilization.

```python
import diffusers
import torch

device = "cuda"
seed = 2024

generator = torch.Generator(device=device).manual_seed(seed)
pipe = diffusers.MarigoldDepthPipeline.from_pretrained("prs-eth/marigold-depth-v1-1").to(device)

image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")

depth = pipe(
    image, 
    num_inference_steps=4,  # set according to the evaluation protocol from the paper
    ensemble_size=10,       # set according to the evaluation protocol from the paper
    generator=generator,
)

# evaluate metrics
```

## Using Predictive Uncertainty

The ensembling mechanism built into Marigold pipelines combines multiple predictions obtained from different random 
latents.
As a side effect, it can be used to quantify epistemic (model) uncertainty; simply specify `ensemble_size` greater 
or equal than 3 and set `output_uncertainty=True`.
The resulting uncertainty will be available in the `uncertainty` field of the output.
It can be visualized as follows:

```python
import diffusers
import torch

pipe = diffusers.MarigoldDepthPipeline.from_pretrained(
    "prs-eth/marigold-depth-v1-1", variant="fp16", torch_dtype=torch.float16
).to("cuda")

image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")

depth = pipe(
	image,
	ensemble_size=10,  # any number >= 3
	output_uncertainty=True,
)

uncertainty = pipe.image_processor.visualize_uncertainty(depth.uncertainty)
uncertainty[0].save("einstein_depth_uncertainty.png")
```

<div class="flex gap-4">
  <div style="flex: 1 1 33%; max-width: 33%;">
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_einstein_depth_uncertainty.png"/>
    <figcaption class="mt-1 text-center text-sm text-gray-500">
      Depth uncertainty
    </figcaption>
  </div>
  <div style="flex: 1 1 33%; max-width: 33%;">
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_einstein_normals_uncertainty.png"/>
    <figcaption class="mt-1 text-center text-sm text-gray-500">
      Surface normals uncertainty
    </figcaption>
  </div>
  <div style="flex: 1 1 33%; max-width: 33%;">
    <img class="rounded-xl" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/4f83035d84a24e5ec44fdda129b1d51eba12ce04/marigold/marigold_einstein_albedo_uncertainty.png"/>
    <figcaption class="mt-1 text-center text-sm text-gray-500">
      Albedo uncertainty
    </figcaption>
  </div>
</div>

The interpretation of uncertainty is easy: higher values (white) correspond to pixels, where the model struggles to 
make consistent predictions.
- The depth model exhibits the most uncertainty around discontinuities, where object depth changes abruptly.
- The surface normals model is least confident in fine-grained structures like hair and in dark regions such as the 
collar area.
- Albedo uncertainty is represented as an RGB image, as it captures uncertainty independently for each color channel, 
unlike depth and surface normals. It is also higher in shaded regions and at discontinuities.

## Conclusion

We hope Marigold proves valuable for your downstream tasks, whether as part of a broader generative workflow or for 
perception-based applications like 3D reconstruction.