Skip to content

GitLab

Unverified Commit 6398fbc3 authored by Hengyue-Bi's avatar Hengyue-Bi Committed by GitHub
Browse files

Fix: Align VAE processing in ControlNet SD3 training with inference (#11909) 

Fix: Apply vae_shift_factor in ControlNet SD3 training
parent 3c8b67b3
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
examples/controlnet/train_controlnet_sd3.py
View file @ 6398fbc3
...@@ -1330,7 +1330,7 @@ def main(args): ...@@ -1330,7 +1330,7 @@ def main(args):
# controlnet(s) inference # controlnet(s) inference
controlnet_image = batch["conditioning_pixel_values"].to(dtype=weight_dtype) controlnet_image = batch["conditioning_pixel_values"].to(dtype=weight_dtype)
controlnet_image = vae.encode(controlnet_image).latent_dist.sample() controlnet_image = vae.encode(controlnet_image).latent_dist.sample()
controlnet_image = controlnet_image * vae.config.scaling_factor controlnet_image = (controlnet_image - vae.config.shift_factor) * vae.config.scaling_factor
control_block_res_samples = controlnet( control_block_res_samples = controlnet(
hidden_states=noisy_model_input, hidden_states=noisy_model_input,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment