GitLab

* Implement option for rescaling betas to zero terminal SNR

* Implement rescale classifier free guidance in pipeline_stable_diffusion.py

* focus on DDIM

* make style

* make style

* make style

* make style

* Apply suggestions from Peter Lin

* Apply suggestions from Peter Lin

* make style

* Apply suggestions from code review

* Apply suggestions from code review

* make style

* make style

---------
Co-authored-by: MaxWe00 <gitlab.9v1lq@slmail.me>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* feat: add lora attention processor for pt 2.0.

* explicit context manager for SDPA.

* switch to flash attention

* make shapes compatible to work optimally with SDPA.

* fix: circular import problem.

* explicitly specify the flash attention kernel in sdpa

* fall back to efficient attention context manager.

* remove explicit dispatch.

* fix: removed processor.

* fix: remove optional from type annotation.

* feat: make changes regarding LoRAAttnProcessor2_0.

* remove confusing warning.

* formatting.

* relax tolerance for PT 2.0

* fix: loading message.

* remove unnecessary logging.

* add: entry to the docs.

* add: network_alpha argument.

* relax tolerance.

Correct multi gpu

Expose num_cycles kwarg of get_schedule() through args.lr_num_cycles.

* dreambooth if docs - stage II, more info

* Update docs/source/en/training/dreambooth.mdx
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update docs/source/en/training/dreambooth.mdx
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update docs/source/en/training/dreambooth.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* download instructions for downsized images

* update source README to match docs

---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* add tensorrt inpaint pipeline
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* run make style
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

---------
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Updated the filename and readme file.

* reformatter

* reformetter

fixed typo

* add _convert_kohya_lora_to_diffusers

* make style

* add scaffold

* match result: unet attention only

* fix monkey-patch for text_encoder

* with CLIPAttention

While the terrible images are no longer produced,
the results do not match those from the hook ver.
This may be due to not setting the network_alpha value.

* add to support network_alpha

* generate diff image

* fix monkey-patch for text_encoder

* add test_text_encoder_lora_monkey_patch()

* verify that it's okay to release the attn_procs

* fix closure version

* add comment

* Revert "fix monkey-patch for text_encoder"

This reverts commit bb9c61e6faecc1935c9c4319c77065837655d616.

* Fix to reuse utility functions

* make LoRAAttnProcessor targets to self_attn

* fix LoRAAttnProcessor target

* make style

* fix split key

* Update src/diffusers/loaders.py

* remove TEXT_ENCODER_TARGET_MODULES loop

* add print memory usage

* remove test_kohya_loras_scaffold.py

* add: doc on LoRA civitai

* remove print statement and refactor in the doc.

* fix state_dict test for kohya-ss style lora

* Apply suggestions from code review
Co-authored-by: Takuma Mori <takuma104@gmail.com>

---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

include the instruction-tuning blog link.

* update dreambooth lora to work with IF stage II

* Update dreambooth script for IF stage II upscaler

* update code to reflect latest changes as of May 30th

* update text to image example

* reflect changes to textual inversion

* make style

* fix typo

* Revert unnecessary readme changes

---------
Co-authored-by: root <root@orttrainingdev8.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
Co-authored-by: Prathik Rao <prathikrao@microsoft.com@orttrainingdev8.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>

Make sure we also change the config when setting `encoder_hid_dim_type=="text_proj"` and allow xformers (#3615)

* fix if

* make style

* make style

* add tests for xformers

* make style

* update

* added clip_guided_images_mixing_stable_diffusion file and readme description

* apply pre-commit

---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

add reference tricks

* added mixture pipeline

* added docstring

* update docstring

refactor save_model_card utility in dreambooth examples.

add controlnet reference and bugfix
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* add stable_diffusion_ipex community pipeline

* Update readme.md

* reformat

* reformat

* Update examples/community/README.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/community/README.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/community/README.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update examples/community/README.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Apply suggestions from code review
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update README.md

* Update README.md

* Apply suggestions from code review
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* style

---------
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

do not scale the initial global step by gradient accumulation steps when loading from checkpoint (#3506)

* add reference only control

* add reference only control

* add reference only control

* fix lint

* fix lint

* reference adain

* bugfix EulerAncestralDiscreteScheduler

* fix style fidelity rule

* fix default output size

* del unused line

* fix deterministic

* DataLoader will now bake in any transforms or image manipulations contained in the EXIF

Images may have rotations stored in EXIF. Training using such images will cause those transforms to be ignored while training and thus produce unexpected results

* Fixed the Dataloading EXIF issue in main DreamBooth training as well

* Run make style (black & isort)

* dreambooth docs torch.compile note

* Update examples/dreambooth/README.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update examples/dreambooth/README.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

add min snr to text2img lora training script

* Make dreambooth lora more robust to orig unet

* up

* add stable diffusion tensorrt img2img pipeline
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* update docstrings
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

---------
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* Improve checkpointing lora

* fix more

* Improve doc string

* Update src/diffusers/loaders.py

* make stytle

* Apply suggestions from code review

* Update src/diffusers/loaders.py

* Apply suggestions from code review

* Apply suggestions from code review

* better

* Fix all

* Fix multi-GPU dreambooth

* Apply suggestions from code review
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Fix all

* make style

* make style

---------
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* [deepspeed] partial ZeRO-3 support

* cleanup

* improve deepspeed fixes

* Improve

* make style

---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* update IF stage I pipelines

add fixed variance schedulers and lora loading

* added kv lora attn processor

* allow loading into alternative lora attn processor

* make vae optional

* throw away predicted variance

* allow loading into added kv lora layer

* allow load T5

* allow pre compute text embeddings

* set new variance type in schedulers

* fix copies

* refactor all prompt embedding code

class prompts are now included in pre-encoding code
max tokenizer length is now configurable
embedding attention mask is now configurable

* fix for when variance type is not defined on scheduler

* do not pre compute validation prompt if not present

* add example test for if lora dreambooth

* add check for train text encoder and pre compute text embeddings