Commits · e30d3bf5442fbdbee899e8a5da0b11b621d54f1b · renzhc / diffusers_dcu

22 Apr, 2025 1 commit

[LoRA] add LoRA support to HiDream and fine-tuning script (#11281) · e30d3bf5

Linoy Tsaban authored Apr 22, 2025



* initial commit

* initial commit

* initial commit

* initial commit

* initial commit

* initial commit

* Update examples/dreambooth/train_dreambooth_lora_hidream.py
Co-authored-by: Bagheera <59658056+bghira@users.noreply.github.com>

* move prompt embeds, pooled embeds outside

* Update examples/dreambooth/train_dreambooth_lora_hidream.py
Co-authored-by: hlky <hlky@hlky.ac>

* Update examples/dreambooth/train_dreambooth_lora_hidream.py
Co-authored-by: hlky <hlky@hlky.ac>

* fix import

* fix import and tokenizer 4, text encoder 4 loading

* te

* prompt embeds

* fix naming

* shapes

* initial commit to add HiDreamImageLoraLoaderMixin

* fix init

* add tests

* loader

* fix model input

* add code example to readme

* fix default max length of text encoders

* prints

* nullify training cond in unpatchify for temp fix to incompatible shaping of transformer output during training

* smol fix

* unpatchify

* unpatchify

* fix validation

* flip pred and loss

* fix shift!!!

* revert unpatchify changes (for now)

* smol fix

* Apply style fixes

* workaround moe training

* workaround moe training

* remove prints

* to reduce some memory, keep vae in `weight_dtype` same as we have for flux (as it's the same vae)
https://github.com/huggingface/diffusers/blob/bbd0c161b55ba2234304f1e6325832dd69c60565/examples/dreambooth/train_dreambooth_lora_flux.py#L1207



* refactor to align with HiDream refactor

* refactor to align with HiDream refactor

* refactor to align with HiDream refactor

* add support for cpu offloading of text encoders

* Apply style fixes

* adjust lr and rank for train example

* fix copies

* Apply style fixes

* update README

* update README

* update README

* fix license

* keep prompt2,3,4 as None in validation

* remove reverse ode comment

* Update examples/dreambooth/train_dreambooth_lora_hidream.py
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update examples/dreambooth/train_dreambooth_lora_hidream.py
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* vae offload change

* fix text encoder offloading

* Apply style fixes

* cleaner to_kwargs

* fix module name in copied from

* add requirements

* fix offloading

* fix offloading

* fix offloading

* update transformers version in reqs

* try AutoTokenizer

* try AutoTokenizer

* Apply style fixes

* empty commit

* Delete tests/lora/test_lora_layers_hidream.py

* change tokenizer_4 to load with AutoTokenizer as well

* make text_encoder_four and tokenizer_four configurable

* save model card

* save model card

* revert T5

* fix test

* remove non diffusers lumina2 conversion

---------
Co-authored-by: Bagheera <59658056+bghira@users.noreply.github.com>
Co-authored-by: hlky <hlky@hlky.ac>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

e30d3bf5

17 Apr, 2025 1 commit
- [chore] fix lora docs utils (#11338) · efc9d68b
  Sayak Paul authored Apr 17, 2025
```
fix lora docs utils
```
  efc9d68b
15 Apr, 2025 1 commit

[LoRA] Add LoRA support to AuraFlow (#10216) · 9352a5ca

Hameer Abbasi authored Apr 15, 2025



* Add AuraFlowLoraLoaderMixin

* Add comments, remove qkv fusion

* Add Tests

* Add AuraFlowLoraLoaderMixin to documentation

* Add Suggested changes

* Change attention_kwargs->joint_attention_kwargs

* Rebasing derp.

* fix

* fix

* Quality fixes.

* make style

* `make fix-copies`

* `ruff check --fix`

* Attept 1 to fix tests.

* Attept 2 to fix tests.

* Attept 3 to fix tests.

* Address review comments.

* Rebasing derp.

* Get more tests passing by copying from Flux. Address review comments.

* `joint_attention_kwargs`->`attention_kwargs`

* Add `lora_scale` property for te LoRAs.

* Make test better.

* Remove useless property.

* Skip TE-only tests for AuraFlow.

* Support LoRA for non-CLIP TEs.

* Restore LoRA tests.

* Undo adding LoRA support for non-CLIP TEs.

* Undo support for TE in AuraFlow LoRA.

* `make fix-copies`

* Sync with upstream changes.

* Remove unneeded stuff.

* Mirror `Lumina2`.

* Skip for MPS.

* Address review comments.

* Remove duplicated code.

* Remove unnecessary code.

* Remove repeated docs.

* Propagate attention.

* Fix TE target modules.

* MPS fix for LoRA tests.

* Unrelated TE LoRA tests fix.

* Fix AuraFlow LoRA tests by applying to the right denoiser layers.
Co-authored-by: AstraliteHeart <81396681+AstraliteHeart@users.noreply.github.com>

* Apply style fixes

* empty commit

* Fix the repo consistency issues.

* Remove unrelated changes.

* Style.

* Fix `test_lora_fuse_nan`.

* fix quality issues.

* `pytest.xfail` -> `ValueError`.

* Add back `skip_mps`.

* Apply style fixes

* `make fix-copies`

---------
Co-authored-by: Warlord-K <warlordk28@gmail.com>
Co-authored-by: hlky <hlky@hlky.ac>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: AstraliteHeart <81396681+AstraliteHeart@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

9352a5ca

20 Feb, 2025 1 commit

[LoRA] add LoRA support to Lumina2 and fine-tuning script (#10818) · f10d3c6d

Sayak Paul authored Feb 20, 2025

* feat: lora support for Lumina2.

* fix-copies.

* updates

* updates

* docs.

* fix

* add: training script.

* tests

* updates

* updates

* major updates.

* updates

* fixes

* docs.

* updates

* updates

f10d3c6d

18 Feb, 2025 1 commit
- [docs] add missing entries to the lora docs. (#10819) · 924f880d
  Sayak Paul authored Feb 18, 2025
```
add missing entries to the lora docs.
```
  924f880d
20 Dec, 2024 1 commit

[WIP] SD3.5 IP-Adapter Pipeline Integration (#9987) · 31912484

Daniel Regado authored Dec 20, 2024




* Added support for single IPAdapter on SD3.5 pipeline



---------
Co-authored-by: hlky <hlky@hlky.ac>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>

31912484

16 Dec, 2024 1 commit
- [Docs] add rest of the lora loader mixins to the docs. (#10230) · ea893a9a
  Sayak Paul authored Dec 16, 2024
```
add rest of the lora loader mixins to the docs.
```
  ea893a9a
17 Sep, 2024 1 commit
- Remove CogVideoX mentions from single file docs; Test updates (#9444) · ba06124e
  Aryan authored Sep 18, 2024
```
* remove mentions from single file

* update tests

* update
```
  ba06124e
16 Sep, 2024 1 commit

CogVideoX-5b-I2V support (#9418) · 8336405e

Yuxuan.Zhang authored Sep 16, 2024



* draft Init

* draft

* vae encode image

* make style

* image latents preparation

* remove image encoder from conversion script

* fix minor bugs

* make pipeline work

* make style

* remove debug prints

* fix imports

* update example

* make fix-copies

* add fast tests

* fix import

* update vae

* update docs

* update image link

* apply suggestions from review

* apply suggestions from review

* add slow test

* make use of learned positional embeddings

* apply suggestions from review

* doc change

* Update convert_cogvideox_to_diffusers.py

* make style

* final changes

* make style

* fix tests

---------
Co-authored-by: Aryan <aryan@huggingface.co>

8336405e

07 Aug, 2024 1 commit

Add CogVideoX text-to-video generation model (#9082) · 2dad462d

zR authored Aug 07, 2024



* add CogVideoX

---------
Co-authored-by: Aryan <aryan@huggingface.co>
Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
Co-authored-by: Aryan <contact.aryanvs@gmail.com>
Co-authored-by: yiyixuxu <yixu310@gmail.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

2dad462d

06 Aug, 2024 1 commit
- update · 4c6152c2
  Dhruv Nair authored Aug 06, 2024
  
  4c6152c2
26 Jul, 2024 1 commit

[Chore] add `LoraLoaderMixin` to the inits (#8981) · d87fe95f

Sayak Paul authored Jul 26, 2024



* introduce  to promote reusability.

* up

* add more tests

* up

* remove comments.

* fix fuse_nan test

* clarify the scope of fuse_lora and unfuse_lora

* remove space

* rewrite fuse_lora a bit.

* feedback

* copy over load_lora_into_text_encoder.

* address dhruv's feedback.

* fix-copies

* fix issubclass.

* num_fused_loras

* fix

* fix

* remove mapping

* up

* fix

* style

* fix-copies

* change to SD3TransformerLoRALoadersMixin

* Apply suggestions from code review
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* up

* handle wuerstchen

* up

* move lora to lora_pipeline.py

* up

* fix-copies

* fix documentation.

* comment set_adapters().

* fix-copies

* fix set_adapters() at the model level.

* fix?

* fix

* loraloadermixin.

---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

d87fe95f

25 Jul, 2024 2 commits

Revert "[LoRA] introduce LoraBaseMixin to promote reusability." (#8976) · 62863bb1
YiYi Xu authored Jul 25, 2024
```
Revert "[LoRA] introduce LoraBaseMixin to promote reusability. (#8774)"

This reverts commit 527430d0.
```
62863bb1

[LoRA] introduce LoraBaseMixin to promote reusability. (#8774) · 527430d0

Sayak Paul authored Jul 25, 2024



* introduce  to promote reusability.

* up

* add more tests

* up

* remove comments.

* fix fuse_nan test

* clarify the scope of fuse_lora and unfuse_lora

* remove space

* rewrite fuse_lora a bit.

* feedback

* copy over load_lora_into_text_encoder.

* address dhruv's feedback.

* fix-copies

* fix issubclass.

* num_fused_loras

* fix

* fix

* remove mapping

* up

* fix

* style

* fix-copies

* change to SD3TransformerLoRALoadersMixin

* Apply suggestions from code review
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* up

* handle wuerstchen

* up

* move lora to lora_pipeline.py

* up

* fix-copies

* fix documentation.

* comment set_adapters().

* fix-copies

* fix set_adapters() at the model level.

* fix?

* fix

---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

527430d0

03 Jul, 2024 2 commits

Revert "[LoRA] introduce `LoraBaseMixin` to promote reusability." (#8773) · 984d3405
Sayak Paul authored Jul 03, 2024
```
Revert "[LoRA] introduce `LoraBaseMixin` to promote reusability. (#8670)"

This reverts commit a2071a18.
```
984d3405

[LoRA] introduce `LoraBaseMixin` to promote reusability. (#8670) · a2071a18

Sayak Paul authored Jul 03, 2024

* introduce  to promote reusability.

* up

* add more tests

* up

* remove comments.

* fix fuse_nan test

* clarify the scope of fuse_lora and unfuse_lora

* remove space

a2071a18

13 Jun, 2024 1 commit
- Expand Single File support in SD3 Pipeline (#8517) · b1a2c0d5
  Dhruv Nair authored Jun 13, 2024
```
* update

* update
```
  b1a2c0d5
06 Jun, 2024 1 commit
- [docs] Single file usage (#8412) · 151a56b8
  Steven Liu authored Jun 06, 2024
```
* single file usage

* edit
```
  151a56b8
24 May, 2024 1 commit
- Clean up `from_single_file` docs (#8268) · cef4a512
  Dhruv Nair authored May 24, 2024
```
* update

* update
```
  cef4a512
09 May, 2024 1 commit

[Refactor] Better align `from_single_file` logic with `from_pretrained` (#7496) · cb0f3b49

Dhruv Nair authored May 09, 2024



* refactor unet single file loading a bit.

* retrieve the unet from create_diffusers_unet_model_from_ldm

* update

* update

* updae

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* tests

* update

* update

* update

* Update docs/source/en/api/single_file.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update docs/source/en/api/single_file.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* Update docs/source/en/api/loaders/single_file.md
Co-authored-by: YiYi Xu <yixu310@gmail.com>

* Update src/diffusers/loaders/single_file.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>

* Update docs/source/en/api/loaders/single_file.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update docs/source/en/api/loaders/single_file.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update docs/source/en/api/loaders/single_file.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update docs/source/en/api/loaders/single_file.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

---------
Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>

cb0f3b49

08 Mar, 2024 1 commit

[docs] IP-Adapter image embedding (#7226) · 1f22c988

Steven Liu authored Mar 08, 2024



* update

* fix parameter name

* feedback

* add no mask version

---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

1f22c988

14 Feb, 2024 1 commit

[docs] IP-Adapter (#6897) · 9efe1e52

Steven Liu authored Feb 14, 2024

* use cases

* first draft

* fix image links

* lcm-lora

* feedback

* review

* feedback

* feedback

9efe1e52

08 Feb, 2024 1 commit
- change to 2024 in the license (#6902) · 30e5e81d
  Sayak Paul authored Feb 08, 2024
```
change to 2024
```
  30e5e81d
23 Jan, 2024 1 commit

[Refactor] Update from single file (#6428) · fee93c81

Dhruv Nair authored Jan 23, 2024

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update'

* update

* update

* update

* update

* update

* update

* up

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* up

* update

* update

* update

* update

* update'

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* clean

* update

* update

* clean up

* clean up

* update

* clean

* clean

* update

* updaet

* clean up

* fix docs

* update

* update

* Revert "update"

This reverts commit dbfb8f1ea9c61a2b4e02f926245be2b3d387e577.

* update

* update

* update

* update

* fix controlnet

* fix scheduler

* fix controlnet tests

fee93c81

09 Jan, 2024 1 commit
- [docs] PEFT adapter API (#6499) · 3c79dd9d
  Steven Liu authored Jan 09, 2024
```
follow up
```
  3c79dd9d
05 Jan, 2024 1 commit

[Core] introduce `PeftAdapterMixin` module. (#6416) · 585f9413

Sayak Paul authored Jan 05, 2024

* introduce integrations module.

* remove duplicate methods.

* better imports.

* move to loaders.py

* remove peftadaptermixin from modelmixin.

* add: peftadaptermixin selectively.

* add: entry to _toctree

* Empty-Commit

585f9413

14 Dec, 2023 1 commit
- [docs] IP-Adapter API doc (#6140) · 8ccc76ab
  Steven Liu authored Dec 14, 2023
```
add ip-adapter
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
```
  8ccc76ab
20 Nov, 2023 1 commit

[docs] Loader APIs (#5813) · 7457aa67

Steven Liu authored Nov 20, 2023

* first draft

* remove old loader doc

* start adding lora code examples

* finish

* add link to loralinearlayer

* feedback

* fix

7457aa67