GitLab

* WIP: flax FlaxDiffusionPipeline & FlaxStableDiffusionPipeline

* todo comment

* Fix imports

* Fix imports

* add dummies

* Fix empty init

* make pipeline work

* up

* Use Flax schedulers (typing, docstring)

* Wrap model imports inside availability checks.

* more updates

* make sure flax is not broken

* make style

* more fixes

* up
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@latenitesoft.com>

* use FlaxPreTrainedModel for flax safety module

* fix name

* fix one more

* Apply suggestions from code review

* Add the K-LMS scheduler to the inpainting pipeline + tests

* Remove redundant casts

* Fix BaseOutput initialization from dict

* style

* Simplify post-init, add tests

* remove debug

* rename weights to align with PT

* DiagonalGaussianDistribution => FlaxDiagonalGaussianDistribution

* fix name

* first commit:

- add `from_pt` argument in `from_pretrained` function
- add `modeling_flax_pytorch_utils.py` file

* small nit

- fix a small nit - to not enter in the second if condition

* major changes

- modify FlaxUnet modules
- first conversion script
- more keys to be matched

* keys match

- now all keys match
- change module names for correct matching
- upsample module name changed

* working v1

- test pass with atol and rtol= `4e-02`

* replace unsued arg

* make quality

* add small docstring

* add more comments

- add TODO for embedding layers

* small change

- use `jnp.expand_dims` for converting `timesteps` in case it is a 0-dimensional array

* add more conditions on conversion

- add better test to check for keys conversion

* make shapes consistent

- output `img_w x img_h x n_channels` from the VAE

* Revert "make shapes consistent"

This reverts commit 4cad1aeb4aeb224402dad13c018a5d42e96267f6.

* fix unet shape

- channels first!

* Fix a setting bug

* Fix typos

* Reverted params to parms

* Fix CrossAttention._sliced_attention
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* add CLIPGuidedStableDiffusion

* add credits

* add readme

* style

* add clip prompt

* fnfix cond_n

* fix cond fn

* fix cond fn for lms

* [Tests] Mark the ncsnpp model tests as slow

* style

* make_reports

* add test utils

* style

* style

* [Flax] Add Vae

* correct

* Apply suggestions from code review
Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Finish
Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Fix _upsample_2d
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Flax: ignore dtype for configuration.

This makes it possible to save models and configuration files.

* Starting to integrate safety checker.

* Fix initialization of CLIPVisionConfig

* Remove commented lines.

* make style

* Remove unused import

* Pass dtype to modules
Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Pass dtype to modules
Co-authored-by: Suraj Patil <surajp815@gmail.com>
Co-authored-by: Suraj Patil <surajp815@gmail.com>

* remove match_shape

* ported fixes from #479 to flax

* remove unused argument

* typo

* remove warnings

* add `get_modified_files.py`

- file copied from https://github.com/huggingface/transformers/blob/main/utils/get_modified_files.py

* make fixup

* [Config] improve logging

* finish

* Unify offset configuration in DDIM and PNDM schedulers

* Format

Add missing variables

* Fix pipeline test

* Update src/diffusers/schedulers/scheduling_ddim.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Default set_alpha_to_one to false

* Format

* Add tests

* Format

* add deprecation warning
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* [Download] Smart downloading

* add test

* finish test

* update

* make style

* [WIP] add LMSDiscreteSchedulerTest

* fixes for comments

* add torch numpy test

* rebase

* Update tests/test_scheduler.py

* Update tests/test_scheduler.py

* style

* return residuals
Co-authored-by: Anton Lozhkov <anton@huggingface.co>

Revert "adding more typehints to DDIM scheduler (#456)"

This reverts commit a0558b11.

* adding more typehints

* resolving mypy issues

* resolving formatting issue

* fixing isort issue
Co-authored-by: V Vishnu Anirudh <git.vva@gmail.com>
Co-authored-by: V Vishnu Anirudh <vvani@kth.se>

* accept tensors

* fix mask handling

* make device placement cleaner

* update doc for mask image

* Quick fix for the img2img tests

* Remove debug lines

* Fix typos

* Add a typo check action

* Fix a bug

* Changed to manual typo check currently

Ref: https://github.com/huggingface/diffusers/pull/483#pullrequestreview-1104468010

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

* Removed a confusing message

* Renamed "nin_shortcut" to "in_shortcut"

* Add memo about NIN
Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

* Fix PT up/down sample_2d

* empty commit

* style

* style
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Finally fix the image-based SD tests

* Remove autocast

* Remove autocast in image tests

* add test for AttentionBlock, SpatialTransformer

* add context_dim, handle device

* removed dropout test

* fixes, add dropout test

* Fix is_onnx_available

Fix: If user install onnxruntime-gpu, is_onnx_available() will return False.

* add more onnxruntime candidates

* Run `make style`
Co-authored-by: anton-l <anton@huggingface.co>

* Add stalebot

* style

* Remove the closing logic

* Make sure not to spam

* begin text2img conversion script

* add fn to convert config

* create config if not provided

* update imports and use UNet2DConditionModel

* fix imports, layer names

* fix unet coversion

* add function to convert VAE

* fix vae conversion

* update main

* create text model

* update config creating logic for unet

* fix config creation

* update script to create and save pipeline

* remove unused imports

* fix checkpoint loading

* better name

* save progress

* finish

* up

* up
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>