1. 06 Mar, 2024 1 commit
    • Kashif Rasul's avatar
      [Pipiline] Wuerstchen v3 aka Stable Cascasde pipeline (#6487) · 40aa47b9
      Kashif Rasul authored
      
      
      * initial diffNext v3
      
      * move to v3 folder
      
      * imports
      
      * dry up the unets
      
      * no switch_level
      
      * fix init
      
      * add switch_level tp config
      
      * Fixed some things
      
      * Added pooled text embeddings
      
      * Initial work on adding image encoder
      
      * changes from @dome272
      
      * Stuff for the image encoder processing and variable naming in decoder
      
      * fix arg name
      
      * inference fixes
      
      * inference fixes
      
      * default TimestepBlock without conds
      
      * c_skip=0 by default
      
      * fix bfloat16 to cpu
      
      * use config
      
      * undo temp change
      
      * fix gen_c_embeddings args
      
      * change text encoding
      
      * text encoding
      
      * undo print
      
      * undo .gitignore change
      
      * Allow WuerstchenV3PriorPipeline to use the base DDPM & DDIM schedulers
      
      * use WuerstchenV3Unet in both pipelines
      
      * fix imports
      
      * initial failing tests
      
      * cleanup
      
      * use scheduler.timesterps
      
      * some fixes to the tests, still not fully working
      
      * fix tests
      
      * fix prior tests
      
      * add dropout to the model_kwargs
      
      * more tests passing
      
      * update expected_slice
      
      * initial rename
      
      * rename tests
      
      * rename class names
      
      * make fix-copies
      
      * initial docs
      
      * autodocs
      
      * typos
      
      * fix arg docs
      
      * add text_encoder info
      
      * combined pipeline has optional image arg
      
      * fix documentation
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * use self.config
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * c_in -> in_channels
      
      * removed kwargs from unet's forward
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * remove older callback api
      
      * removed kwargs and fixed decoder guidance > 1
      
      * decoder takes emeds
      
      * check and use image_embeds
      
      * fixed all but one decoder test
      
      * fix decoder tests
      
      * update callback api
      
      * fix some more combined tests
      
      * push combined pipeline
      
      * initial docs
      
      * fix doc_string
      
      * update combined api
      
      * no test_callback_inputs test for combined pipeline
      
      * add optional components
      
      * fix ordering of components
      
      * fix combined tests
      
      * update convert script
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade_prior.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade_prior.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade_prior.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * fix imports
      
      * move effnet out of deniosing loop
      
      * prompt_embeds_pooled only when doing guidance
      
      * Fix repeat shape
      
      * move StableCascadeUnet to models/unets/
      
      * more descriptive names
      
      * converted when numpy()
      
      * StableCascadePriorPipelineOutput docs
      
      * rename StableCascadeUNet
      
      * add slow tests
      
      * fix slow tests
      
      * update
      
      * update
      
      * updated model_path
      
      * add args for weights
      
      * set push_to_hub to false
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      ---------
      Co-authored-by: default avatarDominic Rampas <d6582533@gmail.com>
      Co-authored-by: default avatarPablo Pernias <pablo@pernias.com>
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      Co-authored-by: default avatar99991 <99991@users.noreply.github.com>
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      40aa47b9
  2. 08 Feb, 2024 1 commit
  3. 23 Jan, 2024 1 commit
    • Sayak Paul's avatar
      [Big refactor] move unets to `unets` module 🦋 (#6630) · 1f0705ad
      Sayak Paul authored
      * move unets to  module 🦋
      
      * parameterize unet-level import.
      
      * fix flax unet2dcondition model import
      
      * models __init__
      
      * mildly depcrecating models.unet_2d_blocks in favor of models.unets.unet_2d_blocks.
      
      * noqa
      
      * correct depcrecation behaviour
      
      * inherit from the actual classes.
      
      * Empty-Commit
      
      * backwards compatibility for unet_2d.py
      
      * backward compatibility for unet_2d_condition
      
      * bc for unet_1d
      
      * bc for unet_1d_blocks
      1f0705ad
  4. 05 Jan, 2024 1 commit
    • Sayak Paul's avatar
      [Core] introduce `PeftAdapterMixin` module. (#6416) · 585f9413
      Sayak Paul authored
      * introduce integrations module.
      
      * remove duplicate methods.
      
      * better imports.
      
      * move to loaders.py
      
      * remove peftadaptermixin from modelmixin.
      
      * add: peftadaptermixin selectively.
      
      * add: entry to _toctree
      
      * Empty-Commit
      585f9413
  5. 03 Jan, 2024 1 commit
  6. 18 Dec, 2023 1 commit
  7. 14 Dec, 2023 1 commit
  8. 13 Nov, 2023 2 commits
  9. 05 Nov, 2023 1 commit
    • YiYi Xu's avatar
      add callbacks to denoising step (#5427) · 2b23ec82
      YiYi Xu authored
      
      
      * draft1
      
      * update
      
      * style
      
      * move to the end of loop
      
      * update
      
      * update callbak_on_step_end_inputs
      
      * Revert "update"
      
      This reverts commit 5f9b153183d0cde3b850f14024d2e37ae8c19576.
      
      * Revert "update callbak_on_step_end_inputs"
      
      This reverts commit 44889f4dabad95b7ebb330faa5f1955b5d008c88.
      
      * update
      
      * update test required_optional_params
      
      * remove self.lora_scale
      
      * img2img
      
      * inpaint
      
      * Apply suggestions from code review
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * fix
      
      * apply feedbacks on img2img + inpaint: keep only important pipeline attributes
      
      * depth
      
      * pix2pix
      
      * make _callback_tensor_inputs an class variable so that we can use it for testing
      
      * add a basic tst for callback
      
      * add a read-only tensor input timesteps + fix tests
      
      * add second test for callback cfg
      
      * sdxl
      
      * sdxl img2img
      
      * sdxl inpaint
      
      * kandinsky prior
      
      * kandinsky decoder
      
      * kandinsky img2img + combined
      
      * kandinsky inpaint
      
      * fix copies
      
      * fix
      
      * consistent default inputs
      
      * fix copies
      
      * wuerstchen_prior prior
      
      * test_wuerstchen_decoder + fix test for prior
      
      * wuerstchen_combined pipeline + skip tests
      
      * skip test for kandinsky combined
      
      * lcm
      
      * remove timesteps etc
      
      * add doc string
      
      * copies
      
      * Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * make style and improve tests
      
      * up
      
      * up
      
      * fix more
      
      * fix cfg test
      
      * tests for callbacks
      
      * fix for real
      
      * update
      
      * lcm img2img
      
      * add doc
      
      * add doc page to index
      
      ---------
      Co-authored-by: default avataryiyixuxu <yixu310@gmail,com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      2b23ec82
  10. 16 Oct, 2023 1 commit
  11. 06 Oct, 2023 1 commit
    • vedant2003's avatar
      [Hacktoberfest]Fixing issues #5241 (#5255) · dd25ef56
      vedant2003 authored
      
      
      * Update pipeline_wuerstchen_prior.py
      
      * prior_num_inference_steps updated
      
      * height, width, num_inference_steps, and guidance_scale synced
      
      * parameters synced
      
      * latent_mean, latent_std, and resolution_multiple synced
      
      * prior_num_inference_steps changed
      
      * Formatted pipeline_wuerstchen_prior.py
      
      * Update src/diffusers/pipelines/wuerstchen/pipeline_wuerstchen_prior.py
      
      ---------
      Co-authored-by: default avatarKashif Rasul <kashif.rasul@gmail.com>
      dd25ef56
  12. 02 Oct, 2023 1 commit
  13. 25 Sep, 2023 3 commits
  14. 15 Sep, 2023 1 commit
  15. 14 Sep, 2023 1 commit
  16. 13 Sep, 2023 1 commit
  17. 12 Sep, 2023 3 commits
  18. 11 Sep, 2023 2 commits
    • Kashif Rasul's avatar
      Wuerstchen fixes (#4942) · 16a056a7
      Kashif Rasul authored
      
      
      * fix arguments and make example code work
      
      * change arguments in combined test
      
      * Add default timesteps
      
      * style
      
      * fixed test
      
      * fix broken test
      
      * formatting
      
      * fix docstrings
      
      * fix  num_images_per_prompt
      
      * fix doc styles
      
      * please dont change this
      
      * fix tests
      
      * rename to DEFAULT_STAGE_C_TIMESTEPS
      
      ---------
      Co-authored-by: default avatarDominic Rampas <d6582533@gmail.com>
      16a056a7
    • Dhruv Nair's avatar
      Lazy Import for Diffusers (#4829) · b6e0b016
      Dhruv Nair authored
      
      
      * initial commit
      
      * move modules to import struct
      
      * add dummy objects and _LazyModule
      
      * add lazy import to schedulers
      
      * clean up unused imports
      
      * lazy import on models module
      
      * lazy import for schedulers module
      
      * add lazy import to pipelines module
      
      * lazy import altdiffusion
      
      * lazy import audio diffusion
      
      * lazy import audioldm
      
      * lazy import consistency model
      
      * lazy import controlnet
      
      * lazy import dance diffusion ddim ddpm
      
      * lazy import deepfloyd
      
      * lazy import kandinksy
      
      * lazy imports
      
      * lazy import semantic diffusion
      
      * lazy imports
      
      * lazy import stable diffusion
      
      * move sd output to its own module
      
      * clean up
      
      * lazy import t2iadapter
      
      * lazy import unclip
      
      * lazy import versatile and vq diffsuion
      
      * lazy import vq diffusion
      
      * helper to fetch objects from modules
      
      * lazy import sdxl
      
      * lazy import txt2vid
      
      * lazy import stochastic karras
      
      * fix model imports
      
      * fix bug
      
      * lazy import
      
      * clean up
      
      * clean up
      
      * fixes for tests
      
      * fixes for tests
      
      * clean up
      
      * remove import of torch_utils from utils module
      
      * clean up
      
      * clean up
      
      * fix mistake import statement
      
      * dedicated modules for exporting and loading
      
      * remove testing utils from utils module
      
      * fixes from  merge conflicts
      
      * Update src/diffusers/pipelines/kandinsky2_2/__init__.py
      
      * fix docs
      
      * fix alt diffusion copied from
      
      * fix check dummies
      
      * fix more docs
      
      * remove accelerate import from utils module
      
      * add type checking
      
      * make style
      
      * fix check dummies
      
      * remove torch import from xformers check
      
      * clean up error message
      
      * fixes after upstream merges
      
      * dummy objects fix
      
      * fix tests
      
      * remove unused module import
      
      ---------
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      b6e0b016
  19. 06 Sep, 2023 1 commit
    • Kashif Rasul's avatar
      Würstchen model (#3849) · 541bb6ee
      Kashif Rasul authored
      
      
      * initial
      
      * initial
      
      * added initial convert script for paella vqmodel
      
      * initial wuerstchen pipeline
      
      * add LayerNorm2d
      
      * added modules
      
      * fix typo
      
      * use model_v2
      
      * embed clip caption amd negative_caption
      
      * fixed name of var
      
      * initial modules in one place
      
      * WuerstchenPriorPipeline
      
      * inital shape
      
      * initial denoising prior loop
      
      * fix output
      
      * add WuerstchenPriorPipeline to __init__.py
      
      * use the noise ratio in the Prior
      
      * try to save pipeline
      
      * save_pretrained working
      
      * Few additions
      
      * add _execution_device
      
      * shape is int
      
      * fix batch size
      
      * fix shape of ratio
      
      * fix shape of ratio
      
      * fix output dataclass
      
      * tests folder
      
      * fix formatting
      
      * fix float16 + started with generator
      
      * Update pipeline_wuerstchen.py
      
      * removed vqgan code
      
      * add WuerstchenGeneratorPipeline
      
      * fix WuerstchenGeneratorPipeline
      
      * fix docstrings
      
      * fix imports
      
      * convert generator pipeline
      
      * fix convert
      
      * Work on Generator Pipeline. WIP
      
      * Pipeline works with our diffuzz code
      
      * apply scale factor
      
      * removed vqgan.py
      
      * use cosine schedule
      
      * redo the denoising loop
      
      * Update src/diffusers/models/resnet.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * use torch.lerp
      
      * use warp-diffusion org
      
      * clip_sample=False,
      
      * some refactoring
      
      * use model_v3_stage_c
      
      * c_cond size
      
      * use clip-bigG
      
      * allow stage b clip to be None
      
      * add dummy
      
      * würstchen scheduler
      
      * minor changes
      
      * set clip=None in the pipeline
      
      * fix attention mask
      
      * add attention_masks to text_encoder
      
      * make fix-copies
      
      * add back clip
      
      * add text_encoder
      
      * gen_text_encoder and tokenizer
      
      * fix import
      
      * updated pipeline test
      
      * undo changes to pipeline test
      
      * nip
      
      * fix typo
      
      * fix output name
      
      * set guidance_scale=0 and remove diffuze
      
      * fix doc strings
      
      * make style
      
      * nip
      
      * removed unused
      
      * initial docs
      
      * rename
      
      * toc
      
      * cleanup
      
      * remvoe test script
      
      * fix-copies
      
      * fix multi images
      
      * remove dup
      
      * remove unused modules
      
      * undo changes for debugging
      
      * no  new line
      
      * remove dup conversion script
      
      * fix doc string
      
      * cleanup
      
      * pass default args
      
      * dup permute
      
      * fix some tests
      
      * fix prepare_latents
      
      * move Prior class to modules
      
      * offload only the text encoder and vqgan
      
      * fix resolution calculation for prior
      
      * nip
      
      * removed testing script
      
      * fix shape
      
      * fix argument to set_timesteps
      
      * do not change .gitignore
      
      * fix resolution calculations + readme
      
      * resolution calculation fix + readme
      
      * small fixes
      
      * Add combined pipeline
      
      * rename generator -> decoder
      
      * Update .gitignore
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * removed efficient_net
      
      * create combined WuerstchenPipeline
      
      * make arguments consistent with VQ model
      
      * fix var names
      
      * no need to return text_encoder_hidden_states
      
      * add latent_dim_scale to config
      
      * split model into its own file
      
      * add WuerschenPipeline to docs
      
      * remove unused latent_size
      
      * register latent_dim_scale
      
      * update script
      
      * update docstring
      
      * use Attention preprocessor
      
      * concat with normed input
      
      * fix-copies
      
      * add docs
      
      * fix test
      
      * fix style
      
      * add to cpu_offloaded_model
      
      * updated type
      
      * remove 1-line func
      
      * updated type
      
      * initial decoder test
      
      * formatting
      
      * formatting
      
      * fix autodoc link
      
      * num_inference_steps is int
      
      * remove comments
      
      * fix example in docs
      
      * Update src/diffusers/pipelines/wuerstchen/diffnext.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * rename layernorm to WuerstchenLayerNorm
      
      * rename DiffNext to WuerstchenDiffNeXt
      
      * added comment about MixingResidualBlock
      
      * move paella vq-vae to pipelines' folder
      
      * initial decoder test
      
      * increased test_float16_inference expected diff
      
      * self_attn is always true
      
      * more passing decoder tests
      
      * batch image_embeds
      
      * fix failing tests
      
      * set the correct dtype
      
      * relax inference test
      
      * update prior
      
      * added combined pipeline test
      
      * faster test
      
      * faster test
      
      * Update src/diffusers/pipelines/wuerstchen/pipeline_wuerstchen_combined.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * fix issues from review
      
      * update wuerstchen.md + change generator name
      
      * resolve issues
      
      * fix copied from usage and add back batch_size
      
      * fix API
      
      * fix arguments
      
      * fix combined test
      
      * Added timesteps argument + fixes
      
      * Update tests/pipelines/test_pipelines_common.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update tests/pipelines/wuerstchen/test_wuerstchen_prior.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/wuerstchen/pipeline_wuerstchen_combined.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/wuerstchen/pipeline_wuerstchen_combined.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/wuerstchen/pipeline_wuerstchen_combined.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/wuerstchen/pipeline_wuerstchen_combined.py
      
      * up
      
      * Fix more
      
      * failing tests
      
      * up
      
      * up
      
      * correct naming
      
      * correct docs
      
      * correct docs
      
      * fix test params
      
      * correct docs
      
      * fix classifier free guidance
      
      * fix classifier free guidance
      
      * fix more
      
      * fix all
      
      * make tests faster
      
      ---------
      Co-authored-by: default avatarDominic Rampas <d6582533@gmail.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarDominic Rampas <61938694+dome272@users.noreply.github.com>
      541bb6ee