1. 25 Aug, 2024 1 commit
  2. 13 Aug, 2024 1 commit
  3. 07 Aug, 2024 1 commit
  4. 01 Aug, 2024 1 commit
  5. 30 Jul, 2024 1 commit
    • Yoach Lacombe's avatar
      Stable Audio integration (#8716) · 69e72b1d
      Yoach Lacombe authored
      
      
      * WIP modeling code and pipeline
      
      * add custom attention processor + custom activation + add to init
      
      * correct ProjectionModel forward
      
      * add stable audio to __initèè
      
      * add autoencoder and update pipeline and modeling code
      
      * add half Rope
      
      * add partial rotary v2
      
      * add temporary modfis to scheduler
      
      * add EDM DPM Solver
      
      * remove TODOs
      
      * clean GLU
      
      * remove att.group_norm to attn processor
      
      * revert back src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
      
      * refactor GLU -> SwiGLU
      
      * remove redundant args
      
      * add channel multiples in autoencoder docstrings
      
      * changes in docsrtings and copyright headers
      
      * clean pipeline
      
      * further cleaning
      
      * remove peft and lora and fromoriginalmodel
      
      * Delete src/diffusers/pipelines/stable_audio/diffusers.code-workspace
      
      * make style
      
      * dummy models
      
      * fix copied from
      
      * add fast oobleck tests
      
      * add brownian tree
      
      * oobleck autoencoder slow tests
      
      * remove TODO
      
      * fast stable audio pipeline tests
      
      * add slow tests
      
      * make style
      
      * add first version of docs
      
      * wrap is_torchsde_available to the scheduler
      
      * fix slow test
      
      * test with input waveform
      
      * add input waveform
      
      * remove some todos
      
      * create stableaudio gaussian projection + make style
      
      * add pipeline to toctree
      
      * fix copied from
      
      * make quality
      
      * refactor timestep_features->time_proj
      
      * refactor joint_attention_kwargs->cross_attention_kwargs
      
      * remove forward_chunk
      
      * move StableAudioDitModel to transformers folder
      
      * correct convert + remove partial rotary embed
      
      * apply suggestions from yiyixuxu -> removing attn.kv_heads
      
      * remove temb
      
      * remove cross_attention_kwargs
      
      * further removal of cross_attention_kwargs
      
      * remove text encoder autocast to fp16
      
      * continue removing autocast
      
      * make style
      
      * refactor how text and audio are embedded
      
      * add paper
      
      * update example code
      
      * make style
      
      * unify projection model forward + fix device placement
      
      * make style
      
      * remove fuse qkv
      
      * apply suggestions from review
      
      * Update src/diffusers/pipelines/stable_audio/pipeline_stable_audio.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * make style
      
      * smaller models in fast tests
      
      * pass sequential offloading fast tests
      
      * add docs for vae and autoencoder
      
      * make style and update example
      
      * remove useless import
      
      * add cosine scheduler
      
      * dummy classes
      
      * cosine scheduler docs
      
      * better description of scheduler
      
      ---------
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      69e72b1d
  6. 26 Jul, 2024 1 commit
    • Aryan's avatar
      [core] AnimateDiff SparseCtrl (#8897) · 5c53ca5e
      Aryan authored
      * initial sparse control model draft
      
      * remove unnecessary implementation
      
      * copy animatediff pipeline
      
      * remove deprecated callbacks
      
      * update
      
      * update pipeline implementation progress
      
      * make style
      
      * make fix-copies
      
      * update progress
      
      * add partially working pipeline
      
      * remove debug prints
      
      * add model docs
      
      * dummy objects
      
      * improve motion lora conversion script
      
      * fix bugs
      
      * update docstrings
      
      * remove unnecessary model params; docs
      
      * address review comment
      
      * add copied from to zero_module
      
      * copy animatediff test
      
      * add fast tests
      
      * update docs
      
      * update
      
      * update pipeline docs
      
      * fix expected slice values
      
      * fix license
      
      * remove get_down_block usage
      
      * remove temporal_double_self_attention from get_down_block
      
      * update
      
      * update docs with org and documentation images
      
      * make from_unet work in sparsecontrolnetmodel
      
      * add latest freeinit test from #8969
      
      * make fix-copies
      
      * LoraLoaderMixin -> StableDiffsuionLoraLoaderMixin
      5c53ca5e
  7. 11 Jul, 2024 2 commits
    • Sayak Paul's avatar
      [Core] Add AuraFlow (#8796) · 2261510b
      Sayak Paul authored
      
      
      * add lavender flow transformer
      
      ---------
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      2261510b
    • Xin Ma's avatar
      Latte: Latent Diffusion Transformer for Video Generation (#8404) · b8cf84a3
      Xin Ma authored
      
      
      * add Latte to diffusers
      
      * remove print
      
      * remove print
      
      * remove print
      
      * remove unuse codes
      
      * remove layer_norm_latte and add a flag
      
      * remove layer_norm_latte and add a flag
      
      * update latte_pipeline
      
      * update latte_pipeline
      
      * remove unuse squeeze
      
      * add norm_hidden_states.ndim == 2: # for Latte
      
      * fixed test latte pipeline bugs
      
      * fixed test latte pipeline bugs
      
      * delete sh
      
      * add doc for latte
      
      * add licensing
      
      * Move Transformer3DModelOutput to modeling_outputs
      
      * give a default value to sample_size
      
      * remove the einops dependency
      
      * change norm2 for latte
      
      * modify pipeline of latte
      
      * update test for Latte
      
      * modify some codes for latte
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * video_length -> num_frames; update prepare_latents copied from
      
      * make fix-copies
      
      * make style
      
      * typo: videe -> video
      
      * update
      
      * modify for Latte pipeline
      
      * modify latte pipeline
      
      * modify latte pipeline
      
      * modify latte pipeline
      
      * modify latte pipeline
      
      * modify for Latte pipeline
      
      * Delete .vscode directory
      
      * make style
      
      * make fix-copies
      
      * add latte transformer 3d to docs _toctree.yml
      
      * update example
      
      * reduce frames for test
      
      * fixed bug of _text_preprocessing
      
      * set num frame to 1 for testing
      
      * remove unuse print
      
      * add text = self._clean_caption(text) again
      
      ---------
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      Co-authored-by: default avatarAryan <contact.aryanvs@gmail.com>
      Co-authored-by: default avatarAryan <aryan@huggingface.co>
      b8cf84a3
  8. 08 Jul, 2024 1 commit
  9. 26 Jun, 2024 1 commit
  10. 19 Jun, 2024 1 commit
  11. 12 Jun, 2024 2 commits
  12. 01 Jun, 2024 1 commit
  13. 31 May, 2024 1 commit
    • Sayak Paul's avatar
      [Core] Introduce class variants for `Transformer2DModel` (#7647) · 983dec3b
      Sayak Paul authored
      * init for patches
      
      * finish patched model.
      
      * continuous transformer
      
      * vectorized transformer2d.
      
      * style.
      
      * inits.
      
      * fix-copies.
      
      * introduce DiTTransformer2DModel.
      
      * fixes
      
      * use REMAPPING as suggested by @DN6
      
      * better logging.
      
      * add pixart transformer model.
      
      * inits.
      
      * caption_channels.
      
      * attention masking.
      
      * fix use_additional_conditions.
      
      * remove print.
      
      * debug
      
      * flatten
      
      * fix: assertion for sigma
      
      * handle remapping for modeling_utils
      
      * add tests for dit transformer2d
      
      * quality
      
      * placeholder for pixart tests
      
      * pixart tests
      
      * add _no_split_modules
      
      * add docs.
      
      * check
      
      * check
      
      * check
      
      * check
      
      * fix tests
      
      * fix tests
      
      * move Transformer output to modeling_output
      
      * move errors better and bring back use_additional_conditions attribute.
      
      * add unnecessary things from DiT.
      
      * clean up pixart
      
      * fix remapping
      
      * fix device_map things in pixart2d.
      
      * replace Transformer2DModel with appropriate classes in dit, pixart tests
      
      * empty
      
      * legacy mixin classes./
      
      * use a remapping dict for fetching class names.
      
      * change to specifc model types in the pipeline implementations.
      
      * move _fetch_remapped_cls_from_config to modeling_loading_utils.py
      
      * fix dependency problems.
      
      * add deprecation note.
      983dec3b
  14. 29 May, 2024 1 commit
  15. 16 Apr, 2024 1 commit
    • UmerHA's avatar
      Fixing implementation of ControlNet-XS (#6772) · fda1531d
      UmerHA authored
      
      
      * CheckIn - created DownSubBlocks
      
      * Added extra channels, implemented subblock fwd
      
      * Fixed connection sizes
      
      * checkin
      
      * Removed iter, next in forward
      
      * Models for SD21 & SDXL run through
      
      * Added back pipelines, cleared up connections
      
      * Cleaned up connection creation
      
      * added debug logs
      
      * updated logs
      
      * logs: added input loading
      
      * Update umer_debug_logger.py
      
      * log: Loading hint
      
      * Update umer_debug_logger.py
      
      * added logs
      
      * Changed debug logging
      
      * debug: added more logs
      
      * Fixed num_norm_groups
      
      * Debug: Logging all of SDXL input
      
      * Update umer_debug_logger.py
      
      * debug: updated logs
      
      * checkim
      
      * Readded tests
      
      * Removed debug logs
      
      * Fixed Slow Tests
      
      * Added value ckecks | Updated model_cpu_offload_seq
      
      * accelerate-offloading works ; fast tests work
      
      * Made unet & addon explicit in controlnet
      
      * Updated slow tests
      
      * Added dtype/device to ControlNetXS
      
      * Filled in test model paths
      
      * Added image_encoder/feature_extractor to XL pipe
      
      * Fixed fast tests
      
      * Added comments and docstrings
      
      * Fixed copies
      
      * Added docs ; Updates slow tests
      
      * Moved changes to UNetMidBlock2DCrossAttn
      
      * tiny cleanups
      
      * Removed stray prints
      
      * Removed ip adapters + freeU
      
      - Removed ip adapters + freeU as they don't make sense for ControlNet-XS
      - Fixed imports of UNet components
      
      * Fixed test_save_load_float16
      
      * Make style, quality, fix-copies
      
      * Changed loading/saving API for ControlNetXS
      
      - Changed loading/saving API for ControlNetXS
      - other small fixes
      
      * Removed ControlNet-XS from research examples
      
      * Make style, quality, fix-copies
      
      * Small fixes
      
      - deleted ControlNetXSModel.init_original
      - added time_embedding_mix to StableDiffusionControlNetXSPipeline .from_pretrained / StableDiffusionXLControlNetXSPipeline.from_pretrained
      - fixed copy hints
      
      * checkin May 11 '23
      
      * CheckIn Mar 12 '24
      
      * Fixed tests for SD
      
      * Added tests for UNetControlNetXSModel
      
      * Fixed SDXL tests
      
      * cleanup
      
      * Delete Pipfile
      
      * CheckIn Mar 20
      
      Started replacing sub blocks  by `ControlNetXSCrossAttnDownBlock2D` and `ControlNetXSCrossAttnUplock2D`
      
      * check-in Mar 23
      
      * checkin 24 Mar
      
      * Created init for UNetCnxs and CnxsAddon
      
      * CheckIn
      
      * Made from_modules, from_unet and no_control work
      
      * make style,quality,fix-copies & small changes
      
      * Fixed freezing
      
      * Added gradient ckpt'ing; fixed tests
      
      * Fix slow tests(+compile) ; clear naming confusion
      
      * Don't create UNet in init ; removed class_emb
      
      * Incorporated review feedback
      
      - Deleted get_base_pipeline /  get_controlnet_addon for pipes
      - Pipes inherit from StableDiffusionXLPipeline
      - Made module dicts for cnxs-addon's down/mid/up classes
      - Added support for qkv fusion and freeU
      
      * Make style, quality, fix-copies
      
      * Implemented review feedback
      
      * Removed compatibility check for vae/ctrl embedding
      
      * make style, quality, fix-copies
      
      * Delete Pipfile
      
      * Integrated review feedback
      
      - Importing ControlNetConditioningEmbedding now
      - get_down/mid/up_block_addon now outside class
      - renamed `do_control` to `apply_control`
      
      * Reduced size of test tensors
      
      For this, added `norm_num_groups` as parameter everywhere
      
      * Renamed cnxs-`Addon` to cnxs-`Adapter`
      
      - `ControlNetXSAddon` -> `ControlNetXSAdapter`
      - `ControlNetXSAddonDownBlockComponents` -> `DownBlockControlNetXSAdapter`, and similarly for mid/up
      - `get_mid_block_addon` -> `get_mid_block_adapter`, and similarly for mid/up
      
      * Fixed save_pretrained/from_pretrained bug
      
      * Removed redundant code
      
      ---------
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      fda1531d
  16. 06 Mar, 2024 1 commit
    • Kashif Rasul's avatar
      [Pipiline] Wuerstchen v3 aka Stable Cascasde pipeline (#6487) · 40aa47b9
      Kashif Rasul authored
      
      
      * initial diffNext v3
      
      * move to v3 folder
      
      * imports
      
      * dry up the unets
      
      * no switch_level
      
      * fix init
      
      * add switch_level tp config
      
      * Fixed some things
      
      * Added pooled text embeddings
      
      * Initial work on adding image encoder
      
      * changes from @dome272
      
      * Stuff for the image encoder processing and variable naming in decoder
      
      * fix arg name
      
      * inference fixes
      
      * inference fixes
      
      * default TimestepBlock without conds
      
      * c_skip=0 by default
      
      * fix bfloat16 to cpu
      
      * use config
      
      * undo temp change
      
      * fix gen_c_embeddings args
      
      * change text encoding
      
      * text encoding
      
      * undo print
      
      * undo .gitignore change
      
      * Allow WuerstchenV3PriorPipeline to use the base DDPM & DDIM schedulers
      
      * use WuerstchenV3Unet in both pipelines
      
      * fix imports
      
      * initial failing tests
      
      * cleanup
      
      * use scheduler.timesterps
      
      * some fixes to the tests, still not fully working
      
      * fix tests
      
      * fix prior tests
      
      * add dropout to the model_kwargs
      
      * more tests passing
      
      * update expected_slice
      
      * initial rename
      
      * rename tests
      
      * rename class names
      
      * make fix-copies
      
      * initial docs
      
      * autodocs
      
      * typos
      
      * fix arg docs
      
      * add text_encoder info
      
      * combined pipeline has optional image arg
      
      * fix documentation
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * use self.config
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * c_in -> in_channels
      
      * removed kwargs from unet's forward
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * remove older callback api
      
      * removed kwargs and fixed decoder guidance > 1
      
      * decoder takes emeds
      
      * check and use image_embeds
      
      * fixed all but one decoder test
      
      * fix decoder tests
      
      * update callback api
      
      * fix some more combined tests
      
      * push combined pipeline
      
      * initial docs
      
      * fix doc_string
      
      * update combined api
      
      * no test_callback_inputs test for combined pipeline
      
      * add optional components
      
      * fix ordering of components
      
      * fix combined tests
      
      * update convert script
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade_prior.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade_prior.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade_prior.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * fix imports
      
      * move effnet out of deniosing loop
      
      * prompt_embeds_pooled only when doing guidance
      
      * Fix repeat shape
      
      * move StableCascadeUnet to models/unets/
      
      * more descriptive names
      
      * converted when numpy()
      
      * StableCascadePriorPipelineOutput docs
      
      * rename StableCascadeUNet
      
      * add slow tests
      
      * fix slow tests
      
      * update
      
      * update
      
      * updated model_path
      
      * add args for weights
      
      * set push_to_hub to false
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      ---------
      Co-authored-by: default avatarDominic Rampas <d6582533@gmail.com>
      Co-authored-by: default avatarPablo Pernias <pablo@pernias.com>
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      Co-authored-by: default avatar99991 <99991@users.noreply.github.com>
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      40aa47b9
  17. 08 Feb, 2024 1 commit
  18. 31 Jan, 2024 1 commit
  19. 29 Jan, 2024 1 commit
  20. 23 Jan, 2024 1 commit
    • Sayak Paul's avatar
      [Big refactor] move unets to `unets` module 🦋 (#6630) · 1f0705ad
      Sayak Paul authored
      * move unets to  module 🦋
      
      * parameterize unet-level import.
      
      * fix flax unet2dcondition model import
      
      * models __init__
      
      * mildly depcrecating models.unet_2d_blocks in favor of models.unets.unet_2d_blocks.
      
      * noqa
      
      * correct depcrecation behaviour
      
      * inherit from the actual classes.
      
      * Empty-Commit
      
      * backwards compatibility for unet_2d.py
      
      * backward compatibility for unet_2d_condition
      
      * bc for unet_1d
      
      * bc for unet_1d_blocks
      1f0705ad
  21. 27 Dec, 2023 1 commit
  22. 21 Dec, 2023 1 commit
    • Will Berman's avatar
      open muse (#5437) · 40398152
      Will Berman authored
      
      
      amused
      
      rename
      
      Update docs/source/en/api/pipelines/amused.md
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      AdaLayerNormContinuous default values
      
      custom micro conditioning
      
      micro conditioning docs
      
      put lookup from codebook in constructor
      
      fix conversion script
      
      remove manual fused flash attn kernel
      
      add training script
      
      temp remove training script
      
      add dummy gradient checkpointing func
      
      clarify temperatures is an instance variable by setting it
      
      remove additional SkipFF block args
      
      hardcode norm args
      
      rename tests folder
      
      fix paths and samples
      
      fix tests
      
      add training script
      
      training readme
      
      lora saving and loading
      
      non-lora saving/loading
      
      some readme fixes
      
      guards
      
      Update docs/source/en/api/pipelines/amused.md
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      
      Update examples/amused/README.md
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      
      Update examples/amused/train_amused.py
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      
      vae upcasting
      
      add fp16 integration tests
      
      use tuple for micro cond
      
      copyrights
      
      remove casts
      
      delegate to torch.nn.LayerNorm
      
      move temperature to pipeline call
      
      upsampling/downsampling changes
      40398152
  23. 18 Dec, 2023 1 commit
  24. 06 Dec, 2023 2 commits
  25. 04 Dec, 2023 1 commit
    • takuoko's avatar
      [Feature] Support IP-Adapter Plus (#5915) · 0a08d419
      takuoko authored
      
      
      * Support IP-Adapter Plus
      
      * fix format
      
      * restore before black format
      
      * restore before black format
      
      * generic
      
      * Refactor PerceiverAttention
      
      * format
      
      * fix test and refactor PerceiverAttention
      
      * generic encode_image
      
      * keep attention implementation
      
      * merge tests
      
      * encode_image backward compatible
      
      * code quality
      
      * fix controlnet inpaint pipeline
      
      * refactor FFN
      
      * refactor FFN
      
      ---------
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      0a08d419
  26. 01 Dec, 2023 1 commit
  27. 29 Nov, 2023 1 commit
    • Suraj Patil's avatar
      Add SVD (#5895) · 63f767ef
      Suraj Patil authored
      
      
      * begin model
      
      * finish blocks
      
      * add_embedding
      
      * addition_time_embed_dim
      
      * use TimestepEmbedding
      
      * fix temporal res block
      
      * fix time_pos_embed
      
      * fix add_embedding
      
      * add conversion script
      
      * fix model
      
      * up
      
      * add new resnet blocks
      
      * make forward work
      
      * return sample in original shape
      
      * fix temb shape in TemporalResnetBlock
      
      * add spatio temporal transformers
      
      * add vae blocks
      
      * fix blocks
      
      * update
      
      * update
      
      * fix shapes in Alphablender and add time activation in res blcok
      
      * use new blocks
      
      * style
      
      * fix temb shape
      
      * fix SpatioTemporalResBlock
      
      * reuse TemporalBasicTransformerBlock
      
      * fix TemporalBasicTransformerBlock
      
      * use TransformerSpatioTemporalModel
      
      * fix TransformerSpatioTemporalModel
      
      * fix time_context dim
      
      * clean up
      
      * make temb optional
      
      * add blocks
      
      * rename model
      
      * update conversion script
      
      * remove UNetMidBlockSpatioTemporal
      
      * add in init
      
      * remove unused arg
      
      * remove unused arg
      
      * remove more unsed args
      
      * up
      
      * up
      
      * check for None
      
      * update vae
      
      * update up/mid blocks for decoder
      
      * begin pipeline
      
      * adapt scheduler
      
      * add guidance scalings
      
      * fix norm eps in temporal transformers
      
      * add temporal autoencoder
      
      * make pipeline run
      
      * fix frame decodig
      
      * decode in float32
      
      * decode n frames at a time
      
      * pass decoding_t to decode_latents
      
      * fix decode_latents
      
      * vae encode/decode in fp32
      
      * fix dtype in TransformerSpatioTemporalModel
      
      * type image_latents same as image_embeddings
      
      * allow using differnt eps in temporal block for video decoder
      
      * fix default values in vae
      
      * pass num frames in decode
      
      * switch spatial to temporal for mixing in VAE
      
      * fix num frames during split decoding
      
      * cast alpha to sample dtype
      
      * fix attention in MidBlockTemporalDecoder
      
      * fix typo
      
      * fix guidance_scales dtype
      
      * fix missing activation in TemporalDecoder
      
      * skip_post_quant_conv
      
      * add vae conversion
      
      * style
      
      * take guidance scale as input
      
      * up
      
      * allow passing PIL to export_video
      
      * accept fps as arg
      
      * add pipeline and vae in init
      
      * remove hack
      
      * use AutoencoderKLTemporalDecoder
      
      * don't scale image latents
      
      * add unet tests
      
      * clean up unet
      
      * clean TransformerSpatioTemporalModel
      
      * add slow svd test
      
      * clean up
      
      * make temb optional in Decoder mid block
      
      * fix norm eps in TransformerSpatioTemporalModel
      
      * clean up temp decoder
      
      * clean up
      
      * clean up
      
      * use c_noise values for timesteps
      
      * use math for log
      
      * update
      
      * fix copies
      
      * doc
      
      * upcast vae
      
      * update forward pass for gradient checkpointing
      
      * make added_time_ids is tensor
      
      * up
      
      * fix upcasting
      
      * remove post quant conv
      
      * add _resize_with_antialiasing
      
      * fix _compute_padding
      
      * cleanup model
      
      * more cleanup
      
      * more cleanup
      
      * more cleanup
      
      * remove freeu
      
      * remove attn slice
      
      * small clean
      
      * up
      
      * up
      
      * remove extra step kwargs
      
      * remove eta
      
      * remove dropout
      
      * remove callback
      
      * remove merge factor args
      
      * clean
      
      * clean up
      
      * move to dedicated folder
      
      * remove attention_head_dim
      
      * docstr and small fix
      
      * update unet doc strings
      
      * rename decoding_t
      
      * correct linting
      
      * store c_skip and c_out
      
      * cleanup
      
      * clean TemporalResnetBlock
      
      * more cleanup
      
      * clean up vae
      
      * clean up
      
      * begin doc
      
      * more cleanup
      
      * up
      
      * up
      
      * doc
      
      * Improve
      
      * better naming
      
      * better naming
      
      * better naming
      
      * better naming
      
      * better naming
      
      * better naming
      
      * better naming
      
      * better naming
      
      * Apply suggestions from code review
      
      * Default chunk size to None
      
      * add example
      
      * Better
      
      * Apply suggestions from code review
      
      * update doc
      
      * Update src/diffusers/pipelines/stable_diffusion_video/pipeline_stable_diffusion_video.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * style
      
      * Get torch compile working
      
      * up
      
      * rename
      
      * fix doc
      
      * add chunking
      
      * torch compile
      
      * torch compile
      
      * add modelling outputs
      
      * torch compile
      
      * Improve chunking
      
      * Apply suggestions from code review
      
      * Update docs/source/en/using-diffusers/svd.md
      
      * Close diff tag
      
      * remove slicing
      
      * resnet docstr
      
      * add docstr in resnet
      
      * rename
      
      * Apply suggestions from code review
      
      * update tests
      
      * Fix output type latents
      
      * fix more
      
      * fix more
      
      * Update docs/source/en/using-diffusers/svd.md
      
      * fix more
      
      * add pipeline tests
      
      * remove unused arg
      
      * clean  up
      
      * make sure get_scaling receives tensors
      
      * fix euler scheduler
      
      * fix get_scalings
      
      * simply euler for now
      
      * remove old test file
      
      * use randn_tensor to create noise
      
      * fix device for rand tensor
      
      * increase expected_max_difference
      
      * fix test_inference_batch_single_identical
      
      * actually fix test_inference_batch_single_identical
      
      * disable test_save_load_float16
      
      * skip test_float16_inference
      
      * skip test_inference_batch_single_identical
      
      * fix test_xformers_attention_forwardGenerator_pass
      
      * Apply suggestions from code review
      
      * update StableVideoDiffusionPipelineSlowTests
      
      * update image
      
      * add diffusers example
      
      * fix more
      
      ---------
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarapolinário <joaopaulo.passos@gmail.com>
      63f767ef
  28. 24 Nov, 2023 1 commit
    • Patrick von Platen's avatar
      [@cene555][Kandinsky 3.0] Add Kandinsky 3.0 (#5913) · b978334d
      Patrick von Platen authored
      * finalize
      
      * finalize
      
      * finalize
      
      * add slow test
      
      * add slow test
      
      * add slow test
      
      * Fix more
      
      * add slow test
      
      * fix more
      
      * fix more
      
      * fix more
      
      * fix more
      
      * fix more
      
      * fix more
      
      * fix more
      
      * fix more
      
      * fix more
      
      * Better
      
      * Fix more
      
      * Fix more
      
      * add slow test
      
      * Add auto pipelines
      
      * add slow test
      
      * Add all
      
      * add slow test
      
      * add slow test
      
      * add slow test
      
      * add slow test
      
      * add slow test
      
      * Apply suggestions from code review
      
      * add slow test
      
      * add slow test
      b978334d
  29. 09 Nov, 2023 1 commit
  30. 02 Nov, 2023 1 commit
    • Dhruv Nair's avatar
      Animatediff Proposal (#5413) · 2a8cf8e3
      Dhruv Nair authored
      * draft design
      
      * clean up
      
      * clean up
      
      * clean up
      
      * clean up
      
      * clean up
      
      * clean  up
      
      * clean up
      
      * clean up
      
      * clean up
      
      * update pipeline
      
      * clean up
      
      * clean up
      
      * clean up
      
      * add tests
      
      * change motion block
      
      * clean up
      
      * clean up
      
      * clean up
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * clean up
      
      * update
      
      * update
      
      * update model test
      
      * update
      
      * update
      
      * update
      
      * update
      
      * make style
      
      * update
      
      * fix embeddings
      
      * update
      
      * merge upstream
      
      * max fix copies
      
      * fix bug
      
      * fix mistake
      
      * add docs
      
      * update
      
      * clean up
      
      * update
      
      * clean up
      
      * clean up
      
      * fix docstrings
      
      * fix docstrings
      
      * update
      
      * update
      
      * clean  up
      
      * update
      2a8cf8e3
  31. 25 Sep, 2023 1 commit
  32. 12 Sep, 2023 1 commit
  33. 11 Sep, 2023 1 commit
    • Dhruv Nair's avatar
      Lazy Import for Diffusers (#4829) · b6e0b016
      Dhruv Nair authored
      
      
      * initial commit
      
      * move modules to import struct
      
      * add dummy objects and _LazyModule
      
      * add lazy import to schedulers
      
      * clean up unused imports
      
      * lazy import on models module
      
      * lazy import for schedulers module
      
      * add lazy import to pipelines module
      
      * lazy import altdiffusion
      
      * lazy import audio diffusion
      
      * lazy import audioldm
      
      * lazy import consistency model
      
      * lazy import controlnet
      
      * lazy import dance diffusion ddim ddpm
      
      * lazy import deepfloyd
      
      * lazy import kandinksy
      
      * lazy imports
      
      * lazy import semantic diffusion
      
      * lazy imports
      
      * lazy import stable diffusion
      
      * move sd output to its own module
      
      * clean up
      
      * lazy import t2iadapter
      
      * lazy import unclip
      
      * lazy import versatile and vq diffsuion
      
      * lazy import vq diffusion
      
      * helper to fetch objects from modules
      
      * lazy import sdxl
      
      * lazy import txt2vid
      
      * lazy import stochastic karras
      
      * fix model imports
      
      * fix bug
      
      * lazy import
      
      * clean up
      
      * clean up
      
      * fixes for tests
      
      * fixes for tests
      
      * clean up
      
      * remove import of torch_utils from utils module
      
      * clean up
      
      * clean up
      
      * fix mistake import statement
      
      * dedicated modules for exporting and loading
      
      * remove testing utils from utils module
      
      * fixes from  merge conflicts
      
      * Update src/diffusers/pipelines/kandinsky2_2/__init__.py
      
      * fix docs
      
      * fix alt diffusion copied from
      
      * fix check dummies
      
      * fix more docs
      
      * remove accelerate import from utils module
      
      * add type checking
      
      * make style
      
      * fix check dummies
      
      * remove torch import from xformers check
      
      * clean up error message
      
      * fixes after upstream merges
      
      * dummy objects fix
      
      * fix tests
      
      * remove unused module import
      
      ---------
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      b6e0b016
  34. 02 Aug, 2023 1 commit
    • Sayak Paul's avatar
      [Feat] add tiny Autoencoder for (almost) instant decoding (#4384) · 18fc40c1
      Sayak Paul authored
      
      
      * add: model implementation of tiny autoencoder.
      
      * add: inits.
      
      * push the latest devs.
      
      * add: conversion script and finish.
      
      * add: scaling factor args.
      
      * debugging
      
      * fix denormalization.
      
      * fix: positional argument.
      
      * handle use_torch_2_0_or_xformers.
      
      * handle post_quant_conv
      
      * handle dtype
      
      * fix: sdxl image processor for tiny ae.
      
      * fix: sdxl image processor for tiny ae.
      
      * unify upcasting logic.
      
      * copied from madness.
      
      * remove trailing whitespace.
      
      * set is_tiny_vae = False
      
      * address PR comments.
      
      * change to AutoencoderTiny
      
      * make act_fn an str throughout
      
      * fix: apply_forward_hook decorator call
      
      * get rid of the special is_tiny_vae flag.
      
      * directly scale the output.
      
      * fix dummies?
      
      * fix: act_fn.
      
      * get rid of the Clamp() layer.
      
      * bring back copied from.
      
      * movement of the blocks to appropriate modules.
      
      * add: docstrings to AutoencoderTiny
      
      * add: documentation.
      
      * changes to the conversion script.
      
      * add doc entry.
      
      * settle tests.
      
      * style
      
      * add one slow test.
      
      * fix
      
      * fix 2
      
      * fix 2
      
      * fix: 4
      
      * fix: 5
      
      * finish integration tests
      
      * Apply suggestions from code review
      Co-authored-by: default avatarSteven Liu <59462357+stevhliu@users.noreply.github.com>
      
      * style
      
      ---------
      Co-authored-by: default avatarSteven Liu <59462357+stevhliu@users.noreply.github.com>
      18fc40c1
  35. 20 Jul, 2023 1 commit
  36. 17 Jul, 2023 1 commit
    • Will Berman's avatar
      t2i pipeline (#3932) · a0597f33
      Will Berman authored
      
      
      * Quick implementation of t2i-adapter
      
      Load adapter module with from_pretrained
      
      Prototyping generalized adapter framework
      
      Writeup doc string for sideload framework(WIP) + some minor update on implementation
      
      Update adapter models
      
      Remove old adapter optional args in UNet
      
      Add StableDiffusionAdapterPipeline unit test
      
      Handle cpu offload in StableDiffusionAdapterPipeline
      
      Auto correct coding style
      
      Update model repo name to "RzZ/sd-v1-4-adapter-pipeline"
      
      Refactor MultiAdapter to better compatible with config system
      
      Export MultiAdapter
      
      Create pipeline document template from controlnet
      
      Create dummy objects
      
      Supproting new AdapterLight model
      
      Fix StableDiffusionAdapterPipeline common pipeline test
      
      [WIP] Update adapter pipeline document
      
      Handle num_inference_steps in StableDiffusionAdapterPipeline
      
      Update definition of Adapter "channels_in"
      
      Update documents
      
      Apply code style
      
      Fix doc typo and merge error
      
      Update doc string and example
      
      Quality of life improvement
      
      Remove redundant code and file from prototyping
      
      Remove unused pageage
      
      Remove comments
      
      Fix title
      
      Fix typo
      
      Add conditioning scale arg
      
      Bring back old implmentation
      
      Offload sideload
      
      Add supply info on document
      
      Update src/diffusers/models/adapter.py
      Co-authored-by: default avatarWill Berman <wlbberman@gmail.com>
      
      Update MultiAdapter constructor
      
      Swap out custom checkpoint and update pipeline constructor
      
      Update docment
      
      Apply suggestions from code review
      Co-authored-by: default avatarWill Berman <wlbberman@gmail.com>
      
      Correcting style
      
      Following single-file policy
      
      Update auto size in image preprocess func
      
      Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_adapter.py
      Co-authored-by: default avatarWill Berman <wlbberman@gmail.com>
      
      fix copies
      
      Update adapter pipeline behavior
      
      Add adapter_conditioning_scale doc string
      
      Add the missing doc string
      
      Apply suggestions from code review
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      Fix few bugs from suggestion
      
      Handle L-mode PIL image as control image
      
      Rename to differentiate adapter resblock
      
      Update src/diffusers/models/adapter.py
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      Fix typo
      
      Update adapter parameter name
      
      Update test case and code style
      
      Fix copies
      
      Fix typo
      
      Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_adapter.py
      Co-authored-by: default avatarWill Berman <wlbberman@gmail.com>
      
      Update Adapter class name
      
      Add checkpoint converting script
      
      Fix style
      
      Fix-copies
      
      Remove dev script
      
      Apply suggestions from code review
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      Updates for parameter rename
      
      Fix convert_adapter
      
      remove main
      
      fix diff
      
      more
      
      refactoring
      
      more
      
      more
      
      small fixes
      
      refactor
      
      tests
      
      more slow tests
      
      more tests
      
      Update docs/source/en/api/pipelines/overview.mdx
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      add community contributor to docs
      
      Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      fix
      
      remove from_adapters
      
      license
      
      paper link
      
      docs
      
      more url fixes
      
      more docs
      
      fix
      
      fixes
      
      fix
      
      fix
      
      * fix sample inplace add
      
      * additional_kwargs -> additional_residuals
      
      * move t2i adapter pipeline to own module
      
      * preprocess -> _preprocess_adapter_image
      
      * add TencentArc to license
      
      * fix example code links
      
      * add image converter and fix example doc string
      
      * fix links
      
      * clearer additional residual application
      
      ---------
      Co-authored-by: default avatarHimariO <dsfhe49854@gmail.com>
      a0597f33
  37. 23 Mar, 2023 1 commit