1. 05 Aug, 2024 2 commits
    • YiYi Xu's avatar
      add sentencepiece as a soft dependency (#9065) · bc3c73ad
      YiYi Xu authored
      
      
      * add sentencepiece as  soft dependency for kolors
      
      * up
      
      ---------
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      bc3c73ad
    • Aryan's avatar
      PAG variant for HunyuanDiT, PAG refactor (#8936) · b7058d14
      Aryan authored
      
      
      * copy hunyuandit pipeline
      
      * pag variant of hunyuan dit
      
      * add tests
      
      * update docs
      
      * make style
      
      * make fix-copies
      
      * Update src/diffusers/pipelines/pag/pag_utils.py
      
      * remove incorrect copied from
      
      * remove pag hunyuan attn procs to resolve conflicts
      
      * add pag attn procs again
      
      * new implementation for pag_utils
      
      * revert pag changes
      
      * add pag refactor back; update pixart sigma
      
      * update pixart pag tests
      
      * apply suggestions from review
      
      Co-Authored-By: yixu310@gmail.com
      
      * make style
      
      * update docs, fix tests
      
      * fix tests
      
      * fix test_components_function since list not accepted as valid __init__ param
      
      * apply patch to fix broken tests
      Co-Authored-By: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * make style
      
      * fix hunyuan tests
      
      ---------
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      b7058d14
  2. 02 Aug, 2024 1 commit
    • Sayak Paul's avatar
      [Core] Add PAG support for PixArtSigma (#8921) · 7b98c4cc
      Sayak Paul authored
      * feat: add pixart sigma pag.
      
      * inits.
      
      * fixes
      
      * fix
      
      * remove print.
      
      * copy paste methods to the pixart pag mixin
      
      * fix-copies
      
      * add documentation.
      
      * add tests.
      
      * remove correction file.
      
      * remove pag_applied_layers
      
      * empty
      7b98c4cc
  3. 01 Aug, 2024 2 commits
  4. 30 Jul, 2024 2 commits
    • Aryan's avatar
      [core] Move community AnimateDiff ControlNet to core (#8972) · e5b94b4c
      Aryan authored
      
      
      * add animatediff controlnet to core
      
      * make style; remove unused method
      
      * fix copied from comment
      
      * add tests
      
      * changes to make tests work
      
      * add utility function to load videos
      
      * update docs
      
      * update pipeline example
      
      * make style
      
      * update docs with example
      
      * address review comments
      
      * add latest freeinit test from #8969
      
      * LoraLoaderMixin -> StableDiffusionLoraLoaderMixin
      
      * fix docs
      
      * Update src/diffusers/utils/loading_utils.py
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      
      * fix: variable out of scope
      
      ---------
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      e5b94b4c
    • Yoach Lacombe's avatar
      Stable Audio integration (#8716) · 69e72b1d
      Yoach Lacombe authored
      
      
      * WIP modeling code and pipeline
      
      * add custom attention processor + custom activation + add to init
      
      * correct ProjectionModel forward
      
      * add stable audio to __initèè
      
      * add autoencoder and update pipeline and modeling code
      
      * add half Rope
      
      * add partial rotary v2
      
      * add temporary modfis to scheduler
      
      * add EDM DPM Solver
      
      * remove TODOs
      
      * clean GLU
      
      * remove att.group_norm to attn processor
      
      * revert back src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
      
      * refactor GLU -> SwiGLU
      
      * remove redundant args
      
      * add channel multiples in autoencoder docstrings
      
      * changes in docsrtings and copyright headers
      
      * clean pipeline
      
      * further cleaning
      
      * remove peft and lora and fromoriginalmodel
      
      * Delete src/diffusers/pipelines/stable_audio/diffusers.code-workspace
      
      * make style
      
      * dummy models
      
      * fix copied from
      
      * add fast oobleck tests
      
      * add brownian tree
      
      * oobleck autoencoder slow tests
      
      * remove TODO
      
      * fast stable audio pipeline tests
      
      * add slow tests
      
      * make style
      
      * add first version of docs
      
      * wrap is_torchsde_available to the scheduler
      
      * fix slow test
      
      * test with input waveform
      
      * add input waveform
      
      * remove some todos
      
      * create stableaudio gaussian projection + make style
      
      * add pipeline to toctree
      
      * fix copied from
      
      * make quality
      
      * refactor timestep_features->time_proj
      
      * refactor joint_attention_kwargs->cross_attention_kwargs
      
      * remove forward_chunk
      
      * move StableAudioDitModel to transformers folder
      
      * correct convert + remove partial rotary embed
      
      * apply suggestions from yiyixuxu -> removing attn.kv_heads
      
      * remove temb
      
      * remove cross_attention_kwargs
      
      * further removal of cross_attention_kwargs
      
      * remove text encoder autocast to fp16
      
      * continue removing autocast
      
      * make style
      
      * refactor how text and audio are embedded
      
      * add paper
      
      * update example code
      
      * make style
      
      * unify projection model forward + fix device placement
      
      * make style
      
      * remove fuse qkv
      
      * apply suggestions from review
      
      * Update src/diffusers/pipelines/stable_audio/pipeline_stable_audio.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * make style
      
      * smaller models in fast tests
      
      * pass sequential offloading fast tests
      
      * add docs for vae and autoencoder
      
      * make style and update example
      
      * remove useless import
      
      * add cosine scheduler
      
      * dummy classes
      
      * cosine scheduler docs
      
      * better description of scheduler
      
      ---------
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      69e72b1d
  5. 26 Jul, 2024 1 commit
    • Aryan's avatar
      [core] AnimateDiff SparseCtrl (#8897) · 5c53ca5e
      Aryan authored
      * initial sparse control model draft
      
      * remove unnecessary implementation
      
      * copy animatediff pipeline
      
      * remove deprecated callbacks
      
      * update
      
      * update pipeline implementation progress
      
      * make style
      
      * make fix-copies
      
      * update progress
      
      * add partially working pipeline
      
      * remove debug prints
      
      * add model docs
      
      * dummy objects
      
      * improve motion lora conversion script
      
      * fix bugs
      
      * update docstrings
      
      * remove unnecessary model params; docs
      
      * address review comment
      
      * add copied from to zero_module
      
      * copy animatediff test
      
      * add fast tests
      
      * update docs
      
      * update
      
      * update pipeline docs
      
      * fix expected slice values
      
      * fix license
      
      * remove get_down_block usage
      
      * remove temporal_double_self_attention from get_down_block
      
      * update
      
      * update docs with org and documentation images
      
      * make from_unet work in sparsecontrolnetmodel
      
      * add latest freeinit test from #8969
      
      * make fix-copies
      
      * LoraLoaderMixin -> StableDiffsuionLoraLoaderMixin
      5c53ca5e
  6. 12 Jul, 2024 1 commit
  7. 11 Jul, 2024 3 commits
    • Sayak Paul's avatar
      [Core] Add AuraFlow (#8796) · 2261510b
      Sayak Paul authored
      
      
      * add lavender flow transformer
      
      ---------
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      2261510b
    • Álvaro Somoza's avatar
      [Core] Add Kolors (#8812) · 87b9db64
      Álvaro Somoza authored
      * initial draft
      87b9db64
    • Xin Ma's avatar
      Latte: Latent Diffusion Transformer for Video Generation (#8404) · b8cf84a3
      Xin Ma authored
      
      
      * add Latte to diffusers
      
      * remove print
      
      * remove print
      
      * remove print
      
      * remove unuse codes
      
      * remove layer_norm_latte and add a flag
      
      * remove layer_norm_latte and add a flag
      
      * update latte_pipeline
      
      * update latte_pipeline
      
      * remove unuse squeeze
      
      * add norm_hidden_states.ndim == 2: # for Latte
      
      * fixed test latte pipeline bugs
      
      * fixed test latte pipeline bugs
      
      * delete sh
      
      * add doc for latte
      
      * add licensing
      
      * Move Transformer3DModelOutput to modeling_outputs
      
      * give a default value to sample_size
      
      * remove the einops dependency
      
      * change norm2 for latte
      
      * modify pipeline of latte
      
      * update test for Latte
      
      * modify some codes for latte
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * modify for Latte pipeline
      
      * video_length -> num_frames; update prepare_latents copied from
      
      * make fix-copies
      
      * make style
      
      * typo: videe -> video
      
      * update
      
      * modify for Latte pipeline
      
      * modify latte pipeline
      
      * modify latte pipeline
      
      * modify latte pipeline
      
      * modify latte pipeline
      
      * modify for Latte pipeline
      
      * Delete .vscode directory
      
      * make style
      
      * make fix-copies
      
      * add latte transformer 3d to docs _toctree.yml
      
      * update example
      
      * reduce frames for test
      
      * fixed bug of _text_preprocessing
      
      * set num frame to 1 for testing
      
      * remove unuse print
      
      * add text = self._clean_caption(text) again
      
      ---------
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      Co-authored-by: default avatarAryan <contact.aryanvs@gmail.com>
      Co-authored-by: default avatarAryan <aryan@huggingface.co>
      b8cf84a3
  8. 09 Jul, 2024 1 commit
  9. 08 Jul, 2024 1 commit
  10. 29 Jun, 2024 1 commit
  11. 26 Jun, 2024 1 commit
  12. 25 Jun, 2024 1 commit
  13. 19 Jun, 2024 1 commit
  14. 13 Jun, 2024 1 commit
  15. 12 Jun, 2024 1 commit
  16. 01 Jun, 2024 1 commit
  17. 31 May, 2024 1 commit
    • Sayak Paul's avatar
      [Core] Introduce class variants for `Transformer2DModel` (#7647) · 983dec3b
      Sayak Paul authored
      * init for patches
      
      * finish patched model.
      
      * continuous transformer
      
      * vectorized transformer2d.
      
      * style.
      
      * inits.
      
      * fix-copies.
      
      * introduce DiTTransformer2DModel.
      
      * fixes
      
      * use REMAPPING as suggested by @DN6
      
      * better logging.
      
      * add pixart transformer model.
      
      * inits.
      
      * caption_channels.
      
      * attention masking.
      
      * fix use_additional_conditions.
      
      * remove print.
      
      * debug
      
      * flatten
      
      * fix: assertion for sigma
      
      * handle remapping for modeling_utils
      
      * add tests for dit transformer2d
      
      * quality
      
      * placeholder for pixart tests
      
      * pixart tests
      
      * add _no_split_modules
      
      * add docs.
      
      * check
      
      * check
      
      * check
      
      * check
      
      * fix tests
      
      * fix tests
      
      * move Transformer output to modeling_output
      
      * move errors better and bring back use_additional_conditions attribute.
      
      * add unnecessary things from DiT.
      
      * clean up pixart
      
      * fix remapping
      
      * fix device_map things in pixart2d.
      
      * replace Transformer2DModel with appropriate classes in dit, pixart tests
      
      * empty
      
      * legacy mixin classes./
      
      * use a remapping dict for fetching class names.
      
      * change to specifc model types in the pipeline implementations.
      
      * move _fetch_remapped_cls_from_config to modeling_loading_utils.py
      
      * fix dependency problems.
      
      * add deprecation note.
      983dec3b
  18. 29 May, 2024 1 commit
  19. 27 May, 2024 1 commit
    • Anton Obukhov's avatar
      [Pipeline] Marigold depth and normals estimation (#7847) · b3d10d6d
      Anton Obukhov authored
      
      
      * implement marigold depth and normals pipelines in diffusers core
      
      * remove bibtex
      
      * remove deprecations
      
      * remove save_memory argument
      
      * remove validate_vae
      
      * remove config output
      
      * remove batch_size autodetection
      
      * remove presets logic
      move default denoising_steps and processing_resolution into the model config
      make default ensemble_size 1
      
      * remove no_grad
      
      * add fp16 to the example usage
      
      * implement is_matplotlib_available
      use is_matplotlib_available, is_scipy_available for conditional imports in the marigold depth pipeline
      
      * move colormap, visualize_depth, and visualize_normals into export_utils.py
      
      * make the denoising loop more lucid
      fix the outputs to always be 4d tensors or lists of pil images
      support a 4d input_image case
      attempt to support model_cpu_offload_seq
      move check_inputs into a separate function
      change default batch_size to 1, remove any logic to make it bigger implicitly
      
      * style
      
      * rename denoising_steps into num_inference_steps
      
      * rename input_image into image
      
      * rename input_latent into latents
      
      * remove decode_image
      change decode_prediction to use the AutoencoderKL.decode method
      
      * move clean_latent outside of progress_bar
      
      * refactor marigold-reusable image processing bits into MarigoldImageProcessor class
      
      * clean up the usage example docstring
      
      * make ensemble functions members of the pipelines
      
      * add early checks in check_inputs
      rename E into ensemble_size in depth ensembling
      
      * fix vae_scale_factor computation
      
      * better compatibility with torch.compile
      better variable naming
      
      * move export_depth_to_png to export_utils
      
      * remove encode_prediction
      
      * improve visualize_depth and visualize_normals to accept multi-dimensional data and lists
      remove visualization functions from the pipelines
      move exporting depth as 16-bit PNGs functionality from the depth pipeline
      update example docstrings
      
      * do not shortcut vae.config variables
      
      * change all asserts to raise ValueError
      
      * rename output_prediction_type to output_type
      
      * better variable names
      clean up variable deletion code
      
      * better variable names
      
      * pass desc and leave kwargs into the diffusers progress_bar
      implement nested progress bar for images and steps loops
      
      * implement scale_invariant and shift_invariant flags in the ensemble_depth function
      add scale_invariant and shift_invariant flags readout from the model config
      further refactor ensemble_depth
      support ensembling without alignment
      add ensemble_depth docstring
      
      * fix generator device placement checks
      
      * move encode_empty_text body into the pipeline call
      
      * minor empty text encoding simplifications
      
      * adjust pipelines' class docstrings to explain the added construction arguments
      
      * improve the scipy failure condition
      add comments
      improve docstrings
      change the default use_full_z_range to True
      
      * make input image values range check configurable in the preprocessor
      refactor load_image_canonical in preprocessor to reject unknown types and return the image in the expected 4D format of tensor and on right device
      support a list of everything as inputs to the pipeline, change type to PipelineImageInput
      implement a check that all input list elements have the same dimensions
      improve docstrings of pipeline outputs
      remove check_input pipeline argument
      
      * remove forgotten print
      
      * add prediction_type model config
      
      * add uncertainty visualization into export utils
      fix NaN values in normals uncertainties
      
      * change default of output_uncertainty to False
      better handle the case of an attempt to export or visualize none
      
      * fix `output_uncertainty=False`
      
      * remove kwargs
      fix check_inputs according to the new inputs of the pipeline
      
      * rename prepare_latent into prepare_latents as in other pipelines
      annotate prepare_latents in normals pipeline with "Copied from"
      annotate encode_image in normals pipeline with "Copied from"
      
      * move nested-capable `progress_bar` method into the pipelines
      revert the original `progress_bar` method in pipeline_utils
      
      * minor message improvement
      
      * fix cpu offloading
      
      * move colormap, visualize_depth, export_depth_to_16bit_png, visualize_normals, visualize_uncertainty to marigold_image_processing.py
      update example docstrings
      
      * fix missing comma
      
      * change torch.FloatTensor to torch.Tensor
      
      * fix importing of MarigoldImageProcessor
      
      * fix vae offloading
      fix batched image encoding
      remove separate encode_image function and use vae.encode instead
      
      * implement marigold's intial tests
      relax generator checks in line with other pipelines
      implement return_dict __call__ argument in line with other pipelines
      
      * fix num_images computation
      
      * remove MarigoldImageProcessor and outputs from import structure
      update tests
      
      * update docstrings
      
      * update init
      
      * update
      
      * style
      
      * fix
      
      * fix
      
      * up
      
      * up
      
      * up
      
      * add simple test
      
      * up
      
      * update expected np input/output to be channel last
      
      * move expand_tensor_or_array into the MarigoldImageProcessor
      
      * rewrite tests to follow conventions - hardcoded slices instead of image artifacts
      write more smoke tests
      
      * add basic docs.
      
      * add anton's contribution statement
      
      * remove todos.
      
      * fix assertion values for marigold depth slow tests
      
      * fix assertion values for depth normals.
      
      * remove print
      
      * support AutoencoderTiny in the pipelines
      
      * update documentation page
      add Available Pipelines section
      add Available Checkpoints section
      add warning about num_inference_steps
      
      * fix missing import in docstring
      fix wrong value in visualize_depth docstring
      
      * [doc] add marigold to pipelines overview
      
      * [doc] add section "usage examples"
      
      * fix an issue with latents check in the pipelines
      
      * add "Frame-by-frame Video Processing with Consistency" section
      
      * grammarly
      
      * replace tables with images with css-styled images (blindly)
      
      * style
      
      * print
      
      * fix the assertions.
      
      * take from the github runner.
      
      * take the slices from action artifacts
      
      * style.
      
      * update with the slices from the runner.
      
      * remove unnecessary code blocks.
      
      * Revert "[doc] add marigold to pipelines overview"
      
      This reverts commit a505165150afd8dab23c474d1a054ea505a56a5f.
      
      * remove invitation for new modalities
      
      * split out marigold usage examples
      
      * doc cleanup
      
      ---------
      Co-authored-by: default avataryiyixuxu <yixu310@gmail.com>
      Co-authored-by: default avataryiyixuxu <yixu310@gmail,com>
      Co-authored-by: default avatarsayakpaul <spsayakpaul@gmail.com>
      b3d10d6d
  20. 09 May, 2024 1 commit
    • Dhruv Nair's avatar
      [Refactor] Better align `from_single_file` logic with `from_pretrained` (#7496) · cb0f3b49
      Dhruv Nair authored
      
      
      * refactor unet single file loading a bit.
      
      * retrieve the unet from create_diffusers_unet_model_from_ldm
      
      * update
      
      * update
      
      * updae
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * tests
      
      * update
      
      * update
      
      * update
      
      * Update docs/source/en/api/single_file.md
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * Update docs/source/en/api/single_file.md
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * Update docs/source/en/api/loaders/single_file.md
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/loaders/single_file.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update docs/source/en/api/loaders/single_file.md
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * Update docs/source/en/api/loaders/single_file.md
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * Update docs/source/en/api/loaders/single_file.md
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * Update docs/source/en/api/loaders/single_file.md
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      ---------
      Co-authored-by: default avatarsayakpaul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      cb0f3b49
  21. 08 May, 2024 1 commit
    • Aryan's avatar
      [Pipeline] AnimateDiff SDXL (#6721) · 818f7607
      Aryan authored
      
      
      * update conversion script to handle motion adapter sdxl checkpoint
      
      * add animatediff xl
      
      * handle addition_embed_type
      
      * fix output
      
      * update
      
      * add imports
      
      * make fix-copies
      
      * add decode latents
      
      * update docstrings
      
      * add animatediff sdxl to docs
      
      * remove unnecessary lines
      
      * update example
      
      * add test
      
      * revert conv_in conv_out kernel param
      
      * remove unused param addition_embed_type_num_heads
      
      * latest IPAdapter impl
      
      * make fix-copies
      
      * fix return
      
      * add IPAdapterTesterMixin to tests
      
      * fix return
      
      * revert based on suggestion
      
      * add freeinit
      
      * fix test_to_dtype test
      
      * use StableDiffusionMixin instead of different helper methods
      
      * fix progress bar iterations
      
      * apply suggestions from review
      
      * hardcode flip_sin_to_cos and freq_shift
      
      * make fix-copies
      
      * fix ip adapter implementation
      
      * fix last failing test
      
      * make style
      
      * Update docs/source/en/api/pipelines/animatediff.md
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      
      * remove todo
      
      * fix doc-builder errors
      
      ---------
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      818f7607
  22. 24 Apr, 2024 1 commit
  23. 16 Apr, 2024 1 commit
    • UmerHA's avatar
      Fixing implementation of ControlNet-XS (#6772) · fda1531d
      UmerHA authored
      
      
      * CheckIn - created DownSubBlocks
      
      * Added extra channels, implemented subblock fwd
      
      * Fixed connection sizes
      
      * checkin
      
      * Removed iter, next in forward
      
      * Models for SD21 & SDXL run through
      
      * Added back pipelines, cleared up connections
      
      * Cleaned up connection creation
      
      * added debug logs
      
      * updated logs
      
      * logs: added input loading
      
      * Update umer_debug_logger.py
      
      * log: Loading hint
      
      * Update umer_debug_logger.py
      
      * added logs
      
      * Changed debug logging
      
      * debug: added more logs
      
      * Fixed num_norm_groups
      
      * Debug: Logging all of SDXL input
      
      * Update umer_debug_logger.py
      
      * debug: updated logs
      
      * checkim
      
      * Readded tests
      
      * Removed debug logs
      
      * Fixed Slow Tests
      
      * Added value ckecks | Updated model_cpu_offload_seq
      
      * accelerate-offloading works ; fast tests work
      
      * Made unet & addon explicit in controlnet
      
      * Updated slow tests
      
      * Added dtype/device to ControlNetXS
      
      * Filled in test model paths
      
      * Added image_encoder/feature_extractor to XL pipe
      
      * Fixed fast tests
      
      * Added comments and docstrings
      
      * Fixed copies
      
      * Added docs ; Updates slow tests
      
      * Moved changes to UNetMidBlock2DCrossAttn
      
      * tiny cleanups
      
      * Removed stray prints
      
      * Removed ip adapters + freeU
      
      - Removed ip adapters + freeU as they don't make sense for ControlNet-XS
      - Fixed imports of UNet components
      
      * Fixed test_save_load_float16
      
      * Make style, quality, fix-copies
      
      * Changed loading/saving API for ControlNetXS
      
      - Changed loading/saving API for ControlNetXS
      - other small fixes
      
      * Removed ControlNet-XS from research examples
      
      * Make style, quality, fix-copies
      
      * Small fixes
      
      - deleted ControlNetXSModel.init_original
      - added time_embedding_mix to StableDiffusionControlNetXSPipeline .from_pretrained / StableDiffusionXLControlNetXSPipeline.from_pretrained
      - fixed copy hints
      
      * checkin May 11 '23
      
      * CheckIn Mar 12 '24
      
      * Fixed tests for SD
      
      * Added tests for UNetControlNetXSModel
      
      * Fixed SDXL tests
      
      * cleanup
      
      * Delete Pipfile
      
      * CheckIn Mar 20
      
      Started replacing sub blocks  by `ControlNetXSCrossAttnDownBlock2D` and `ControlNetXSCrossAttnUplock2D`
      
      * check-in Mar 23
      
      * checkin 24 Mar
      
      * Created init for UNetCnxs and CnxsAddon
      
      * CheckIn
      
      * Made from_modules, from_unet and no_control work
      
      * make style,quality,fix-copies & small changes
      
      * Fixed freezing
      
      * Added gradient ckpt'ing; fixed tests
      
      * Fix slow tests(+compile) ; clear naming confusion
      
      * Don't create UNet in init ; removed class_emb
      
      * Incorporated review feedback
      
      - Deleted get_base_pipeline /  get_controlnet_addon for pipes
      - Pipes inherit from StableDiffusionXLPipeline
      - Made module dicts for cnxs-addon's down/mid/up classes
      - Added support for qkv fusion and freeU
      
      * Make style, quality, fix-copies
      
      * Implemented review feedback
      
      * Removed compatibility check for vae/ctrl embedding
      
      * make style, quality, fix-copies
      
      * Delete Pipfile
      
      * Integrated review feedback
      
      - Importing ControlNetConditioningEmbedding now
      - get_down/mid/up_block_addon now outside class
      - renamed `do_control` to `apply_control`
      
      * Reduced size of test tensors
      
      For this, added `norm_num_groups` as parameter everywhere
      
      * Renamed cnxs-`Addon` to cnxs-`Adapter`
      
      - `ControlNetXSAddon` -> `ControlNetXSAdapter`
      - `ControlNetXSAddonDownBlockComponents` -> `DownBlockControlNetXSAdapter`, and similarly for mid/up
      - `get_mid_block_addon` -> `get_mid_block_adapter`, and similarly for mid/up
      
      * Fixed save_pretrained/from_pretrained bug
      
      * Removed redundant code
      
      ---------
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      fda1531d
  24. 18 Mar, 2024 1 commit
  25. 13 Mar, 2024 1 commit
    • Manuel Brack's avatar
      [Pipeline] Add LEDITS++ pipelines (#6074) · 00eca4b8
      Manuel Brack authored
      
      
      * Setup LEdits++ file structure
      
      * Fix import
      
      * LEditsPP Stable Diffusion pipeline
      
      * Include variable image aspect ratios
      
      * Implement LEDITS++ for SDXL
      
      * clean up LEditsPPPipelineStableDiffusion
      
      * Adjust inversion output
      
      * Added docu, more cleanup for LEditsPPPipelineStableDiffusion
      
      * clean up LEditsPPPipelineStableDiffusionXL
      
      * Update documentation
      
      * Fix documentation import
      
      * Add skeleton IF implementation
      
      * Fix documentation typo
      
      * Add LEDTIS docu to toctree
      
      * Add missing title
      
      * Finalize SD documentation
      
      * Finalize SD-XL documentation
      
      * Fix code style and quality
      
      * Fix typo
      
      * Fix return types
      
      * added LEditsPPPipelineIF; minor changes for LEditsPPPipelineStableDiffusion and LEditsPPPipelineStableDiffusionXL
      
      * Fix copy reference
      
      * add documentation for IF
      
      * Add first tests
      
      * Fix batching for SD-XL
      
      * Fix text encoding and perfect reconstruction for SD-XL
      
      * Add tests for SD-XL, minor changes
      
      * move user_mask to correct device, use cross_attention_kwargs also for inversion
      
      * Example docstring
      
      * Fix attention resolution for non-square images
      
      * Refactoring for PR review
      
      * Safely remove ledits_utils.py
      
      * Style fixes
      
      * Replace assertions with ValueError
      
      * Remove LEditsPPPipelineIF
      
      * Remove unecessary input checks
      
      * Refactoring of CrossAttnProcessor
      
      * Revert unecessary changes to scheduler
      
      * Remove first progress-bar in inversion
      
      * Refactor scheduler usage and reset
      
      * Use imageprocessor instead of custom logic
      
      * Fix scheduler init warning
      
      * Fix error when running the pipeline in fp16
      
      * Update documentation wrt perfect inversion
      
      * Update tests
      
      * Fix code quality and copy consistency
      
      * Update LEditsPP import
      
      * Remove enable/disable methods that are now in StableDiffusionMixin
      
      * Change import in docs
      
      * Revert import structure change
      
      * Fix ledits imports
      
      ---------
      Co-authored-by: default avatarKatharina Kornmeier <katharina.kornmeier@stud.tu-darmstadt.de>
      00eca4b8
  26. 06 Mar, 2024 1 commit
    • Kashif Rasul's avatar
      [Pipiline] Wuerstchen v3 aka Stable Cascasde pipeline (#6487) · 40aa47b9
      Kashif Rasul authored
      
      
      * initial diffNext v3
      
      * move to v3 folder
      
      * imports
      
      * dry up the unets
      
      * no switch_level
      
      * fix init
      
      * add switch_level tp config
      
      * Fixed some things
      
      * Added pooled text embeddings
      
      * Initial work on adding image encoder
      
      * changes from @dome272
      
      * Stuff for the image encoder processing and variable naming in decoder
      
      * fix arg name
      
      * inference fixes
      
      * inference fixes
      
      * default TimestepBlock without conds
      
      * c_skip=0 by default
      
      * fix bfloat16 to cpu
      
      * use config
      
      * undo temp change
      
      * fix gen_c_embeddings args
      
      * change text encoding
      
      * text encoding
      
      * undo print
      
      * undo .gitignore change
      
      * Allow WuerstchenV3PriorPipeline to use the base DDPM & DDIM schedulers
      
      * use WuerstchenV3Unet in both pipelines
      
      * fix imports
      
      * initial failing tests
      
      * cleanup
      
      * use scheduler.timesterps
      
      * some fixes to the tests, still not fully working
      
      * fix tests
      
      * fix prior tests
      
      * add dropout to the model_kwargs
      
      * more tests passing
      
      * update expected_slice
      
      * initial rename
      
      * rename tests
      
      * rename class names
      
      * make fix-copies
      
      * initial docs
      
      * autodocs
      
      * typos
      
      * fix arg docs
      
      * add text_encoder info
      
      * combined pipeline has optional image arg
      
      * fix documentation
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * use self.config
      
      * Update src/diffusers/pipelines/stable_cascade/modeling_stable_cascade_common.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * c_in -> in_channels
      
      * removed kwargs from unet's forward
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * remove older callback api
      
      * removed kwargs and fixed decoder guidance > 1
      
      * decoder takes emeds
      
      * check and use image_embeds
      
      * fixed all but one decoder test
      
      * fix decoder tests
      
      * update callback api
      
      * fix some more combined tests
      
      * push combined pipeline
      
      * initial docs
      
      * fix doc_string
      
      * update combined api
      
      * no test_callback_inputs test for combined pipeline
      
      * add optional components
      
      * fix ordering of components
      
      * fix combined tests
      
      * update convert script
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade_prior.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade_prior.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * Update src/diffusers/pipelines/stable_cascade/pipeline_stable_cascade_prior.py
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      
      * fix imports
      
      * move effnet out of deniosing loop
      
      * prompt_embeds_pooled only when doing guidance
      
      * Fix repeat shape
      
      * move StableCascadeUnet to models/unets/
      
      * more descriptive names
      
      * converted when numpy()
      
      * StableCascadePriorPipelineOutput docs
      
      * rename StableCascadeUNet
      
      * add slow tests
      
      * fix slow tests
      
      * update
      
      * update
      
      * updated model_path
      
      * add args for weights
      
      * set push_to_hub to false
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      * update
      
      ---------
      Co-authored-by: default avatarDominic Rampas <d6582533@gmail.com>
      Co-authored-by: default avatarPablo Pernias <pablo@pernias.com>
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarYiYi Xu <yixu310@gmail.com>
      Co-authored-by: default avatar99991 <99991@users.noreply.github.com>
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      40aa47b9
  27. 05 Mar, 2024 1 commit
  28. 28 Feb, 2024 1 commit
  29. 27 Feb, 2024 2 commits
  30. 09 Feb, 2024 1 commit
  31. 31 Jan, 2024 2 commits
  32. 24 Jan, 2024 1 commit
    • Aryan V S's avatar
      AnimateDiff Video to Video (#6328) · a517f665
      Aryan V S authored
      
      
      * begin animatediff img2video and video2video
      
      * revert animatediff to original implementation
      
      * add img2video as pipeline
      
      * update
      
      * add vid2vid pipeline
      
      * update imports
      
      * update
      
      * remove copied from line for check_inputs
      
      * update
      
      * update examples
      
      * add multi-batch support
      
      * fix __init__.py files
      
      * move img2vid to community
      
      * update community readme and examples
      
      * fix
      
      * make fix-copies
      
      * add vid2vid batch params
      
      * apply suggestions from review
      Co-Authored-By: default avatarDhruv Nair <dhruv.nair@gmail.com>
      
      * add test for animatediff vid2vid
      
      * torch.stack -> torch.cat
      Co-Authored-By: default avatarDhruv Nair <dhruv.nair@gmail.com>
      
      * make style
      
      * docs for vid2vid
      
      * update
      
      * fix prepare_latents
      
      * fix docs
      
      * remove img2vid
      
      * update README to :main
      
      * remove slow test
      
      * refactor pipeline output
      
      * update docs
      
      * update docs
      
      * merge community readme from :main
      
      * final fix i promise
      
      * add support for url in animatediff example
      
      * update example
      
      * update callbacks to latest implementation
      
      * Update src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * fix merge
      
      * Apply suggestions from code review
      
      * remove callback and callback_steps as suggested in review
      
      * Update tests/pipelines/animatediff/test_animatediff_video2video.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * fix import error caused due to unet refactor in #6630
      
      * fix numpy import error after tensor2vid refactor in #6626
      
      * make fix-copies
      
      * fix numpy error
      
      * fix progress bar test
      
      ---------
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      a517f665
  33. 23 Jan, 2024 1 commit
    • Sayak Paul's avatar
      [Big refactor] move unets to `unets` module 🦋 (#6630) · 1f0705ad
      Sayak Paul authored
      * move unets to  module 🦋
      
      * parameterize unet-level import.
      
      * fix flax unet2dcondition model import
      
      * models __init__
      
      * mildly depcrecating models.unet_2d_blocks in favor of models.unets.unet_2d_blocks.
      
      * noqa
      
      * correct depcrecation behaviour
      
      * inherit from the actual classes.
      
      * Empty-Commit
      
      * backwards compatibility for unet_2d.py
      
      * backward compatibility for unet_2d_condition
      
      * bc for unet_1d
      
      * bc for unet_1d_blocks
      1f0705ad