1. 19 Sep, 2023 1 commit
  2. 11 Sep, 2023 1 commit
    • Dhruv Nair's avatar
      Lazy Import for Diffusers (#4829) · b6e0b016
      Dhruv Nair authored
      
      
      * initial commit
      
      * move modules to import struct
      
      * add dummy objects and _LazyModule
      
      * add lazy import to schedulers
      
      * clean up unused imports
      
      * lazy import on models module
      
      * lazy import for schedulers module
      
      * add lazy import to pipelines module
      
      * lazy import altdiffusion
      
      * lazy import audio diffusion
      
      * lazy import audioldm
      
      * lazy import consistency model
      
      * lazy import controlnet
      
      * lazy import dance diffusion ddim ddpm
      
      * lazy import deepfloyd
      
      * lazy import kandinksy
      
      * lazy imports
      
      * lazy import semantic diffusion
      
      * lazy imports
      
      * lazy import stable diffusion
      
      * move sd output to its own module
      
      * clean up
      
      * lazy import t2iadapter
      
      * lazy import unclip
      
      * lazy import versatile and vq diffsuion
      
      * lazy import vq diffusion
      
      * helper to fetch objects from modules
      
      * lazy import sdxl
      
      * lazy import txt2vid
      
      * lazy import stochastic karras
      
      * fix model imports
      
      * fix bug
      
      * lazy import
      
      * clean up
      
      * clean up
      
      * fixes for tests
      
      * fixes for tests
      
      * clean up
      
      * remove import of torch_utils from utils module
      
      * clean up
      
      * clean up
      
      * fix mistake import statement
      
      * dedicated modules for exporting and loading
      
      * remove testing utils from utils module
      
      * fixes from  merge conflicts
      
      * Update src/diffusers/pipelines/kandinsky2_2/__init__.py
      
      * fix docs
      
      * fix alt diffusion copied from
      
      * fix check dummies
      
      * fix more docs
      
      * remove accelerate import from utils module
      
      * add type checking
      
      * make style
      
      * fix check dummies
      
      * remove torch import from xformers check
      
      * clean up error message
      
      * fixes after upstream merges
      
      * dummy objects fix
      
      * fix tests
      
      * remove unused module import
      
      ---------
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      b6e0b016
  3. 23 Aug, 2023 1 commit
  4. 16 Aug, 2023 1 commit
  5. 15 Aug, 2023 1 commit
  6. 18 Jul, 2023 1 commit
    • clarencechen's avatar
      Add Recent Timestep Scheduling Improvements to DDIM Inverse Scheduler (#3865) · c6e56e92
      clarencechen authored
      * Add Recent Timestep Scheduling Improvements to DDIM Inverse Scheduler
      
      Roll timesteps by one to reflect origin-destination semantic discrepancy
      
      Restore `set_alpha_to_one` option to handle negative initial timesteps
      
      Remove `set_alpha_to_zero` option not used due to previous truncation
      
      * Bugfix
      
      * Remove unnecessary calls to `detach()`
      
      Use `self.image_processor.preprocess` in DiffEdit pipeline functions
      
      * Preprocess list input for inverted image latents in diffedit pipeline
      
      * Add `timestep_spacing` and `steps_offset` to `DPMSolverMultistepInverseScheduler`
      
      * Update expected test results to account for inverting last forward diffusion step
      
      * Fix inversion progress bar bug
      
      * Add first draft for proper fast tests for DDIMInverseScheduler
      
      * Add deprecated DDIMInverseScheduler kwarg to ConfigMixer registry
      
      * Fix test failure in DPMMultistepInverseScheduler
      
      Invert step specification leads to negative noise variance in SDE-based algs
      
      Add first draft for proper fast tests for DPMMultistepInverseScheduler
      
      * Update expected test results to account for inverting last forward diffusion step
      
      Clean up diffedit fast test
      c6e56e92
  7. 06 Jul, 2023 1 commit
    • YiYi Xu's avatar
      Add Shap-E (#3742) · 45f6d52b
      YiYi Xu authored
      
      
      * refactor prior_transformer
      
      adding conversion script
      
      add pipeline
      
      add step_index from pipeline, + remove permute
      
      add zero pad token
      
      remove copy from statement for betas_for_alpha_bar function
      
      * add
      
      * add
      
      * update conversion script for renderer model
      
      * refactor camera a little bit
      
      * clean up
      
      * style
      
      * fix copies
      
      * Update src/diffusers/schedulers/scheduling_heun_discrete.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * alpha_transform_type
      
      * remove step_index argument
      
      * remove get_sigmas_karras
      
      * remove _yiyi_sigma_to_t
      
      * move the rescale prompt_embeds from prior_transformer to pipeline
      
      * replace baddbmm with einsum to match origial repo
      
      * Revert "replace baddbmm with einsum to match origial repo"
      
      This reverts commit 3f6b435d65dad3e5514cad2f5dd9e4419ca78e0b.
      
      * add step_index to scale_model_input
      
      * Revert "move the rescale prompt_embeds from prior_transformer to pipeline"
      
      This reverts commit 5b5a8e6be918fefd114a2945ed89d8e8fa8be21b.
      
      * move rescale from prior_transformer to pipeline
      
      * correct step_index in scale_model_input
      
      * remove print lines
      
      * refactor prior - reduce arguments
      
      * make style
      
      * add prior_image
      
      * arg embedding_proj_norm -> norm_embedding_proj
      
      * add pre-norm for proj_embedding
      
      * move rescale prompt from pipeline to _encode_prompt
      
      * add img2img pipeline
      
      * style
      
      * copies
      
      * Update src/diffusers/models/prior_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      
      add arg: encoder_hid_proj
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      
      add new config: norm_in_type
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      
      add new config: added_emb_type
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      
      rename out_dim -> clip_embed_dim
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      
      rename config: out_dim -> clip_embed_dim
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * finish refactor prior_tranformer
      
      * make style
      
      * refactor renderer
      
      * fix
      
      * make style
      
      * refactor img2img
      
      * remove params_proj
      
      * add test
      
      * add upcast_softmax to prior_transformer
      
      * enable num_images_per_prompt, add save_gif utility
      
      * add
      
      * add fast test
      
      * make style
      
      * add slow test
      
      * style
      
      * add test for img2img
      
      * refactor
      
      * enable batching
      
      * style
      
      * refactor scheduler
      
      * update test
      
      * style
      
      * attempt to solve batch related tests timeout
      
      * add doc
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * hardcode rendering related config
      
      * update betas_for_alpha_bar on ddpm_scheduler
      
      * fix copies
      
      * fix
      
      * export_to_gif
      
      * style
      
      * second attempt to speed up batching tests
      
      * add doc page to index
      
      * Remove intermediate clipping
      
      * 3rd attempt to speed up batching tests
      
      * Remvoe time index
      
      * simplify scheduler
      
      * Fix more
      
      * Fix more
      
      * fix more
      
      * make style
      
      * fix schedulers
      
      * fix some more tests
      
      * finish
      
      * add one more test
      
      * Apply suggestions from code review
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarPedro Cuenca <pedro@huggingface.co>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * style
      
      * apply feedbacks
      
      * style
      
      * fix copies
      
      * add one example
      
      * style
      
      * add example for img2img
      
      * fix doc
      
      * fix more doc strings
      
      * size -> frame_size
      
      * style
      
      * update doc
      
      * style
      
      * fix on doc
      
      * update repo name
      
      * improve the usage example in shap-e img2img
      
      * add usage examples in the shap-e docs.
      
      * consolidate examples.
      
      * minor fix.
      
      * update doc
      
      * Apply suggestions from code review
      
      * Apply suggestions from code review
      
      * remove upcast
      
      * Make sure background is white
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
      
      * Apply suggestions from code review
      
      * Finish
      
      * Apply suggestions from code review
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
      
      * Make style
      
      ---------
      Co-authored-by: default avataryiyixuxu <yixu310@gmail,com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarPedro Cuenca <pedro@huggingface.co>
      45f6d52b
  8. 05 Jul, 2023 2 commits
    • dg845's avatar
      Add Consistency Models Pipeline (#3492) · aed7499a
      dg845 authored
      
      
      * initial commit
      
      * Improve consistency models sampling implementation.
      
      * Add CMStochasticIterativeScheduler, which implements the multi-step sampler (stochastic_iterative_sampler) in the original code, and make further improvements to sampling.
      
      * Add Unet blocks for consistency models
      
      * Add conversion script for Unet
      
      * Fix bug in new unet blocks
      
      * Fix attention weight loading
      
      * Make design improvements to ConsistencyModelPipeline and CMStochasticIterativeScheduler and add initial version of tests.
      
      * make style
      
      * Make small random test UNet class conditional and set resnet_time_scale_shift to 'scale_shift' to better match consistency model checkpoints.
      
      * Add support for converting a test UNet and non-class-conditional UNets to the consistency models conversion script.
      
      * make style
      
      * Change num_class_embeds to 1000 to better match the original consistency models implementation.
      
      * Add support for distillation in pipeline_consistency_models.py.
      
      * Improve consistency model tests:
      	- Get small testing checkpoints from hub
      	- Modify tests to take into account "distillation" parameter of ConsistencyModelPipeline
      	- Add onestep, multistep tests for distillation and distillation + class conditional
      	- Add expected image slices for onestep tests
      
      * make style
      
      * Improve ConsistencyModelPipeline:
      	- Add initial support for class-conditional generation
      	- Fix initial sigma for onestep generation
      	- Fix some sigma shape issues
      
      * make style
      
      * Improve ConsistencyModelPipeline:
      	- add latents __call__ argument and prepare_latents method
      	- add check_inputs method
      	- add initial docstrings for ConsistencyModelPipeline.__call__
      
      * make style
      
      * Fix bug when randomly generating class labels for class-conditional generation.
      
      * Switch CMStochasticIterativeScheduler to configuring a sigma schedule and make related changes to the pipeline and tests.
      
      * Remove some unused code and make style.
      
      * Fix small bug in CMStochasticIterativeScheduler.
      
      * Add expected slices for multistep sampling tests and make them pass.
      
      * Work on consistency model fast tests:
      	- in pipeline, call self.scheduler.scale_model_input before denoising
      	- get expected slices for Euler and Heun scheduler tests
      	- make Euler test pass
      	- mark Heun test as expected fail because it doesn't support prediction_type "sample" yet
      	- remove DPM and Euler Ancestral tests because they don't support use_karras_sigmas
      
      * make style
      
      * Refactor conversion script to make it easier to add more model architectures to convert in the future.
      
      * Work on ConsistencyModelPipeline tests:
      	- Fix device bug when handling class labels in ConsistencyModelPipeline.__call__
      	- Add slow tests for onestep and multistep sampling and make them pass
      	- Refactor fast tests
      	- Refactor ConsistencyModelPipeline.__init__
      
      * make style
      
      * Remove the add_noise and add_noise_to_input methods from CMStochasticIterativeScheduler for now.
      
      * Run python utils/check_copies.py --fix_and_overwrite
      python utils/check_dummies.py --fix_and_overwrite to make dummy objects for new pipeline and scheduler.
      
      * Make fast tests from PipelineTesterMixin pass.
      
      * make style
      
      * Refactor consistency models pipeline and scheduler:
      	- Remove support for Karras schedulers (only support CMStochasticIterativeScheduler)
      	- Move sigma manipulation, input scaling, denoising from pipeline to scheduler
      	- Make corresponding changes to tests and ensure they pass
      
      * make style
      
      * Add docstrings and further refactor pipeline and scheduler.
      
      * make style
      
      * Add initial version of the consistency models documentation.
      
      * Refactor custom timesteps logic following DDPMScheduler/IFPipeline and temporarily add torch 2.0 SDPA kernel selection logic for debugging.
      
      * make style
      
      * Convert current slow tests to use fp16 and flash attention.
      
      * make style
      
      * Add slow tests for normal attention on cuda device.
      
      * make style
      
      * Fix attention weights loading
      
      * Update consistency model fast tests for new test checkpoints with attention fix.
      
      * make style
      
      * apply suggestions
      
      * Add add_noise method to CMStochasticIterativeScheduler (copied from EulerDiscreteScheduler).
      
      * Conversion script now outputs pipeline instead of UNet and add support for LSUN-256 models and different schedulers.
      
      * When both timesteps and num_inference_steps are supplied, raise warning instead of error (timesteps take precedence).
      
      * make style
      
      * Add remaining diffusers model checkpoints for models in the original consistency model release and update usage example.
      
      * apply suggestions from review
      
      * make style
      
      * fix attention naming
      
      * Add tests for CMStochasticIterativeScheduler.
      
      * make style
      
      * Make CMStochasticIterativeScheduler tests pass.
      
      * make style
      
      * Override test_step_shape in CMStochasticIterativeSchedulerTest instead of modifying it in SchedulerCommonTest.
      
      * make style
      
      * rename some models
      
      * Improve API
      
      * rename some models
      
      * Remove duplicated block
      
      * Add docstring and make torch compile work
      
      * More fixes
      
      * Fixes
      
      * Apply suggestions from code review
      
      * Apply suggestions from code review
      
      * add more docstring
      
      * update consistency conversion script
      
      ---------
      Co-authored-by: default avatarayushmangal <ayushmangal@microsoft.com>
      Co-authored-by: default avatarAyush Mangal <43698245+ayushtues@users.noreply.github.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      aed7499a
    • Pedro Cuenca's avatar
      Add `timestep_spacing` and `steps_offset` to schedulers (#3947) · 07c9a08e
      Pedro Cuenca authored
      
      
      * Add timestep_spacing to DDPM, LMSDiscrete, PNDM.
      
      * Remove spurious line.
      
      * More easy schedulers.
      
      * Add `linspace` to DDIM
      
      * Noise sigma for `trailing`.
      
      * Add timestep_spacing to DEISMultistepScheduler.
      
      Not sure the range is the way it was intended.
      
      * Fix: remove line used to debug.
      
      * Support timestep_spacing in DPMSolverMultistep, DPMSolverSDE, UniPC
      
      * Fix: convert to numpy.
      
      * Use sched. defaults when instantiating from_config
      
      For params not present in the original configuration.
      
      This makes it possible to switch pipeline schedulers even if they use
      different timestep_spacing (or any other param).
      
      * Apply suggestions from code review
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Missing args in DPMSolverMultistep
      
      * Test: default args not in config
      
      * Style
      
      * Fix scheduler name in test
      
      * Remove duplicated entries
      
      * Add test for solver_type
      
      This test currently fails in main. When switching from DEIS to UniPC,
      solver_type is "logrho" (the default value from DEIS), which gets
      translated to "bh1" by UniPC. This is different to the default value for
      UniPC: "bh2". This is where the translation happens: https://github.com/huggingface/diffusers/blob/36d22d0709dc19776e3016fb3392d0f5578b0ab2/src/diffusers/schedulers/scheduling_unipc_multistep.py#L171
      
      
      
      * UniPC: use same default for solver_type
      
      Fixes a bug when switching from UniPC from another scheduler (i.e.,
      DEIS) that uses a different solver type. The solver is now the same as
      if we had instantiated the scheduler directly.
      
      * do not save use default values
      
      * fix more
      
      * fix all
      
      * fix schedulers
      
      * fix more
      
      * finish for real
      
      * finish for real
      
      * flaky tests
      
      * Update tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py
      
      * Default steps_offset to 0.
      
      * Add missing docstrings
      
      * Apply suggestions from code review
      
      ---------
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      07c9a08e
  9. 20 Jun, 2023 1 commit
  10. 07 Jun, 2023 1 commit
  11. 22 May, 2023 2 commits
  12. 20 May, 2023 1 commit
    • Pedro Cuenca's avatar
      mps & onnx tests rework (#3449) · f7b4f51c
      Pedro Cuenca authored
      * Remove ONNX tests from PR.
      
      They are already a part of push_tests.yml.
      
      * Remove mps tests from PRs.
      
      They are already performed on push.
      
      * Fix workflow name for fast push tests.
      
      * Extract mps tests to a workflow.
      
      For better control/filtering.
      
      * Remove --extra-index-url from mps tests
      
      * Increase tolerance of mps test
      
      This test passes in my Mac (Ventura 13.3) but fails in the CI hardware
      (Ventura 13.2). I ran the local tests following the same steps that
      exist in the CI workflow.
      
      * Temporarily run mps tests on pr
      
      So we can test.
      
      * Revert "Temporarily run mps tests on pr"
      
      Tests passed, go back to running on push.
      f7b4f51c
  13. 06 May, 2023 1 commit
  14. 05 May, 2023 1 commit
  15. 03 May, 2023 1 commit
  16. 27 Apr, 2023 2 commits
  17. 21 Apr, 2023 1 commit
  18. 17 Apr, 2023 1 commit
  19. 14 Apr, 2023 1 commit
    • Will Berman's avatar
      ddpm custom timesteps (#3007) · b811964a
      Will Berman authored
      add custom timesteps test
      
      add custom timesteps descending order check
      
      docs
      
      timesteps -> custom_timesteps
      
      can only pass one of num_inference_steps and timesteps
      b811964a
  20. 12 Apr, 2023 1 commit
  21. 10 Apr, 2023 2 commits
  22. 06 Apr, 2023 1 commit
  23. 09 Mar, 2023 1 commit