1. 06 Jul, 2023 1 commit
    • YiYi Xu's avatar
      Add Shap-E (#3742) · 45f6d52b
      YiYi Xu authored
      
      
      * refactor prior_transformer
      
      adding conversion script
      
      add pipeline
      
      add step_index from pipeline, + remove permute
      
      add zero pad token
      
      remove copy from statement for betas_for_alpha_bar function
      
      * add
      
      * add
      
      * update conversion script for renderer model
      
      * refactor camera a little bit
      
      * clean up
      
      * style
      
      * fix copies
      
      * Update src/diffusers/schedulers/scheduling_heun_discrete.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * alpha_transform_type
      
      * remove step_index argument
      
      * remove get_sigmas_karras
      
      * remove _yiyi_sigma_to_t
      
      * move the rescale prompt_embeds from prior_transformer to pipeline
      
      * replace baddbmm with einsum to match origial repo
      
      * Revert "replace baddbmm with einsum to match origial repo"
      
      This reverts commit 3f6b435d65dad3e5514cad2f5dd9e4419ca78e0b.
      
      * add step_index to scale_model_input
      
      * Revert "move the rescale prompt_embeds from prior_transformer to pipeline"
      
      This reverts commit 5b5a8e6be918fefd114a2945ed89d8e8fa8be21b.
      
      * move rescale from prior_transformer to pipeline
      
      * correct step_index in scale_model_input
      
      * remove print lines
      
      * refactor prior - reduce arguments
      
      * make style
      
      * add prior_image
      
      * arg embedding_proj_norm -> norm_embedding_proj
      
      * add pre-norm for proj_embedding
      
      * move rescale prompt from pipeline to _encode_prompt
      
      * add img2img pipeline
      
      * style
      
      * copies
      
      * Update src/diffusers/models/prior_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      
      add arg: encoder_hid_proj
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      
      add new config: norm_in_type
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      
      add new config: added_emb_type
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      
      rename out_dim -> clip_embed_dim
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      
      rename config: out_dim -> clip_embed_dim
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/prior_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * finish refactor prior_tranformer
      
      * make style
      
      * refactor renderer
      
      * fix
      
      * make style
      
      * refactor img2img
      
      * remove params_proj
      
      * add test
      
      * add upcast_softmax to prior_transformer
      
      * enable num_images_per_prompt, add save_gif utility
      
      * add
      
      * add fast test
      
      * make style
      
      * add slow test
      
      * style
      
      * add test for img2img
      
      * refactor
      
      * enable batching
      
      * style
      
      * refactor scheduler
      
      * update test
      
      * style
      
      * attempt to solve batch related tests timeout
      
      * add doc
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * hardcode rendering related config
      
      * update betas_for_alpha_bar on ddpm_scheduler
      
      * fix copies
      
      * fix
      
      * export_to_gif
      
      * style
      
      * second attempt to speed up batching tests
      
      * add doc page to index
      
      * Remove intermediate clipping
      
      * 3rd attempt to speed up batching tests
      
      * Remvoe time index
      
      * simplify scheduler
      
      * Fix more
      
      * Fix more
      
      * fix more
      
      * make style
      
      * fix schedulers
      
      * fix some more tests
      
      * finish
      
      * add one more test
      
      * Apply suggestions from code review
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarPedro Cuenca <pedro@huggingface.co>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * style
      
      * apply feedbacks
      
      * style
      
      * fix copies
      
      * add one example
      
      * style
      
      * add example for img2img
      
      * fix doc
      
      * fix more doc strings
      
      * size -> frame_size
      
      * style
      
      * update doc
      
      * style
      
      * fix on doc
      
      * update repo name
      
      * improve the usage example in shap-e img2img
      
      * add usage examples in the shap-e docs.
      
      * consolidate examples.
      
      * minor fix.
      
      * update doc
      
      * Apply suggestions from code review
      
      * Apply suggestions from code review
      
      * remove upcast
      
      * Make sure background is white
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
      
      * Apply suggestions from code review
      
      * Finish
      
      * Apply suggestions from code review
      
      * Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
      
      * Make style
      
      ---------
      Co-authored-by: default avataryiyixuxu <yixu310@gmail,com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarPedro Cuenca <pedro@huggingface.co>
      45f6d52b
  2. 05 Jul, 2023 1 commit
    • Pedro Cuenca's avatar
      Add `timestep_spacing` and `steps_offset` to schedulers (#3947) · 07c9a08e
      Pedro Cuenca authored
      
      
      * Add timestep_spacing to DDPM, LMSDiscrete, PNDM.
      
      * Remove spurious line.
      
      * More easy schedulers.
      
      * Add `linspace` to DDIM
      
      * Noise sigma for `trailing`.
      
      * Add timestep_spacing to DEISMultistepScheduler.
      
      Not sure the range is the way it was intended.
      
      * Fix: remove line used to debug.
      
      * Support timestep_spacing in DPMSolverMultistep, DPMSolverSDE, UniPC
      
      * Fix: convert to numpy.
      
      * Use sched. defaults when instantiating from_config
      
      For params not present in the original configuration.
      
      This makes it possible to switch pipeline schedulers even if they use
      different timestep_spacing (or any other param).
      
      * Apply suggestions from code review
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Missing args in DPMSolverMultistep
      
      * Test: default args not in config
      
      * Style
      
      * Fix scheduler name in test
      
      * Remove duplicated entries
      
      * Add test for solver_type
      
      This test currently fails in main. When switching from DEIS to UniPC,
      solver_type is "logrho" (the default value from DEIS), which gets
      translated to "bh1" by UniPC. This is different to the default value for
      UniPC: "bh2". This is where the translation happens: https://github.com/huggingface/diffusers/blob/36d22d0709dc19776e3016fb3392d0f5578b0ab2/src/diffusers/schedulers/scheduling_unipc_multistep.py#L171
      
      
      
      * UniPC: use same default for solver_type
      
      Fixes a bug when switching from UniPC from another scheduler (i.e.,
      DEIS) that uses a different solver type. The solver is now the same as
      if we had instantiated the scheduler directly.
      
      * do not save use default values
      
      * fix more
      
      * fix all
      
      * fix schedulers
      
      * fix more
      
      * finish for real
      
      * finish for real
      
      * flaky tests
      
      * Update tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py
      
      * Default steps_offset to 0.
      
      * Add missing docstrings
      
      * Apply suggestions from code review
      
      ---------
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      07c9a08e
  3. 10 Apr, 2023 1 commit
  4. 01 Mar, 2023 1 commit
  5. 16 Feb, 2023 2 commits
  6. 17 Jan, 2023 1 commit
    • Kashif Rasul's avatar
      DiT Pipeline (#1806) · 37d113cc
      Kashif Rasul authored
      
      
      * added dit model
      
      * import
      
      * initial pipeline
      
      * initial convert script
      
      * initial pipeline
      
      * make style
      
      * raise valueerror
      
      * single function
      
      * rename classes
      
      * use DDIMScheduler
      
      * timesteps embedder
      
      * samples to cpu
      
      * fix var names
      
      * fix numpy type
      
      * use timesteps class for proj
      
      * fix typo
      
      * fix arg name
      
      * flip_sin_to_cos and better var names
      
      * fix C shape cal
      
      * make style
      
      * remove unused imports
      
      * cleanup
      
      * add back patch_size
      
      * initial dit doc
      
      * typo
      
      * Update docs/source/api/pipelines/dit.mdx
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      
      * added copyright license headers
      
      * added example usage and toc
      
      * fix variable names asserts
      
      * remove comment
      
      * added docs
      
      * fix typo
      
      * upstream changes
      
      * set proper device for drop_ids
      
      * added initial dit pipeline test
      
      * update docs
      
      * fix imports
      
      * make fix-copies
      
      * isort
      
      * fix imports
      
      * get rid of more magic numbers
      
      * fix code when guidance is off
      
      * remove block_kwargs
      
      * cleanup script
      
      * removed to_2tuple
      
      * use FeedForward class instead of another MLP
      
      * style
      
      * work on mergint DiTBlock with BasicTransformerBlock
      
      * added missing final_dropout and args to BasicTransformerBlock
      
      * use norm from block
      
      * fix arg
      
      * remove unused arg
      
      * fix call to class_embedder
      
      * use timesteps
      
      * make style
      
      * attn_output gets multiplied
      
      * removed commented code
      
      * use Transformer2D
      
      * use self.is_input_patches
      
      * fix flags
      
      * fixed conversion to use Transformer2DModel
      
      * fixes for pipeline
      
      * remove dit.py
      
      * fix timesteps device
      
      * use randn_tensor and fix fp16 inf.
      
      * timesteps_emb already the right dtype
      
      * fix dit test class
      
      * fix test and style
      
      * fix norm2 usage in vq-diffusion
      
      * added author names to pipeline and lmagenet labels link
      
      * fix tests
      
      * use norm_type as string
      
      * rename dit to transformer
      
      * fix name
      
      * fix test
      
      * set  norm_type = "layer" by default
      
      * fix tests
      
      * do not skip common tests
      
      * Update src/diffusers/models/attention.py
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      
      * revert AdaLayerNorm API
      
      * fix norm_type name
      
      * make sure all components are in eval mode
      
      * revert norm2 API
      
      * compact
      
      * finish deprecation
      
      * add slow tests
      
      * remove @
      
      * refactor some stuff
      
      * upload
      
      * Update src/diffusers/pipelines/dit/pipeline_dit.py
      
      * finish more
      
      * finish docs
      
      * improve docs
      
      * finish docs
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      Co-authored-by: default avatarWilliam Berman <WLBberman@gmail.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      37d113cc
  7. 12 Jan, 2023 2 commits
  8. 02 Dec, 2022 1 commit
  9. 01 Dec, 2022 1 commit
  10. 30 Nov, 2022 1 commit
  11. 28 Nov, 2022 1 commit
    • Patrick von Platen's avatar
      Add 2nd order heun scheduler (#1336) · 4c54519e
      Patrick von Platen authored
      * Add heun
      
      * Finish first version of heun
      
      * remove bogus
      
      * finish
      
      * finish
      
      * improve
      
      * up
      
      * up
      
      * fix more
      
      * change progress bar
      
      * Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py
      
      * finish
      
      * up
      
      * up
      
      * up
      4c54519e
  12. 15 Nov, 2022 1 commit
  13. 06 Nov, 2022 1 commit
    • Cheng Lu's avatar
      Add multistep DPM-Solver discrete scheduler (#1132) · b4a1ed85
      Cheng Lu authored
      
      
      * add dpmsolver discrete pytorch scheduler
      
      * fix some typos in dpm-solver pytorch
      
      * add dpm-solver pytorch in stable-diffusion pipeline
      
      * add jax/flax version dpm-solver
      
      * change code style
      
      * change code style
      
      * add docs
      
      * add `add_noise` method for dpmsolver
      
      * add pytorch unit test for dpmsolver
      
      * add dummy object for pytorch dpmsolver
      
      * Update src/diffusers/schedulers/scheduling_dpmsolver_discrete.py
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      
      * Update tests/test_config.py
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      
      * Update tests/test_config.py
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      
      * resolve the code comments
      
      * rename the file
      
      * change class name
      
      * fix code style
      
      * add auto docs for dpmsolver multistep
      
      * add more explanations for the stabilizing trick (for steps < 15)
      
      * delete the dummy file
      
      * change the API name of predict_epsilon, algorithm_type and solver_type
      
      * add compatible lists
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      b4a1ed85
  14. 31 Oct, 2022 1 commit
  15. 20 Oct, 2022 2 commits
  16. 14 Oct, 2022 1 commit
  17. 07 Oct, 2022 2 commits
  18. 05 Oct, 2022 2 commits
  19. 03 Oct, 2022 1 commit
  20. 29 Sep, 2022 1 commit
  21. 27 Sep, 2022 3 commits
    • Kashif Rasul's avatar
      [Pytorch] add dep. warning for pytorch schedulers (#651) · 85494e88
      Kashif Rasul authored
      * add dep. warning for schedulers
      
      * fix format
      85494e88
    • Suraj Patil's avatar
      [DDIM, DDPM] fix add_noise (#648) · 33045382
      Suraj Patil authored
      fix add noise
      33045382
    • Kashif Rasul's avatar
      [Pytorch] Pytorch only schedulers (#534) · bd8df2da
      Kashif Rasul authored
      
      
      * pytorch only schedulers
      
      * fix style
      
      * remove match_shape
      
      * pytorch only ddpm
      
      * remove SchedulerMixin
      
      * remove numpy from karras_ve
      
      * fix types
      
      * remove numpy from lms_discrete
      
      * remove numpy from pndm
      
      * fix typo
      
      * remove mixin and numpy from sde_vp and ve
      
      * remove remaining tensor_format
      
      * fix style
      
      * sigmas has to be torch tensor
      
      * removed set_format in readme
      
      * remove set format from docs
      
      * remove set_format from pipelines
      
      * update tests
      
      * fix typo
      
      * continue to use mixin
      
      * fix imports
      
      * removed unsed imports
      
      * match shape instead of assuming image shapes
      
      * remove import typo
      
      * update call to add_noise
      
      * use math instead of numpy
      
      * fix t_index
      
      * removed commented out numpy tests
      
      * timesteps needs to be discrete
      
      * cast timesteps to int in flax scheduler too
      
      * fix device mismatch issue
      
      * small fix
      
      * Update src/diffusers/schedulers/scheduling_pndm.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      bd8df2da
  22. 17 Sep, 2022 1 commit
  23. 15 Sep, 2022 1 commit
    • Kashif Rasul's avatar
      Karras VE, DDIM and DDPM flax schedulers (#508) · b34be039
      Kashif Rasul authored
      * beta never changes removed from state
      
      * fix typos in docs
      
      * removed unused var
      
      * initial ddim flax scheduler
      
      * import
      
      * added dummy objects
      
      * fix style
      
      * fix typo
      
      * docs
      
      * fix typo in comment
      
      * set return type
      
      * added flax ddom
      
      * fix style
      
      * remake
      
      * pass PRNG key as argument and split before use
      
      * fix doc string
      
      * use config
      
      * added flax Karras VE scheduler
      
      * make style
      
      * fix dummy
      
      * fix ndarray type annotation
      
      * replace returns a new state
      
      * added lms_discrete scheduler
      
      * use self.config
      
      * add_noise needs state
      
      * use config
      
      * use config
      
      * docstring
      
      * added flax score sde ve
      
      * fix imports
      
      * fix typos
      b34be039
  24. 14 Sep, 2022 1 commit
  25. 13 Sep, 2022 2 commits
  26. 08 Sep, 2022 3 commits
    • Patrick von Platen's avatar
      [Outputs] Improve syntax (#423) · f6fb3282
      Patrick von Platen authored
      
      
      * [Outputs] Improve syntax
      
      * improve more
      
      * fix docstring return
      
      * correct all
      
      * uP
      Co-authored-by: default avatarMishig Davaadorj <dmishig@gmail.com>
      f6fb3282
    • Pedro Cuenca's avatar
      Inference support for `mps` device (#355) · 5dda1735
      Pedro Cuenca authored
      * Initial support for mps in Stable Diffusion pipeline.
      
      * Initial "warmup" implementation when using mps.
      
      * Make some deterministic tests pass with mps.
      
      * Disable training tests when using mps.
      
      * SD: generate latents in CPU then move to device.
      
      This is especially important when using the mps device, because
      generators are not supported there. See for example
      https://github.com/pytorch/pytorch/issues/84288.
      
      In addition, the other pipelines seem to use the same approach: generate
      the random samples then move to the appropriate device.
      
      After this change, generating an image in MPS produces the same result
      as when using the CPU, if the same seed is used.
      
      * Remove prints.
      
      * Pass AutoencoderKL test_output_pretrained with mps.
      
      Sampling from `posterior` must be done in CPU.
      
      * Style
      
      * Do not use torch.long for log op in mps device.
      
      * Perform incompatible padding ops in CPU.
      
      UNet tests now pass.
      See https://github.com/pytorch/pytorch/issues/84535
      
      
      
      * Style: fix import order.
      
      * Remove unused symbols.
      
      * Remove MPSWarmupMixin, do not apply automatically.
      
      We do apply warmup in the tests, but not during normal use.
      This adopts some PR suggestions by @patrickvonplaten.
      
      * Add comment for mps fallback to CPU step.
      
      * Add README_mps.md for mps installation and use.
      
      * Apply `black` to modified files.
      
      * Restrict README_mps to SD, show measures in table.
      
      * Make PNDM indexing compatible with mps.
      
      Addresses #239.
      
      * Do not use float64 when using LDMScheduler.
      
      Fixes #358.
      
      * Fix typo identified by @patil-suraj
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      
      * Adapt example to new output style.
      
      * Restore 1:1 results reproducibility with CompVis.
      
      However, mps latents need to be generated in CPU because generators
      don't work in the mps device.
      
      * Move PyTorch nightly to requirements.
      
      * Adapt `test_scheduler_outputs_equivalence` ton MPS.
      
      * mps: skip training tests instead of ignoring silently.
      
      * Make VQModel tests pass on mps.
      
      * mps ddim tests: warmup, increase tolerance.
      
      * ScoreSdeVeScheduler indexing made mps compatible.
      
      * Make ldm pipeline tests pass using warmup.
      
      * Style
      
      * Simplify casting as suggested in PR.
      
      * Add Known Issues to readme.
      
      * `isort` import order.
      
      * Remove _mps_warmup helpers from ModelMixin.
      
      And just make changes to the tests.
      
      * Skip tests using unittest decorator for consistency.
      
      * Remove temporary var.
      
      * Remove spurious blank space.
      
      * Remove unused symbol.
      
      * Remove README_mps.
      Co-authored-by: default avatarSuraj Patil <surajp815@gmail.com>
      Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> 
      5dda1735
    • Nathan Lambert's avatar
      [docs sprint] schedulers docs, will update (#376) · e6110f68
      Nathan Lambert authored
      
      
      * init schedulers docs
      
      * add some docstrings, fix sidebar formatting
      
      * add docstrings
      
      * [Type hint] PNDM schedulers (#335)
      
      * [Type hint] PNDM Schedulers
      
      * ran make style
      
      * updated timesteps type hint
      
      * apply suggestions from code review
      
      * ran make style
      
      * removed unused import
      
      * [Type hint] scheduling ddim (#343)
      
      * [Type hint] scheduling ddim
      
      * apply suggestions from code review
      
      apply suggestions to also return the return type
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * make style
      
      * update class docstrings
      
      * add docstrings
      
      * missed merge edit
      
      * add general docs page
      
      * modify headings for right sidebar
      Co-authored-by: default avatarPartho <parthodas6176@gmail.com>
      Co-authored-by: default avatarSantiago Víquez <santi.viquez@gmail.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      e6110f68
  27. 05 Sep, 2022 1 commit
  28. 04 Sep, 2022 1 commit
    • Partho's avatar
      [Type hint] PNDM schedulers (#335) · dea5ec50
      Partho authored
      * [Type hint] PNDM Schedulers
      
      * ran make style
      
      * updated timesteps type hint
      
      * apply suggestions from code review
      
      * ran make style
      
      * removed unused import
      dea5ec50
  29. 31 Aug, 2022 2 commits