1. 24 Jan, 2024 1 commit
    • Aryan V S's avatar
      AnimateDiff Video to Video (#6328) · a517f665
      Aryan V S authored
      
      
      * begin animatediff img2video and video2video
      
      * revert animatediff to original implementation
      
      * add img2video as pipeline
      
      * update
      
      * add vid2vid pipeline
      
      * update imports
      
      * update
      
      * remove copied from line for check_inputs
      
      * update
      
      * update examples
      
      * add multi-batch support
      
      * fix __init__.py files
      
      * move img2vid to community
      
      * update community readme and examples
      
      * fix
      
      * make fix-copies
      
      * add vid2vid batch params
      
      * apply suggestions from review
      Co-Authored-By: default avatarDhruv Nair <dhruv.nair@gmail.com>
      
      * add test for animatediff vid2vid
      
      * torch.stack -> torch.cat
      Co-Authored-By: default avatarDhruv Nair <dhruv.nair@gmail.com>
      
      * make style
      
      * docs for vid2vid
      
      * update
      
      * fix prepare_latents
      
      * fix docs
      
      * remove img2vid
      
      * update README to :main
      
      * remove slow test
      
      * refactor pipeline output
      
      * update docs
      
      * update docs
      
      * merge community readme from :main
      
      * final fix i promise
      
      * add support for url in animatediff example
      
      * update example
      
      * update callbacks to latest implementation
      
      * Update src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/animatediff/pipeline_animatediff_video2video.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * fix merge
      
      * Apply suggestions from code review
      
      * remove callback and callback_steps as suggested in review
      
      * Update tests/pipelines/animatediff/test_animatediff_video2video.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * fix import error caused due to unet refactor in #6630
      
      * fix numpy import error after tensor2vid refactor in #6626
      
      * make fix-copies
      
      * fix numpy error
      
      * fix progress bar test
      
      ---------
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      a517f665
  2. 05 Nov, 2023 1 commit
    • YiYi Xu's avatar
      add callbacks to denoising step (#5427) · 2b23ec82
      YiYi Xu authored
      
      
      * draft1
      
      * update
      
      * style
      
      * move to the end of loop
      
      * update
      
      * update callbak_on_step_end_inputs
      
      * Revert "update"
      
      This reverts commit 5f9b153183d0cde3b850f14024d2e37ae8c19576.
      
      * Revert "update callbak_on_step_end_inputs"
      
      This reverts commit 44889f4dabad95b7ebb330faa5f1955b5d008c88.
      
      * update
      
      * update test required_optional_params
      
      * remove self.lora_scale
      
      * img2img
      
      * inpaint
      
      * Apply suggestions from code review
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * fix
      
      * apply feedbacks on img2img + inpaint: keep only important pipeline attributes
      
      * depth
      
      * pix2pix
      
      * make _callback_tensor_inputs an class variable so that we can use it for testing
      
      * add a basic tst for callback
      
      * add a read-only tensor input timesteps + fix tests
      
      * add second test for callback cfg
      
      * sdxl
      
      * sdxl img2img
      
      * sdxl inpaint
      
      * kandinsky prior
      
      * kandinsky decoder
      
      * kandinsky img2img + combined
      
      * kandinsky inpaint
      
      * fix copies
      
      * fix
      
      * consistent default inputs
      
      * fix copies
      
      * wuerstchen_prior prior
      
      * test_wuerstchen_decoder + fix test for prior
      
      * wuerstchen_combined pipeline + skip tests
      
      * skip test for kandinsky combined
      
      * lcm
      
      * remove timesteps etc
      
      * add doc string
      
      * copies
      
      * Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      
      * make style and improve tests
      
      * up
      
      * up
      
      * fix more
      
      * fix cfg test
      
      * tests for callbacks
      
      * fix for real
      
      * update
      
      * lcm img2img
      
      * add doc
      
      * add doc page to index
      
      ---------
      Co-authored-by: default avataryiyixuxu <yixu310@gmail,com>
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarSayak Paul <spsayakpaul@gmail.com>
      Co-authored-by: default avatarDhruv Nair <dhruv.nair@gmail.com>
      2b23ec82
  3. 01 May, 2023 1 commit
  4. 13 Apr, 2023 1 commit
  5. 23 Mar, 2023 2 commits
    • Sanchit Gandhi's avatar
      Add AudioLDM (#2232) · b94880e5
      Sanchit Gandhi authored
      
      
      * Add AudioLDM
      
      * up
      
      * add vocoder
      
      * start unet
      
      * unconditional unet
      
      * clap, vocoder and vae
      
      * clean-up: conversion scripts
      
      * fix: conversion script token_type_ids
      
      * clean-up: pipeline docstring
      
      * tests: from SD
      
      * clean-up: cpu offload vocoder instead of safety checker
      
      * feat: adapt tests to audioldm
      
      * feat: add docs
      
      * clean-up: amend pipeline docstrings
      
      * clean-up: make style
      
      * clean-up: make fix-copies
      
      * fix: add doc path to toctree
      
      * clean-up: args for conversion script
      
      * clean-up: paths to checkpoints
      
      * fix: use conditional unet
      
      * clean-up: make style
      
      * fix: type hints for UNet
      
      * clean-up: docstring for UNet
      
      * clean-up: make style
      
      * clean-up: remove duplicate in docstring
      
      * clean-up: make style
      
      * clean-up: make fix-copies
      
      * clean-up: move imports to start in code snippet
      
      * fix: pass cross_attention_dim as a list/tuple to unet
      
      * clean-up: make fix-copies
      
      * fix: update checkpoint path
      
      * fix: unet cross_attention_dim in tests
      
      * film embeddings -> class embeddings
      
      * Apply suggestions from code review
      Co-authored-by: default avatarWill Berman <wlbberman@gmail.com>
      
      * fix: unet film embed to use existing args
      
      * fix: unet tests to use existing args
      
      * fix: make style
      
      * fix: transformers import and version in init
      
      * clean-up: make style
      
      * Revert "clean-up: make style"
      
      This reverts commit 5d6d1f8b324f5583e7805dc01e2c86e493660d66.
      
      * clean-up: make style
      
      * clean-up: use pipeline tester mixin tests where poss
      
      * clean-up: skip attn slicing test
      
      * fix: add torch dtype to docs
      
      * fix: remove conversion script out of src
      
      * fix: remove .detach from 1d waveform
      
      * fix: reduce default num inf steps
      
      * fix: swap height/width -> audio_length_in_s
      
      * clean-up: make style
      
      * fix: remove nightly tests
      
      * fix: imports in conversion script
      
      * clean-up: slim-down to two slow tests
      
      * clean-up: slim-down fast tests
      
      * fix: batch consistent tests
      
      * clean-up: make style
      
      * clean-up: remove vae slicing fast test
      
      * clean-up: propagate changes to doc
      
      * fix: increase test tol to 1e-2
      
      * clean-up: finish docs
      
      * clean-up: make style
      
      * feat: vocoder / VAE compatibility check
      
      * feat: possibly expand / cut audio waveform
      
      * fix: pipeline call signature test
      
      * fix: slow tests output len
      
      * clean-up: make style
      
      * make style
      
      ---------
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarWilliam Berman <WLBberman@gmail.com>
      b94880e5
    • Kashif Rasul's avatar
      Music Spectrogram diffusion pipeline (#1044) · 2ef9bdd7
      Kashif Rasul authored
      
      
      * initial TokenEncoder and ContinuousEncoder
      
      * initial modules
      
      * added ContinuousContextTransformer
      
      * fix copy paste error
      
      * use numpy for get_sequence_length
      
      * initial terminal relative positional encodings
      
      * fix weights keys
      
      * fix assert
      
      * cross attend style: concat encodings
      
      * make style
      
      * concat once
      
      * fix formatting
      
      * Initial SpectrogramPipeline
      
      * fix input_tokens
      
      * make style
      
      * added mel output
      
      * ignore weights for config
      
      * move mel to numpy
      
      * import pipeline
      
      * fix class names and import
      
      * moved models to models folder
      
      * import ContinuousContextTransformer and SpectrogramDiffusionPipeline
      
      * initial spec diffusion converstion script
      
      * renamed config to t5config
      
      * added weight loading
      
      * use arguments instead of t5config
      
      * broadcast noise time to batch dim
      
      * fix call
      
      * added scale_to_features
      
      * fix weights
      
      * transpose laynorm weight
      
      * scale is a vector
      
      * scale the query outputs
      
      * added comment
      
      * undo scaling
      
      * undo depth_scaling
      
      * inital get_extended_attention_mask
      
      * attention_mask is none in self-attention
      
      * cleanup
      
      * manually invert attention
      
      * nn.linear need bias=False
      
      * added T5LayerFFCond
      
      * remove to fix conflict
      
      * make style and dummy
      
      * remove unsed variables
      
      * remove predict_epsilon
      
      * Move accelerate to a soft-dependency (#1134)
      
      * finish
      
      * finish
      
      * Update src/diffusers/modeling_utils.py
      
      * Update src/diffusers/pipeline_utils.py
      Co-authored-by: default avatarAnton Lozhkov <anton@huggingface.co>
      
      * more fixes
      
      * fix
      Co-authored-by: default avatarAnton Lozhkov <anton@huggingface.co>
      
      * fix order
      
      * added initial midi to note token data pipeline
      
      * added int to int tokenizer
      
      * remove duplicate
      
      * added logic for segments
      
      * add melgan to pipeline
      
      * move autoregressive gen into pipeline
      
      * added note_representation_processor_chain
      
      * fix dtypes
      
      * remove immutabledict req
      
      * initial doc
      
      * use np.where
      
      * require note_seq
      
      * fix typo
      
      * update dependency
      
      * added note-seq to test
      
      * added is_note_seq_available
      
      * fix import
      
      * added toc
      
      * added example usage
      
      * undo for now
      
      * moved docs
      
      * fix merge
      
      * fix imports
      
      * predict first segment
      
      * avoid un-needed copy to and from cpu
      
      * make style
      
      * Copyright
      
      * fix style
      
      * add test and fix inference steps
      
      * remove bogus files
      
      * reorder models
      
      * up
      
      * remove transformers dependency
      
      * make work with diffusers cross attention
      
      * clean more
      
      * remove @
      
      * improve further
      
      * up
      
      * uP
      
      * Apply suggestions from code review
      
      * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py
      
      * loop over all tokens
      
      * make style
      
      * Added a section on the model
      
      * fix formatting
      
      * grammer
      
      * formatting
      
      * make fix-copies
      
      * Update src/diffusers/pipelines/__init__.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/pipelines/spectrogram_diffusion/pipeline_spectrogram_diffusion.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * added callback ad optional ionnx
      
      * do not squeeze batch dim
      
      * clean up more
      
      * upload
      
      * convert jax to nnumpy
      
      * make style
      
      * fix warning
      
      * make fix-copies
      
      * fix warning
      
      * add initial fast tests
      
      * add initial pipeline_params
      
      * eval mode due to dropout
      
      * skip batch tests as pipeline runs on a single file
      
      * make style
      
      * fix relative path
      
      * fix doc tests
      
      * Update src/diffusers/models/t5_film_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update src/diffusers/models/t5_film_transformer.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update docs/source/en/api/pipelines/spectrogram_diffusion.mdx
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      
      * add MidiProcessor
      
      * format
      
      * fix org
      
      * Apply suggestions from code review
      
      * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py
      
      * make style
      
      * pin protobuf to <4
      
      * fix formatting
      
      * white space
      
      * tensorboard needs protobuf
      
      ---------
      Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      Co-authored-by: default avatarAnton Lozhkov <anton@huggingface.co>
      2ef9bdd7
  6. 01 Mar, 2023 1 commit