Skip to content

GitLab

Commit d34de4c4 authored by mashun1's avatar mashun1
Browse files

add_loop

parent 4352b6e6
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 365 additions and 474 deletions
LICENSE
View file @ d34de4c4
This diff is collapsed.
README.md
View file @ d34de4c4
...@@ -81,6 +81,7 @@ Tips：以上dtk驱动、python、torch等DCU相关工具版本需要严格一 ...@@ -81,6 +81,7 @@ Tips：以上dtk驱动、python、torch等DCU相关工具版本需要严格一
|DynamiCrafter1024|576x1024|18.3GB & 75s (`perframe_ae=True`)|https://huggingface.co/Doubiiu/DynamiCrafter_1024/blob/main/model.ckpt| |DynamiCrafter1024|576x1024|18.3GB & 75s (`perframe_ae=True`)|https://huggingface.co/Doubiiu/DynamiCrafter_1024/blob/main/model.ckpt|
|DynamiCrafter512|320x512|12.8GB & 20s (`perframe_ae=True`)|https://huggingface.co/Doubiiu/DynamiCrafter_512/blob/main/model.ckpt| |DynamiCrafter512|320x512|12.8GB & 20s (`perframe_ae=True`)|https://huggingface.co/Doubiiu/DynamiCrafter_512/blob/main/model.ckpt|
|DynamiCrafter256|256x256|11.9GB & 10s (`perframe_ae=False`)|https://huggingface.co/Doubiiu/DynamiCrafter/blob/main/model.ckpt| |DynamiCrafter256|256x256|11.9GB & 10s (`perframe_ae=False`)|https://huggingface.co/Doubiiu/DynamiCrafter/blob/main/model.ckpt|
|DynamiCrafter512_interp| 320x512 | 12.8GB & 20s (`perframe_ae=True`)|https://huggingface.co/Doubiiu/DynamiCrafter_512_Interp/blob/main/model.ckpt|
注意：若无法访问`huggingface`，可使用镜像`hf-mirror`（替换`huggingface.co`）。若无法访问`huggingface`，需要执行`export HF_ENDPOINT=https://hf-mirror.com`设置环境变量，用以自动下载其他必要模型。 注意：若无法访问`huggingface`，可使用镜像`hf-mirror`（替换`huggingface.co`）。若无法访问`huggingface`，需要执行`export HF_ENDPOINT=https://hf-mirror.com`设置环境变量，用以自动下载其他必要模型。
...@@ -91,8 +92,9 @@ Tips：以上dtk驱动、python、torch等DCU相关工具版本需要严格一 ...@@ -91,8 +92,9 @@ Tips：以上dtk驱动、python、torch等DCU相关工具版本需要严格一
└── model.ckpt └── model.ckpt
|── dynamicrafter_1024_v1 |── dynamicrafter_1024_v1
└── model.ckpt └── model.ckpt
└── dynamicrafter_256_v1 |── dynamicrafter_256_v1
└── model.ckpt └── model.ckpt
└── ...
### 命令行 ### 命令行
...@@ -103,18 +105,36 @@ Tips：以上dtk驱动、python、torch等DCU相关工具版本需要严格一 ...@@ -103,18 +105,36 @@ Tips：以上dtk驱动、python、torch等DCU相关工具版本需要严格一
# Run on multiple GPUs for parallel inference: # Run on multiple GPUs for parallel inference:
sh scripts/run_mp.sh 512 sh scripts/run_mp.sh 512
sh scripts/run_application.sh interp # Generate frame interpolation
sh scripts/run_application.sh loop # Looping video generation
### gradio页面 ### gradio页面
python gradio_app.py --res 512 python gradio_app.py --res 512
## result python gradio_app_interp_and_loop.py
## result
### normal
||输入|输出| ||输入|输出|
|:---|:---|:---| |:---|:---|:---|
|image|![alt text](readme_imgs/bloom01.png)|![Alt text](readme_imgs/image-3.gif)| |image|![alt text](readme_imgs/bloom01.png)|![Alt text](readme_imgs/image-3.gif)|
|prompt|time-lapse of a blooming flower with leaves and a stem|| |prompt|time-lapse of a blooming flower with leaves and a stem||
### interp
||输入1|输入2|结果|
|:---|:---|:---|:---|
|image|![alt text](readme_imgs/smile_01.png)|![alt text](readme_imgs/smile_02.png)|![alt text](readme_imgs/r2.gif)|
|prompt|a smiling girl|||
### loop
||输入|结果|
|:---|:---|:---|
|image|![alt text](readme_imgs/24.png)|![alt text](readme_imgs/r3.gif)|
|prompt|a beach with waves and clouds at sunset||
### 精度 ### 精度
无 无
......
assets/application/smile.gif 0 → 100644
View file @ d34de4c4
{"payload":{"allShortcutsEnabled":false,"fileTree":{"assets/application":{"items":[{"name":"05.gif","path":"assets/application/05.gif","contentType":"file"},{"name":"25.gif","path":"assets/application/25.gif","contentType":"file"},{"name":"34.gif","path":"assets/application/34.gif","contentType":"file"},{"name":"35.gif","path":"assets/application/35.gif","contentType":"file"},{"name":"36.gif","path":"assets/application/36.gif","contentType":"file"},{"name":"60.gif","path":"assets/application/60.gif","contentType":"file"},{"name":"gkxX0kb8mE8.gif","path":"assets/application/gkxX0kb8mE8.gif","contentType":"file"},{"name":"gkxX0kb8mE8_input_end.png","path":"assets/application/gkxX0kb8mE8_input_end.png","contentType":"file"},{"name":"gkxX0kb8mE8_input_start.png","path":"assets/application/gkxX0kb8mE8_input_start.png","contentType":"file"},{"name":"smile.gif","path":"assets/application/smile.gif","contentType":"file"},{"name":"smile_end.png","path":"assets/application/smile_end.png","contentType":"file"},{"name":"smile_start.png","path":"assets/application/smile_start.png","contentType":"file"},{"name":"stone01.gif","path":"assets/application/stone01.gif","contentType":"file"},{"name":"stone01_end.png","path":"assets/application/stone01_end.png","contentType":"file"},{"name":"stone01_start.png","path":"assets/application/stone01_start.png","contentType":"file"},{"name":"storytellingvideo.gif","path":"assets/application/storytellingvideo.gif","contentType":"file"}],"totalCount":16},"assets":{"items":[{"name":"application","path":"assets/application","contentType":"directory"},{"name":"showcase","path":"assets/showcase","contentType":"directory"},{"name":"logo_long.png","path":"assets/logo_long.png","contentType":"file"},{"name":"logo_long_dark.png","path":"assets/logo_long_dark.png","contentType":"file"}],"totalCount":4},"":{"items":[{"name":"assets","path":"assets","contentType":"directory"},{"name":"configs","path":"configs","contentType":"directory"},{"name":"lvdm","path":"lvdm","contentType":"directory"},{"name":"prompts","path":"prompts","contentType":"directory"},{"name":"scripts","path":"scripts","contentType":"directory"},{"name":"utils","path":"utils","contentType":"directory"},{"name":".gitignore","path":".gitignore","contentType":"file"},{"name":"LICENSE","path":"LICENSE","contentType":"file"},{"name":"README.md","path":"README.md","contentType":"file"},{"name":"gradio_app.py","path":"gradio_app.py","contentType":"file"},{"name":"gradio_app_interp_and_loop.py","path":"gradio_app_interp_and_loop.py","contentType":"file"},{"name":"requirements.txt","path":"requirements.txt","contentType":"file"}],"totalCount":12}},"fileTreeProcessingTime":4.6787160000000005,"foldersToFetch":[],"repo":{"id":724115598,"defaultBranch":"main","name":"DynamiCrafter","ownerLogin":"Doubiiu","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-11-27T12:34:23.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/43002743?v=4","public":true,"private":false,"isOrgOwned":false},"codeLineWrapEnabled":false,"symbolsExpanded":false,"treeExpanded":true,"refInfo":{"name":"main","listCacheKey":"v0:1701088464.7378628","canEdit":false,"refType":"branch","currentOid":"29733fbd0bf53bd0e4f2fdb7e92dec217e17c06f"},"path":"assets/application/smile.gif","currentUser":null,"blob":{"rawLines":null,"stylingDirectives":null,"colorizedLines":null,"csv":null,"csvError":null,"dependabotInfo":{"showConfigurationBanner":false,"configFilePath":null,"networkDependabotPath":"/Doubiiu/DynamiCrafter/network/updates","dismissConfigurationNoticePath":"/settings/dismiss-notice/dependabot_configuration_notice","configurationNoticeDismissed":null},"displayName":"smile.gif","displayUrl":"https://github.com/Doubiiu/DynamiCrafter/blob/main/assets/application/smile.gif?raw=true","headerInfo":{"blobSize":"1.39 MB","deleteTooltip":"You must be signed in to make or propose changes","editTooltip":"You must be signed in to make or propose changes","ghDesktopPath":"https://desktop.github.com","isGitLfs":false,"onBranch":true,"shortPath":"2105b97","siteNavLoginPath":"/login?return_to=https%3A%2F%2Fgithub.com%2FDoubiiu%2FDynamiCrafter%2Fblob%2Fmain%2Fassets%2Fapplication%2Fsmile.gif","isCSV":false,"isRichtext":false,"toc":null,"lineInfo":{"truncatedLoc":null,"truncatedSloc":null},"mode":"file"},"image":true,"isCodeownersFile":null,"isPlain":false,"isValidLegacyIssueTemplate":false,"issueTemplate":null,"discussionTemplate":null,"language":null,"languageID":null,"large":false,"planSupportInfo":{"repoIsFork":null,"repoOwnedByCurrentUser":null,"requestFullPath":"/Doubiiu/DynamiCrafter/blob/main/assets/application/smile.gif","showFreeOrgGatedFeatureMessage":null,"showPlanSupportBanner":null,"upgradeDataAttributes":null,"upgradePath":null},"publishBannersInfo":{"dismissActionNoticePath":"/settings/dismiss-notice/publish_action_from_dockerfile","releasePath":"/Doubiiu/DynamiCrafter/releases/new?marketplace=true","showPublishActionBanner":false},"rawBlobUrl":"https://github.com/Doubiiu/DynamiCrafter/raw/main/assets/application/smile.gif","renderImageOrRaw":true,"richText":null,"renderedFileInfo":null,"shortPath":null,"symbolsEnabled":true,"tabSize":8,"topBannersInfo":{"overridingGlobalFundingFile":false,"globalPreferredFundingPath":null,"showInvalidCitationWarning":false,"citationHelpUrl":"https://docs.github.com/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/about-citation-files","actionsOnboardingTip":null},"truncated":false,"viewable":false,"workflowRedirectUrl":null,"symbols":null},"copilotInfo":null,"copilotAccessAllowed":false,"csrf_tokens":{"/Doubiiu/DynamiCrafter/branches":{"post":"ybQcuXFxJMy_O5zrrzx-a-fay--Ix6Ew1E4VkTnVNTGSweQRfUi1kHrwfmBLXlGiaCc_e3Q4aiC6_wP-zGicRA"},"/repos/preferences":{"post":"_GB8XTD4Rh6FrdrOv2-9BsOenCAsKxjxUEfeh0YQ2MwuxuMlWxwMX9i0XrIdW2VIzwfxR_xleXajPUjqj8M32Q"}}},"title":"DynamiCrafter/assets/application/smile.gif at main · Doubiiu/DynamiCrafter"}
\ No newline at end of file
gradio_app.py
View file @ d34de4c4
...@@ -4,6 +4,9 @@ import gradio as gr ...@@ -4,6 +4,9 @@ import gradio as gr
from scripts.gradio.i2v_test import Image2Video from scripts.gradio.i2v_test import Image2Video
sys.path.insert(1, os.path.join(sys.path[0], 'lvdm')) sys.path.insert(1, os.path.join(sys.path[0], 'lvdm'))
import tempfile
tempfile.tempdir = "./tmp"
i2v_examples_1024 = [ i2v_examples_1024 = [
['prompts/1024/astronaut04.png', 'a man in an astronaut suit playing a guitar', 50, 7.5, 1.0, 6, 123], ['prompts/1024/astronaut04.png', 'a man in an astronaut suit playing a guitar', 50, 7.5, 1.0, 6, 123],
['prompts/1024/bloom01.png', 'time-lapse of a blooming flower with leaves and a stem', 50, 7.5, 1.0, 10, 123], ['prompts/1024/bloom01.png', 'time-lapse of a blooming flower with leaves and a stem', 50, 7.5, 1.0, 10, 123],
...@@ -32,7 +35,7 @@ i2v_examples_256 = [ ...@@ -32,7 +35,7 @@ i2v_examples_256 = [
] ]
def dynamicrafter_demo(result_dir='./tmp/', res=1024): def dynamicrafter_demo(result_dir='./tmp', res=1024):
if res == 1024: if res == 1024:
resolution = '576_1024' resolution = '576_1024'
css = """#input_img {max-width: 1024px !important} #output_vid {max-width: 1024px; max-height:576px}""" css = """#input_img {max-width: 1024px !important} #output_vid {max-width: 1024px; max-height:576px}"""
...@@ -170,4 +173,4 @@ if __name__ == "__main__": ...@@ -170,4 +173,4 @@ if __name__ == "__main__":
dynamicrafter_iface = dynamicrafter_demo(result_dir, args.res) dynamicrafter_iface = dynamicrafter_demo(result_dir, args.res)
dynamicrafter_iface.queue(max_size=12) dynamicrafter_iface.queue(max_size=12)
# dynamicrafter_iface.launch(max_threads=1) # dynamicrafter_iface.launch(max_threads=1)
dynamicrafter_iface.launch(server_name='0.0.0.0', server_port=8033, max_threads=1) dynamicrafter_iface.launch(server_name='0.0.0.0', server_port=8099, max_threads=1)
\ No newline at end of file
gradio_app_interp_and_loop.py 0 → 100755
View file @ d34de4c4
import os, argparse
import sys
import gradio as gr
from scripts.gradio.i2v_test_application import Image2Video
sys.path.insert(1, os.path.join(sys.path[0], 'lvdm'))
i2v_examples_interp_512 = [
['prompts/512_interp/smile_01.png', 'a smiling girl', 50, 7.5, 1.0, 5, 12306, 'prompts/512_interp/smile_02.png'],
['prompts/512_interp/stone01_01.png', 'rotating view', 50, 7.5, 1.0, 5, 123, 'prompts/512_interp/stone01_02.png'],
['prompts/512_interp/walk_01.png', 'man walking', 50, 7.5, 1.0, 5, 345, 'prompts/512_interp/walk_02.png'],
]
i2v_examples_loop_512 = [
['prompts/512_loop/24.png', 'a beach with waves and clouds at sunset', 50, 7.5, 1.0, 5, 234],
['prompts/512_loop/36.png', 'clothes swaying in the wind', 50, 7.5, 1.0, 5, 123],
['prompts/512_loop/40.png', 'flowers swaying in the wind', 50, 7.5, 1.0, 5, 234],
]
def dynamicrafter_demo(result_dir='./tmp/', res=512):
if res == 1024:
resolution = '576_1024'
css = """#input_img {max-width: 1024px !important} #output_vid {max-width: 1024px; max-height:576px}"""
elif res == 512:
resolution = '320_512'
css = """#input_img {max-width: 512px !important} #output_vid {max-width: 512px; max-height: 320px} #input_img2 {max-width: 512px !important} #output_vid {max-width: 512px; max-height: 320px}"""
elif res == 256:
resolution = '256_256'
css = """#input_img {max-width: 256px !important} #output_vid {max-width: 256px; max-height: 256px}"""
else:
raise NotImplementedError(f"Unsupported resolution: {res}")
image2video = Image2Video(result_dir, resolution=resolution)
with gr.Blocks(analytics_enabled=False, css=css) as dynamicrafter_iface:
gr.Markdown("<div align='center'> <h1> DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors </span> </h1> \
<h2 style='font-weight: 450; font-size: 1rem; margin: 0rem'>\
<a href='https://doubiiu.github.io/'>Jinbo Xing</a>, \
<a href='https://menghanxia.github.io/'>Menghan Xia</a>, <a href='https://yzhang2016.github.io/'>Yong Zhang</a>, \
<a href=''>Haoxin Chen</a>, <a href=''> Wangbo Yu</a>,\
<a href='https://github.com/hyliu'>Hanyuan Liu</a>, <a href='https://xinntao.github.io/'>Xintao Wang</a>,\
<a href='https://www.cse.cuhk.edu.hk/~ttwong/myself.html'>Tien-Tsin Wong</a>,\
<a href='https://scholar.google.com/citations?user=4oXBp9UAAAAJ&hl=zh-CN'>Ying Shan</a>\
</h2> \
<a style='font-size:18px;color: #000000' href='https://arxiv.org/abs/2310.12190'> [ArXiv] </a>\
<a style='font-size:18px;color: #000000' href='https://doubiiu.github.io/projects/DynamiCrafter/'> [Project Page] </a> \
<a style='font-size:18px;color: #000000' href='https://github.com/Doubiiu/DynamiCrafter'> [Github] </a> </div>")
#######generative frame interpolation and looping video generation######
with gr.Tab(label='Generative Frame Interpolation_320x512'):
with gr.Column():
with gr.Row():
with gr.Column():
with gr.Row():
i2v_input_image = gr.Image(label="Input Image1",elem_id="input_img")
with gr.Row():
i2v_input_text = gr.Text(label='Prompts')
with gr.Row():
i2v_seed = gr.Slider(label='Random Seed', minimum=0, maximum=50000, step=1, value=123)
i2v_eta = gr.Slider(minimum=0.0, maximum=1.0, step=0.1, label='ETA', value=1.0, elem_id="i2v_eta")
i2v_cfg_scale = gr.Slider(minimum=1.0, maximum=15.0, step=0.5, label='CFG Scale', value=7.5, elem_id="i2v_cfg_scale")
with gr.Row():
i2v_steps = gr.Slider(minimum=1, maximum=60, step=1, elem_id="i2v_steps", label="Sampling steps", value=50)
i2v_motion = gr.Slider(minimum=5, maximum=30, step=1, elem_id="i2v_motion", label="FPS", value=10)
i2v_end_btn = gr.Button("Generate")
with gr.Column():
with gr.Row():
i2v_input_image2 = gr.Image(label="Input Image2",elem_id="input_img2")
with gr.Row():
i2v_output_video = gr.Video(label="Generated Video",elem_id="output_vid",autoplay=True,show_share_button=True)
gr.Examples(examples=i2v_examples_interp_512,
inputs=[i2v_input_image, i2v_input_text, i2v_steps, i2v_cfg_scale, i2v_eta, i2v_motion, i2v_seed, i2v_input_image2],
outputs=[i2v_output_video],
fn = image2video.get_image,
cache_examples=False,
)
i2v_end_btn.click(inputs=[i2v_input_image, i2v_input_text, i2v_steps, i2v_cfg_scale, i2v_eta, i2v_motion, i2v_seed, i2v_input_image2],
outputs=[i2v_output_video],
fn = image2video.get_image
)
#######generative frame interpolation and looping video generation######
with gr.Tab(label='Looping Video Generation_320x512'):
with gr.Column():
with gr.Row():
with gr.Column():
with gr.Row():
i2v_input_image = gr.Image(label="Input Image",elem_id="input_img")
with gr.Row():
i2v_input_text = gr.Text(label='Prompts')
with gr.Row():
i2v_seed = gr.Slider(label='Random Seed', minimum=0, maximum=50000, step=1, value=123)
i2v_eta = gr.Slider(minimum=0.0, maximum=1.0, step=0.1, label='ETA', value=1.0, elem_id="i2v_eta")
i2v_cfg_scale = gr.Slider(minimum=1.0, maximum=15.0, step=0.5, label='CFG Scale', value=7.5, elem_id="i2v_cfg_scale")
with gr.Row():
i2v_steps = gr.Slider(minimum=1, maximum=60, step=1, elem_id="i2v_steps", label="Sampling steps", value=50)
i2v_motion = gr.Slider(minimum=5, maximum=30, step=1, elem_id="i2v_motion", label="FPS", value=5)
i2v_end_btn = gr.Button("Generate")
# with gr.Tab(label='Result'):
with gr.Row():
i2v_output_video = gr.Video(label="Generated Video",elem_id="output_vid",autoplay=True,show_share_button=True)
gr.Examples(examples=i2v_examples_loop_512,
inputs=[i2v_input_image, i2v_input_text, i2v_steps, i2v_cfg_scale, i2v_eta, i2v_motion, i2v_seed],
outputs=[i2v_output_video],
fn = image2video.get_image,
cache_examples=False,
)
i2v_end_btn.click(inputs=[i2v_input_image, i2v_input_text, i2v_steps, i2v_cfg_scale, i2v_eta, i2v_motion, i2v_seed],
outputs=[i2v_output_video],
fn = image2video.get_image
)
return dynamicrafter_iface
def get_parser():
parser = argparse.ArgumentParser()
return parser
if __name__ == "__main__":
parser = get_parser()
args = parser.parse_args()
result_dir = os.path.join('./', 'results')
dynamicrafter_iface = dynamicrafter_demo(result_dir)
dynamicrafter_iface.queue(max_size=12)
# dynamicrafter_iface.launch(max_threads=1)
dynamicrafter_iface.launch(server_name='0.0.0.0', server_port=10203, max_threads=1)
nohup.txt 0 → 100644
View file @ d34de4c4
nohup: ignoring input
Global seed set to 123
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.1.1+cu121 with CUDA 1201 (you have 2.1.0a0)
Python 3.8.18 (you have 3.8.12)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
scripts/run.sh: line 56: 866 Killed CUDA_VISIBLE_DEVICES=0 python3 scripts/evaluation/inference.py --seed ${seed} --ckpt_path $ckpt --config $config --savedir $res_dir/$name --n_samples 1 --bs 1 --height ${H} --width $1 --unconditional_guidance_scale 7.5 --ddim_steps 50 --ddim_eta 1.0 --prompt_dir $prompt_dir --text_input --video_length 16 --frame_stride ${FS} --timestep_spacing 'uniform_trailing' --guidance_rescale 0.7 --perframe_ae
prompts/1024/astronaut04.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/1024/bike_chineseink.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/1024/bloom01.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/1024/firework03.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/1024/girl07.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/1024/pour_bear.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/1024/robot01.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/1024/test_prompts.txt 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/1024/zreal_penguin.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/256/art.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/256/bear.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/256/boy.png 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/256/dance1.jpeg 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
prompts/256/fire_and_beach.jpg 100644 → 100755
View file @ d34de4c4
File mode changed from 100644 to 100755
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment