InstantID: Zero-shot Identity-Preserving Generation in Seconds

[**Qixun Wang**](https://github.com/wangqixun)12 · [**Xu Bai**](https://huggingface.co/baymin0220)12 · [**Haofan Wang**](https://haofanwang.github.io/)12* · [**Zekui Qin**](https://github.com/ZekuiQin)12 · [**Anthony Chen**](https://antonioo-c.github.io/)123 · Huaxia Li2 · Xu Tang2 1InstantX Team · 2Xiaohongshu Inc · 3Peking University *corresponding authors {haofanwang.ai, wangqixun.ai}@gmail.com [![GitHub](https://img.shields.io/github/stars/InstantID/InstantID?style=social)](https://github.com/InstantID/InstantID) [![ModelScope](https://img.shields.io/badge/ModelScope-Studios-blue)](https://modelscope.cn/studios/instantx/InstantID/summary) [![Open in OpenXLab](https://cdn-static.openxlab.org.cn/app-center/openxlab_app.svg)](https://openxlab.org.cn/apps/detail/InstantX/InstantID)
InstantID is a new state-of-the-art tuning-free method to achieve ID-Preserving generation with only single image, supporting various downstream tasks. ## Release - [2024/01/23] 🔥 Our pipeline has been merged into [diffusers](https://github.com/huggingface/diffusers/blob/main/examples/community/pipeline_stable_diffusion_xl_instantid.py)! - [2024/01/22] 🔥 We release the [pre-trained checkpoints](https://huggingface.co/InstantX/InstantID), [inference code](https://github.com/InstantID/InstantID/blob/main/infer.py) and [gradio demo](https://huggingface.co/spaces/InstantX/InstantID)! - [2024/01/15] 🔥 We release the [technical report](https://arxiv.org/abs/2401.07519). - [2023/12/11] 🔥 We launch the [project page](https://instantid.github.io/). ## Demos ### Stylized Synthesis

### Comparison with Previous Works

Comparison with existing tuning-free state-of-the-art techniques. InstantID achieves better fidelity and retain good text editability (faces and styles blend better).

Comparison with pre-trained character LoRAs. We don't need multiple images and still can achieve competitive results as LoRAs without any training.

Comparison with InsightFace Swapper (also known as ROOP or Refactor). However, in non-realistic style, our work is more flexible on the integration of face and background. ## Download You can directly download the model from [Huggingface](https://huggingface.co/InstantX/InstantID). You also can download the model in python script: ```python from huggingface_hub import hf_hub_download hf_hub_download(repo_id="InstantX/InstantID", filename="ControlNetModel/config.json", local_dir="./checkpoints") hf_hub_download(repo_id="InstantX/InstantID", filename="ControlNetModel/diffusion_pytorch_model.safetensors", local_dir="./checkpoints") hf_hub_download(repo_id="InstantX/InstantID", filename="ip-adapter.bin", local_dir="./checkpoints") ``` If you cannot access to Huggingface, you can use [hf-mirror](https://hf-mirror.com/) to download models. ```python export HF_ENDPOINT=https://hf-mirror.com huggingface-cli download --resume-download InstantX/InstantID --local-dir checkpoints ``` For face encoder, you need to manually download via this [URL](https://github.com/deepinsight/insightface/issues/1896#issuecomment-1023867304) to `models/antelopev2` as the default link is invalid. Once you have prepared all models, the folder tree should be like: ``` . ├── models ├── checkpoints ├── ip_adapter ├── pipeline_stable_diffusion_xl_instantid.py └── README.md ``` ## Usage ```python # !pip install opencv-python transformers accelerate insightface import diffusers from diffusers.utils import load_image from diffusers.models import ControlNetModel import cv2 import torch import numpy as np from PIL import Image from insightface.app import FaceAnalysis from pipeline_stable_diffusion_xl_instantid import StableDiffusionXLInstantIDPipeline, draw_kps # prepare 'antelopev2' under ./models app = FaceAnalysis(name='antelopev2', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider']) app.prepare(ctx_id=0, det_size=(640, 640)) # prepare models under ./checkpoints face_adapter = f'./checkpoints/ip-adapter.bin' controlnet_path = f'./checkpoints/ControlNetModel' # load IdentityNet controlnet = ControlNetModel.from_pretrained(controlnet_path, torch_dtype=torch.float16) base_model = 'wangqixun/YamerMIX_v8' # from https://civitai.com/models/84040?modelVersionId=196039 pipe = StableDiffusionXLInstantIDPipeline.from_pretrained( base_model, controlnet=controlnet, torch_dtype=torch.float16 ) pipe.cuda() # load adapter pipe.load_ip_adapter_instantid(face_adapter) ``` Then, you can customized your own face images ```python # load an image face_image = load_image("./examples/yann-lecun_resize.jpg") # prepare face emb face_info = app.get(cv2.cvtColor(np.array(face_image), cv2.COLOR_RGB2BGR)) face_info = sorted(face_info, key=lambda x:(x['bbox'][2]-x['bbox'][0])*x['bbox'][3]-x['bbox'][1])[-1] # only use the maximum face face_emb = face_info['embedding'] face_kps = draw_kps(face_image, face_info['kps']) # prompt prompt = "film noir style, ink sketch|vector, male man, highly detailed, sharp focus, ultra sharpness, monochrome, high contrast, dramatic shadows, 1940s style, mysterious, cinematic" negative_prompt = "ugly, deformed, noisy, blurry, low contrast, realism, photorealistic, vibrant, colorful" # generate image image = pipe( prompt, image_embeds=face_emb, image=face_kps, controlnet_conditioning_scale=0.8, ip_adapter_scale=0.8, ).images[0] ``` ## Speed Up with LCM-LoRA Our work is compatible with [LCM-LoRA](https://github.com/luosiallen/latent-consistency-model). First, download the model. ```python from huggingface_hub import hf_hub_download hf_hub_download(repo_id="latent-consistency/lcm-lora-sdxl", filename="pytorch_lora_weights.safetensors", local_dir="./checkpoints") ``` To use it, you just need to load it and infer with a small num_inference_steps. Note that it is recommendated to set guidance_scale between [0, 1]. ```python from diffusers import LCMScheduler lcm_lora_path = "./checkpoints/pytorch_lora_weights.safetensors" pipe.load_lora_weights(lcm_lora_path) pipe.fuse_lora() pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config) num_inference_steps = 10 guidance_scale = 0 ``` ## Start a local gradio demo Run the following command: ```python python gradio_demo/app.py ``` ## Usage Tips - For higher similarity, increase the weight of controlnet_conditioning_scale (IdentityNet) and ip_adapter_scale (Adapter). - For over-saturation, decrease the ip_adapter_scale. If not work, decrease controlnet_conditioning_scale. - For higher text control ability, decrease ip_adapter_scale. - For specific styles, choose corresponding base model makes differences. - We have not supported multi-person yet, will only use the largest face as reference pose. - We provide a [style template](https://github.com/ahgsql/StyleSelectorXL/blob/main/sdxl_styles.json) for reference. ## Community Resources ### Replicate Demo - [zsxkib/instant-id](https://replicate.com/zsxkib/instant-id) ### WebUI - [Mikubill/sd-webui-controlnet](https://github.com/Mikubill/sd-webui-controlnet/discussions/2589) ### ComfyUI - [ZHO-ZHO-ZHO/ComfyUI-InstantID](https://github.com/ZHO-ZHO-ZHO/ComfyUI-InstantID) - [huxiuhan/ComfyUI-InstantID](https://github.com/huxiuhan/ComfyUI-InstantID) ### Windows - [sdbds/InstantID-for-windows](https://github.com/sdbds/InstantID-for-windows) ## Acknowledgements - InstantID is developed by InstantX Team at Xiaohongshu Inc, all copyright reserved. - Our work is highly inspired by [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter) and [ControlNet](https://github.com/lllyasviel/ControlNet). Thanks for their great works! - Thanks [Yamer](https://civitai.com/user/Yamer) for developing [YamerMIX](https://civitai.com/models/84040?modelVersionId=196039), we use it as base model in our demo. - Thanks [ZHO-ZHO-ZHO](https://github.com/ZHO-ZHO-ZHO), [huxiuhan](https://github.com/huxiuhan), [sdbds](https://github.com/sdbds), [zsxkib](https://replicate.com/zsxkib) for their generous contributions. - Thanks to the [HuggingFace](https://github.com/huggingface) gradio team for their free GPU support! - Thanks to the [ModelScope](https://github.com/modelscope/modelscope) team for their free GPU support! - Thanks to the [OpenXLab](https://openxlab.org.cn/apps/detail/InstantX/InstantID) team for their free GPU support! ## Disclaimer The code of InstantID is released under [Apache License](https://github.com/InstantID/InstantID?tab=Apache-2.0-1-ov-file#readme) for both academic and commercial usage. **However, both manual-downloading and auto-downloading face models from insightface are for non-commercial research purposes only** accoreding to their [license](https://github.com/deepinsight/insightface?tab=readme-ov-file#license). Users are granted the freedom to create images using this tool, but they are obligated to comply with local laws and utilize it responsibly. The developers will not assume any responsibility for potential misuse by users. ## Declaration 🚨🚨🚨 We solemnly clarify that [FAKE][FAKE][FAKE] http://instantid.org [FAKE][FAKE][FAKE] is not authorized and has no relationship with us. It is infringing and quite misleading, and has never contacted us for official cooperation. Please be aware of your personal privacy and subscription fraud. We reserve all legal rights. ## Star History [![Star History Chart](https://api.star-history.com/svg?repos=InstantID/InstantID&type=Date)](https://star-history.com/#InstantID/InstantID&Date) ## Cite If you find InstantID useful for your research and applications, please cite us using this BibTeX: ```bibtex @article{wang2024instantid, title={InstantID: Zero-shot Identity-Preserving Generation in Seconds}, author={Wang, Qixun and Bai, Xu and Wang, Haofan and Qin, Zekui and Chen, Anthony}, journal={arXiv preprint arXiv:2401.07519}, year={2024} } ``` For any question, please feel free to contact us via haofanwang.ai@gmail.com or wangqixun.ai@gmail.com.