# Shap-E ## Overview The Shap-E model was proposed in [Shap-E: Generating Conditional 3D Implicit Functions](https://arxiv.org/abs/2305.02463) by Alex Nichol and Heewon Jun from [OpenAI](https://github.com/openai). The abstract of the paper is the following: *We present Shap-E, a conditional generative model for 3D assets. Unlike recent work on 3D generative models which produce a single output representation, Shap-E directly generates the parameters of implicit functions that can be rendered as both textured meshes and neural radiance fields. We train Shap-E in two stages: first, we train an encoder that deterministically maps 3D assets into the parameters of an implicit function; second, we train a conditional diffusion model on outputs of the encoder. When trained on a large dataset of paired 3D and text data, our resulting models are capable of generating complex and diverse 3D assets in a matter of seconds. When compared to Point-E, an explicit generative model over point clouds, Shap-E converges faster and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space.* The original codebase can be found [here](https://github.com/openai/shap-e). ## Available Pipelines: | Pipeline | Tasks | |---|---| | [pipeline_shap_e.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/shap_e/pipeline_shap_e.py) | *Text-to-Image Generation* | | [pipeline_shap_e_img2img.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py) | *Image-to-Image Generation* | ## Available checkpoints * [`openai/shap-e`](https://huggingface.co/openai/shap-e) * [`openai/shap-e-img2img`](https://huggingface.co/openai/shap-e-img2img) ## Usage Examples In the following, we will walk you through some examples of how to use Shap-E pipelines to create 3D objects in gif format. ### Text-to-3D image generation We can use [`ShapEPipeline`] to create 3D object based on a text prompt. In this example, we will make a birthday cupcake for :firecracker: diffusers library's 1 year birthday. The workflow to use the Shap-E text-to-image pipeline is same as how you would use other text-to-image pipelines in diffusers. ```python import torch from diffusers import DiffusionPipeline device = torch.device("cuda" if torch.cuda.is_available() else "cpu") repo = "openai/shap-e" pipe = DiffusionPipeline.from_pretrained(repo, torch_dtype=torch.float16) pipe = pipe.to(device) guidance_scale = 15.0 prompt = ["A firecracker", "A birthday cupcake"] images = pipe( prompt, guidance_scale=guidance_scale, num_inference_steps=64, frame_size=256, ).images ``` The output of [`ShapEPipeline`] is a list of lists of images frames. Each list of frames can be used to create a 3D object. Let's use the `export_to_gif` utility function in diffusers to make a 3D cupcake! ```python from diffusers.utils import export_to_gif export_to_gif(images[0], "firecracker_3d.gif") export_to_gif(images[1], "cake_3d.gif") ```   ### Image-to-Image generation You can use [`ShapEImg2ImgPipeline`] along with other text-to-image pipelines in diffusers and turn your 2D generation into 3D. In this example, We will first genrate a cheeseburger with a simple prompt "A cheeseburger, white background" ```python from diffusers import DiffusionPipeline import torch pipe_prior = DiffusionPipeline.from_pretrained("kandinsky-community/kandinsky-2-1-prior", torch_dtype=torch.float16) pipe_prior.to("cuda") t2i_pipe = DiffusionPipeline.from_pretrained("kandinsky-community/kandinsky-2-1", torch_dtype=torch.float16) t2i_pipe.to("cuda") prompt = "A cheeseburger, white background" image_embeds, negative_image_embeds = pipe_prior(prompt, guidance_scale=1.0).to_tuple() image = t2i_pipe( prompt, image_embeds=image_embeds, negative_image_embeds=negative_image_embeds, ).images[0] image.save("burger.png") ```  we will then use the Shap-E image-to-image pipeline to turn it into a 3D cheeseburger :) ```python from PIL import Image from diffusers.utils import export_to_gif repo = "openai/shap-e-img2img" pipe = DiffusionPipeline.from_pretrained(repo, torch_dtype=torch.float16) pipe = pipe.to("cuda") guidance_scale = 3.0 image = Image.open("burger.png").resize((256, 256)) images = pipe( image, guidance_scale=guidance_scale, num_inference_steps=64, frame_size=256, ).images gif_path = export_to_gif(images[0], "burger_3d.gif") ```  ## ShapEPipeline [[autodoc]] ShapEPipeline - all - __call__ ## ShapEImg2ImgPipeline [[autodoc]] ShapEImg2ImgPipeline - all - __call__