main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-27T03:28:46.042713Z",
     "iopub.status.busy": "2022-10-27T03:28:46.041900Z",
     "iopub.status.idle": "2022-10-27T03:28:46.056473Z",
     "shell.execute_reply": "2022-10-27T03:28:46.055255Z",
     "shell.execute_reply.started": "2022-10-27T03:28:46.042679Z"
    },
    "tags": []
   },
   "source": [
    "# 想定制自己的文图生成模型吗？想画什么画什么\n",
    "\n",
    "**文图生成有多火，已经不用介绍了。今天主要来分享如何定制自己的文图生成模型，只需要几张图片，即可定制开发自己想要的文图生成模型哦。有问题的话，文末有交流群，欢迎加入！**\n",
    "\n",
    "文图生成任务要求模型根据所提供的描述性文本生成一张与之相对应的图片。这极大地释放了AI的想象力，也激发了人类的创意，给视觉内容创作者、文字内容创作者和大众用户带来了方便。用户可以生成多样化创意图片，并从中汲取创意灵感，打破创意瓶颈，从而可以进行创作出更优质的作品。\n",
    "\n",
    "## 感谢 @风飏 开发者开发的UI界面！\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 一、Fork项目，进入 Notebook！\n",
    "\n",
    "点击“运行一下”后，再点击“启动环境\"，选择合适的GPU后即可进入项目。 AI Studio每天自动**赠送 8 小时**的GPU算力，显存更大的GPU能够生成尺寸更大的图片哦。\n",
    "<div>\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/9319c725e55445458e0536a254e0825fa09f332bef524855899801ba0b12879b\" width=\"40%\" height=\"40%\"> \n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/d07ebc951af14889bbb100e2714fe5c7203f0d2f2cf94f4680146d612e7e9d81\" width=\"40%\" height=\"40%\">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 二、运行下面的代码，并且重启内核！\n",
    "\n",
    "进入之后，点击下边的框里左上角的“点击运行”按钮（或者点进下面的框内用快捷键 Ctrl + Enter）。\n",
    "\n",
    "**提示**：下面安装环境的代码，只需要在你**第一次进入本项目**时运行！\n",
    "\n",
    "等到显示“加载完毕, 请重启内核”后，请重启内核。\n",
    "<div>\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/12bb4a5a24fa4111b70b85aa89d64bceb7c98ea24fa54d0dba32c0e8610b412d\" width=\"40%\" height=\"40%\"> \n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/d9487d2c0b9d4467ad34a96abd60d6349802ced3ca2d483cb76bffe8415a24aa\" width=\"40%\" height=\"40%\">\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-12T14:35:14.706990Z",
     "iopub.status.busy": "2023-01-12T14:35:14.706668Z",
     "iopub.status.idle": "2023-01-12T14:35:19.300615Z",
     "shell.execute_reply": "2023-01-12T14:35:19.299887Z",
     "shell.execute_reply.started": "2023-01-12T14:35:14.706966Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "加载完毕, 请重启内核\r\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "from ui.utils import diffusers_auto_update\n",
    "diffusers_auto_update()\n",
    "clear_output() # 清理很长的内容\n",
    "print('加载完毕, 请重启内核')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# 三、运行下面的代码快速体验\n",
    "> 最后一步，点击左上角的“点击运行”后，就会自动运行下面的代码，等几秒加载模型就可以玩耍啦~ **以后每次进来这个项目，就可以直接从这里开始运行啦~**\n",
    "\n",
    "**下面推荐了部分二次元模型，想要了解更多的模型可以 [点开这里的链接](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/ppdiffusers#ppdiffusers%E6%A8%A1%E5%9E%8B%E6%94%AF%E6%8C%81%E7%9A%84%E6%9D%83%E9%87%8D)！**\n",
    "\n",
    "| ppdiffusers支持的模型名称    | huggingface对应的模型地址                           | Tips备注                                                     |\n",
    "| ---------------------------------------- | ---------------------------------------------------------- | -------------------------------------------------------------- |\n",
    "| **Linaqruf/anything-v3.0**       （推荐！） | https://huggingface.co/Linaqruf/anything-v3.0 | 二次元模型！ |\n",
    "| **hakurei/waifu-diffusion-v1-3**         （推荐！）            | https://huggingface.co/hakurei/waifu-diffusion             | Waifu v1-3的模型，主要适合画二次元图像！（对比v1-2更好！）         |\n",
    "| **MoososCap/NOVEL-MODEL**        （推荐！）| https://huggingface.co/MoososCap/NOVEL-MODEL | 二次元模型！ |\n",
    "| **Baitian/momocha**       （推荐！） | 无 | 二次元模型！ |\n",
    "| **Baitian/momoco**         （推荐！）            | 无             | 二次元模型！         |\n",
    "| **hequanshaguo/monoko-e**         （推荐！）            | 无             | 二次元模型！         |\n",
    "\n",
    "> Tips:\n",
    "！！！🔥 删除原有的模型名称后，我们可以手动输入新的模型名称时会出现智能提示！例如：我输入了any，他会自动提示出 Linaqruf/anything-v3.0 和 ruisi/anything。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0 在线转换 ckpt 或者 safetensors 权重\n",
    "我们可以使用下面的**UI**，在线转换**pytorch**的**ckpt权重**，当前仅支持**SD-V1**的权重进行转换！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-18T05:48:23.313309Z",
     "iopub.status.busy": "2023-02-18T05:48:23.312544Z",
     "iopub.status.idle": "2023-02-18T05:48:23.324141Z",
     "shell.execute_reply": "2023-02-18T05:48:23.323413Z",
     "shell.execute_reply.started": "2023-02-18T05:48:23.313249Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a1e71a804ae548cd8095dd4596ba69e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Box(children=(Box(children=(Text(value='', description='ckpt或safetensors模型文件位置', description_tooltip='你要转换的模型位…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ui import gui_convert # 点开始可能没反应，多等一会就好了，千万别多点\n",
    "display(gui_convert.gui)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 文生图UI使用\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-18T05:49:03.271708Z",
     "iopub.status.busy": "2023-02-18T05:49:03.271128Z",
     "iopub.status.idle": "2023-02-18T05:49:03.299380Z",
     "shell.execute_reply": "2023-02-18T05:49:03.298796Z",
     "shell.execute_reply.started": "2023-02-18T05:49:03.271665Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f6990bc9e654499ca46aa3c0e379e9fd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Box(children=(HTML(value='<style>\\n@media (max-width:576px) {\\n    .StableDiffusionUI_txt2imgF651 {\\n        m…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ui import gui_txt2img # 文生图\n",
    "display(gui_txt2img.gui) # 生成的图片自动保存到左侧的 outputs/txt2img 的文件夹里\n",
    "# 记得点击“生成图片”，才能出图"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 图生图UI使用\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-28T11:20:59.613296Z",
     "iopub.status.busy": "2022-12-28T11:20:59.612695Z",
     "iopub.status.idle": "2022-12-28T11:20:59.665567Z",
     "shell.execute_reply": "2022-12-28T11:20:59.664643Z",
     "shell.execute_reply.started": "2022-12-28T11:20:59.613256Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "56e47acaa1534a198e9255eba7b11ee6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Box(children=(HTML(value=\"<style>\\n@media (max-width:576px) {\\n    .StableDiffusionUI_img2img91AD {\\n        m…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ui import gui_img2img # 图生图, 在左侧上传图片, 然后修改 \"需要转换的图片路径\"\n",
    "display(gui_img2img.gui) # 生成的图片自动保存到左侧的 outputs/img2img 的文件夹里\n",
    "\n",
    "# 记得点击“生成图片”，才能出图"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 超分UI使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-28T11:21:15.413637Z",
     "iopub.status.busy": "2022-12-28T11:21:15.412816Z",
     "iopub.status.idle": "2022-12-28T11:21:15.423444Z",
     "shell.execute_reply": "2022-12-28T11:21:15.422625Z",
     "shell.execute_reply.started": "2022-12-28T11:21:15.413590Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "78cafc2bdc934ba0a6e2f242852c8fe1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Box(children=(Text(value='resources/image_Kurisu.png', description='需要超分的图片路径', layout=Layout(align_items='cen…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ui import gui_superres # 超分 (图片放大一倍), 在左侧“文件”目录上传图片, 然后修改 \"需要超分的图片路径\"\n",
    "display(gui_superres.gui) # 生成的图片自动保存到左侧的 outputs/highres 的文件夹里"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 定制化训练（让模型认识新的物体或学习到新的风格）\n",
    "\n",
    "除了直接调用外，**PaddleNLP**还提供好用的二次开发能力，可以参照**下方**或[**PaddleNLP仓库**](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/ppdiffusers/examples)的教程，只需要几张图片，就可以定制属于自己的文图生成模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "在下面**GIF**例子中，我们进行了如下操作，更多信息可进入[PaddleNLP examples](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/ppdiffusers/examples/textual_inversion)中查看更多例子。\n",
    "\n",
    "**(1)** 输入了代表该人物的**新单词**：\\<Alice\\>，新词用<>括起来，主要是为了避免与已有的单词混淆；\n",
    "\n",
    "**(2)** 指定了与该人物**比较接近的单词**：**girl**, 该单词作为先验（已知）知识，可以帮助模型快速理解新单词，从而加快模型训练；\n",
    "\n",
    "**(3)** 提供了含有下面**6张图片**的文件夹地址：**resources/Alices**，在这里我们指定了resources目录下的Alices文件夹。\n",
    "\n",
    "<center> <img src=\"https://ai-studio-static-online.cdn.bcebos.com/dcdfa7f8c35f4d5f9e0eeab7e590f5f4b576bb1728e94bb4a889b34d833397d2\" width=\"40%\" height=\"40%\"> <br /></center>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-28T11:21:18.513434Z",
     "iopub.status.busy": "2022-12-28T11:21:18.512755Z",
     "iopub.status.idle": "2022-12-28T11:21:18.533021Z",
     "shell.execute_reply": "2022-12-28T11:21:18.531913Z",
     "shell.execute_reply.started": "2022-12-28T11:21:18.513398Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7489b2c29d1440398723c69fec907f38",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Box(children=(Box(children=(Dropdown(description='训练目标', description_tooltip='训练目标是什么？风格还是实体？', index=1, layou…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ui import gui_train_text_inversion # 训练\n",
    "display(gui_train_text_inversion.gui) # 训练结果会保存到 outputs/textual_inversion 的文件夹"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**模型训练完成之后，可以使用新模型预测了！**\n",
    "\n",
    "现在模型已经认识 \\<Alice\\> 这个object（人物、物体，everything...）了。\n",
    "\n",
    "比如输入Prompt：\\<Alice\\> at the lake，就可以生成一张站在湖边的 \\<Alice\\> 图像了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-28T11:21:21.214243Z",
     "iopub.status.busy": "2022-12-28T11:21:21.213559Z",
     "iopub.status.idle": "2022-12-28T11:21:21.231119Z",
     "shell.execute_reply": "2022-12-28T11:21:21.230250Z",
     "shell.execute_reply.started": "2022-12-28T11:21:21.214206Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7489b2c29d1440398723c69fec907f38",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Box(children=(Box(children=(Dropdown(description='训练目标', description_tooltip='训练目标是什么？风格还是实体？', index=1, layou…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ui import gui_train_text_inversion # 使用训练好的权重进行加载预测\n",
    "display(gui_train_text_inversion.gui) # 生成的图片自动保存到左侧的 outputs/text_inversion_txt2img 的文件夹里\n",
    "# 输入 <Alice> 相关的Prompt试试吧，例如：<Alice> at the lake，看看模型是否学到了<Alice>这个object。记得删除默认给出的negative_prompt反面描述。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是Dreambooth训练部分\n",
    "可以参考 https://github.com/PaddlePaddle/PaddleNLP/tree/develop/ppdiffusers/examples/dreambooth 配置下面的选项\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-28T11:39:34.406174Z",
     "iopub.status.busy": "2022-12-28T11:39:34.405796Z",
     "iopub.status.idle": "2022-12-28T11:39:39.239954Z",
     "shell.execute_reply": "2022-12-28T11:39:39.238856Z",
     "shell.execute_reply.started": "2022-12-28T11:39:34.406148Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0ec6a2aba73d4b099a11f65685342e5b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Box(children=(Box(children=(Combobox(value='MoososCap/NOVEL-MODEL', description='需要训练的模型名称', layout=Layout(ali…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ui import gui_dreambooth #训练dreambooth\n",
    "display(gui_dreambooth.gui) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**现在你已经学会了如何让模型认识新的物体，模型不仅可以学习新物体还可以学习新风格。风格的学习与物体的学习方法与流程一致，我们仅需在UI界面中设置学习“style”，开始训练即可。**\n",
    "\n",
    "**欢迎各位贡献自己训练好的模型权重到PaddleNLP，比如\"xxx艺术家风格的模型\"，让更多开发者了解、使用自己的作品。欢迎加入交流群，了解如何贡献。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 四、 加入交流群，一起创造吧\n",
    "\n",
    "以上实现基于PaddleNLP，开源不易，希望大家多多支持~ \n",
    "**如果对您有帮助，记得给[PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP)点个小小的Star⭐，收藏起来，不易走丢~**\n",
    "\n",
    "GitHub地址：[https://github.com/PaddlePaddle/PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP)\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/7f287240b16f4c4081d0b57ca808dce62769b13369434c64b9a26cfc48b6a978\" width=\"80%\" height=\"80%\"> <br />\n",
    "\n",
    "- **加入交流群，一起创造吧**\n",
    "\n",
    "如果对NLP模型二次开发技术感兴趣，欢迎加入PaddleNLP**微信群**交流：\n",
    "\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/0be17ea733e341638404261e2321d4304c9fbfade5ed426389884bf8a7ff5867\" width=\"25%\" height=\"25%\"> <br />\n",
    "\n",
    "也可以加入**QQ群**，交流文图趣味应用。\n",
    "\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/de62713000a641d28c74d3174aaf517d78a4801e422f4e2788d5f890b272760b\" width=\"25%\" height=\"25%\"> <br />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 五、 更多相关产品与服务\n",
    "\n",
    "## [文心一格](https://yige.baidu.com/)\n",
    "\n",
    "文心一格是百度推出的AI艺术和创意辅助平台，是百度依托于飞桨、文心大模型持续的技术创新，在“AI作图”方面发布的产品和技术工具。定位为面向有设计需求和创意的人群，基于文心大模型智能生成多样化AI创意图片，辅助创意设计，打破创意瓶颈。用户只需输入自己的创想文字，并选择期望的画作风格，即可快速获取由一格生成的相应画作。\n",
    "\n",
    "- 文心一格官网：[yige.baidu.com](https://yige.baidu.com/)\n",
    "\n",
    "## [文心 ERNIE-ViLG AI 作画大模型](https://wenxin.baidu.com/ernie-vilg)\n",
    "\n",
    "文心 ERNIE-ViLG 是目前全球规模最大的中文跨模态生成模型，可通过自然语言实现图像生成与编辑。开发者可以通过申请相关API，实现在自己的项目中添加文生图、参考图等文心ERNIE-ViLG AI作画相关能力。\n",
    "\n",
    "- 前往文心 ERNIE-ViLG 官网，申请API服务：[https://wenxin.baidu.com/ernie-vilg](https://wenxin.baidu.com/ernie-vilg)\n",
    "\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/99da398946094eb8baeb3362a63ded8dfd34e1dbb07e44ac923922b54be64810\" width=\"10%\" height=\"10%\"> <br />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 六、参考资料\n",
    "\n",
    "- https://github.com/PaddlePaddle/PaddleNLP\n",
    "- https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features\n",
    "- https://github.com/huggingface/diffusers\n",
    "- https://github.com/CompVis/stable-diffusion\n",
    "- https://arxiv.org/pdf/2006.11239.pdf\n",
    "- https://arxiv.org/pdf/2208.01618.pdf\n",
    "- https://huggingface.co/runwayml/stable-diffusion-v1-5 (请注意里面使用的LICENSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 七、鸣谢\n",
    "感谢 **@风飏** 开发者开发的UI界面！这个界面真的太好用啦！！！"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "02171df25bfaf2d4dd046004a91568a270b3a8e32f8815a8efc9d94f8a69beff"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}