gradio_web_demo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "gpuType": "V100"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## 通过Colab启动Web Demo\n",
        "\n",
        "本notebook提供了使用Colab中的GPU架设基于Gradio的demo示例。\n",
        "\n",
        "项目地址：https://github.com/ymcui/Chinese-LLaMA-Alpaca-2\n",
        "\n",
        "启动前准备：\n",
        "- V100/A100：可直接加载7B模型\n",
        "- T4：需要启用8bit加载（见以下启动说明）"
      ],
      "metadata": {
        "id": "xidqUxwtsBnl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 克隆代码并安装依赖"
      ],
      "metadata": {
        "id": "BNc29-zks3p5"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zPqpnapOr70x",
        "outputId": "e297adf1-b934-492d-ff97-4f1c2c8c738a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'Chinese-LLaMA-Alpaca-2'...\n",
            "remote: Enumerating objects: 242, done.\u001b[K\n",
            "remote: Counting objects: 100% (84/84), done.\u001b[K\n",
            "remote: Compressing objects: 100% (66/66), done.\u001b[K\n",
            "remote: Total 242 (delta 37), reused 39 (delta 18), pack-reused 158\u001b[K\n",
            "Receiving objects: 100% (242/242), 362.76 KiB | 3.42 MiB/s, done.\n",
            "Resolving deltas: 100% (104/104), done.\n",
            "Collecting git+https://github.com/huggingface/peft.git@13e53fc (from -r Chinese-LLaMA-Alpaca-2/requirements.txt (line 2))\n",
            "  Cloning https://github.com/huggingface/peft.git (to revision 13e53fc) to /tmp/pip-req-build-edd6aj5p\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/peft.git /tmp/pip-req-build-edd6aj5p\n",
            "\u001b[33m  WARNING: Did not find branch or tag '13e53fc', assuming revision or ref.\u001b[0m\u001b[33m\n",
            "\u001b[0m  Running command git checkout -q 13e53fc\n",
            "  Resolved https://github.com/huggingface/peft.git to commit 13e53fc\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: torch==2.0.1 in /usr/local/lib/python3.10/dist-packages (from -r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (2.0.1+cu118)\n",
            "Collecting transformers==4.31.0 (from -r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3))\n",
            "  Downloading transformers-4.31.0-py3-none-any.whl (7.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m30.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sentencepiece==0.1.97 (from -r Chinese-LLaMA-Alpaca-2/requirements.txt (line 4))\n",
            "  Downloading sentencepiece-0.1.97-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m57.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting bitsandbytes==0.39.1 (from -r Chinese-LLaMA-Alpaca-2/requirements.txt (line 5))\n",
            "  Downloading bitsandbytes-0.39.1-py3-none-any.whl (97.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.1/97.1 MB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (3.12.2)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (4.7.1)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (1.11.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (3.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (2.0.0)\n",
            "Collecting huggingface-hub<1.0,>=0.14.1 (from transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3))\n",
            "  Downloading huggingface_hub-0.16.4-py3-none-any.whl (268 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m31.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (1.22.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (23.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (6.0.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (2022.10.31)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (2.27.1)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3))\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m120.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting safetensors>=0.3.1 (from transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3))\n",
            "  Downloading safetensors-0.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m81.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (4.65.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch==2.0.1->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (3.25.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch==2.0.1->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (16.0.6)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from peft==0.3.0.dev0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 2)) (5.9.5)\n",
            "Collecting accelerate (from peft==0.3.0.dev0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 2))\n",
            "  Downloading accelerate-0.21.0-py3-none-any.whl (244 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m244.2/244.2 kB\u001b[0m \u001b[31m30.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (2023.6.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.0.1->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (2.1.3)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (2023.7.22)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 3)) (3.4)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.0.1->-r Chinese-LLaMA-Alpaca-2/requirements.txt (line 1)) (1.3.0)\n",
            "Building wheels for collected packages: peft\n",
            "  Building wheel for peft (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for peft: filename=peft-0.3.0.dev0-py3-none-any.whl size=40651 sha256=b925814e35e3f918ed8cf768c39e74046f9687af061d572ba75f446b8cdb631d\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-oj3_2rx9/wheels/d9/13/c6/404d5f8a81c5620f65f7fd75b6a66619f013cd79c2875b981c\n",
            "Successfully built peft\n",
            "Installing collected packages: tokenizers, sentencepiece, safetensors, bitsandbytes, huggingface-hub, transformers, accelerate, peft\n",
            "Successfully installed accelerate-0.21.0 bitsandbytes-0.39.1 huggingface-hub-0.16.4 peft-0.3.0.dev0 safetensors-0.3.1 sentencepiece-0.1.97 tokenizers-0.13.3 transformers-4.31.0\n",
            "Collecting gradio\n",
            "  Downloading gradio-3.39.0-py3-none-any.whl (19.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m19.9/19.9 MB\u001b[0m \u001b[31m72.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting aiofiles<24.0,>=22.0 (from gradio)\n",
            "  Downloading aiofiles-23.1.0-py3-none-any.whl (14 kB)\n",
            "Requirement already satisfied: aiohttp~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (3.8.5)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (4.2.2)\n",
            "Collecting fastapi (from gradio)\n",
            "  Downloading fastapi-0.100.1-py3-none-any.whl (65 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m65.8/65.8 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ffmpy (from gradio)\n",
            "  Downloading ffmpy-0.3.1.tar.gz (5.5 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting gradio-client>=0.3.0 (from gradio)\n",
            "  Downloading gradio_client-0.3.0-py3-none-any.whl (294 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m294.2/294.2 kB\u001b[0m \u001b[31m31.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpx (from gradio)\n",
            "  Downloading httpx-0.24.1-py3-none-any.whl (75 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.4/75.4 kB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: huggingface-hub>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (0.16.4)\n",
            "Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (3.1.2)\n",
            "Requirement already satisfied: markdown-it-py[linkify]>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (3.0.0)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (2.1.3)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (3.7.1)\n",
            "Collecting mdit-py-plugins<=0.3.3 (from gradio)\n",
            "  Downloading mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.5/50.5 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy~=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (1.22.4)\n",
            "Collecting orjson~=3.0 (from gradio)\n",
            "  Downloading orjson-3.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.7/138.7 kB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from gradio) (23.1)\n",
            "Requirement already satisfied: pandas<3.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (1.5.3)\n",
            "Requirement already satisfied: pillow<11.0,>=8.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (9.4.0)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from gradio) (1.10.12)\n",
            "Collecting pydub (from gradio)\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Collecting python-multipart (from gradio)\n",
            "  Downloading python_multipart-0.0.6-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.7/45.7 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml<7.0,>=5.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (6.0.1)\n",
            "Requirement already satisfied: requests~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (2.27.1)\n",
            "Collecting semantic-version~=2.0 (from gradio)\n",
            "  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (4.7.1)\n",
            "Collecting uvicorn>=0.14.0 (from gradio)\n",
            "  Downloading uvicorn-0.23.2-py3-none-any.whl (59 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.5/59.5 kB\u001b[0m \u001b[31m7.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting websockets<12.0,>=10.0 (from gradio)\n",
            "  Downloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp~=3.0->gradio) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp~=3.0->gradio) (2.0.12)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp~=3.0->gradio) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp~=3.0->gradio) (4.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp~=3.0->gradio) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp~=3.0->gradio) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp~=3.0->gradio) (1.3.1)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio) (4.3.3)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio) (0.12.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client>=0.3.0->gradio) (2023.6.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.14.0->gradio) (3.12.2)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.14.0->gradio) (4.65.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio) (0.1.2)\n",
            "Requirement already satisfied: linkify-it-py<3,>=1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio) (2.0.2)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (0.11.0)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (4.41.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (1.4.4)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (3.1.0)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (2.8.2)\n",
            "INFO: pip is looking at multiple versions of mdit-py-plugins to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting mdit-py-plugins<=0.3.3 (from gradio)\n",
            "  Downloading mdit_py_plugins-0.3.2-py3-none-any.whl (50 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.4/50.4 kB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading mdit_py_plugins-0.3.1-py3-none-any.whl (46 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.5/46.5 kB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading mdit_py_plugins-0.3.0-py3-none-any.whl (43 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.7/43.7 kB\u001b[0m \u001b[31m5.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading mdit_py_plugins-0.2.8-py3-none-any.whl (41 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.0/41.0 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading mdit_py_plugins-0.2.7-py3-none-any.whl (41 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.0/41.0 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading mdit_py_plugins-0.2.6-py3-none-any.whl (39 kB)\n",
            "  Downloading mdit_py_plugins-0.2.5-py3-none-any.whl (39 kB)\n",
            "INFO: pip is looking at multiple versions of mdit-py-plugins to determine which version is compatible with other requirements. This could take a while.\n",
            "  Downloading mdit_py_plugins-0.2.4-py3-none-any.whl (39 kB)\n",
            "  Downloading mdit_py_plugins-0.2.3-py3-none-any.whl (39 kB)\n",
            "  Downloading mdit_py_plugins-0.2.2-py3-none-any.whl (39 kB)\n",
            "  Downloading mdit_py_plugins-0.2.1-py3-none-any.whl (38 kB)\n",
            "  Downloading mdit_py_plugins-0.2.0-py3-none-any.whl (38 kB)\n",
            "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n",
            "  Downloading mdit_py_plugins-0.1.0-py3-none-any.whl (37 kB)\n",
            "Collecting markdown-it-py[linkify]>=2.0.0 (from gradio)\n",
            "  Downloading markdown_it_py-3.0.0-py3-none-any.whl (87 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m87.5/87.5 kB\u001b[0m \u001b[31m11.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading markdown_it_py-2.2.0-py3-none-any.whl (84 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.5/84.5 kB\u001b[0m \u001b[31m11.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas<3.0,>=1.0->gradio) (2022.7.1)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (2023.7.22)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (3.4)\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.14.0->gradio) (8.1.6)\n",
            "Collecting h11>=0.8 (from uvicorn>=0.14.0->gradio)\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting starlette<0.28.0,>=0.27.0 (from fastapi->gradio)\n",
            "  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.0/67.0 kB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpcore<0.18.0,>=0.15.0 (from httpx->gradio)\n",
            "  Downloading httpcore-0.17.3-py3-none-any.whl (74 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m74.5/74.5 kB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio) (1.3.0)\n",
            "Requirement already satisfied: anyio<5.0,>=3.0 in /usr/local/lib/python3.10/dist-packages (from httpcore<0.18.0,>=0.15.0->httpx->gradio) (3.7.1)\n",
            "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.19.3)\n",
            "Requirement already satisfied: uc-micro-py in /usr/local/lib/python3.10/dist-packages (from linkify-it-py<3,>=1->markdown-it-py[linkify]>=2.0.0->gradio) (1.0.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio) (1.16.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5.0,>=3.0->httpcore<0.18.0,>=0.15.0->httpx->gradio) (1.1.2)\n",
            "Building wheels for collected packages: ffmpy\n",
            "  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ffmpy: filename=ffmpy-0.3.1-py3-none-any.whl size=5579 sha256=57ade8007aa0450b3a03c935d39c4bf930a2cd6896c0e9d5c0687136d78ccfcc\n",
            "  Stored in directory: /root/.cache/pip/wheels/01/a6/d1/1c0828c304a4283b2c1639a09ad86f83d7c487ef34c6b4a1bf\n",
            "Successfully built ffmpy\n",
            "Installing collected packages: pydub, ffmpy, websockets, semantic-version, python-multipart, orjson, markdown-it-py, h11, aiofiles, uvicorn, starlette, mdit-py-plugins, httpcore, httpx, fastapi, gradio-client, gradio\n",
            "  Attempting uninstall: markdown-it-py\n",
            "    Found existing installation: markdown-it-py 3.0.0\n",
            "    Uninstalling markdown-it-py-3.0.0:\n",
            "      Successfully uninstalled markdown-it-py-3.0.0\n",
            "  Attempting uninstall: mdit-py-plugins\n",
            "    Found existing installation: mdit-py-plugins 0.4.0\n",
            "    Uninstalling mdit-py-plugins-0.4.0:\n",
            "      Successfully uninstalled mdit-py-plugins-0.4.0\n",
            "Successfully installed aiofiles-23.1.0 fastapi-0.100.1 ffmpy-0.3.1 gradio-3.39.0 gradio-client-0.3.0 h11-0.14.0 httpcore-0.17.3 httpx-0.24.1 markdown-it-py-2.2.0 mdit-py-plugins-0.3.3 orjson-3.9.2 pydub-0.25.1 python-multipart-0.0.6 semantic-version-2.10.0 starlette-0.27.0 uvicorn-0.23.2 websockets-11.0.3\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/ymcui/Chinese-LLaMA-Alpaca-2.git\n",
        "!pip install -r Chinese-LLaMA-Alpaca-2/requirements.txt\n",
        "!pip install gradio"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 下载Alpaca-2-7B完整模型"
      ],
      "metadata": {
        "id": "aqD4OUDBtaKM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://huggingface.co/ziqingyang/chinese-alpaca-2-7b"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4R7_FhEUtfFq",
        "outputId": "321d12ae-9a56-47bc-8d0e-c2adc35e3c1d"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'chinese-alpaca-2-7b'...\n",
            "remote: Enumerating objects: 18, done.\u001b[K\n",
            "remote: Counting objects:   5% (1/18)\u001b[K\rremote: Counting objects:  11% (2/18)\u001b[K\rremote: Counting objects:  16% (3/18)\u001b[K\rremote: Counting objects:  22% (4/18)\u001b[K\rremote: Counting objects:  27% (5/18)\u001b[K\rremote: Counting objects:  33% (6/18)\u001b[K\rremote: Counting objects:  38% (7/18)\u001b[K\rremote: Counting objects:  44% (8/18)\u001b[K\rremote: Counting objects:  50% (9/18)\u001b[K\rremote: Counting objects:  55% (10/18)\u001b[K\rremote: Counting objects:  61% (11/18)\u001b[K\rremote: Counting objects:  66% (12/18)\u001b[K\rremote: Counting objects:  72% (13/18)\u001b[K\rremote: Counting objects:  77% (14/18)\u001b[K\rremote: Counting objects:  83% (15/18)\u001b[K\rremote: Counting objects:  88% (16/18)\u001b[K\rremote: Counting objects:  94% (17/18)\u001b[K\rremote: Counting objects: 100% (18/18)\u001b[K\rremote: Counting objects: 100% (18/18), done.\u001b[K\n",
            "remote: Compressing objects:   5% (1/17)\u001b[K\rremote: Compressing objects:  11% (2/17)\u001b[K\rremote: Compressing objects:  17% (3/17)\u001b[K\rremote: Compressing objects:  23% (4/17)\u001b[K\rremote: Compressing objects:  29% (5/17)\u001b[K\rremote: Compressing objects:  35% (6/17)\u001b[K\rremote: Compressing objects:  41% (7/17)\u001b[K\rremote: Compressing objects:  47% (8/17)\u001b[K\rremote: Compressing objects:  52% (9/17)\u001b[K\rremote: Compressing objects:  58% (10/17)\u001b[K\rremote: Compressing objects:  64% (11/17)\u001b[K\rremote: Compressing objects:  70% (12/17)\u001b[K\rremote: Compressing objects:  76% (13/17)\u001b[K\rremote: Compressing objects:  82% (14/17)\u001b[K\rremote: Compressing objects:  88% (15/17)\u001b[K\rremote: Compressing objects:  94% (16/17)\u001b[K\rremote: Compressing objects: 100% (17/17)\u001b[K\rremote: Compressing objects: 100% (17/17), done.\u001b[K\n",
            "remote: Total 18 (delta 2), reused 0 (delta 0), pack-reused 0\u001b[K\n",
            "Unpacking objects: 100% (18/18), 4.58 KiB | 1.14 MiB/s, done.\n",
            "Filtering content: 100% (3/3), 4.90 GiB | 24.03 MiB/s, done.\n",
            "Encountered 1 file(s) that may not have been copied correctly on Windows:\n",
            "\tpytorch_model-00001-of-00002.bin\n",
            "\n",
            "See: `git lfs help smudge` for more details.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 启动Demo\n"
      ],
      "metadata": {
        "id": "EZCv3HS3tohu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python Chinese-LLaMA-Alpaca-2/scripts/inference/gradio_demo.py --base_model /content/chinese-alpaca-2-7b --load_in_8bit"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l9lURhKSuJVM",
        "outputId": "ef5ea01e-e5bb-4ceb-dcc3-4af5e4083027"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "===================================BUG REPORT===================================\n",
            "Welcome to bitsandbytes. For bug reports, please run\n",
            "\n",
            "python -m bitsandbytes\n",
            "\n",
            " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
            "================================================================================\n",
            "bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/lib64-nvidia did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')}\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('8013'), PosixPath('http'), PosixPath('//172.28.0.1')}\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-v100-hm-2ecbo83kb4j2y --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true'), PosixPath('--logtostderr --listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https')}\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')}\n",
            "  warn(msg)\n",
            "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.\n",
            "Either way, this might cause trouble in the future:\n",
            "If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.\n",
            "  warn(msg)\n",
            "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0\n",
            "CUDA SETUP: Highest compute capability among GPUs detected: 7.0\n",
            "CUDA SETUP: Detected CUDA version 118\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!\n",
            "  warn(msg)\n",
            "CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so...\n",
            "2023-08-02 06:10:09.023871: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "USE_MEM_EFF_ATTENTION:  True\n",
            "STORE_KV_BEFORE_ROPE: False\n",
            "Apply NTK scaling with ALPHA=1.0\n",
            "You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565\n",
            "Loading checkpoint shards: 100% 2/2 [01:08<00:00, 34.32s/it]\n",
            "Vocab of the base model: 55296\n",
            "Vocab of the tokenizer: 55296\n",
            "/content/Chinese-LLaMA-Alpaca-2/scripts/inference/gradio_demo.py:444: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.\n",
            "  user_input = gr.Textbox(\n",
            "Running on local URL:  http://0.0.0.0:19324\n",
            "Running on public URL: https://70d599ba8bb44ae10a.gradio.live\n",
            "\n",
            "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n",
            "Keyboard interruption in main thread... closing server.\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/blocks.py\", line 2133, in block_thread\n",
            "    time.sleep(0.1)\n",
            "KeyboardInterrupt\n",
            "\n",
            "During handling of the above exception, another exception occurred:\n",
            "\n",
            "Traceback (most recent call last):\n",
            "  File \"/content/Chinese-LLaMA-Alpaca-2/scripts/inference/gradio_demo.py\", line 538, in <module>\n",
            "    demo.queue().launch(\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/blocks.py\", line 2049, in launch\n",
            "    self.block_thread()\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/blocks.py\", line 2137, in block_thread\n",
            "    self.server.close()\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/gradio/networking.py\", line 49, in close\n",
            "    self.thread.join()\n",
            "  File \"/usr/lib/python3.10/threading.py\", line 1096, in join\n",
            "    self._wait_for_tstate_lock()\n",
            "  File \"/usr/lib/python3.10/threading.py\", line 1116, in _wait_for_tstate_lock\n",
            "    if lock.acquire(block, timeout):\n",
            "KeyboardInterrupt\n",
            "Killing tunnel 0.0.0.0:19324 <> https://70d599ba8bb44ae10a.gradio.live\n",
            "^C\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "启动之后，日志中的\n",
        "```\n",
        "Running on public URL: https://70d599ba8bb44ae10a.gradio.live\n",
        "```\n",
        "就是可以在线体验的demo链接（每次都不一样）。用完记得及时关闭和删除运行时，以免不必要的浪费。\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "S4BLss6S0AuD"
      }
    }
  ]
}