quickstart.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "private_outputs": true,
      "toc_visible": true,
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Quickstart\n",
        "\n",
        "The tutorial provides a quick walkthrough of the classes and operators provided by the `dgl.sparse` package.\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/sparse/quickstart.ipynb) [![GitHub](https://img.shields.io/badge/-View%20on%20GitHub-181717?logo=github&logoColor=ffffff)](https://github.com/dmlc/dgl/blob/master/notebooks/sparse/quickstart.ipynb)"
      ],
      "metadata": {
        "id": "E0DAKDMuWz7I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install the required packages.\n",
        "\n",
        "import os\n",
        "# Uncomment following commands to download Pytorch and DGL\n",
        "# !pip install torch==2.0.0+cpu torchvision==0.15.1+cpu torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cpu > /dev/null\n",
        "# !pip install  dgl==1.1.0 -f https://data.dgl.ai/wheels/repo.html > /dev/null\n",
        "import torch\n",
        "os.environ['TORCH'] = torch.__version__\n",
        "os.environ['DGLBACKEND'] = \"pytorch\"\n",
        "\n",
        "\n",
        "try:\n",
        "    import dgl.sparse as dglsp\n",
        "    installed = True\n",
        "except ImportError:\n",
        "    installed = False\n",
        "print(\"DGL installed!\" if installed else \"DGL not found!\")"
      ],
      "metadata": {
        "id": "19UZd7wyWzpT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Sparse Matrix\n",
        "\n",
        "The core abstraction of DGL's sparse package is the `SparseMatrix` class. Compared with other sparse matrix libraries (such as `scipy.sparse` and `torch.sparse`), DGL's `SparseMatrix` is specialized for the deep learning workloads on structure data (e.g., Graph Neural Networks), with the following features:\n",
        "\n",
        "* **Auto sparse format.** Don't bother choosing between different sparse formats. There is only one `SparseMatrix` and it will select the best format for the operation to be performed.\n",
        "* **Non-zero elements can be scalar or vector.** Easy for modeling relations (e.g., edges) by vector representation.\n",
        "* **Fully PyTorch compatible.** The package is built upon PyTorch and is natively compatible with other tools in the PyTorch ecosystem.\n"
      ],
      "metadata": {
        "id": "GsWoAGC4RpHw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating a DGL Sparse Matrix\n",
        "\n",
        "The simplest way to create a sparse matrix is using the `spmatrix` API by providing the indices of the non-zero elements. The indices are stored in a tensor of shape `(2, nnz)`, where the `i`-th non-zero element is stored at position `(indices[0][i], indices[1][i])`. The code below creates a 3x3 sparse matrix.\n"
      ],
      "metadata": {
        "id": "_q4HYodcWenB"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h-ryVEs1PuIP"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import dgl.sparse as dglsp\n",
        "\n",
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "A = dglsp.spmatrix(i)  # 1.0 is default value for nnz elements.\n",
        "\n",
        "print(A)\n",
        "print(\"\")\n",
        "print(\"In dense format:\")\n",
        "print(A.to_dense())"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If not specified, the shape is inferred automatically from the indices but you can specify it explicitly too."
      ],
      "metadata": {
        "id": "W1JJg-eZ7K3t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[0, 0, 1],\n",
        "                  [0, 2, 0]])\n",
        "\n",
        "A1 = dglsp.spmatrix(i)\n",
        "print(f\"Implicit Shape: {A1.shape}\")\n",
        "print(A1.to_dense())\n",
        "print(\"\")\n",
        "\n",
        "A2 = dglsp.spmatrix(i, shape=(3, 3))\n",
        "print(f\"Explicit Shape: {A2.shape}\")\n",
        "print(A2.to_dense())"
      ],
      "metadata": {
        "id": "80NNSQfd7L5V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Both scalar values and vector values can be set for nnz elements in Sparse Matrix."
      ],
      "metadata": {
        "id": "zdNgUf0ShfCe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "# The length of the value should match the nnz elements represented by the\n",
        "# sparse matrix format.\n",
        "scalar_val = torch.tensor([1., 2., 3.])\n",
        "vector_val = torch.tensor([[1., 1.], [2., 2.], [3., 3.]])\n",
        "\n",
        "print(\"-----Scalar Values-----\")\n",
        "A = dglsp.spmatrix(i, scalar_val)\n",
        "print(A)\n",
        "print(\"\")\n",
        "print(\"In dense format:\")\n",
        "print(A.to_dense())\n",
        "print(\"\")\n",
        "\n",
        "print(\"-----Vector Values-----\")\n",
        "A = dglsp.spmatrix(i, vector_val)\n",
        "print(A)\n",
        "print(\"\")\n",
        "print(\"In dense format:\")\n",
        "print(A.to_dense())"
      ],
      "metadata": {
        "id": "buE9ZkKvhp1f"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "*Duplicated indices*"
      ],
      "metadata": {
        "id": "7ufTCDAVsrmP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[0, 0, 0, 1],\n",
        "                  [0, 2, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3., 4])\n",
        "A = dglsp.spmatrix(i, val)\n",
        "print(A)\n",
        "print(f\"Whether A contains duplicate indices: {A.has_duplicate()}\")\n",
        "print(\"\")\n",
        "\n",
        "B = A.coalesce()\n",
        "print(B)\n",
        "print(f\"Whether B contains duplicate indices: {B.has_duplicate()}\")"
      ],
      "metadata": {
        "id": "ilSAlFLOs0o8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**val_like**\n",
        "\n",
        "You can create a new sparse matrix by retaining the non-zero indices of a given sparse matrix but with different non-zero values."
      ],
      "metadata": {
        "id": "ZJ09qM5NaxuI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3.])\n",
        "A = dglsp.spmatrix(i, val)\n",
        "\n",
        "new_val = torch.tensor([4., 5., 6.])\n",
        "B = dglsp.val_like(A, new_val)\n",
        "print(B)"
      ],
      "metadata": {
        "id": "UB3lKJVBbsUD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Create a sparse matrix from various sparse formats**\n",
        "\n",
        "*   `from_coo()`: Create a sparse matrix from [COO](https://en.wikipedia.org/wiki/Sparse_matrix#Coordinate_list_(COO)) format.\n",
        "*   `from_csr()`: Create a sparse matrix from [CSR](https://en.wikipedia.org/wiki/Sparse_matrix#Compressed_sparse_row_(CSR,_CRS_or_Yale_format)) format.\n",
        "*   `from_csc()`: Create a sparse matrix from [CSC](https://en.wikipedia.org/wiki/Sparse_matrix#Compressed_sparse_column_(CSC_or_CCS)) format."
      ],
      "metadata": {
        "id": "nWjBSFDBXDPJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "row = torch.tensor([0, 1, 2, 2, 2])\n",
        "col = torch.tensor([1, 2, 0, 1, 2])\n",
        "\n",
        "print(\"-----Create from COO format-----\")\n",
        "A = dglsp.from_coo(row, col)\n",
        "print(A)\n",
        "print(\"\")\n",
        "print(\"In dense format:\")\n",
        "print(A.to_dense())\n",
        "print(\"\")\n",
        "\n",
        "indptr = torch.tensor([0, 1, 2, 5])\n",
        "indices = torch.tensor([1, 2, 0, 1, 2])\n",
        "\n",
        "print(\"-----Create from CSR format-----\")\n",
        "A = dglsp.from_csr(indptr, indices)\n",
        "print(A)\n",
        "print(\"\")\n",
        "print(\"In dense format:\")\n",
        "print(A.to_dense())\n",
        "print(\"\")\n",
        "\n",
        "print(\"-----Create from CSC format-----\")\n",
        "B = dglsp.from_csc(indptr, indices)\n",
        "print(B)\n",
        "print(\"\")\n",
        "print(\"In dense format:\")\n",
        "print(B.to_dense())"
      ],
      "metadata": {
        "id": "3puXyMFsvdlj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Attributes and methods of a DGL Sparse Matrix"
      ],
      "metadata": {
        "id": "nd4hJ9ysd4St"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[0, 1, 1, 2],\n",
        "                  [1, 0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3., 4.])\n",
        "A = dglsp.spmatrix(i, val)\n",
        "\n",
        "print(f\"Shape of sparse matrix: {A.shape}\")\n",
        "print(f\"The number of nonzero elements of sparse matrix: {A.nnz}\")\n",
        "print(f\"Datatype of sparse matrix: {A.dtype}\")\n",
        "print(f\"Device sparse matrix is stored on: {A.device}\")\n",
        "print(f\"Get the values of the nonzero elements: {A.val}\")\n",
        "print(f\"Get the row indices of the nonzero elements: {A.row}\")\n",
        "print(f\"Get the column indices of the nonzero elements: {A.col}\")\n",
        "print(f\"Get the coordinate (COO) representation: {A.coo()}\")\n",
        "print(f\"Get the compressed sparse row (CSR) representation: {A.csr()}\")\n",
        "print(f\"Get the compressed sparse column (CSC) representation: {A.csc()}\")"
      ],
      "metadata": {
        "id": "OKbFiWKIzZVe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**dtype and/or device conversion**"
      ],
      "metadata": {
        "id": "VzosM7i3yQPK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[0, 1, 1, 2],\n",
        "                  [1, 0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3., 4.])\n",
        "A = dglsp.spmatrix(i, val)\n",
        "\n",
        "B = A.to(device='cpu', dtype=torch.int32)\n",
        "print(f\"Device sparse matrix is stored on: {B.device}\")\n",
        "print(f\"Datatype of sparse matrix: {B.dtype}\")"
      ],
      "metadata": {
        "id": "y_RJihw-ypXp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Similar to pytorch, we also provide various fine-grained APIs ([Doc](https://docs.dgl.ai/en/latest/api/python/dgl.sparse_v0.html)) for dtype and/or device conversion."
      ],
      "metadata": {
        "id": "U26arLlJzfkN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Diagonal Matrix\n",
        "\n",
        "Diagonal Matrix is a special type of Sparse Matrix, in which the entries outside the main diagonal are all zero.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "EFe9ABRuWHqf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Initializing a DGL Diagonal Sparse Matrix\n",
        "A DGL Diagonal Sparse Matrix can be initiate by `dglsp.diag()`.\n",
        "\n",
        "Identity Matrix is a special type of Diagonal Sparse Matrix, in which all the value on the diagonal are 1.0. Use `dglsp.identity()` to initiate a Diagonal Sparse Matrix."
      ],
      "metadata": {
        "id": "1CeCoE2Fgl_x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "val = torch.tensor([1., 2., 3., 4.])\n",
        "D = dglsp.diag(val)\n",
        "print(D)\n",
        "\n",
        "I = dglsp.identity(shape=(3, 3))\n",
        "print(I)"
      ],
      "metadata": {
        "id": "9wzJNApahXAR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Operations on Sparse Matrix\n",
        "*   Elementwise operations\n",
        "    *   `A + B`\n",
        "    *   `A - B`\n",
        "    *   `A * B`\n",
        "    *   `A / B`\n",
        "    *   `A ** scalar`\n",
        "*   Broadcast operations\n",
        "    *   `sp_<op>_v()`\n",
        "*   Reduce operations\n",
        "    *   `reduce()`\n",
        "    *   `sum()`\n",
        "    *   `smax()`\n",
        "    *   `smin()`\n",
        "    *   `smean()`\n",
        "*   Matrix transformations\n",
        "    *   `SparseMatrix.transpose()` or `SparseMatrix.T`\n",
        "    *   `SparseMatrix.neg()`\n",
        "    *   `SparseMatrix.inv()`\n",
        "*   Matrix multiplication\n",
        "    *   `matmul()`\n",
        "    *   `sddmm()`\n",
        "\n",
        "\n",
        "*We are using dense format to print sparse matrix in this tutorial since it is more intuitive to read.*"
      ],
      "metadata": {
        "id": "Tjsapqp6zSFR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### *Elementwise operations*"
      ],
      "metadata": {
        "id": "psvGwcIqYvC2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**add(A, B), equivalent to A + B**\n",
        "\n",
        "Element-wise addition on two sparse matrices, returning a sparse matrix."
      ],
      "metadata": {
        "id": "39YJitpW-K9v"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3.])\n",
        "A1 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A1:\")\n",
        "print(A1.to_dense())\n",
        "\n",
        "i = torch.tensor([[0, 1, 2],\n",
        "                  [0, 2, 1]])\n",
        "val = torch.tensor([4., 5., 6.])\n",
        "A2 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A2:\")\n",
        "print(A2.to_dense())\n",
        "\n",
        "val = torch.tensor([-1., -2., -3.])\n",
        "D1 = dglsp.diag(val)\n",
        "print(\"D1:\")\n",
        "print(D1.to_dense())\n",
        "\n",
        "val = torch.tensor([-4., -5., -6.])\n",
        "D2 = dglsp.diag(val)\n",
        "print(\"D2:\")\n",
        "print(D2.to_dense())\n",
        "\n",
        "print(\"A1 + A2:\")\n",
        "print((A1 + A2).to_dense())\n",
        "\n",
        "print(\"A1 + D1:\")\n",
        "print((A1 + D1).to_dense())\n",
        "\n",
        "print(\"D1 + D2:\")\n",
        "print((D1 + D2).to_dense())"
      ],
      "metadata": {
        "id": "pj3Ckx41-BSu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**sub(A, B), equivalent to A - B**\n",
        "\n",
        "Element-wise substraction on two sparse matrices, returning a sparse matrix."
      ],
      "metadata": {
        "id": "i25N0JHUTUX9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3.])\n",
        "A1 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A1:\")\n",
        "print(A1.to_dense())\n",
        "\n",
        "i = torch.tensor([[0, 1, 2],\n",
        "                  [0, 2, 1]])\n",
        "val = torch.tensor([4., 5., 6.])\n",
        "A2 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A2:\")\n",
        "print(A2.to_dense())\n",
        "\n",
        "val = torch.tensor([-1., -2., -3.])\n",
        "D1 = dglsp.diag(val)\n",
        "print(\"D1:\")\n",
        "print(D1.to_dense())\n",
        "\n",
        "val = torch.tensor([-4., -5., -6.])\n",
        "D2 = dglsp.diag(val)\n",
        "print(\"D2:\")\n",
        "print(D2.to_dense())\n",
        "\n",
        "print(\"A1 - A2:\")\n",
        "print((A1 - A2).to_dense())\n",
        "\n",
        "print(\"A1 - D1:\")\n",
        "print((A1 - D1).to_dense())\n",
        "\n",
        "print(\"D1 - A1:\")\n",
        "print((D1 - A1).to_dense())\n",
        "\n",
        "print(\"D1 - D2:\")\n",
        "print((D1 - D2).to_dense())"
      ],
      "metadata": {
        "id": "GMxfz-cyT129"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**mul(A, B), equivalent to A * B**\n",
        "\n",
        "Element-wise multiplication on two sparse matrices or on a sparse matrix and a scalar, returning a sparse matrix."
      ],
      "metadata": {
        "id": "bg45jnq8T9EJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3.])\n",
        "A1 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A1:\")\n",
        "print(A1.to_dense())\n",
        "\n",
        "i = torch.tensor([[0, 1, 2, 2],\n",
        "                  [0, 2, 0, 1]])\n",
        "val = torch.tensor([1., 2., 3., 4.])\n",
        "A2 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "\n",
        "print(\"A2:\")\n",
        "print(A2.to_dense())\n",
        "\n",
        "print(\"A1 * 3:\")\n",
        "print((A1 * 3).to_dense())\n",
        "print(\"3 * A1:\")\n",
        "print((3 * A1).to_dense())\n",
        "\n",
        "print(\"A1 * A2\")\n",
        "print((A1 * A2).to_dense())\n",
        "\n",
        "val = torch.tensor([-1., -2., -3.])\n",
        "D1 = dglsp.diag(val)\n",
        "print(\"D1:\")\n",
        "print(D1.to_dense())\n",
        "\n",
        "print(\"D1 * A2\")\n",
        "print((D1 * A2).to_dense())\n",
        "\n",
        "val = torch.tensor([-4., -5., -6.])\n",
        "D2 = dglsp.diag(val)\n",
        "print(\"D2:\")\n",
        "print(D2.to_dense())\n",
        "\n",
        "print(\"D1 * -2:\")\n",
        "print((D1 * -2).to_dense())\n",
        "print(\"-2 * D1:\")\n",
        "print((-2 * D1).to_dense())\n",
        "\n",
        "print(\"D1 * D2:\")\n",
        "print((D1 * D2).to_dense())"
      ],
      "metadata": {
        "id": "4PAITJqHUB8J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**div(A, B), equivalent to A / B**\n",
        "\n",
        "Element-wise multiplication on two sparse matrices or on a sparse matrix and a scalar, returning a sparse matrix. If both `A` and `B` are sparse matrices, both of them must have the same sparsity. And the returned matrix has the same order of non-zero entries as `A`."
      ],
      "metadata": {
        "id": "Xb2RU6H4UBCs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3.])\n",
        "A1 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A1:\")\n",
        "print(A1.to_dense())\n",
        "\n",
        "i = torch.tensor([[1, 2, 1],\n",
        "                  [0, 0, 2]])\n",
        "val = torch.tensor([1., 3., 2.])\n",
        "A2 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "\n",
        "print(\"A1 / 2:\")\n",
        "print((A1 / 2).to_dense())\n",
        "\n",
        "print(\"A1 / A2\")\n",
        "print((A1 / A2).to_dense())\n",
        "\n",
        "val = torch.tensor([-1., -2., -3.])\n",
        "D1 = dglsp.diag(val)\n",
        "print(\"D1:\")\n",
        "print(D1.to_dense())\n",
        "\n",
        "val = torch.tensor([-4., -5., -6.])\n",
        "D2 = dglsp.diag(val)\n",
        "print(\"D2:\")\n",
        "print(D2.to_dense())\n",
        "\n",
        "print(\"D1 / D2:\")\n",
        "print((D1 / D2).to_dense())\n",
        "\n",
        "print(\"D1 / 2:\")\n",
        "print((D1 / 2).to_dense())"
      ],
      "metadata": {
        "id": "TFB_UcmEUdr3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**power(A, B), equivalent to A \\*\\* B**\n",
        "\n",
        "Element-wise power of a sparse matrix and a scalar, returning a sparse matrix."
      ],
      "metadata": {
        "id": "2lZbyTYUUgSi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3.])\n",
        "A = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A:\")\n",
        "print(A.to_dense())\n",
        "\n",
        "print(\"A ** 3:\")\n",
        "print((A ** 3).to_dense())\n",
        "\n",
        "val = torch.tensor([-1., -2., -3.])\n",
        "D = dglsp.diag(val)\n",
        "print(\"D:\")\n",
        "print(D.to_dense())\n",
        "\n",
        "print(\"D1 ** 2:\")\n",
        "print((D1 ** 2).to_dense())"
      ],
      "metadata": {
        "id": "ox-XxCnuUqAy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### *Broadcast operations*"
      ],
      "metadata": {
        "id": "VXBz4j5x_wQ4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**sp_\\<op\\>_v(A, v)**\n",
        "\n",
        "Broadcast operations on a sparse matrix and a vector, returning a sparse matrix. `v` is broadcasted to the shape of `A` and then the operator is applied on the non-zero values of `A`. `<op>` can be add, sub, mul, and div. \n",
        "\n",
        "There are two cases regarding the shape of `v`:\n",
        "\n",
        "1. `v` is a vector of shape `(1, A.shape[1])` or `(A.shape[1])`. In this case, `v` is broadcasted on the row dimension of `A`.\n",
        "\n",
        "2. `v` is a vector of shape `(A.shape[0], 1)`. In this case, `v` is broadcasted on the column dimension of `A`."
      ],
      "metadata": {
        "id": "PtnyZdXHAZ6Z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 0, 2], [0, 3, 2]])\n",
        "val = torch.tensor([10, 20, 30])\n",
        "A = dglsp.spmatrix(i, val, shape=(3, 4))\n",
        "\n",
        "v1 = torch.tensor([1, 2, 3, 4])\n",
        "print(\"A:\")\n",
        "print(A.to_dense())\n",
        "\n",
        "print(\"v1:\")\n",
        "print(v1)\n",
        "\n",
        "print(\"sp_add_v(A, v1)\")\n",
        "print(dglsp.sp_add_v(A, v1).to_dense())\n",
        "\n",
        "v2 = v1.reshape(1, -1)\n",
        "print(\"v2:\")\n",
        "print(v2)\n",
        "\n",
        "print(\"sp_add_v(A, v2)\")\n",
        "print(dglsp.sp_add_v(A, v2).to_dense())\n",
        "\n",
        "v3 = torch.tensor([1, 2, 3]).reshape(-1, 1)\n",
        "print(\"v3:\")\n",
        "print(v3)\n",
        "\n",
        "print(\"sp_add_v(A, v3)\")\n",
        "print(dglsp.sp_add_v(A, v3).to_dense())"
      ],
      "metadata": {
        "id": "xxf3s-uWBRR7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### *Reduce operations*\n",
        "\n",
        "All DGL sparse reduce operations only consider non-zero elements. To distinguish them from dense PyTorch reduce operations that consider zero elements, we use name `smax`, `smin` and `smean` (`s` stands for sparse)."
      ],
      "metadata": {
        "id": "TQJJlctZjYPv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[0, 1, 1, 2],\n",
        "                  [1, 0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3., 4.])\n",
        "A = dglsp.spmatrix(i, val)\n",
        "print(A.T.to_dense())\n",
        "print(\"\")\n",
        "\n",
        "# O1, O2 will have the same value.\n",
        "O1 = A.reduce(0, 'sum')\n",
        "O2 = A.sum(0)\n",
        "print(\"Reduce with reducer:sum along dim = 0:\")\n",
        "print(O1)\n",
        "print(\"\")\n",
        "\n",
        "# O3, O4 will have the same value.\n",
        "O3 = A.reduce(0, 'smax')\n",
        "O4 = A.smax(0)\n",
        "print(\"Reduce with reducer:max along dim = 0:\")\n",
        "print(O3)\n",
        "print(\"\")\n",
        "\n",
        "# O5, O6 will have the same value.\n",
        "O5 = A.reduce(0, 'smin')\n",
        "O6 = A.smin(0)\n",
        "print(\"Reduce with reducer:min along dim = 0:\")\n",
        "print(O5)\n",
        "print(\"\")\n",
        "\n",
        "# O7, O8 will have the same value.\n",
        "O7 = A.reduce(0, 'smean')\n",
        "O8 = A.smean(0)\n",
        "print(\"Reduce with reducer:smean along dim = 0:\")\n",
        "print(O7)\n",
        "print(\"\")"
      ],
      "metadata": {
        "id": "GhS49Js1jW4b"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### *Matrix transformations*"
      ],
      "metadata": {
        "id": "kanwnB7LOQui"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "*Sparse Matrix*"
      ],
      "metadata": {
        "id": "NiiXso9elM2p"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[0, 1, 1, 2],\n",
        "                  [1, 0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3., 4.])\n",
        "A = dglsp.spmatrix(i, val)\n",
        "print(A.to_dense())\n",
        "print(\"\")\n",
        "\n",
        "print(\"Get transpose of sparse matrix.\")\n",
        "print(A.T.to_dense())\n",
        "# Alias\n",
        "# A.transpose()\n",
        "# A.t()\n",
        "print(\"\")\n",
        "\n",
        "print(\"Get a sparse matrix with the negation of the original nonzero values.\")\n",
        "print(A.neg().to_dense())\n",
        "print(\"\")"
      ],
      "metadata": {
        "id": "qJcmZHmf-oTY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### *Matrix multiplication*"
      ],
      "metadata": {
        "id": "4uQlDFb0Uzto"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**matmul(A, B), equivalent to A @ B**\n",
        "\n",
        "Matrix multiplication on sparse matrices and/or dense matrix. There are two cases as follows."
      ],
      "metadata": {
        "id": "THWE30v6WpAk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**SparseMatrix @ SparseMatrix -> SparseMatrix:**\n",
        "\n",
        "For a $L \\times M$ sparse matrix A and a $M \\times N$ sparse matrix B, the shape of `A @ B` will be $L \\times N$ sparse matrix."
      ],
      "metadata": {
        "id": "VxyykR-vX7lF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3.])\n",
        "A1 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A1:\")\n",
        "print(A1.to_dense())\n",
        "\n",
        "i = torch.tensor([[0, 1, 2],\n",
        "                  [0, 2, 1]])\n",
        "val = torch.tensor([4., 5., 6.])\n",
        "A2 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A2:\")\n",
        "print(A2.to_dense())\n",
        "\n",
        "val = torch.tensor([-1., -2., -3.])\n",
        "D1 = dglsp.diag(val)\n",
        "print(\"D1:\")\n",
        "print(D1.to_dense())\n",
        "\n",
        "val = torch.tensor([-4., -5., -6.])\n",
        "D2 = dglsp.diag(val)\n",
        "print(\"D2:\")\n",
        "print(D2.to_dense())\n",
        "\n",
        "print(\"A1 @ A2:\")\n",
        "print((A1 @ A2).to_dense())\n",
        "\n",
        "print(\"A1 @ D1:\")\n",
        "print((A1 @ D1).to_dense())\n",
        "\n",
        "print(\"D1 @ A1:\")\n",
        "print((D1 @ A1).to_dense())\n",
        "\n",
        "print(\"D1 @ D2:\")\n",
        "print((D1 @ D2).to_dense())"
      ],
      "metadata": {
        "id": "XRDFC2rOYQM4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**SparseMatrix @ Tensor -> Tensor:**\n",
        "\n",
        "For a $L \\times M$ sparse matrix A and a $M \\times N$ dense matrix B, the shape of `A @ B` will be $L \\times N$ dense matrix."
      ],
      "metadata": {
        "id": "g13fG8nvaVOt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3.])\n",
        "A = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A:\")\n",
        "print(A.to_dense())\n",
        "\n",
        "val = torch.tensor([-1., -2., -3.])\n",
        "D = dglsp.diag(val)\n",
        "print(\"D:\")\n",
        "print(D.to_dense())\n",
        "\n",
        "X = torch.tensor([[11., 22.], [33., 44.], [55., 66.]])\n",
        "print(\"X:\")\n",
        "print(X)\n",
        "\n",
        "print(\"A @ X:\")\n",
        "print(A @ X)\n",
        "\n",
        "print(\"D @ X:\")\n",
        "print(D @ X)"
      ],
      "metadata": {
        "id": "FcQ-CnqdlgWF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This operator also supports batched sparse-dense matrix multiplication. The sparse matrix A should have shape $L \\times M$, where the non-zero values are vectors of length $K$. The dense matrix B should have shape $M \\times N \\times K$. The output is a dense matrix of shape $L \\times N \\times K$."
      ],
      "metadata": {
        "id": "_KZiULLbmEZE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [0, 2, 0]])\n",
        "val = torch.tensor([[1., 1.], [2., 2.], [3., 3.]])\n",
        "A = dglsp.spmatrix(i, val, shape=(3, 3))\n",
        "print(\"A:\")\n",
        "print(A.to_dense())\n",
        "\n",
        "X = torch.tensor([[[1., 1.], [1., 2.]],\n",
        "                  [[1., 3.], [1., 4.]],\n",
        "                  [[1., 5.], [1., 6.]]])\n",
        "print(\"X:\")\n",
        "print(X)\n",
        "\n",
        "print(\"A @ X:\")\n",
        "print(A @ X)"
      ],
      "metadata": {
        "id": "ZUzXQk7Ab2wG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Sampled-Dense-Dense Matrix Multiplication (SDDMM)**\n",
        "\n",
        "``sddmm`` matrix-multiplies two dense matrices X1 and X2, then elementwise-multiplies the result with sparse matrix A at the nonzero locations. This is designed for sparse matrix with scalar values.\n",
        "\n",
        "$$out = (X_1 @ X_2) * A$$\n",
        "\n",
        "For a $L \\times N$ sparse matrix A, a $L \\times M$ dense matrix X1 and a $M \\times N$ dense matrix X2, `sddmm(A, X1, X2)` will be a $L \\times N$ sparse matrix."
      ],
      "metadata": {
        "id": "qO_8f_vhPKtf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [2, 3, 3]])\n",
        "val = torch.tensor([1., 2., 3.])\n",
        "A = dglsp.spmatrix(i, val, (3, 4))\n",
        "print(\"A:\")\n",
        "print(A.to_dense())\n",
        "\n",
        "X1 = torch.randn(3, 5)\n",
        "X2 = torch.randn(5, 4)\n",
        "print(\"X1:\")\n",
        "print(X1)\n",
        "print(\"X2:\")\n",
        "print(X2)\n",
        "\n",
        "O = dglsp.sddmm(A, X1, X2)\n",
        "print(\"dglsp.sddmm(A, X1, X2):\")\n",
        "print(O.to_dense())"
      ],
      "metadata": {
        "id": "3ZIFV0TgPhwH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This operator also supports batched sampled-dense-dense matrix multiplication. For a $L \\times N$ sparse matrix A with non-zero vector values of length $𝐾$, a $L \\times M \\times K$ dense matrix X1 and a $M \\times N \\times K$ dense matrix X2, `sddmm(A, X1, X2)` will be a $L \\times N \\times K$ sparse matrix."
      ],
      "metadata": {
        "id": "RmNmXU_ZqyF7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[1, 1, 2],\n",
        "                  [2, 3, 3]])\n",
        "val = torch.tensor([[1., 1.], [2., 2.], [3., 3.]])\n",
        "A = dglsp.spmatrix(i, val, (3, 4))\n",
        "print(\"A:\")\n",
        "print(A.to_dense())\n",
        "\n",
        "X1 = torch.randn(3, 5, 2)\n",
        "X2 = torch.randn(5, 4, 2)\n",
        "print(\"X1:\")\n",
        "print(X1)\n",
        "print(\"X2:\")\n",
        "print(X2)\n",
        "\n",
        "O = dglsp.sddmm(A, X1, X2)\n",
        "print(\"dglsp.sddmm(A, X1, X2):\")\n",
        "print(O.to_dense())"
      ],
      "metadata": {
        "id": "DuSAjamyrIO_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Non-linear activation functions"
      ],
      "metadata": {
        "id": "fVkbTT28ZzPr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Element-wise functions\n",
        "\n",
        "Most activation functions are element-wise and can be further grouped into two categories:\n",
        "\n",
        "**Sparse-preserving functions** such as `sin()`, `tanh()`, `sigmoid()`, `relu()`, etc. You can directly apply them on the `val` tensor of the sparse matrix and then recreate a new matrix of the same sparsity using `val_like`."
      ],
      "metadata": {
        "id": "XuaNdFO7XG2r"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[0, 1, 1, 2],\n",
        "                  [1, 0, 2, 0]])\n",
        "val = torch.randn(4)\n",
        "A = dglsp.spmatrix(i, val)\n",
        "print(A.to_dense())\n",
        "\n",
        "print(\"Apply tanh.\")\n",
        "A_new = dglsp.val_like(A, torch.tanh(A.val))\n",
        "print(A_new.to_dense())"
      ],
      "metadata": {
        "id": "GZkCJJ0TX0cI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Non-sparse-preserving functions** such as `exp()`, `cos()`, etc. You can first convert the sparse matrix to dense before applying the functions."
      ],
      "metadata": {
        "id": "i92lhMEnYas3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[0, 1, 1, 2],\n",
        "                  [1, 0, 2, 0]])\n",
        "val = torch.randn(4)\n",
        "A = dglsp.spmatrix(i, val)\n",
        "print(A.to_dense())\n",
        "\n",
        "print(\"Apply exp.\")\n",
        "A_new = A.to_dense().exp()\n",
        "print(A_new)"
      ],
      "metadata": {
        "id": "sroJpzRNYZq5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Softmax\n",
        "\n",
        "Apply row-wise softmax to the nonzero entries of the sparse matrix."
      ],
      "metadata": {
        "id": "y8OQZReVXpo3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[0, 1, 1, 2],\n",
        "                  [1, 0, 2, 0]])\n",
        "val = torch.tensor([1., 2., 3., 4.])\n",
        "A = dglsp.spmatrix(i, val)\n",
        "\n",
        "print(A.softmax())\n",
        "print(\"In dense format:\")\n",
        "print(A.softmax().to_dense())\n",
        "print(\"\\n\")"
      ],
      "metadata": {
        "id": "CQaKgzCJULjt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercise \\#1\n",
        "\n",
        "*Let's test what you've learned. Feel free to [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/sparse/quickstart.ipynb).*\n",
        "\n",
        "Given a sparse symmetrical adjacency matrix $A$, calculate its symmetrically normalized adjacency matrix: $$norm = \\bar{D}^{-\\frac{1}{2}}\\bar{A}\\bar{D}^{-\\frac{1}{2}}$$\n",
        "\n",
        "Where $\\bar{A} = A + I$, $I$ is the identity matrix, and $\\bar{D}$ is the diagonal node degree matrix of $\\bar{A}$."
      ],
      "metadata": {
        "id": "1iBNlJVYz3zi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = torch.tensor([[0, 0, 1, 1, 2, 2, 3],\n",
        "                  [1, 3, 2, 5, 3, 5, 4]])\n",
        "asym_A = dglsp.spmatrix(i, shape=(6, 6))\n",
        "# Step 1: create symmetrical adjacency matrix A from asym_A.\n",
        "# A =\n",
        "\n",
        "# Step 2: calculate A_hat from A.\n",
        "# A_hat =\n",
        "\n",
        "# Step 3: diagonal node degree matrix of A_hat\n",
        "# D_hat =\n",
        "\n",
        "# Step 4: calculate the norm from D_hat and A_hat.\n",
        "# norm = "
      ],
      "metadata": {
        "id": "0dDhfbJo0ByV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercise \\#2\n",
        "\n",
        "Let's implement a simplified version of the Graph Attention Network (GAT) layer.\n",
        "\n",
        "A GAT layer has two inputs: the adjacency matrix $A$ and the node input features $X$.  The idea of GAT layer is to update each node's representation with a weighted average of the node's own representation and its neighbors' representations.  In particular, when computing the output for node $i$, the GAT layer does the following:\n",
        "1. Compute the scores $S_{ij}$ representing the attention logit from neighbor $j$ to node $i$.  $S_{ij}$ is a function of $i$ and $j$'s input features $X_i$ and $X_j$: $$S_{ij} = LeakyReLU(X_i^\\top v_1 + X_j^\\top v_2)$$, where $v_1$ and $v_2$ are trainable vectors.\n",
        "2. Compute a softmax attention $R_{ij} = \\exp S_{ij} / \\left( \\sum_{j' \\in \\mathcal{N}_i} s_{ij'} \\right)$, where $\\mathcal{N}_j$ means the neighbors of $j$.  This means that $R$ is a row-wise softmax attention of $S$.\n",
        "3. Compute the weighted average $H_i = \\sum_{j' : j' \\in \\mathcal{N}_i} R_{j'} X_{j'} W$, where $W$ is a trainable matrix.\n",
        "\n",
        "The following code defined all the parameters you need but only completes step 1.  Could you implement step 2 and step 3?"
      ],
      "metadata": {
        "id": "yfEVQBUuI-cE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "class SimplifiedGAT(nn.Module):\n",
        "    def __init__(self, in_size, out_size):\n",
        "        super().__init__()\n",
        "\n",
        "        self.W = nn.Parameter(torch.randn(in_size, out_size))\n",
        "        self.v1 = nn.Parameter(torch.randn(in_size))\n",
        "        self.v2 = nn.Parameter(torch.randn(in_size))\n",
        "\n",
        "    def forward(self, A, X):\n",
        "        # A: A sparse matrix with size (N, N).  A[i, j] represent the edge from j to i.\n",
        "        # X: A dense matrix with size (N, D)\n",
        "        # Step 1: compute S[i, j]\n",
        "        Xv1 = X @ self.v1\n",
        "        Xv2 = X @ self.v2\n",
        "        s = F.leaky_relu(Xv1[A.col] + Xv2[A.row])\n",
        "        S = dglsp.val_like(A, s)\n",
        "\n",
        "        # Step 2: compute R[i, j] which is the row-wise attention of $S$.\n",
        "        # EXERCISE: replace the statement below.\n",
        "        R = S\n",
        "\n",
        "        # Step 3: compute H.\n",
        "        # EXERCISE: replace the statement below.\n",
        "        H = X\n",
        "\n",
        "        return H"
      ],
      "metadata": {
        "id": "pYrgSxq6La5c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Test:\n",
        "# Let's use the symmetric A created above.\n",
        "X = torch.randn(6, 20)\n",
        "module = SimplifiedGAT(20, 10)\n",
        "Y = module(A, X)"
      ],
      "metadata": {
        "id": "qjcXiidYCqGK"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}