[Doc] sparse lib tutorial and colab links (#5181)

* Created using Colaboratory

* nbsphinx & nblink

* Created using Colaboratory

* update notebook

* minor update

docs/source/conf.py

@@ -48,6 +48,8 @@ extensions = [
     'sphinx.ext.graphviz',
     'sphinx_gallery.gen_gallery',
     'sphinx_copybutton',
+    'nbsphinx',
+    'nbsphinx_link',
 ]
 
 # Add any paths that contain templates here, relative to this directory.

docs/source/index.rst

@@ -25,6 +25,7 @@ Welcome to Deep Graph Library Tutorials and Documentation
    guide/index
    guide_cn/index
    guide_ko/index
+   notebooks/sparse/index
    tutorials/large/index
    tutorials/cpu/index
    tutorials/multi/index
@@ -51,10 +52,10 @@ Welcome to Deep Graph Library Tutorials and Documentation
    api/python/dgl.ops
    api/python/dgl.optim
    api/python/dgl.sampling
+   api/python/dgl.sparse_v0
    api/python/dgl.multiprocessing
    api/python/transforms
    api/python/udf
-   api/python/dgl.sparse_v0
 
 .. toctree::
    :maxdepth: 1

docs/source/notebooks/sparse/index.rst

+🆕 Tutorials: dgl.sparse
+========================
+
+TODO(minjie): intro for the new library.
+
+.. toctree::
+  :maxdepth: 2
+  :titlesonly:
+
+  quickstart.nblink

docs/source/notebooks/sparse/quickstart.nblink

+{
+    "path": "../../../../notebooks/sparse/quickstart.ipynb"
+}

notebooks/sparse/quickstart.ipynb

+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "private_outputs": true,
+      "toc_visible": true,
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "gpuClass": "standard",
+    "accelerator": "GPU"
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Quickstart\n",
+        "\n",
+        "The tutorial provides a quick walkthrough of the classes and operators provided by the `dgl.sparse` package.\n",
+        "\n",
+        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/sparse/quickstart.ipynb) [![GitHub](https://img.shields.io/badge/-View%20on%20GitHub-181717?logo=github&logoColor=ffffff)](https://github.com/dmlc/dgl/blob/master/notebooks/sparse/quickstart.ipynb)"
+      ],
+      "metadata": {
+        "id": "E0DAKDMuWz7I"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Uncomment below to install the required packages.\n",
+        "\n",
+        "import os\n",
+        "import torch\n",
+        "os.environ['TORCH'] = torch.__version__\n",
+        "os.environ['DGLBACKEND'] = \"pytorch\"\n",
+        "\n",
+        "# TODO(Steve): change to stable version.\n",
+        "!pip install --pre dgl -I https://data.dgl.ai/wheels-test/repo.html > /dev/null\n",
+        "\n",
+        "try:\n",
+        "    import dgl.sparse as dglsp\n",
+        "    installed = True\n",
+        "except ImportError:\n",
+        "    installed = False\n",
+        "print(\"DGL installed!\" if installed else \"Failed to install DGL!\")"
+      ],
+      "metadata": {
+        "id": "19UZd7wyWzpT"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Sparse Matrix\n",
+        "\n",
+        "The core abstraction of DGL's sparse package is the `SparseMatrix` class. Compared with other sparse matrix libraries (such as `scipy.sparse` and `torch.sparse`), DGL's `SparseMatrix` is specialized for the deep learning workloads on structure data (e.g., Graph Neural Networks), with the following features:\n",
+        "\n",
+        "* **Auto sparse format.** Don't bother choosing between different sparse formats. There is only one `SparseMatrix` and it will select the best format for the operation to be performed.\n",
+        "* **Non-zero elements can be scalar or vector.** Easy for modeling relations (e.g., edges) by vector representation.\n",
+        "* **Fully PyTorch compatible.** The package is built upon PyTorch and is natively compatible with other tools in the PyTorch ecosystem.\n"
+      ],
+      "metadata": {
+        "id": "GsWoAGC4RpHw"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Initializing a DGL Sparse Matrix\n",
+        "\n",
+        "A DGL Sparse Matrix can be initialized in various ways.\n",
+        "\n",
+        "*   `from_coo()`\n",
+        "*   `from_csr()`\n",
+        "*   `from_csc()`\n",
+        "\n",
+        "Take a look at the following examples:\n",
+        "\n",
+        "**From [COO format](https://en.wikipedia.org/wiki/Sparse_matrix#Coordinate_list_(COO))**"
+      ],
+      "metadata": {
+        "id": "_q4HYodcWenB"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "h-ryVEs1PuIP"
+      },
+      "outputs": [],
+      "source": [
+        "import torch\n",
+        "import dgl.sparse as dglsp\n",
+        "\n",
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "A = dglsp.from_coo(row, col)  # 1.0 is default value for nnz elements.\n",
+        "print(A)\n",
+        "print(\"\")\n",
+        "print(\"In dense format:\")\n",
+        "print(A.to_dense())"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "*Compare implicit shape vs explicit shape*"
+      ],
+      "metadata": {
+        "id": "W1JJg-eZ7K3t"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([0, 0, 1])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "\n",
+        "A1 = dglsp.from_coo(row, col)\n",
+        "print(f\"Implicit Shape: {A1.shape}\")\n",
+        "print(A1.to_dense())\n",
+        "print(\"\")\n",
+        "\n",
+        "A2 = dglsp.from_coo(row, col, shape=(3, 3))\n",
+        "print(f\"Explicit Shape: {A2.shape}\")\n",
+        "print(A2.to_dense())"
+      ],
+      "metadata": {
+        "id": "80NNSQfd7L5V"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Both scalar values and vector values can be set for nnz elements in Sparse Matrix."
+      ],
+      "metadata": {
+        "id": "zdNgUf0ShfCe"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "# The length of the value should match the nnz elements represented by the\n",
+        "# sparse matrix format.\n",
+        "scalar_val = torch.tensor([1., 2., 3.])\n",
+        "vector_val = torch.tensor([[1., 1.], [2., 2.], [3., 3.]])\n",
+        "\n",
+        "print(\"-----Scalar Values-----\")\n",
+        "A = dglsp.from_coo(row, col, scalar_val)\n",
+        "print(A)\n",
+        "print(\"\")\n",
+        "print(\"In dense format:\")\n",
+        "print(A.to_dense())\n",
+        "print(\"\")\n",
+        "\n",
+        "print(\"-----Vector Values-----\")\n",
+        "A = dglsp.from_coo(row, col, vector_val)\n",
+        "print(A)\n",
+        "print(\"\")\n",
+        "print(\"In dense format:\")\n",
+        "print(A.to_dense())"
+      ],
+      "metadata": {
+        "id": "buE9ZkKvhp1f"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "*Duplicated indices*"
+      ],
+      "metadata": {
+        "id": "7ufTCDAVsrmP"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([0, 0, 0, 1])\n",
+        "col = torch.tensor([0, 2, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3., 4])\n",
+        "A = dglsp.from_coo(row, col, val)\n",
+        "print(A)\n",
+        "print(f\"Whether A contains duplicate indices: {A.has_duplicate()}\")\n",
+        "print(\"\")\n",
+        "\n",
+        "B = A.coalesce()\n",
+        "print(B)\n",
+        "print(f\"Whether B contains duplicate indices: {B.has_duplicate()}\")"
+      ],
+      "metadata": {
+        "id": "ilSAlFLOs0o8"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**From [CSR format](https://en.wikipedia.org/wiki/Sparse_matrix#Compressed_sparse_row_(CSR,_CRS_or_Yale_format)) and [CSC format](https://en.wikipedia.org/wiki/Sparse_matrix#Compressed_sparse_column_(CSC_or_CCS))**"
+      ],
+      "metadata": {
+        "id": "XfnW7kGnd1lF"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "indptr = torch.tensor([0, 1, 2, 5])\n",
+        "indices = torch.tensor([1, 2, 0, 1, 2])\n",
+        "\n",
+        "print(\"-----Create from CSR format-----\")\n",
+        "A = dglsp.from_csr(indptr, indices)\n",
+        "print(A)\n",
+        "print(\"\")\n",
+        "print(\"In dense format:\")\n",
+        "print(A.to_dense())\n",
+        "print(\"\")\n",
+        "\n",
+        "print(\"-----Create from CSC format-----\")\n",
+        "B = dglsp.from_csc(indptr, indices)\n",
+        "print(B)\n",
+        "print(\"\")\n",
+        "print(\"In dense format:\")\n",
+        "print(B.to_dense())"
+      ],
+      "metadata": {
+        "id": "3puXyMFsvdlj"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**val_like**\n",
+        "\n",
+        "Similar to pytorch, we can create a Sparse Matrix with new values, the same nonzero indices as the given sparse matrix."
+      ],
+      "metadata": {
+        "id": "ZJ09qM5NaxuI"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3.])\n",
+        "A = dglsp.from_coo(row, col, val)\n",
+        "\n",
+        "new_val = torch.tensor([4., 5., 6.])\n",
+        "B = dglsp.val_like(A, new_val)\n",
+        "print(B)"
+      ],
+      "metadata": {
+        "id": "UB3lKJVBbsUD"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Attributes and methods of a DGL Sparse Matrix"
+      ],
+      "metadata": {
+        "id": "nd4hJ9ysd4St"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([0, 1, 1, 2])\n",
+        "col = torch.tensor([1, 0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3., 4.])\n",
+        "A = dglsp.from_coo(row, col, val)\n",
+        "\n",
+        "print(f\"Shape of sparse matrix: {A.shape}\")\n",
+        "print(f\"The number of nonzero elements of sparse matrix: {A.nnz}\")\n",
+        "print(f\"Datatype of sparse matrix: {A.dtype}\")\n",
+        "print(f\"Device sparse matrix is stored on: {A.device}\")\n",
+        "print(f\"Get the values of the nonzero elements: {A.val}\")\n",
+        "print(f\"Get the row indices of the nonzero elements: {A.row}\")\n",
+        "print(f\"Get the column indices of the nonzero elements: {A.col}\")\n",
+        "print(f\"Get the coordinate (COO) representation: {A.coo()}\")\n",
+        "print(f\"Get the compressed sparse row (CSR) representation: {A.csr()}\")\n",
+        "print(f\"Get the compressed sparse column (CSC) representation: {A.csc()}\")"
+      ],
+      "metadata": {
+        "id": "OKbFiWKIzZVe"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**dtype and/or device conversion**"
+      ],
+      "metadata": {
+        "id": "VzosM7i3yQPK"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([0, 1, 1, 2])\n",
+        "col = torch.tensor([1, 0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3., 4.])\n",
+        "A = dglsp.from_coo(row, col, val)\n",
+        "\n",
+        "B = A.to(device='cuda:0', dtype=torch.int32)\n",
+        "print(f\"Device sparse matrix is stored on: {B.device}\")\n",
+        "print(f\"Datatype of sparse matrix: {B.dtype}\")"
+      ],
+      "metadata": {
+        "id": "y_RJihw-ypXp"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Similar to pytorch, we also provide various fine-grained APIs ([Doc](https://docs.dgl.ai/en/latest/api/python/dgl.sparse_v0.html)) for dtype and/or device conversion."
+      ],
+      "metadata": {
+        "id": "U26arLlJzfkN"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Diagonal Matrix\n",
+        "\n",
+        "Diagonal Matrix is a special type of Sparse Matrix, in which the entries outside the main diagonal are all zero.\n",
+        "\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "EFe9ABRuWHqf"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Initializing a DGL Diagonal Matrix\n",
+        "A DGL Diagonal Matrix can be initiate by `dglsp.diag()`.\n",
+        "\n",
+        "Identity Matrix is a special type of Diagonal Matrix, in which all the value on the diagonal are 1.0. Use `dglsp.identity()` to initiate a Diagonal Matrix."
+      ],
+      "metadata": {
+        "id": "1CeCoE2Fgl_x"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "val = torch.tensor([1., 2., 3., 4.])\n",
+        "D = dglsp.diag(val)\n",
+        "print(D)\n",
+        "\n",
+        "I = dglsp.identity(shape=(3, 3))\n",
+        "print(I)"
+      ],
+      "metadata": {
+        "id": "9wzJNApahXAR"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Attributes and methods of a DGL Diagonal Matrix"
+      ],
+      "metadata": {
+        "id": "s-JpSHGLhWlm"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "val = torch.tensor([1., 2., 3., 4.])\n",
+        "D = dglsp.diag(val)\n",
+        "\n",
+        "print(f\"Shape of sparse matrix: {D.shape}\")\n",
+        "print(f\"The number of nonzero elements of sparse matrix: {D.nnz}\")\n",
+        "print(f\"Datatype of sparse matrix: {D.dtype}\")\n",
+        "print(f\"Device sparse matrix is stored on: {D.device}\")\n",
+        "print(f\"Get the values of the nonzero elements: {D.val}\")"
+      ],
+      "metadata": {
+        "id": "QMV0u-kQWsWd"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Operations on Sparse Matrix and Diagonal Matrix\n",
+        "*   Elementwise operations\n",
+        "    *   `A + B`\n",
+        "    *   `A - B`\n",
+        "    *   `A * B`\n",
+        "    *   `A / B`\n",
+        "    *   `A ** scalar`\n",
+        "*   Reduce operations\n",
+        "    *   `reduce()`\n",
+        "    *   `sum()`\n",
+        "    *   `smax()`\n",
+        "    *   `smin()`\n",
+        "    *   `smean()`\n",
+        "*   Matrix transformations\n",
+        "    *   `SparseMatrix.transpose()` or `SparseMatrix.T`\n",
+        "    *   `SparseMatrix.neg()`\n",
+        "    *   `DiagMatrix.transpose()` or `DiagMatrix.T`\n",
+        "    *   `DiagMatrix.neg()`\n",
+        "    *   `DiagMatrix.inv()`\n",
+        "*   Matrix multiplication\n",
+        "    *   `matmul()`\n",
+        "    *   `sddmm()`\n",
+        "\n",
+        "\n",
+        "*We are using dense format to print sparse matrix in this tutorial since it is more intuitive to read.*"
+      ],
+      "metadata": {
+        "id": "Tjsapqp6zSFR"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### *Elementwise operations*"
+      ],
+      "metadata": {
+        "id": "psvGwcIqYvC2"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**add(A, B), equivalent to A + B**\n",
+        "\n",
+        "The supported combinations are shown as follows.\n",
+        "\n",
+        "A \\ B           | **DiagMatrix**|**SparseMatrix**|**scalar**\n",
+        "----------------|---------------|----------------|----------\n",
+        "**DiagMatrix**  |✅             |✅              |🚫\n",
+        "**SparseMatrix**|✅             |✅              |🚫\n",
+        "**scalar**      |🚫             |🚫              |🚫"
+      ],
+      "metadata": {
+        "id": "39YJitpW-K9v"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3.])\n",
+        "A1 = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A1:\")\n",
+        "print(A1.to_dense())\n",
+        "\n",
+        "row = torch.tensor([0, 1, 2])\n",
+        "col = torch.tensor([0, 2, 1])\n",
+        "val = torch.tensor([4., 5., 6.])\n",
+        "A2 = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A2:\")\n",
+        "print(A2.to_dense())\n",
+        "\n",
+        "val = torch.tensor([-1., -2., -3.])\n",
+        "D1 = dglsp.diag(val)\n",
+        "print(\"D1:\")\n",
+        "print(D1.to_dense())\n",
+        "\n",
+        "val = torch.tensor([-4., -5., -6.])\n",
+        "D2 = dglsp.diag(val)\n",
+        "print(\"D2:\")\n",
+        "print(D2.to_dense())\n",
+        "\n",
+        "print(\"A1 + A2:\")\n",
+        "print((A1 + A2).to_dense())\n",
+        "\n",
+        "print(\"A1 + D1:\")\n",
+        "print((A1 + D1).to_dense())\n",
+        "\n",
+        "print(\"D1 + D2:\")\n",
+        "print((D1 + D2).to_dense())"
+      ],
+      "metadata": {
+        "id": "pj3Ckx41-BSu"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**sub(A, B) equivalent to A - B**\n",
+        "\n",
+        "The supported combinations are shown as follows.\n",
+        "\n",
+        "A \\ B           | **DiagMatrix**|**SparseMatrix**|**scalar**\n",
+        "----------------|---------------|----------------|----------\n",
+        "**DiagMatrix**  |✅             |🚫              |🚫\n",
+        "**SparseMatrix**|🚫             |🚫              |🚫\n",
+        "**scalar**      |🚫             |🚫              |🚫"
+      ],
+      "metadata": {
+        "id": "i25N0JHUTUX9"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3.])\n",
+        "A1 = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A1:\")\n",
+        "print(A1.to_dense())\n",
+        "\n",
+        "row = torch.tensor([0, 1, 2])\n",
+        "col = torch.tensor([0, 2, 1])\n",
+        "val = torch.tensor([4., 5., 6.])\n",
+        "A2 = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A2:\")\n",
+        "print(A2.to_dense())\n",
+        "\n",
+        "val = torch.tensor([-1., -2., -3.])\n",
+        "D1 = dglsp.diag(val)\n",
+        "print(\"D1:\")\n",
+        "print(D1.to_dense())\n",
+        "\n",
+        "val = torch.tensor([-4., -5., -6.])\n",
+        "D2 = dglsp.diag(val)\n",
+        "print(\"D2:\")\n",
+        "print(D2.to_dense())\n",
+        "\n",
+        "print(\"A1 - A2:\")\n",
+        "print((A1 - A2).to_dense())\n",
+        "\n",
+        "print(\"A1 - D1:\")\n",
+        "print((A1 - D1).to_dense())\n",
+        "\n",
+        "print(\"D1 - A1:\")\n",
+        "print((D1 - A1).to_dense())\n",
+        "\n",
+        "print(\"D1 - D2:\")\n",
+        "print((D1 - D2).to_dense())"
+      ],
+      "metadata": {
+        "id": "GMxfz-cyT129"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**mul(A, B), equivalent to A * B**\n",
+        "\n",
+        "The supported combinations are shown as follows.\n",
+        "\n",
+        "A \\ B           | **DiagMatrix**|**SparseMatrix**|**scalar**\n",
+        "----------------|---------------|----------------|----------\n",
+        "**DiagMatrix**  |✅             |🚫              |✅\n",
+        "**SparseMatrix**|🚫             |🚫              |✅\n",
+        "**scalar**      |✅             |✅              |🚫"
+      ],
+      "metadata": {
+        "id": "bg45jnq8T9EJ"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3.])\n",
+        "A = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A:\")\n",
+        "print(A.to_dense())\n",
+        "\n",
+        "print(\"A * 3:\")\n",
+        "print((A * 3).to_dense())\n",
+        "print(\"3 * A:\")\n",
+        "print((3 * A).to_dense())\n",
+        "\n",
+        "val = torch.tensor([-1., -2., -3.])\n",
+        "D1 = dglsp.diag(val)\n",
+        "print(\"D1:\")\n",
+        "print(D1.to_dense())\n",
+        "\n",
+        "val = torch.tensor([-4., -5., -6.])\n",
+        "D2 = dglsp.diag(val)\n",
+        "print(\"D2:\")\n",
+        "print(D2.to_dense())\n",
+        "\n",
+        "print(\"D1 * -2:\")\n",
+        "print((D1 * -2).to_dense())\n",
+        "print(\"-2 * D1:\")\n",
+        "print((-2 * D1).to_dense())\n",
+        "\n",
+        "print(\"D1 * D2:\")\n",
+        "print((D1 * D2).to_dense())"
+      ],
+      "metadata": {
+        "id": "4PAITJqHUB8J"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**div(A, B), equivalent to A / B**\n",
+        "\n",
+        "The supported combinations are shown as follows.\n",
+        "\n",
+        "A \\ B           | **DiagMatrix**|**SparseMatrix**|**scalar**\n",
+        "----------------|---------------|----------------|----------\n",
+        "**DiagMatrix**  |✅             |🚫              |✅\n",
+        "**SparseMatrix**|🚫             |🚫              |✅\n",
+        "**scalar**      |🚫             |🚫              |🚫"
+      ],
+      "metadata": {
+        "id": "Xb2RU6H4UBCs"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3.])\n",
+        "A = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A:\")\n",
+        "print(A.to_dense())\n",
+        "\n",
+        "print(\"A / 2:\")\n",
+        "print((A / 2).to_dense())\n",
+        "\n",
+        "val = torch.tensor([-1., -2., -3.])\n",
+        "D1 = dglsp.diag(val)\n",
+        "print(\"D1:\")\n",
+        "print(D1.to_dense())\n",
+        "\n",
+        "val = torch.tensor([-4., -5., -6.])\n",
+        "D2 = dglsp.diag(val)\n",
+        "print(\"D2:\")\n",
+        "print(D2.to_dense())\n",
+        "\n",
+        "print(\"D1 / D2:\")\n",
+        "print((D1 / D2).to_dense())\n",
+        "\n",
+        "print(\"D1 / 2:\")\n",
+        "print((D1 / 2).to_dense())"
+      ],
+      "metadata": {
+        "id": "TFB_UcmEUdr3"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**power(A, B), equivalent to A \\*\\* B**\n",
+        "\n",
+        "The supported combinations are shown as follows.\n",
+        "\n",
+        "A \\ B           | **DiagMatrix**|**SparseMatrix**|**scalar**\n",
+        "----------------|---------------|----------------|----------\n",
+        "**DiagMatrix**  |🚫             |🚫              |✅\n",
+        "**SparseMatrix**|🚫             |🚫              |✅\n",
+        "**scalar**      |🚫             |🚫              |🚫"
+      ],
+      "metadata": {
+        "id": "2lZbyTYUUgSi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3.])\n",
+        "A = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A:\")\n",
+        "print(A.to_dense())\n",
+        "\n",
+        "print(\"A ** 3:\")\n",
+        "print((A ** 3).to_dense())\n",
+        "\n",
+        "val = torch.tensor([-1., -2., -3.])\n",
+        "D = dglsp.diag(val)\n",
+        "print(\"D:\")\n",
+        "print(D.to_dense())\n",
+        "\n",
+        "print(\"D1 ** 2:\")\n",
+        "print((D1 ** 2).to_dense())"
+      ],
+      "metadata": {
+        "id": "ox-XxCnuUqAy"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### *Reduce operations*\n",
+        "\n",
+        "All DGL sparse reduce operations only consider non-zero elements. To distinguish them from dense PyTorch reduce operations that consider zero elements, we use name `smax`, `smin` and `smean` (`s` stands for sparse)."
+      ],
+      "metadata": {
+        "id": "TQJJlctZjYPv"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([0, 1, 1, 2])\n",
+        "col = torch.tensor([1, 0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3., 4.])\n",
+        "A = dglsp.from_coo(row, col, val)\n",
+        "print(A.T.to_dense())\n",
+        "print(\"\")\n",
+        "\n",
+        "# O1, O2 will have the same value.\n",
+        "O1 = A.reduce(0, 'sum')\n",
+        "O2 = A.sum(0)\n",
+        "print(\"Reduce with reducer:sum along dim = 0:\")\n",
+        "print(O1)\n",
+        "print(\"\")\n",
+        "\n",
+        "# O3, O4 will have the same value.\n",
+        "O3 = A.reduce(0, 'smax')\n",
+        "O4 = A.smax(0)\n",
+        "print(\"Reduce with reducer:max along dim = 0:\")\n",
+        "print(O3)\n",
+        "print(\"\")\n",
+        "\n",
+        "# O5, O6 will have the same value.\n",
+        "O5 = A.reduce(0, 'smin')\n",
+        "O6 = A.smin(0)\n",
+        "print(\"Reduce with reducer:min along dim = 0:\")\n",
+        "print(O5)\n",
+        "print(\"\")\n",
+        "\n",
+        "# O7, O8 will have the same value.\n",
+        "O7 = A.reduce(0, 'smean')\n",
+        "O8 = A.smean(0)\n",
+        "print(\"Reduce with reducer:smean along dim = 0:\")\n",
+        "print(O7)\n",
+        "print(\"\")"
+      ],
+      "metadata": {
+        "id": "GhS49Js1jW4b"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### *Matrix transformations*"
+      ],
+      "metadata": {
+        "id": "kanwnB7LOQui"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "*Sparse Matrix*"
+      ],
+      "metadata": {
+        "id": "NiiXso9elM2p"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([0, 1, 1, 2])\n",
+        "col = torch.tensor([1, 0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3., 4.])\n",
+        "A = dglsp.from_coo(row, col, val)\n",
+        "print(A.to_dense())\n",
+        "print(\"\")\n",
+        "\n",
+        "print(\"Get transpose of sparse matrix.\")\n",
+        "print(A.T.to_dense())\n",
+        "# Alias\n",
+        "# A.transpose()\n",
+        "# A.t()\n",
+        "print(\"\")\n",
+        "\n",
+        "print(\"Get a sparse matrix with the negation of the original nonzero values.\")\n",
+        "print(A.neg().to_dense())\n",
+        "print(\"\")"
+      ],
+      "metadata": {
+        "id": "qJcmZHmf-oTY"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "*Diagonal Matrix*"
+      ],
+      "metadata": {
+        "id": "iE3ANjFolIJu"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "val = torch.tensor([1., 2., 3., 4.])\n",
+        "D = dglsp.diag(val)\n",
+        "print(D.to_dense())\n",
+        "print(\"\")\n",
+        "\n",
+        "print(\"Get inverse of diagonal matrix:\")\n",
+        "print(D.inv().to_dense())\n",
+        "print(\"\")\n",
+        "\n",
+        "print(\"Get a diagonal matrix with the negation of the original nonzero values.\")\n",
+        "print(D.neg().to_dense())\n",
+        "print(\"\")"
+      ],
+      "metadata": {
+        "id": "j9kjY9RdlGXx"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### *Matrix multiplication*"
+      ],
+      "metadata": {
+        "id": "4uQlDFb0Uzto"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**matmul(A, B), equivalent to A @ B**\n",
+        "\n",
+        "The supported combinations are shown as follows.\n",
+        "\n",
+        "A \\ B           | **Tensor**|**DiagMatrix**|**SparseMatrix**\n",
+        "----------------|-----------|--------------|----------\n",
+        "**Tensor**      |✅         |🚫            |🚫\n",
+        "**DiagMatrix**. |✅         |✅            |✅\n",
+        "**SparseMatrix**|✅         |✅            |✅"
+      ],
+      "metadata": {
+        "id": "THWE30v6WpAk"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**Union[DiagMatrix, SparseMatrix] @ Union[DiagMatrix, SparseMatrix] -> Union[SparseMatrix, DiagMatrix]:**\n",
+        "\n",
+        "For a $L \\times M$ sparse matrix A and a $M \\times N$ sparse matrix B, the shape of `A @ B` will be $L \\times N$ sparse matrix."
+      ],
+      "metadata": {
+        "id": "VxyykR-vX7lF"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3.])\n",
+        "A1 = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A1:\")\n",
+        "print(A1.to_dense())\n",
+        "\n",
+        "row = torch.tensor([0, 1, 2])\n",
+        "col = torch.tensor([0, 2, 1])\n",
+        "val = torch.tensor([4., 5., 6.])\n",
+        "A2 = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A2:\")\n",
+        "print(A2.to_dense())\n",
+        "\n",
+        "val = torch.tensor([-1., -2., -3.])\n",
+        "D1 = dglsp.diag(val)\n",
+        "print(\"D1:\")\n",
+        "print(D1.to_dense())\n",
+        "\n",
+        "val = torch.tensor([-4., -5., -6.])\n",
+        "D2 = dglsp.diag(val)\n",
+        "print(\"D2:\")\n",
+        "print(D2.to_dense())\n",
+        "\n",
+        "print(\"A1 @ A2:\")\n",
+        "print((A1 @ A2).to_dense())\n",
+        "\n",
+        "print(\"A1 @ D1:\")\n",
+        "print((A1 @ D1).to_dense())\n",
+        "\n",
+        "print(\"D1 @ A1:\")\n",
+        "print((D1 @ A1).to_dense())\n",
+        "\n",
+        "print(\"D1 @ D2:\")\n",
+        "print((D1 @ D2).to_dense())"
+      ],
+      "metadata": {
+        "id": "XRDFC2rOYQM4"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**Union[DiagMatrix, SparseMatrix] @ Tensor -> Tensor:**\n",
+        "\n",
+        "For a $L \\times M$ sparse matrix A and a $M \\times N$ dense matrix B, the shape of `A @ B` will be $L \\times N$ dense matrix."
+      ],
+      "metadata": {
+        "id": "g13fG8nvaVOt"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3.])\n",
+        "A = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A:\")\n",
+        "print(A.to_dense())\n",
+        "\n",
+        "val = torch.tensor([-1., -2., -3.])\n",
+        "D = dglsp.diag(val)\n",
+        "print(\"D:\")\n",
+        "print(D.to_dense())\n",
+        "\n",
+        "X = torch.tensor([[11., 22.], [33., 44.], [55., 66.]])\n",
+        "print(\"X:\")\n",
+        "print(X)\n",
+        "\n",
+        "print(\"A @ X:\")\n",
+        "print(A @ X)\n",
+        "\n",
+        "print(\"D @ X:\")\n",
+        "print(D @ X)"
+      ],
+      "metadata": {
+        "id": "FcQ-CnqdlgWF"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "This operator also supports batched sparse-dense matrix multiplication. The sparse matrix A should have shape $L \\times M$, where the non-zero values are vectors of length $K$. The dense matrix B should have shape $M \\times N \\times K$. The output is a dense matrix of shape $L \\times N \\times K$."
+      ],
+      "metadata": {
+        "id": "_KZiULLbmEZE"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([0, 2, 0])\n",
+        "val = torch.tensor([[1., 1.], [2., 2.], [3., 3.]])\n",
+        "A = dglsp.from_coo(row, col, val, shape=(3, 3))\n",
+        "print(\"A:\")\n",
+        "print(A.to_dense())\n",
+        "\n",
+        "X = torch.tensor([[[1., 1.], [1., 2.]],\n",
+        "                  [[1., 3.], [1., 4.]],\n",
+        "                  [[1., 5.], [1., 6.]]])\n",
+        "print(\"X:\")\n",
+        "print(X)\n",
+        "\n",
+        "print(\"A @ X:\")\n",
+        "print(A @ X)"
+      ],
+      "metadata": {
+        "id": "ZUzXQk7Ab2wG"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**Sampled-Dense-Dense Matrix Multiplication**\n",
+        "\n",
+        "``sddmm`` matrix-multiplies two dense matrices X1 and X2, then elementwise-multiplies the result with sparse matrix A at the nonzero locations. This is designed for sparse matrix with scalar values.\n",
+        "\n",
+        "$$out = (X_1 @ X_2) * A$$\n",
+        "\n",
+        "For a $L \\times N$ sparse matrix A, a $L \\times M$ dense matrix X1 and a $M \\times N$ dense matrix X2, `sddmm(A, X1, X2)` will be a $L \\times N$ sparse matrix."
+      ],
+      "metadata": {
+        "id": "qO_8f_vhPKtf"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([2, 3, 3])\n",
+        "val = torch.tensor([1., 2., 3.])\n",
+        "A = dglsp.from_coo(row, col, val, (3, 4))\n",
+        "print(\"A:\")\n",
+        "print(A.to_dense())\n",
+        "\n",
+        "X1 = torch.randn(3, 5)\n",
+        "X2 = torch.randn(5, 4)\n",
+        "print(\"X1:\")\n",
+        "print(X1)\n",
+        "print(\"X2:\")\n",
+        "print(X2)\n",
+        "\n",
+        "O = dglsp.sddmm(A, X1, X2)\n",
+        "print(\"dglsp.sddmm(A, X1, X2):\")\n",
+        "print(O.to_dense())"
+      ],
+      "metadata": {
+        "id": "3ZIFV0TgPhwH"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "This operator also supports batched sampled-dense-dense matrix multiplication. For a $L \\times N$ sparse matrix A with non-zero vector values of length $𝐾$, a $L \\times M \\times K$ dense matrix X1 and a $M \\times N \\times K$ dense matrix X2, `sddmm(A, X1, X2)` will be a $L \\times N \\times K$ sparse matrix."
+      ],
+      "metadata": {
+        "id": "RmNmXU_ZqyF7"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([1, 1, 2])\n",
+        "col = torch.tensor([2, 3, 3])\n",
+        "val = torch.tensor([[1., 1.], [2., 2.], [3., 3.]])\n",
+        "A = dglsp.from_coo(row, col, val, (3, 4))\n",
+        "print(\"A:\")\n",
+        "print(A.to_dense())\n",
+        "\n",
+        "X1 = torch.randn(3, 5, 2)\n",
+        "X2 = torch.randn(5, 4, 2)\n",
+        "print(\"X1:\")\n",
+        "print(X1)\n",
+        "print(\"X2:\")\n",
+        "print(X2)\n",
+        "\n",
+        "O = dglsp.sddmm(A, X1, X2)\n",
+        "print(\"dglsp.sddmm(A, X1, X2):\")\n",
+        "print(O.to_dense())"
+      ],
+      "metadata": {
+        "id": "DuSAjamyrIO_"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Non-linear activation functions\n",
+        "\n",
+        "*   `softmax()`"
+      ],
+      "metadata": {
+        "id": "XuaNdFO7XG2r"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**Softmax**\n",
+        "\n",
+        "Apply row-wise softmax to the nonzero entries of the sparse matrix."
+      ],
+      "metadata": {
+        "id": "y8OQZReVXpo3"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([0, 1, 1, 2])\n",
+        "col = torch.tensor([1, 0, 2, 0])\n",
+        "val = torch.tensor([1., 2., 3., 4.])\n",
+        "A = dglsp.from_coo(row, col, val)\n",
+        "\n",
+        "print(A.softmax())\n",
+        "print(\"In dense format:\")\n",
+        "print(A.softmax().to_dense())\n",
+        "print(\"\\n\")"
+      ],
+      "metadata": {
+        "id": "CQaKgzCJULjt"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Exercise"
+      ],
+      "metadata": {
+        "id": "1iBNlJVYz3zi"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "*Let's test what you've learned. Feel free to [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/sparse/quickstart.ipynb).*\n",
+        "\n",
+        "Given a sparse symmetrical adjacency matrix $A$, calculate its symmetrically normalized adjacency matrix: $$norm = \\hat{D}^{-\\frac{1}{2}}\\hat{A}\\hat{D}^{-\\frac{1}{2}}$$\n",
+        "\n",
+        "Where $\\hat{A} = A + I$, $I$ is the identity matrix, and $\\hat{D}$ is the diagonal node degree matrix of $\\hat{A}$."
+      ],
+      "metadata": {
+        "id": "yDQ4Kmr_08St"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "row = torch.tensor([0, 0, 1, 1, 2, 2, 3])\n",
+        "col = torch.tensor([1, 3, 2, 5, 3, 5, 4])\n",
+        "asym_A = dglsp.from_coo(row, col, shape=(6, 6))\n",
+        "# Step 1: create symmetrical adjacency matrix A from asym_A.\n",
+        "# A =\n",
+        "\n",
+        "# Step 2: calculate A_hat from A.\n",
+        "# A_hat =\n",
+        "\n",
+        "# Step 3: diagonal node degree matrix of A_hat\n",
+        "# D_hat =\n",
+        "\n",
+        "# Step 4: calculate the norm from D_hat and A_hat.\n",
+        "# norm = "
+      ],
+      "metadata": {
+        "id": "0dDhfbJo0ByV"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}