README.md

DGL Benchmarks
====

Benchmarking DGL with Airspeed Velocity.

Usage
---

Before beginning, ensure that airspeed velocity is installed:

```bash
pip install asv
```

To run all benchmarks locally, build the project first and then run:

```bash
asv run -n -e --python=same --verbose
```

**Due to ASV's restriction, `--python=same` will not write any benchmark results
to disk. It does not support specifying branches and commits either. They are only
available under ASV's managed environment.**

To change the device for benchmarking, set the `DGL_BENCH_DEVICE` environment variable.
Any valid PyTorch device strings are allowed.

```bash
export DGL_BENCH_DEVICE=cuda:0
```

To select which benchmark to run, use the `--bench` flag. For example,

```bash
asv run -n -e --python=same --verbose --bench model_acc.bench_gat
```

Run in docker locally
---

DGL runs all benchmarks automatically in docker container. To run bencmarks in docker locally,

* Git commit your locally changes. No need to push to remote repository.
* To compare commits from different branches. Change the `"branches"` list in `asv.conf.json`.
  The default is `"HEAD"` which is the last commit of the current branch. For example, to
  compare your proposed changes with the master branch, set it to be `["HEAD", "master"]`.
  If your workspace is a forked repository, make sure your local master has synced with
  the upstream.
* Use the `publish.sh` script. It accepts two arguments, a name specifying the identity of
  the test machine and a device name. For example,
  ```bash
  bash publish.sh dev-machine cuda:0
  ```

The script will output two folders `results` and `html`. The `html` folder contains the
generated static web pages. View it by:

```bash
asv preview
```

Please see `publish.sh` for more information on how it works and how to modify it according
to your need.

Adding a new benchmark suite
---

The benchmark folder is organized as follows:

```
|-- benchmarks/
  |-- model_acc/           # benchmarks for model accuracy
    |-- bench_gcn.py
    |-- bench_gat.py
    |-- bench_sage.py
    ...
  |-- model_speed/         # benchmarks for model training speed
    |-- bench_gat.py
    |-- bench_sage.py
    ...
  ...                      # other types of benchmarks
|-- html/                  # generated html files
|-- results/               # generated result files
|-- asv.conf.json          # asv config file
|-- build_dgl_asv.sh       # script for building dgl in asv
|-- install_dgl_asv.sh     # script for installing dgl in asv
|-- publish.sh             # script for running benchmarks in docker
|-- README.md              # this readme
|-- run.sh                 # script for calling asv in docker
|-- ...                    # other aux files
```

To add a new benchmark, pick a suitable benchmark type and create a python script under
it. We prefer to have the prefix `bench_` in the name. Here is a toy example:

```python
# bench_range.py

import time
from .. import utils

@utils.benchmark('time')
@utils.parametrize('l', [10, 100, 1000])
@utils.parametrize('u', [10, 100, 1000])
def track_time(l, u):
    t0 = time.time()
    for i in range(l, u):
        pass
    return time.time() - t0
```

* The main entry point of each benchmark script is a `track_*` function. The function
  can have arbitrary arguments and must return the benchmark result.
* There are two useful decorators: `utils.benchmark` and `utils.parametrize`.
* `utils.benchmark` indicates the type of this benchmark. Currently supported types are:
  `'time'` and `'acc'`. The decorator will perform some necessary setup and finalize
  steps such as fixing the random seed for the `'acc'` type.
* `utils.parametrize` specifies the parameters to test.
  Multiple parametrize decorators mean benchmarking the combination.
* Check out `model_acc/bench_gcn.py` and `model_speed/bench_sage.py`.
* ASV's [official guide on writing benchmarks](https://asv.readthedocs.io/en/stable/writing_benchmarks.html)
  is also very helpful.


Tips
----
* Feed flags `-e --verbose` to `asv run` to print out stderr and more information.
* When running benchmarks locally (e.g., with `--python=same`), ASV will not write results to disk
  so `asv publish` will not generate plots.
* Try make your benchmarks compatible with all the versions being tested.
* For ogbn dataset, put the dataset into /tmp/dataset/