Commits · 71f7e96aeb7ea0ab3270093b7d8e8e8f676082f4 · OpenDAS / dgl

21 Feb, 2023 9 commits

[Misc] Abort previous CI. (#5346) · 71f7e96a

Hongzhi (Steve), Chen authored Feb 21, 2023



* Abort previous CI

* script

* newtry

* more

* fix

* comment

* add notes

---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

71f7e96a

fix (#5349) · 27b3cd87

Hongzhi (Steve), Chen authored Feb 21, 2023


Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

27b3cd87

[Misc] Revert status.py (#5348) · b8ed4d9b
Hongzhi (Steve), Chen authored Feb 21, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
b8ed4d9b
[Misc] Add debug output to avoid silent failure. (#5347) · e4e01329
Hongzhi (Steve), Chen authored Feb 21, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
e4e01329

[Misc] Update Jenkins status. (#5345) · 0127a2ea

Hongzhi (Steve), Chen authored Feb 21, 2023



* update

* polish

* polish

---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

0127a2ea

[Misc] Print jenkins response for test. (#5344) · e330954c
Hongzhi (Steve), Chen authored Feb 21, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
e330954c
[Misc] Revert jenkins update change and add build url in log. (#5343) · deed9456
Hongzhi (Steve), Chen authored Feb 21, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
deed9456
Update Jenkins response status (#5342) · f87e98e4
Hongzhi (Steve), Chen authored Feb 21, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
f87e98e4

autofix (#5336) · 36065bfd

Hongzhi (Steve), Chen authored Feb 21, 2023


Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

36065bfd

20 Feb, 2023 6 commits

auto-fix (#5331) · a566b60b

Hongzhi (Steve), Chen authored Feb 21, 2023


Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

a566b60b

autofix2 (#5333) · d1827488

Hongzhi (Steve), Chen authored Feb 20, 2023


Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

d1827488

[Misc] Autoformat python dgl (Part 2). (#5332) · 3e5137fe
Hongzhi (Steve), Chen authored Feb 20, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
3e5137fe

[Misc] Auto format with manual fix of 3 files and add pylint: disable=... · 6e58f5f1

Hongzhi (Steve), Chen authored Feb 20, 2023


[Misc] Auto format with manual fix of 3 files and add pylint: disable= too-many-lines for functional.py. (#5330)

* blabal

* 2more

---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

6e58f5f1

[CI] add boto3 to interactive with AWS services like batch (#5329) · f17c0d62
Rhett Ying authored Feb 20, 2023
```
* [CI] add boto3 to interactive with AWS services like batch

* Update dgl_dev.yml.template
```
f17c0d62

[Misc] Auto-reformat multiple python folders. (#5325) · dce89919

Hongzhi (Steve), Chen authored Feb 20, 2023



* auto-reformat

* lintrunner

---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

dce89919

19 Feb, 2023 8 commits

[Misc] Auto reformat data/. (#5318) · ab812179

Hongzhi (Steve), Chen authored Feb 19, 2023



* data

* lintrunner

---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

ab812179

[Misc] Auto-reformat test/. (#5324) · f0759a96

Hongzhi (Steve), Chen authored Feb 19, 2023



* auto-format-test

* more

* remove

---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

f0759a96

lintrunner (#5326) · 9fe5092c

Hongzhi (Steve), Chen authored Feb 19, 2023


Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

9fe5092c

autoformat (#5322) · 9836f78e

Hongzhi (Steve), Chen authored Feb 19, 2023


Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

9836f78e

examples (#5323) · 704bcaf6

Hongzhi (Steve), Chen authored Feb 19, 2023


Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

704bcaf6

[Misc] auto-format tools. (#5321) · 6bc82161
Hongzhi (Steve), Chen authored Feb 19, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
6bc82161
auto-reformat-nn (#5319) · 63ac788f
Hongzhi (Steve), Chen authored Feb 19, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
63ac788f
auto format distributed (#5317) · 0b3a447b
Hongzhi (Steve), Chen authored Feb 19, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
0b3a447b

17 Feb, 2023 5 commits

[Misc] Auto-format tests. (#5313) · 74c9d27d

Hongzhi (Steve), Chen authored Feb 17, 2023



* [Misc] Auto-format tests.

* more

---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

74c9d27d

[Misc] Reorg python test (with backend) according to the guideline. (#5312) · 86193c26

Hongzhi (Steve), Chen authored Feb 17, 2023



* rename

* remove test transpose

* udpate

* rename

* cleanup

---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

86193c26

[Misc] Reorg python test according to the guideline. (#5307) · d9e55036
Hongzhi (Steve), Chen authored Feb 17, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
d9e55036

remove (#5309) · 2bf2566f

Hongzhi (Steve), Chen authored Feb 17, 2023


Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

2bf2566f

blabal (#5306) · 03dbe758

Hongzhi (Steve), Chen authored Feb 17, 2023


Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

03dbe758

16 Feb, 2023 9 commits

[DistDGL][Optimizations]Rehash code to optimize for loop (#5224) · 9ce800d2

kylasa authored Feb 16, 2023

* Rehash code to optimize for loop

Reduced number of instructions in for loop, which exchanging edge features. This will reduce the number of times numpy's intersect1d is invoked (saving the runtime and memory overhead needs of numpy).

* Applying lintrunner patch to data_shuffle.py

9ce800d2

[DistDGL][Mem_Optimizations]Edge Ownership processes are computed on the fly when required. (#5225) · e25f47de

kylasa authored Feb 16, 2023

* Edge Ownership processes are computed on the fly when required.

Earlier we were storing Edge ownership processes after the dataset was retrieved from the disk. For massively large datasets, each node can handle upto 5 Billion edges, this means storing owner process-ids will consume 5 * 8 = 40GB. This memory will be hanging around until the edges are exchanged.

To reduce the memory footprint of the pipeline, we no longer store the ownership process-ids in the 'edge_data' dictionary after reading the dataset from the disk. Instead, we compute them on the fly at the time of exchanging edges.

Another optimization is not to send/receive all the messages in a one single large message. Instead we now split the total number edges into chunks, limited by 8 GB per node. And we iterate until all the chunks are exchanged.

Once all the edges are exchanged, as a sanity check, we compute the total number of edges in the system and compare it with the original value before edge shuffling, in a final assert statement before return the result to the caller.

* Applying lintrunner patch.

e25f47de

[Misc] Fix build warnings (#5303) · 1329be96
Songqing Zhang authored Feb 16, 2023
```
Co-authored-by: songqing.zhang <songqing.zhang@shopee.com>
```
1329be96
[Dev] Create a script for kick off python test. (#5298) · 85dfcea0
Hongzhi (Steve), Chen authored Feb 16, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
85dfcea0
[Dev] Create a script for building dgl from source. (#5294) · ccbc30fd
Hongzhi (Steve), Chen authored Feb 16, 2023
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>
```
ccbc30fd

[bugfix] Fix assertions in /src/runtime/workspace.h and expand unit tests for... · 2bbca12a

nv-dlasalle authored Feb 15, 2023

[bugfix] Fix assertions in /src/runtime/workspace.h and expand unit tests for sparse optimizer (#5299)

* Fix assertions for size 0 workspaces

* Expand unit test to cover case of communication

* Fixes

* Format

* Fix c++ formatting

2bbca12a

Set USE_LIBXSMM default to OFF. (#5287) · a5e31391

Hongzhi (Steve), Chen authored Feb 16, 2023



* Set USE_LIBXSMM default to OFF.

* update

---------
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

a5e31391

sort (#5281) · 0552466c

Hongzhi (Steve), Chen authored Feb 16, 2023


Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

0552466c

remove (#5280) · f7a0c4bb

Hongzhi (Steve), Chen authored Feb 16, 2023


Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

f7a0c4bb

15 Feb, 2023 2 commits

[Model] Update `CuGraphRelGraphConv` to use `pylibcugraphops=23.02` (#5217) · 19b3cea1

Tingyu Wang authored Feb 15, 2023



* update cugraph_relgraphconv

* update equality test

* update cugraph rgcn example

* update RelGraphConvAgg based on latest API changes

* enable fallback option to fg when fanout is large

---------
Co-authored-by: Mufei Li <mufeili1996@gmail.com>

19b3cea1

fix build for multiple pytorch versions (#5289) · 5f1babf0
Quan (Andy) Gan authored Feb 15, 2023

5f1babf0

14 Feb, 2023 1 commit

fix (#5293) · 40c968b7

Hongzhi (Steve), Chen authored Feb 14, 2023


Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

40c968b7