Commits · 4be4b134247cf79617480e5f4646dfa07bd96a4e · OpenDAS / dgl

31 Jul, 2020 2 commits

[Distributed] add copy_partitions.py (#1866) · 4be4b134

Da Zheng authored Jul 31, 2020



* fix bugs.

* eval on both vaidation and testing.

* add script.

* update.

* update launch.

* make train_dist.py independent.

* update readme.

* update readme.

* update readme.

* update readme.

* generate undirected graph.

* rename conf_file to part_config

* use rsync

* make train_dist independent.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-115.us-west-2.compute.internal>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>

4be4b134

[KVStore] Fix bug in barrier. (#1904) · e5e1a99d
Chao Ma authored Jul 31, 2020
```
* update

* update

* fix lint

* update

* update
```
e5e1a99d

30 Jul, 2020 1 commit

[KVStore] Support group barrier (#1880) · 5b515cf6

Chao Ma authored Jul 30, 2020



* udpate

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* update

* update

* update

* udpate

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

5b515cf6

29 Jul, 2020 1 commit

[Distributed] Refactor distributed training to new DGLGraph. (#1874) · 3e2f94ed

Da Zheng authored Jul 28, 2020

* fix tests in partition.

* fix DistGraph.

* fix without shared memory.

* fix sampling.

* enable distributed test.

* fix tests.

* fix a bug in shared-mem heterograph.

* print better error messages.

* fix.

* don't specify formats.

* fix.

* fix

* small fix.

3e2f94ed

28 Jul, 2020 2 commits

Revert "[Distributed] DistDataloader (#1870)" (#1876) · 71cf7865
Jinjing Zhou authored Jul 28, 2020
```
This reverts commit 6557291f.
```
71cf7865

[Distributed] DistDataloader (#1870) · 6557291f

Jinjing Zhou authored Jul 28, 2020



* 111

* 111

* fix

* 111

* fix

* 11

* fix

* lint

* Update __init__.py

* lint

* fix

* lint

* fix

* fix

* fix

* fix

* fix

* try fix

* try fix

* fix

* Revert "fix"

This reverts commit a0b954fd4e99b7df92b53db8334dcb583d6e1551.

* fixes.

* fix.

* fix test.

* fix exit.

* fix.

* fix

* fix

* lint

* lint

* lint

* fix

* Update .gitignore
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

6557291f

27 Jul, 2020 2 commits
- [Distributed] Small fixes (#1859) · 015acfd2
  Da Zheng authored Jul 27, 2020
```
* fix node/edge_split.

* fix partition.

* support heterograph interface.

* fix test.

* fix

* fix docstring.
```
  015acfd2
- [Distributed] turn off recording on embeddings in the inference. (#1861) · ff8f7082
  Da Zheng authored Jul 27, 2020
```
* turn on/off recording in sparse embedding.

* add test.
```
  ff8f7082
26 Jul, 2020 1 commit

[Distributed] update Embedding API (#1847) · 6963d796

Da Zheng authored Jul 26, 2020



* update.

* update Embedding.

* add comments.

* fix lint
Co-authored-by: Chao Ma <mctt90@gmail.com>

6963d796

22 Jul, 2020 2 commits

[HeteroGraph] Metis partition for hetero (#1843) · 33abd275
Jinjing Zhou authored Jul 22, 2020
```
* metis for hetero

* fix partition
```
33abd275

[Distributed] add distributed evaluation. (#1810) · 562871e7

Da Zheng authored Jul 22, 2020



* add eval.

* extend DistTensor.

* fix.

* add barrier.

* add more print.

* add more checks in kvstore.

* fix lint.

* get all neighbors for eval.

* reorganize.

* fix.

* fix.

* fix.

* fix test.

* add reuse_if_exist.

* add test for reuse_if_exist.

* fix lint.

* fix bugs.

* fix.

* print errors of tcp socket.

* support delete tensors.

* fix lint.

* fix

* fix example
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>

562871e7

20 Jul, 2020 2 commits

[RPC] Rpc exit with explicit invocation (#1825) · 5c92f6c2

Chao Ma authored Jul 20, 2020

* exit client

* update

* update

* update

* update

* update

* update

* update

* update test

* update

* update

* update

* update

* update

* update

* update

* update

* update

5c92f6c2

[KVStore] Support delete_data() api on client (#1824) · 65e7805e

Chao Ma authored Jul 20, 2020

* delete tensor

* update

* update

* update

* update

* update

* update

* udpate

* update

* update

* update

65e7805e

15 Jul, 2020 1 commit

[Distributed] add the standalone mode in DistGraph (#1800) · cda0abf7

Da Zheng authored Jul 14, 2020



* add standalone mode

* add comments.

* add tests for sampling.

* fix.

* make the code to run the standalone mode

* fix

* fix

* fix readme.

* fix.

* fix test
Co-authored-by: Chao Ma <mctt90@gmail.com>

cda0abf7

14 Jul, 2020 1 commit

[Distributed] Run distributed graph server inside DGL (#1801) · a1472bcf

Da Zheng authored Jul 14, 2020

* run dist server in dgl.

* fix bugs.

* fix example.

* check environment variables and fix lint.

* fix lint

a1472bcf

08 Jul, 2020 1 commit

[Distributed] add in_subgraph on DistGraph. (#1755) · 167216af

Da Zheng authored Jul 08, 2020

* add in_subgraph on DistGraph.

* check in more.

* fix test.

* add comments.

* fix test.

* update test.

* update.

* rename.

* update comment

* fix test

167216af

07 Jul, 2020 1 commit
- [RPC] Client will retry connection until the server starts. (#1758) · bdc1e649
  Chao Ma authored Jul 07, 2020
```
* update

* update

* update
```
  bdc1e649
06 Jul, 2020 1 commit

[Distributed] fix init_data in kvstore. (#1750) · afb7f5b8

Da Zheng authored Jul 06, 2020



* fix

* fix.

* update.

* fix

* add assert
Co-authored-by: Chao Ma <mctt90@gmail.com>

afb7f5b8

03 Jul, 2020 2 commits

[Feature] add sparse embedding. (#1497) · cadcc1c2

Da Zheng authored Jul 03, 2020

* add sparse embedding.

* fix

* add test.

* man fixes.

* many fixes

* fix sparse emb.

* fix.

* fix lint.

* fix lint.

* fix kvstore.

* expose DistTensor.

* test sparse embeddings.

* add attach_grad to the backends.

* remove part_id

* fix.

* move backward computation.

* move more computation to backend.

* fix a bug when applying learning rate.

* fix a few things.

* fix a few things.

* add docstring

* fix.

* apply no_grad.

* fix tests.

* fix for other frameworks.

* add examples in docstring.

cadcc1c2

[Test] Fix test of distributed graph (#1748) · 17701174

Da Zheng authored Jul 03, 2020

* Revert "Serialize server and client start for dist_graph_store test (#1736)"

This reverts commit da16ebf0.

* sleep

17701174

02 Jul, 2020 2 commits
- Serialize server and client start for dist_graph_store test (#1736) · da16ebf0
  xiang song(charlie.song) authored Jul 02, 2020
```
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal>
```
  da16ebf0
- [KVStore] Test multi-server and multi-client (#1732) · 9eb0efcf
  Chao Ma authored Jul 02, 2020
```
* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update
```
  9eb0efcf
01 Jul, 2020 1 commit

[Distributed] Split data to ensure data locality for multiple clients (#1710) · 2c88f7c0

Da Zheng authored Jul 01, 2020

* fix.

* fix tests.

* fix

* add tests.

* fix.

* have default rank.

* add comment.

* fix test.

* remove check

* simplify code.

* add test.

* split data evenly.

* simplify the distributed training code.

* add comments.

* add comments.

2c88f7c0

29 Jun, 2020 1 commit

[rpc] Add num_clients and sort client ID by machine-id (#1713) · dd8d5289

Chao Ma authored Jun 29, 2020



* add num_clients to kvstore

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* update

* update

* add test

* update

* update

* update

* update

* fix test

* update

* update

* update

* update

* update

* update

* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

dd8d5289

27 Jun, 2020 1 commit

[Distributed] Optimize distributed sampling (#1700) · 45e1333e

Da Zheng authored Jun 26, 2020



* optimize sampling neighbors from the local partition.

* fix minior bugs.

* fix lint

* overlap local sampling and remote sampling.

* fix.

* fix lint.

* fix a potential bug.
Co-authored-by: Chao Ma <mctt90@gmail.com>

45e1333e

26 Jun, 2020 1 commit

[BUGFIX] Fix bugs in range partitioning. (#1703) · 4b96addc

Da Zheng authored Jun 26, 2020

* Fix bugs in partitioning.

* add comment.

* fix lint.

* fix test.

* add more tests.

* simplify the fix

* fix.

4b96addc

24 Jun, 2020 1 commit
- [BUGFIX] fix a bug in partition book. (#1698) · fa1478a4
  Da Zheng authored Jun 24, 2020
```
* fix a bug.

* add msg for assert
```
  fa1478a4
18 Jun, 2020 3 commits

[RPC] Sampling over RPC (#1616) · fb248b67

Jinjing Zhou authored Jun 19, 2020



* fix

* test1111

* 111

* 111

* fff

* lint

* 111

* lint

* lint

* 111

* fijx

* 111

* fix

* 111

* commit

* 111

* 111

* lint

* fix typo

* fix

* lint

* fix

* 111

* support mxnet

* support mxnet

* lint

* remove print

* fix

* fix test.

* fix test.

* fix test.

* try to fix an undetermistic error.
Co-authored-by: Chao Ma <mctt90@gmail.com>
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

fb248b67

[Distributed] support no shared memory. (#1663) · ff5b5a4a

Da Zheng authored Jun 18, 2020

* support no shared memory.

* add test.

* add CAPI to check existence of shared memory.

* revert the change in src/runtime/ndarray.cc

* update docstring.

* fix compile.

ff5b5a4a

[BUGFIX] reshuffle node features in range partitioning. (#1659) · b9ec3e36
Da Zheng authored Jun 17, 2020
```
* fix a bug.

* fix test.

* simplify.

* fix test.

* test edge features.

* fix.
```
b9ec3e36

17 Jun, 2020 1 commit

[KVStore] Add fast-pull for kvstore (#1647) · b372b3c7

Chao Ma authored Jun 17, 2020

* add fast-pull

* update

* add fast-pull

* update

* update

* update

* update test

* update

* update

* update

* update

* update

* update

* update

* update

* add omp

* update

* update

b372b3c7

16 Jun, 2020 1 commit
- [KVStore] make pull/push handler per tensor. (#1646) · e8a56dc1
  Da Zheng authored Jun 15, 2020
```
* make pull/push handler per tensor.

* update.
```
  e8a56dc1
15 Jun, 2020 1 commit

[Feature] Range partition (#1522) · 41349dce

Da Zheng authored Jun 15, 2020



* add reorder immutable graph.

* add python API.

* add reorder for csr.

* remove gk version.

* fix

* add cpp test.

* bug fixes

* fix tests.

* fix bugs and add check

* fix test.

* add omp.

* add comments.

* add coo reorder.

* fix a bug.

* handle reorder for different graph structues.

* fix lint.

* fix.

* add original ids.

* reshuffle nodes before metis partition.

* inner nodes are in contiguous Id range.

* reshuffle nodes/edges when partitioning.

* load partition return graph partition book.

* use inner_node/inner_edges

* add and test range partition book.

* count inner_edge correctly.

* fix lint.

* fix lint.

* fix lint.

* fix errors.

* fix errors.

* fix for TF.

* fix.

* fix.

* change docstring.

* support logical and.

* add comments.

* avoid copy.

* fix

* update docstring.

* fix a bug.

* add range search.

* fix

* fix a bug.

* add more tests.

* load graph partition book.

* support shared memory for range partition book.

* fix a bug.

* fix.

* fix lint.

* remove check

* fix test.

* remove num_nodes and num_edges

* fix lint.

* fix graph partition book.

* address comments.

* use makedirs.

* fix compile
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
Co-authored-by: Chao Ma <mctt90@gmail.com>

41349dce

11 Jun, 2020 3 commits

[CI] Use spawn in unittest instead of fork (#1627) · 77006db2

Jinjing Zhou authored Jun 11, 2020



* update

* update

* update

* update

* update

* update

* fix ci using spawn
Co-authored-by: aksnzhy <mctt90@gmail.com>

77006db2

[rpc] Move test_rpc.py to distributed and use dynamic port binding (#1623) · 0c313e51
Chao Ma authored Jun 11, 2020
```
* update

* update

* update

* update

* update

* update

* update

* update
```
0c313e51

Refactor distributed graph store (#1597) · 50962c43

Da Zheng authored Jun 10, 2020

* refactor graph store for new kvstore.

* fix kvstore.

* fix.

* fix lint complains.

* add docstring.

* small fix in graph partition book.

* fix.

* remove tests

* disable sampling.

* Revert "disable sampling."

This reverts commit 70451008f61ff1481d5dadbf8bd199470aee559d.

* Revert "remove tests"

This reverts commit 1394364243bdd73b669abde6193a74e2cda5521d.

50962c43

09 Jun, 2020 1 commit
- [KVStore] Update kvstore test (#1607) · 48cc8fa8
  Chao Ma authored Jun 09, 2020
```
* remove freeze

* update

* update

* fix lint

* update test

* update
```
  48cc8fa8
04 Jun, 2020 1 commit

[KVStore] Re-write kvstore using DGL RPC infrastructure (#1569) · 64f49703

Chao Ma authored Jun 04, 2020

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update init_data

* update server_state

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* debug init_data

* update

* update

* update

* update

* update

* update

* test get_meta_data

* update

* update

* update

* update

* update

* debug push

* update

* update

* update

* update

* update

* update

* update

* update

* update

* use F.reverse_data_type_dict

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* fix lint

* update

* fix lint

* update

* update

* update

* update

* fix test

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* set random seed

* update

64f49703

14 May, 2020 1 commit

[Feature] Split data for distributed training (#1505) · 96984fac

Da Zheng authored May 14, 2020



* fix tests in graph partition book.

* implement node_split and edge_split.

* fix a bug.

* add tests.

* remove unnecessary code.

* avoid storing graph structure in partition book.

* add partition book in DistGraph.

* Revert "avoid storing graph structure in partition book."

This reverts commit 55b11fbf9293efcae5f8c97e93f1e121399dadae.

* small fixes.

* add a test for boolean mask vector.

* fix test.

* fix bugs.

* make it work for all different cases.

* fix tests.

* fix boolean mask

* fix for TF

* fix for tensorflow.

* fix test for TF

* only support boolean mask for now.

* fix tests.

* make the code more readable.

* fix test.
Co-authored-by: Chao Ma <mctt90@gmail.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>

96984fac

08 May, 2020 1 commit

[Distributed] Add GraphPartitionBook (#1496) · ef7e4750

Chao Ma authored May 08, 2020

* Add graph-partition-book

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* udpate

* fix lint

* update

* update

* update

* update

* update

* update

* update

* skip test on GPU

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* update

* update

* update

* fix unittest

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

ef7e4750