Commits · 25ac33440334fd2fdfe3fe4e4dd3dd1428601a47 · OpenDAS / dgl

25 Jan, 2021 1 commit

[Distributed] Heterogeneous graph support (#2457) · 25ac3344

Da Zheng authored Jan 24, 2021



* Distributed heterograph (#3)

* heterogeneous graph partition.

* fix graph partition book for heterograph.

* load heterograph partitions.

* update DistGraphServer to support heterograph.

* make DistGraph runnable for heterograph.

* partition a graph and store parts with homogeneous graph structure.

* update DistGraph server&client to use homogeneous graph.

* shuffle node Ids based on node types.

* load mag in heterograph.

* fix per-node-type mapping.

* balance node types.

* fix for homogeneous graph

* store etype for now.

* fix data name.

* fix a bug in example.

* add profiler in rgcn.

* heterogeneous RGCN.

* map homogeneous node ids to hetero node ids.

* fix graph partition book.

* fix DistGraph.

* shuffle eids.

* verify eids and their mappings when loading a partition.

* Id map from homogneous Ids to per-type Ids.

* verify partitioned results.

* add test for distributed sampler.

* add mapping from per-type Ids to homogeneous Ids.

* update example.

* fix DistGraph.

* Revert "add profiler in rgcn."

This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.

* add tests for homogeneous graphs.

* fix a bug.

* fix test.

* fix for one partition.

* fix for standalone training and evaluation.

* small fix.

* fix two bugs.

* initialize projection matrix.

* small fix on RGCN.

* Fix rgcn performance (#17)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal>

* fix lint.

* fix lint.

* fix lint.

* fix lint.

* fix lint.

* fix lint.

* fix.

* fix test.

* fix lint.

* test partitions.

* remove redundant test for partitioning.

* remove commented code.

* fix partition.

* fix tests.

* fix RGCN.

* fix test.

* fix test.

* fix test.

* fix.

* fix a bug.

* update dmlc-core.

* fix.

* fix rgcn.

* update readme.

* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal>

* fix.

* fix.

* add div_int.

* fix.

* fix.

* fix lint.

* fix.

* fix.

* fix.

* adjust.

* move code.

* handle heterograph.

* return pytorch tensor in GPB.

* remove some tests in example.

* add to_block for distributed training.

* use distributed to_block.

* remove unnecessary function in DistGraph.

* remove distributed to_block.

* use pytorch tensor.

* fix a bug in ntypes and etypes.

* enable norm.

* make the data loader compatible with the old format.

* fix.

* add comments.

* fix a bug.

* add test for heterograph.

* support partition without reshuffle.

* add test.

* support partition without reshuffle.

* fix.

* add test.

* fix bugs.

* fix lint.

* fix dataset.

* fix for mxnet.

* update docstring.

* rename to floor_div

* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.

* fix docstring.

* fix error.

* fixes.

* fix comments.

* rename.

* rename.

* explain IdMap.

* fix docstring.

* fix docstring.

* update docstring.

* remove the code of returning heterograph.

* remove argument.

* fix example.

* make GraphPartitionBook an abstract class.

* fix.

* fix.

* fix a bug.

* fix a bug in example

* fix a bug

* reverse heterograph sampling.

* temp fix.

* fix lint.

* Revert "temp fix."

This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.

* compute norm.

* Revert "reverse heterograph sampling."

This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.

* fix.

* move id_map.py

* remove check

* add more comments.

* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal>

25ac3344

18 Aug, 2020 1 commit

[Doc] Update the docstring of distributed APIs. (#2025) · 75ffc31f

Da Zheng authored Aug 18, 2020



* add doc.

* update DistGraph.

* add DistTensor.

* update DistEmbedding.

* add partition.py

* add sampling.

* fix.

* add graph partition book and create a base class.

* fix test.

* add rst.

* update doc rst.

* update.

* fix.

* fix docs

* update distributed tensor and embeddings.

* add checks.

* update DistGraph.

* update initialization.

* fix graph partition book.

* update graph partition book.

* update partition.

* update partition.

* fix.

* add example code.

* update DistGraph

* Update python/dgl/distributed/dist_context.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Update python/dgl/distributed/dist_context.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* update initialize.

* update dataloader.

* update distgraph.

* update DistGraph.

* update DistTensor.

* update.

* more updates.

* fix lint.

* add num_nodes and num_edges
Co-authored-by: Chao Ma <mctt90@gmail.com>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>

75ffc31f

12 Aug, 2020 1 commit

[Distributed] adjust various APIs. (#1993) · d1cf5c38

Da Zheng authored Aug 11, 2020

* rename get_data_size.

* remove g from DistTensor.

* remove g from DistEmbedding.

* clean up API of graph partition book.

* fix DistGraph

* fix lint.

* collect all part policies.

* fix.

* fix.

* support distributed sampler.

* remove partition.py

d1cf5c38

11 Aug, 2020 1 commit

[Distributed] Remove server_count from ip_config.txt (#1985) · d340ea3a

Chao Ma authored Aug 11, 2020



* remove server_count from ip_config.txt

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* lint

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* Update dist_context.py

* fix lint.

* make it work for multiple spaces.

* update ip_config.txt.

* fix examples.

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* udpate

* update

* update

* update

* update

* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>

d340ea3a

08 Aug, 2020 1 commit

[Distributed] Add roles (#1971) · 5454471f

Da Zheng authored Aug 08, 2020

* distinguish roles.

* add comments.

* fix lint.

* move roles to server_state.

* fix text.

* fix tests.

* fix tests.

* Revert "fix tests."

This reverts commit 5baa136b872a4550d4e612bfb1dfe363d7814adf.

5454471f

03 Aug, 2020 1 commit

[Distributed] Support multiple servers (#1886) · a4c931a9

Da Zheng authored Aug 02, 2020



* client init graph on the backup servers.

* fix.

* test multi-server.

* fix anonymous dist tensors.

* check #parts.

* fix init_data

* add multi-server multi-client tests.

* update tests in kvstore.

* fix.

* verify the loaded partition.

* fix a bug.

* fix lint.

* fix.

* fix example.

* fix rpc.

* fix pull/push handler for backup kvstore

* fix example readme.

* change ip.

* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>

a4c931a9

31 Jul, 2020 1 commit
- [KVStore] Fix bug in barrier. (#1904) · e5e1a99d
  Chao Ma authored Jul 31, 2020
```
* update

* update

* fix lint

* update

* update
```
  e5e1a99d
30 Jul, 2020 1 commit

[KVStore] Support group barrier (#1880) · 5b515cf6

Chao Ma authored Jul 30, 2020



* udpate

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* update

* update

* update

* udpate

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

5b515cf6

29 Jul, 2020 1 commit

[Distributed] Refactor distributed training to new DGLGraph. (#1874) · 3e2f94ed

Da Zheng authored Jul 28, 2020

* fix tests in partition.

* fix DistGraph.

* fix without shared memory.

* fix sampling.

* enable distributed test.

* fix tests.

* fix a bug in shared-mem heterograph.

* print better error messages.

* fix.

* don't specify formats.

* fix.

* fix

* small fix.

3e2f94ed

20 Jul, 2020 2 commits

[RPC] Rpc exit with explicit invocation (#1825) · 5c92f6c2

Chao Ma authored Jul 20, 2020

* exit client

* update

* update

* update

* update

* update

* update

* update

* update test

* update

* update

* update

* update

* update

* update

* update

* update

* update

5c92f6c2

[KVStore] Support delete_data() api on client (#1824) · 65e7805e

Chao Ma authored Jul 20, 2020

* delete tensor

* update

* update

* update

* update

* update

* update

* udpate

* update

* update

* update

65e7805e

07 Jul, 2020 1 commit
- [RPC] Client will retry connection until the server starts. (#1758) · bdc1e649
  Chao Ma authored Jul 07, 2020
```
* update

* update

* update
```
  bdc1e649
06 Jul, 2020 1 commit

[Distributed] fix init_data in kvstore. (#1750) · afb7f5b8

Da Zheng authored Jul 06, 2020



* fix

* fix.

* update.

* fix

* add assert
Co-authored-by: Chao Ma <mctt90@gmail.com>

afb7f5b8

03 Jul, 2020 1 commit

[Feature] add sparse embedding. (#1497) · cadcc1c2

Da Zheng authored Jul 03, 2020

* add sparse embedding.

* fix

* add test.

* man fixes.

* many fixes

* fix sparse emb.

* fix.

* fix lint.

* fix lint.

* fix kvstore.

* expose DistTensor.

* test sparse embeddings.

* add attach_grad to the backends.

* remove part_id

* fix.

* move backward computation.

* move more computation to backend.

* fix a bug when applying learning rate.

* fix a few things.

* fix a few things.

* add docstring

* fix.

* apply no_grad.

* fix tests.

* fix for other frameworks.

* add examples in docstring.

cadcc1c2

02 Jul, 2020 1 commit

[KVStore] Test multi-server and multi-client (#1732) · 9eb0efcf

Chao Ma authored Jul 02, 2020

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

9eb0efcf

29 Jun, 2020 1 commit

[rpc] Add num_clients and sort client ID by machine-id (#1713) · dd8d5289

Chao Ma authored Jun 29, 2020



* add num_clients to kvstore

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* update

* update

* add test

* update

* update

* update

* update

* fix test

* update

* update

* update

* update

* update

* update

* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

dd8d5289

18 Jun, 2020 1 commit

[RPC] Sampling over RPC (#1616) · fb248b67

Jinjing Zhou authored Jun 19, 2020



* fix

* test1111

* 111

* 111

* fff

* lint

* 111

* lint

* lint

* 111

* fijx

* 111

* fix

* 111

* commit

* 111

* 111

* lint

* fix typo

* fix

* lint

* fix

* 111

* support mxnet

* support mxnet

* lint

* remove print

* fix

* fix test.

* fix test.

* fix test.

* try to fix an undetermistic error.
Co-authored-by: Chao Ma <mctt90@gmail.com>
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

fb248b67

17 Jun, 2020 1 commit

[KVStore] Add fast-pull for kvstore (#1647) · b372b3c7

Chao Ma authored Jun 17, 2020

* add fast-pull

* update

* add fast-pull

* update

* update

* update

* update test

* update

* update

* update

* update

* update

* update

* update

* update

* add omp

* update

* update

b372b3c7

16 Jun, 2020 1 commit
- [KVStore] make pull/push handler per tensor. (#1646) · e8a56dc1
  Da Zheng authored Jun 15, 2020
```
* make pull/push handler per tensor.

* update.
```
  e8a56dc1
11 Jun, 2020 2 commits

[CI] Use spawn in unittest instead of fork (#1627) · 77006db2

Jinjing Zhou authored Jun 11, 2020



* update

* update

* update

* update

* update

* update

* fix ci using spawn
Co-authored-by: aksnzhy <mctt90@gmail.com>

77006db2

Refactor distributed graph store (#1597) · 50962c43

Da Zheng authored Jun 10, 2020

* refactor graph store for new kvstore.

* fix kvstore.

* fix.

* fix lint complains.

* add docstring.

* small fix in graph partition book.

* fix.

* remove tests

* disable sampling.

* Revert "disable sampling."

This reverts commit 70451008f61ff1481d5dadbf8bd199470aee559d.

* Revert "remove tests"

This reverts commit 1394364243bdd73b669abde6193a74e2cda5521d.

50962c43

09 Jun, 2020 1 commit
- [KVStore] Update kvstore test (#1607) · 48cc8fa8
  Chao Ma authored Jun 09, 2020
```
* remove freeze

* update

* update

* fix lint

* update test

* update
```
  48cc8fa8
04 Jun, 2020 1 commit

[KVStore] Re-write kvstore using DGL RPC infrastructure (#1569) · 64f49703

Chao Ma authored Jun 04, 2020

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update init_data

* update server_state

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* debug init_data

* update

* update

* update

* update

* update

* update

* test get_meta_data

* update

* update

* update

* update

* update

* debug push

* update

* update

* update

* update

* update

* update

* update

* update

* update

* use F.reverse_data_type_dict

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* fix lint

* update

* fix lint

* update

* update

* update

* update

* fix test

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* set random seed

* update

64f49703