GitLab

* remove num_workers.

* remove num_workers.

* remove num_workers.

* remove num-servers.

* update error message.

* update docstring.

* fix docs.

* fix tests.

* fix test.

* fix.

* print messages in test.

* fix.

* fix test.

* fix.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal>

* Distributed heterograph (#3)

* heterogeneous graph partition.

* fix graph partition book for heterograph.

* load heterograph partitions.

* update DistGraphServer to support heterograph.

* make DistGraph runnable for heterograph.

* partition a graph and store parts with homogeneous graph structure.

* update DistGraph server&client to use homogeneous graph.

* shuffle node Ids based on node types.

* load mag in heterograph.

* fix per-node-type mapping.

* balance node types.

* fix for homogeneous graph

* store etype for now.

* fix data name.

* fix a bug in example.

* add profiler in rgcn.

* heterogeneous RGCN.

* map homogeneous node ids to hetero node ids.

* fix graph partition book.

* fix DistGraph.

* shuffle eids.

* verify eids and their mappings when loading a partition.

* Id map from homogneous Ids to per-type Ids.

* verify partitioned results.

* add test for distributed sampler.

* add mapping from per-type Ids to homogeneous Ids.

* update example.

* fix DistGraph.

* Revert "add profiler in rgcn."

This reverts commit 36daaed8b660933dac8f61a39faec3da2467d676.

* add tests for homogeneous graphs.

* fix a bug.

* fix test.

* fix for one partition.

* fix for standalone training and evaluation.

* small fix.

* fix two bugs.

* initialize projection matrix.

* small fix on RGCN.

* Fix rgcn performance (#17)
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal>

* fix lint.

* fix lint.

* fix lint.

* fix lint.

* fix lint.

* fix lint.

* fix.

* fix test.

* fix lint.

* test partitions.

* remove redundant test for partitioning.

* remove commented code.

* fix partition.

* fix tests.

* fix RGCN.

* fix test.

* fix test.

* fix test.

* fix.

* fix a bug.

* update dmlc-core.

* fix.

* fix rgcn.

* update readme.

* add comments.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal>

* fix.

* fix.

* add div_int.

* fix.

* fix.

* fix lint.

* fix.

* fix.

* fix.

* adjust.

* move code.

* handle heterograph.

* return pytorch tensor in GPB.

* remove some tests in example.

* add to_block for distributed training.

* use distributed to_block.

* remove unnecessary function in DistGraph.

* remove distributed to_block.

* use pytorch tensor.

* fix a bug in ntypes and etypes.

* enable norm.

* make the data loader compatible with the old format.

* fix.

* add comments.

* fix a bug.

* add test for heterograph.

* support partition without reshuffle.

* add test.

* support partition without reshuffle.

* fix.

* add test.

* fix bugs.

* fix lint.

* fix dataset.

* fix for mxnet.

* update docstring.

* rename to floor_div

* avoid exposing NodePartitionPolicy and EdgePartitionPolicy.

* fix docstring.

* fix error.

* fixes.

* fix comments.

* rename.

* rename.

* explain IdMap.

* fix docstring.

* fix docstring.

* update docstring.

* remove the code of returning heterograph.

* remove argument.

* fix example.

* make GraphPartitionBook an abstract class.

* fix.

* fix.

* fix a bug.

* fix a bug in example

* fix a bug

* reverse heterograph sampling.

* temp fix.

* fix lint.

* Revert "temp fix."

This reverts commit c450717b9f578b8c48769c675f2a19d6c1e64381.

* compute norm.

* Revert "reverse heterograph sampling."

This reverts commit bd6deb7f52998de76508f800441ff518e2fadcb9.

* fix.

* move id_map.py

* remove check

* add more comments.

* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-202.us-west-1.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-9-132.us-west-1.compute.internal>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-62-171.ec2.internal>

* add doc.

* update DistGraph.

* add DistTensor.

* update DistEmbedding.

* add partition.py

* add sampling.

* fix.

* add graph partition book and create a base class.

* fix test.

* add rst.

* update doc rst.

* update.

* fix.

* fix docs

* update distributed tensor and embeddings.

* add checks.

* update DistGraph.

* update initialization.

* fix graph partition book.

* update graph partition book.

* update partition.

* update partition.

* fix.

* add example code.

* update DistGraph

* Update python/dgl/distributed/dist_context.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Update python/dgl/distributed/dist_context.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Update python/dgl/distributed/dist_dataloader.py
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* update initialize.

* update dataloader.

* update distgraph.

* update DistGraph.

* update DistTensor.

* update.

* more updates.

* fix lint.

* add num_nodes and num_edges
Co-authored-by: Chao Ma <mctt90@gmail.com>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>

* Update graph

* Fix for dgl.graph

* from_scipy

* Replace canonical_etypes with relations

* from_networkx

* Update for hetero_from_relations

* Roll back the change of canonical_etypes to relations

* heterograph

* bipartite

* Update doc

* Fix lint

* Fix lint

* Fix test cases

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Update

* Fix test

* Fix

* Update

* Use DGLError

* Update

* Update

* Update

* Update

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Update

* Fix

* Update

* Fix

* Fix

* Fix

* Update

* Fix

* Update

* Fix

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Fix

* Fix

* Update

* Update

* Update

* Update

* Update

* Update

* rewrite sanity checks

* delete unnecessary checks

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Fix

* Update

* Update

* Update

* Fix

* Fix

* Fix

* Update

* Fix

* Update

* Fix

* Fix

* Update

* Fix

* Update

* Fix
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>

* fix.

* add tests.

* fix.

* fix.

* fix.

* doesn't support EdgeDataLoader

* add test.

* fix.

* fix test.

* catch errors in the test.

* fix.

* fix error message

* add print.

* join the server processes.

* update test.

* fix test.

* rename get_data_size.

* remove g from DistTensor.

* remove g from DistEmbedding.

* clean up API of graph partition book.

* fix DistGraph

* fix lint.

* collect all part policies.

* fix.

* fix.

* support distributed sampler.

* remove partition.py

* remove server_count from ip_config.txt

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* lint

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* Update dist_context.py

* fix lint.

* make it work for multiple spaces.

* update ip_config.txt.

* fix examples.

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* udpate

* update

* update

* update

* update

* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>

* fix tests.

* fix.

* remove a test.

* make code work in the standalone mode.

* fix example.

* more fix.

* make DistDataloader work with num_workers=0

* fix DistDataloader tests.

* fix.

* fix lint.

* fix cleanup.

* fix test

* remove unnecessary code.

* remove tests.

* fix.

* fix.

* fix.

* fix example

* fix.

* fix.

* fix launch script.
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>

* fix perf.

* fix.

* accelerate metis.

* fix lint.

* use gklib.

* fix perf.

* fix.

* update metis.

* update launch script

* handle synchronized API.

* fix.

* fix example.

* fix dataloader.

* temp fix.

* temp fix omp.

* distinguish roles.

* initialize iterator of DistDataloader correctly.

* check the correctness of launch script.

* move feature copy to sampler.

* measure mem/network copy time.

* remove

* Revert "measure mem/network copy time."

This reverts commit 86cefdc14b7815fcf5aad6496af912dba48e4aa6.

* fix.

* fix

* fix.

* fix cmake.

* disable metis in windows.

* disable metis tests in windows.

* remove test for multigraph.

* fix test.

* fix.

* fix cmake.

* fix.

* revert.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-115.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>

* distinguish roles.

* add comments.

* fix lint.

* move roles to server_state.

* fix text.

* fix tests.

* fix tests.

* Revert "fix tests."

This reverts commit 5baa136b872a4550d4e612bfb1dfe363d7814adf.

* fix dataloader.

* initialize iterator of DistDataloader correctly.

* update test.

* 111

* 111

* fix

* 111

* fix

* 11

* fix

* lint

* Update __init__.py

* lint

* fix

* lint

* fix

* fix

* fix

* fix

* fix

* try fix

* try fix

* fix

* Revert "fix"

This reverts commit a0b954fd4e99b7df92b53db8334dcb583d6e1551.

* fixes.

* fix.

* fix test.

* fix exit.

* fix.

* fix

* fix

* lint

* lint

* lint

* fix

* Update .gitignore

* 111

* fix

* 111

* 111

* fff

* 1111

* 111

* 1325315

* ffff

* f???

* fff

* 1111

* 111

* fix

* 111

* asda

* 1111

* 11

* 123

* 啊啊啊啊啊啊啊啊啊啊啊啊啊啊啊啊

* spawn

* 1231231

* up

* 111

* fix

* fix

* Revert "fix"

This reverts commit 7373f95312fdcaa36d2fc330bf242339e89c045d.

* fix

* fix

* 1111

* fix

* fix tests

* start kvclient as early as possible.

* lint

* fix test

* lint

* 1111

* fix

* fix

* 111

* fix

* fix

* 1

* fix

* fix

* lint

* fix

* lint

* lint

* remove quit

* fix

* lint

* fix

* fix several

* lint

* fix minor

* fix

* lint
Co-authored-by: Da Zheng <zhengda1936@gmail.com>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>

* client init graph on the backup servers.

* fix.

* test multi-server.

* fix anonymous dist tensors.

* check #parts.

* fix init_data

* add multi-server multi-client tests.

* update tests in kvstore.

* fix.

* verify the loaded partition.

* fix a bug.

* fix lint.

* fix.

* fix example.

* fix rpc.

* fix pull/push handler for backup kvstore

* fix example readme.

* change ip.

* update docstring.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>

* Standalone can run

* fix

* Add save

* Fix

* Fix

* Fix

* Fix

* debug

* test

* test

* Fix

* Fix

* log

* Fix

* fix

* Profile

* auto sync grad

* update

* add test for unsupervised dist training

* upd

* Fix lr

* Fix update

* sync

* fix

* Revert "fix"

This reverts commit d5caa7398b36125f6d6e2c742a95c6ff4298c9e9.

* Fix

* unsupervised

* Fix

* remove debug

* Add test case for dist_graph find_edges()

* Fix

* skip tensorflow test for find_edges

* Update readme

* remove some test

* upd

* Update partition_graph.py
Co-authored-by: Ubuntu <ubuntu@ip-172-31-68-185.ec2.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-24-210.ec2.internal>
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

* fix bugs.

* eval on both vaidation and testing.

* add script.

* update.

* update launch.

* make train_dist.py independent.

* update readme.

* update readme.

* update readme.

* update readme.

* generate undirected graph.

* rename conf_file to part_config

* use rsync

* make train_dist independent.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-115.us-west-2.compute.internal>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>

* update

* update

* fix lint

* update

* update

* udpate

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* update

* update

* update

* udpate

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

* fix tests in partition.

* fix DistGraph.

* fix without shared memory.

* fix sampling.

* enable distributed test.

* fix tests.

* fix a bug in shared-mem heterograph.

* print better error messages.

* fix.

* don't specify formats.

* fix.

* fix

* small fix.

This reverts commit 6557291f.

* 111

* 111

* fix

* 111

* fix

* 11

* fix

* lint

* Update __init__.py

* lint

* fix

* lint

* fix

* fix

* fix

* fix

* fix

* try fix

* try fix

* fix

* Revert "fix"

This reverts commit a0b954fd4e99b7df92b53db8334dcb583d6e1551.

* fixes.

* fix.

* fix test.

* fix exit.

* fix.

* fix

* fix

* lint

* lint

* lint

* fix

* Update .gitignore
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

* fix node/edge_split.

* fix partition.

* support heterograph interface.

* fix test.

* fix

* fix docstring.

* turn on/off recording in sparse embedding.

* add test.

* update.

* update Embedding.

* add comments.

* fix lint
Co-authored-by: Chao Ma <mctt90@gmail.com>

* metis for hetero

* fix partition

* add eval.

* extend DistTensor.

* fix.

* add barrier.

* add more print.

* add more checks in kvstore.

* fix lint.

* get all neighbors for eval.

* reorganize.

* fix.

* fix.

* fix.

* fix test.

* add reuse_if_exist.

* add test for reuse_if_exist.

* fix lint.

* fix bugs.

* fix.

* print errors of tcp socket.

* support delete tensors.

* fix lint.

* fix

* fix example
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-1.us-west-2.compute.internal>

* exit client

* update

* update

* update

* update

* update

* update

* update

* update test

* update

* update

* update

* update

* update

* update

* update

* update

* update

* delete tensor

* update

* update

* update

* update

* update

* update

* udpate

* update

* update

* update

* add standalone mode

* add comments.

* add tests for sampling.

* fix.

* make the code to run the standalone mode

* fix

* fix

* fix readme.

* fix.

* fix test
Co-authored-by: Chao Ma <mctt90@gmail.com>

* run dist server in dgl.

* fix bugs.

* fix example.

* check environment variables and fix lint.

* fix lint

* add in_subgraph on DistGraph.

* check in more.

* fix test.

* add comments.

* fix test.

* update test.

* update.

* rename.

* update comment

* fix test

* update

* update

* update

* fix

* fix.

* update.

* fix

* add assert
Co-authored-by: Chao Ma <mctt90@gmail.com>

* add sparse embedding.

* fix

* add test.

* man fixes.

* many fixes

* fix sparse emb.

* fix.

* fix lint.

* fix lint.

* fix kvstore.

* expose DistTensor.

* test sparse embeddings.

* add attach_grad to the backends.

* remove part_id

* fix.

* move backward computation.

* move more computation to backend.

* fix a bug when applying learning rate.

* fix a few things.

* fix a few things.

* add docstring

* fix.

* apply no_grad.

* fix tests.

* fix for other frameworks.

* add examples in docstring.

* Revert "Serialize server and client start for dist_graph_store test (#1736)"

This reverts commit da16ebf0.

* sleep

Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal>

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix.

* fix tests.

* fix

* add tests.

* fix.

* have default rank.

* add comment.

* fix test.

* remove check

* simplify code.

* add test.

* split data evenly.

* simplify the distributed training code.

* add comments.

* add comments.

* add num_clients to kvstore

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* update

* update

* add test

* update

* update

* update

* update

* fix test

* update

* update

* update

* update

* update

* update

* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

* optimize sampling neighbors from the local partition.

* fix minior bugs.

* fix lint

* overlap local sampling and remote sampling.

* fix.

* fix lint.

* fix a potential bug.
Co-authored-by: Chao Ma <mctt90@gmail.com>