GitLab

* add standalone mode

* add comments.

* add tests for sampling.

* fix.

* make the code to run the standalone mode

* fix

* fix

* fix readme.

* fix.

* fix test
Co-authored-by: Chao Ma <mctt90@gmail.com>

* run dist server in dgl.

* fix bugs.

* fix example.

* check environment variables and fix lint.

* fix lint

* add in_subgraph on DistGraph.

* check in more.

* fix test.

* add comments.

* fix test.

* update test.

* update.

* rename.

* update comment

* fix test

* update

* update

* update

* fix

* fix.

* update.

* fix

* add assert
Co-authored-by: Chao Ma <mctt90@gmail.com>

* add sparse embedding.

* fix

* add test.

* man fixes.

* many fixes

* fix sparse emb.

* fix.

* fix lint.

* fix lint.

* fix kvstore.

* expose DistTensor.

* test sparse embeddings.

* add attach_grad to the backends.

* remove part_id

* fix.

* move backward computation.

* move more computation to backend.

* fix a bug when applying learning rate.

* fix a few things.

* fix a few things.

* add docstring

* fix.

* apply no_grad.

* fix tests.

* fix for other frameworks.

* add examples in docstring.

* Revert "Serialize server and client start for dist_graph_store test (#1736)"

This reverts commit da16ebf0.

* sleep

Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal>

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix.

* fix tests.

* fix

* add tests.

* fix.

* have default rank.

* add comment.

* fix test.

* remove check

* simplify code.

* add test.

* split data evenly.

* simplify the distributed training code.

* add comments.

* add comments.

* add num_clients to kvstore

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* update

* update

* add test

* update

* update

* update

* update

* fix test

* update

* update

* update

* update

* update

* update

* update
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

* optimize sampling neighbors from the local partition.

* fix minior bugs.

* fix lint

* overlap local sampling and remote sampling.

* fix.

* fix lint.

* fix a potential bug.
Co-authored-by: Chao Ma <mctt90@gmail.com>

* Fix bugs in partitioning.

* add comment.

* fix lint.

* fix test.

* add more tests.

* simplify the fix

* fix.

* fix a bug.

* add msg for assert

* fix

* test1111

* 111

* 111

* fff

* lint

* 111

* lint

* lint

* 111

* fijx

* 111

* fix

* 111

* commit

* 111

* 111

* lint

* fix typo

* fix

* lint

* fix

* 111

* support mxnet

* support mxnet

* lint

* remove print

* fix

* fix test.

* fix test.

* fix test.

* try to fix an undetermistic error.
Co-authored-by: Chao Ma <mctt90@gmail.com>
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

* support no shared memory.

* add test.

* add CAPI to check existence of shared memory.

* revert the change in src/runtime/ndarray.cc

* update docstring.

* fix compile.

* fix a bug.

* fix test.

* simplify.

* fix test.

* test edge features.

* fix.

* add fast-pull

* update

* add fast-pull

* update

* update

* update

* update test

* update

* update

* update

* update

* update

* update

* update

* update

* add omp

* update

* update

* make pull/push handler per tensor.

* update.

* add reorder immutable graph.

* add python API.

* add reorder for csr.

* remove gk version.

* fix

* add cpp test.

* bug fixes

* fix tests.

* fix bugs and add check

* fix test.

* add omp.

* add comments.

* add coo reorder.

* fix a bug.

* handle reorder for different graph structues.

* fix lint.

* fix.

* add original ids.

* reshuffle nodes before metis partition.

* inner nodes are in contiguous Id range.

* reshuffle nodes/edges when partitioning.

* load partition return graph partition book.

* use inner_node/inner_edges

* add and test range partition book.

* count inner_edge correctly.

* fix lint.

* fix lint.

* fix lint.

* fix errors.

* fix errors.

* fix for TF.

* fix.

* fix.

* change docstring.

* support logical and.

* add comments.

* avoid copy.

* fix

* update docstring.

* fix a bug.

* add range search.

* fix

* fix a bug.

* add more tests.

* load graph partition book.

* support shared memory for range partition book.

* fix a bug.

* fix.

* fix lint.

* remove check

* fix test.

* remove num_nodes and num_edges

* fix lint.

* fix graph partition book.

* address comments.

* use makedirs.

* fix compile
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>
Co-authored-by: Chao Ma <mctt90@gmail.com>

* update

* update

* update

* update

* update

* update

* fix ci using spawn
Co-authored-by: aksnzhy <mctt90@gmail.com>

* update

* update

* update

* update

* update

* update

* update

* update

* refactor graph store for new kvstore.

* fix kvstore.

* fix.

* fix lint complains.

* add docstring.

* small fix in graph partition book.

* fix.

* remove tests

* disable sampling.

* Revert "disable sampling."

This reverts commit 70451008f61ff1481d5dadbf8bd199470aee559d.

* Revert "remove tests"

This reverts commit 1394364243bdd73b669abde6193a74e2cda5521d.

* remove freeze

* update

* update

* fix lint

* update test

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update init_data

* update server_state

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* debug init_data

* update

* update

* update

* update

* update

* update

* test get_meta_data

* update

* update

* update

* update

* update

* debug push

* update

* update

* update

* update

* update

* update

* update

* update

* update

* use F.reverse_data_type_dict

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* fix lint

* update

* fix lint

* update

* update

* update

* update

* fix test

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* set random seed

* update

* fix tests in graph partition book.

* implement node_split and edge_split.

* fix a bug.

* add tests.

* remove unnecessary code.

* avoid storing graph structure in partition book.

* add partition book in DistGraph.

* Revert "avoid storing graph structure in partition book."

This reverts commit 55b11fbf9293efcae5f8c97e93f1e121399dadae.

* small fixes.

* add a test for boolean mask vector.

* fix test.

* fix bugs.

* make it work for all different cases.

* fix tests.

* fix boolean mask

* fix for TF

* fix for tensorflow.

* fix test for TF

* only support boolean mask for now.

* fix tests.

* make the code more readable.

* fix test.
Co-authored-by: Chao Ma <mctt90@gmail.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>

* Add graph-partition-book

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* udpate

* fix lint

* update

* update

* update

* update

* update

* update

* update

* skip test on GPU

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* update

* update

* update

* fix unittest

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* add test.

* move test code.

* remvoe unnecessary test.

* fix.

* turn on tests for TF.

* Revert "move test code."

This reverts commit e7b4f36395b2121a7be030bd4364a704d0e357bf.

* fix.

* fix.

* skip test for tensorflow.
Co-authored-by: Chao Ma <mctt90@gmail.com>

* initial version from distributed training.

This is copied from multiprocessing training.

* modify for distributed training.

* it's runnable now.

* measure time in neighbor sampling.

* simplify neighbor sampling.

* fix a bug in distributed neighbor sampling.

* allow single-machine training.

* fix a bug.

* fix a bug.

* fix openmp.

* make some improvement.

* fix.

* add prepare in the sampler.

* prepare nodeflow async.

* fix a bug.

* get id.

* simplify the code.

* improve.

* fix partition.py

* fix the example.

* add more features.

* fix the example.

* allow one partition

* use distributed kvstore.

* do g2l map manually.

* fix commandline.

* a temp script to save reddit.

* fix pull_handler.

* add pytorch version.

* estimate the time for copying data.

* delete unused code.

* fix a bug.

* print id.

* fix a bug

* fix a bug

* fix a bug.

* remove redundent code.

* revert modify in sampler.

* fix temp script.

* remove pytorch version.

* fix.

* distributed training with pytorch.

* add distributed graph store.

* fix.

* add metis_partition_assignment.

* fix a few bugs in distributed graph store.

* fix test.

* fix bugs in distributed graph store.

* fix tests.

* remove code of defining DistGraphStore.

* fix partition.

* fix example.

* update run.sh.

* only read necessary node data.

* batching data fetch of multiple NodeFlows.

* simplify gcn.

* remove unnecessary code.

* use the new copy_from_kvstore.

* update training script.

* print time in graphsage.

* make distributed training runnable.

* use val_nid.

* fix train_sampling.

* add distributed training.

* add run.sh

* add more timing.

* fix a bug.

* save graph metadata when partition.

* create ndata and edata in distributed graph store.

* add timing in minibatch training of GraphSage.

* use pytorch distributed.

* add checks.

* fix a bug in global vs. local ids.

* remove fast pull

* fix a compile error.

* update and add new APIs.

* implement more methods in DistGraphStore.

* update more APIs.

* rename it to DistGraph.

* rename to DistTensor

* remove some unnecessary API.

* remove unnecessary files.

* revert changes in sampler.

* Revert "simplify gcn."

This reverts commit 0ed3a34ca714203a5b45240af71555d4227ce452.

* Revert "simplify neighbor sampling."

This reverts commit 551c72d20f05a029360ba97f312c7a7a578aacec.

* Revert "measure time in neighbor sampling."

This reverts commit 63ae80c7b402bb626e24acbbc8fdfe9fffd0bc64.

* Revert "add timing in minibatch training of GraphSage."

This reverts commit e59dc8957a414c7df5c316f51d78bce822bdef5e.

* Revert "fix train_sampling."

This reverts commit ea6aea9a4aabb8ba0ff63070aa51e7ca81536ad9.

* fix lint.

* add comments and small update.

* add more comments.

* add more unit tests and fix bugs.

* check the existence of shared-mem graph index.

* use new partitioned graph storage.

* fix bugs.

* print error in fast pull.

* fix lint

* fix a compile error.

* save absolute path after partitioning.

* small fixes in the example

* Revert "[kvstore] support any data type for init_data() (#1465)"

This reverts commit 87b6997b

.

* fix a bug.

* disable evaluation.

* Revert "Revert "[kvstore] support any data type for init_data() (#1465)""

This reverts commit f5b8039c6326eb73bad8287db3d30d93175e5bee.

* support set and init data.

* support set and init data.

* Revert "Revert "[kvstore] support any data type for init_data() (#1465)""

This reverts commit f5b8039c6326eb73bad8287db3d30d93175e5bee.

* fix bugs.

* fix unit test.

* move to dgl.distributed.

* fix lint.

* fix lint.

* remove local_nids.

* fix lint.

* fix test.

* remove train_dist.

* revert train_sampling.

* rename funcs.

* address comments.

* address comments.

Use NodeDataView/EdgeDataView to keep track of data.

* address comments.

* address comments.

* revert.

* save data with DGL serializer.

* use the right way of getting shape.

* fix lint.

* address comments.

* address comments.

* fix an error in mxnet.

* address comments.

* add edge_map.

* add more test and fix bugs.
Co-authored-by: Zheng <dzzhen@186590dc80ff.ant.amazon.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-6-131.us-east-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-26-167.us-east-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-150.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-250.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-135.us-west-2.compute.internal>

* simplify shared memory graph index.

* fix.

* remove edge_dir in SharedMemGraphStore.

* avoid creating shared-mem graph store with from_csr.

* simplify from_csr.

* add comments.

* fix lint.

* remove the test.

* fix compilation error.

* fix a bug.

* fix a bug.
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-150.us-west-2.compute.internal>

* Deprecate multi-graph

* Handle heterograph and edge_ids

* lint

* Fix

* Remove multigraph in C++ end

* Fix lint

* Add some test and fix something

* Fix

* Fix

* upd

* Fix some test case

* Fix

* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-51-214.ec2.internal>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

* upd

* fig edgebatch edges

* add test

* trigger

* Update README.md for pytorch PinSage example.

Add noting that the PinSage model example under
example/pytorch/recommendation only work with Python 3.6+
as its dataset loader depends on stanfordnlp package
which work only with Python 3.6+.

* Provid a frame agnostic API to test nn modules on both CPU and CUDA side.

1. make dgl.nn.xxx frame agnostic
2. make test.backend include dgl.nn modules
3. modify test_edge_softmax of test/mxnet/test_nn.py and
    test/pytorch/test_nn.py work on both CPU and GPU

* Fix style

* Delete unused code

* Make agnostic test only related to tests/backend

1. clear all agnostic related code in dgl.nn
2. make test_graph_conv agnostic to cpu/gpu

* Fix code style

* fix

* doc

* Make all test code under tests.mxnet/pytorch.test_nn.py
work on both CPU and GPU.

* Fix syntex

* Remove rand

* Add TAGCN nn.module and example

* Now tagcn can run on CPU.

* Add unitest for TGConv

* Fix style

* For pubmed dataset, using --lr=0.005 can achieve better acc

* Fix style

* Fix some descriptions

* trigger

* Fix doc

* Add nn.TGConv and example

* Fix bug

* Update data in mxnet.tagcn test acc.

* Fix some comments and code

* delete useless code

* Fix namming

* Fix bug

* Fix bug

* Add test for mxnet TAGCov

* Add test code for mxnet TAGCov

* Update some docs

* Fix some code

* Update docs dgl.nn.mxnet

* Update weight init

* Fix

* reproduce the bug

* Fix concurrency bug reported at #755.
Also make test_shared_mem_store.py more deterministic.

* Update test_shared_mem_store.py

* Update dmlc/core

* upd

* up

* upd

* upd

* copy graph index to shared memory.

* fix.

* fix.

* fix.

* use a diff name for in-csr and out-csr.

* fix lint.

* remove print.

* add test.

* add comments.

* fix graph store for Pytorch.

* add test.

* fix dtype error in test

* disable test on GPU.

* test avoid windows.

* fix shared-memory test.

* use script to control testing environment.

* update test.

* enable all tests.

* fix test script.