Commits · 901e0c24d699217a6e442f1ef5001b70a65649e5 · OpenDAS / dgl

27 May, 2020 1 commit

[Example][Optimization] Performance optimization for graphsage unsupervised example (#1531) · 901e0c24

xiang song(charlie.song) authored May 27, 2020



* test

* profile

* opt

* Some fix

* upd

* upd

* Add multigpu training support for graphsage unsupervised

* Add share neg

* Fix

* Add profile

* turn on eval

* upd

* Fix

* performance opt
Co-authored-by: Ubuntu <ubuntu@ip-172-31-12-103.ec2.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-14-53.ec2.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-19-29.ec2.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-50.ec2.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-52-181.ec2.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-87-240.ec2.internal>

901e0c24

26 May, 2020 1 commit

[Fix] New StreamWithBuffer interface (#1557) · 7639b5e7

Jinjing Zhou authored May 27, 2020



* WIP: rpc components

* client & server

* move network package to rpc

* fix include

* fix compile

* c api

* wip: test

* add basic tests

* missing file

* [RPC] Zero copy serializer (#1517)

* zerocopy serialization

* add test for HeteroGraph

* fix lint

* remove unnecessary codes

* add comment

* lint

* lint

* disable pylint for now

* add include for win

* windows guard

* lint

* lint

* skip test on windows

* refactor

* add comment

* fix

* comment

* 1111

* fix

* Update Jenkinsfile

* [RPC] Implementation of RPC infra (#1544)

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* remove client.cc and server.cc

* fix lint

* update

* update

* fix linr

* update

* fix lint

* update

* update

* update

* update

* update

* update

* update test

* update

* update test

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update comment

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix lint

* fix lint

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* Refactor StreamWithBuffer (#1550)

* refactor

* fix with new interface

* remove copy

* fix

* remove comment
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Chao Ma <mctt90@gmail.com>

7639b5e7

25 May, 2020 2 commits
- [Readme] Add cuda 10.2 to docs (#1560) · cc07bd44
  Jinjing Zhou authored May 25, 2020
```
* add cuda 10.2

* type
```
  cc07bd44
- [RPC] Init server_state during get_server_state() (#1559) · 6a363378
  Chao Ma authored May 25, 2020
```
* init server_state

* update

* update
```
  6a363378
24 May, 2020 1 commit

[DGL-LifeSci] Pre-trained GIN (#1558) · 165c67cc

Mufei Li authored May 25, 2020

* Update

* Fix

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

165c67cc

22 May, 2020 1 commit

[RPC] New RPC infrastructure. (#1549) · 3e696922

Chao Ma authored May 22, 2020



* WIP: rpc components

* client & server

* move network package to rpc

* fix include

* fix compile

* c api

* wip: test

* add basic tests

* missing file

* [RPC] Zero copy serializer (#1517)

* zerocopy serialization

* add test for HeteroGraph

* fix lint

* remove unnecessary codes

* add comment

* lint

* lint

* disable pylint for now

* add include for win

* windows guard

* lint

* lint

* skip test on windows

* refactor

* add comment

* fix

* comment

* 1111

* fix

* Update Jenkinsfile

* [RPC] Implementation of RPC infra (#1544)

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* remove client.cc and server.cc

* fix lint

* update

* update

* fix linr

* update

* fix lint

* update

* update

* update

* update

* update

* update

* update test

* update

* update test

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update comment

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix lint

* fix lint

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>

3e696922

19 May, 2020 2 commits
- [Docs] Fix docstrings and documentations (#1420) · 7ba9cbc6
  Quan (Andy) Gan authored May 19, 2020
```
* fix docs of to_block

* fix

* oops

* more fixes
```
  7ba9cbc6
- [CI] Give Windows more time to run (#1541) · e3793a5c
  Quan (Andy) Gan authored May 19, 2020
  
  e3793a5c
18 May, 2020 1 commit

[Performance] Reduces OpenMP atomic additions with zero inputs (#1527) · a296c1ef

pawelpiotrowicz authored May 18, 2020



* cirtical performance fix - reduce atomic

 * training faster up to 50%

* leave a TODO
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

a296c1ef

17 May, 2020 2 commits
- [Bug][Model] Bugfix in PinSAGE code (#1533) · 0b902d03
  Quan (Andy) Gan authored May 18, 2020
  
  0b902d03
- Update (#1534) · 7b6af87a
  Mufei Li authored May 17, 2020
  
  7b6af87a
16 May, 2020 1 commit

[DGL-LifeSci] WLN for Reaction Prediction (#1530) · 039fefc2

Mufei Li authored May 16, 2020

* Update

* Update

* Update

* Update

* Update

* Fix bug

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* UPdate

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Udpate

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Finalize

039fefc2

15 May, 2020 2 commits
- [Feature] Add a request_format api and enrich related docstring. (#1528) · 70dc2ee9
  Zihao Ye authored May 16, 2020
```
* upd

* upd

* lint

* fix

* bloody lint
```
  70dc2ee9
- removing old pinsage implementation (#1525) · ccd4eab0
  Quan (Andy) Gan authored May 15, 2020
  
  ccd4eab0
14 May, 2020 5 commits

[Feature] Split data for distributed training (#1505) · 96984fac

Da Zheng authored May 14, 2020



* fix tests in graph partition book.

* implement node_split and edge_split.

* fix a bug.

* add tests.

* remove unnecessary code.

* avoid storing graph structure in partition book.

* add partition book in DistGraph.

* Revert "avoid storing graph structure in partition book."

This reverts commit 55b11fbf9293efcae5f8c97e93f1e121399dadae.

* small fixes.

* add a test for boolean mask vector.

* fix test.

* fix bugs.

* make it work for all different cases.

* fix tests.

* fix boolean mask

* fix for TF

* fix for tensorflow.

* fix test for TF

* only support boolean mask for now.

* fix tests.

* make the code more readable.

* fix test.
Co-authored-by: Chao Ma <mctt90@gmail.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>

96984fac

[hotfix] Several bug fix for remove nodes/edges in DGLGraph. (#1521) · cfb24790
Zihao Ye authored May 15, 2020
```
* upd

* upd

* better
```
cfb24790

[Bug?] Fixing compatibility of pickles across DGL versions (#1507) · 20ec7bb0

Quan (Andy) Gan authored May 15, 2020



* pickle compatibility before 0.4.2

* fixing for utils.Index

* add TODOs

* fix

* fix

* more compatibility checks for DGLGraph and DGLHeteroGraph objects

* lint
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

20ec7bb0

[Refactor] Restrict sparse format for DGLHeteroGraph (#1474) · c2e61ce1

Zihao Ye authored May 14, 2020



* upd

* simplify

* further simplify

* lint

* doc

* upd

* upd

* upd

* upd

* upd

* upd

* upd

* lint

* rename format

* upd

* lint

* upd

* upd

* upd

* udp

* debug

* upd

* upd

* upd

* upd

* upd

* 无可厚非吧

* 中三边肥

* 你一定要喊吗

* lint

* upd

* upd

* upd

* upd

* upd

* upd

* upd

* Update unit_graph.h
Co-authored-by: Zihao Ye <yzh119@192.168.0.110>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

c2e61ce1

[Model] PinSAGE model with new sampler (#1334) · 1e84168e

Quan (Andy) Gan authored May 14, 2020



* another first commit

* test commit

* porting to nowplaying

* fixes

* more updates

* update readme

* add performance

* update

* switch to return_uv

* update result

* Update examples/pytorch/pinsage/process_nowplaying_rs.py
Co-authored-by: Xiagkun Hu <huxk_hit@qq.com>

* update with comments
Co-authored-by: Xiagkun Hu <huxk_hit@qq.com>

1e84168e

13 May, 2020 2 commits
- [Doc] write warning for remove_edges. (#1509) · 59aefe6e
  Da Zheng authored May 13, 2020
```
* write warning for remove_edges.

* fix.
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
```
  59aefe6e
- [Performance] fixing performance on heterogeneous graph creation in specific cases (#1506) · e5b10b10
  Quan (Andy) Gan authored May 13, 2020
```
* [Performance] fixing performance on heterogeneous graph creation in specific cases

* fix for mxnet
```
  e5b10b10
12 May, 2020 2 commits

[Model] Spatial-temporal Graph Neural Networks for Traffic Prediction (#1445) · 9b34b1c2

jianmohuo authored May 12, 2020



* stgcn_wave model

* fix readme

* rm data file

* split sensors2graph

* rm dead code

* fix README

* rename class

* rm seed & dead code

* Update README.md

* rm dead code & networkx

* add num_layer papram, make model structure adjustable

* fix

* add model structure controller string, make code easier to understand and make model strcture more flexible

* Update main.py

* Update model.py

* fix

* Update README.md
Co-authored-by: Ubuntu <ubuntu@ip-172-31-14-255.ap-northeast-1.compute.internal>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

9b34b1c2

[Model] OGB products submission (#1512) · ed60f401
Quan (Andy) Gan authored May 12, 2020

ed60f401

11 May, 2020 1 commit
- [Backend] Turn to official dlpack for Tensorflow (#1511) · 6d048853
  Jinjing Zhou authored May 11, 2020
```
* Turn to official dlpack

* fix

* fix
```
  6d048853
09 May, 2020 1 commit

Fix bug in GIN Dataset degree OHE logic (#1498) · 02850212

Mohammad Amin Khashkhashi Moghaddam authored May 09, 2020

In the previous code, the same vector was being assigned to all the nodes in each graph.
`:` was used incorrectly assuming that only one row will be selected for each column but all rows were selected for each column.

02850212

08 May, 2020 4 commits

[bugfix] Fix the behavior of dgl.batch when batch size is 1 (#1483) · 1af0d806
Zihao Ye authored May 08, 2020
```
* fix

* upd

* upd

* upd

* upd

* clone

* upd
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
```
1af0d806

[Distributed] Add GraphPartitionBook (#1496) · ef7e4750

Chao Ma authored May 08, 2020

* Add graph-partition-book

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* udpate

* fix lint

* update

* update

* update

* update

* update

* update

* update

* skip test on GPU

* update

* update

* update

* update

* update

* update

* update

* fix lint

* update

* update

* update

* update

* fix unittest

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

ef7e4750

[Refactor] Explicit dtype for HeteroGraph (#1467) · dc8ca88e

Jinjing Zhou authored May 08, 2020



* 111

* 111

* lint

* lint

* lint

* lint

* fix

* lint

* try

* fix

* lint

* lint

* test

* fix

* ttt

* test

* fix

* fix

* fix

* mxnet

* 111

* fix 64bits computation

* pylint

* roll back

* fix

* lint

* fix hetero_from_relations

* remove index_dtype in to_homo and to_hetero

* fix

* fix

* fix

* fix

* remove default

* fix

* lint

* fix

* fix error message

* fix error

* lint

* macro dispatch

* try

* lint

* remove nbits

* error message

* fix

* fix

* lint

* lint

* lint

* fix

* lint

* fix

* fix random walk

* lint

* lint

* fix

* fix

* fix

* lint

* fix

* lint
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

dc8ca88e

fix metapath2vec on custom datasets (#1499) · de34e15a
Quan (Andy) Gan authored May 08, 2020

de34e15a

06 May, 2020 2 commits

[Test] Add tests for TensorFlow (#1501) · 16561a2e

Da Zheng authored May 06, 2020



* add test.

* move test code.

* remvoe unnecessary test.

* fix.

* turn on tests for TF.

* Revert "move test code."

This reverts commit e7b4f36395b2121a7be030bd4364a704d0e357bf.

* fix.

* fix.

* skip test for tensorflow.
Co-authored-by: Chao Ma <mctt90@gmail.com>

16561a2e

[Model] GATNE-T (#1470) · 6ae440db

Linfang He authored May 06, 2020



* Add an example for GATNE-T

* Update README.md

* Add links for datasets

* Update README.md with running results

* Update README.md

* Add comments

* Update main.py

* Remove node type name `user`
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

6ae440db

03 May, 2020 1 commit

[Feature] Distributed graph store (#1383) · 2190c39d

Da Zheng authored May 02, 2020

* initial version from distributed training.

This is copied from multiprocessing training.

* modify for distributed training.

* it's runnable now.

* measure time in neighbor sampling.

* simplify neighbor sampling.

* fix a bug in distributed neighbor sampling.

* allow single-machine training.

* fix a bug.

* fix a bug.

* fix openmp.

* make some improvement.

* fix.

* add prepare in the sampler.

* prepare nodeflow async.

* fix a bug.

* get id.

* simplify the code.

* improve.

* fix partition.py

* fix the example.

* add more features.

* fix the example.

* allow one partition

* use distributed kvstore.

* do g2l map manually.

* fix commandline.

* a temp script to save reddit.

* fix pull_handler.

* add pytorch version.

* estimate the time for copying data.

* delete unused code.

* fix a bug.

* print id.

* fix a bug

* fix a bug

* fix a bug.

* remove redundent code.

* revert modify in sampler.

* fix temp script.

* remove pytorch version.

* fix.

* distributed training with pytorch.

* add distributed graph store.

* fix.

* add metis_partition_assignment.

* fix a few bugs in distributed graph store.

* fix test.

* fix bugs in distributed graph store.

* fix tests.

* remove code of defining DistGraphStore.

* fix partition.

* fix example.

* update run.sh.

* only read necessary node data.

* batching data fetch of multiple NodeFlows.

* simplify gcn.

* remove unnecessary code.

* use the new copy_from_kvstore.

* update training script.

* print time in graphsage.

* make distributed training runnable.

* use val_nid.

* fix train_sampling.

* add distributed training.

* add run.sh

* add more timing.

* fix a bug.

* save graph metadata when partition.

* create ndata and edata in distributed graph store.

* add timing in minibatch training of GraphSage.

* use pytorch distributed.

* add checks.

* fix a bug in global vs. local ids.

* remove fast pull

* fix a compile error.

* update and add new APIs.

* implement more methods in DistGraphStore.

* update more APIs.

* rename it to DistGraph.

* rename to DistTensor

* remove some unnecessary API.

* remove unnecessary files.

* revert changes in sampler.

* Revert "simplify gcn."

This reverts commit 0ed3a34ca714203a5b45240af71555d4227ce452.

* Revert "simplify neighbor sampling."

This reverts commit 551c72d20f05a029360ba97f312c7a7a578aacec.

* Revert "measure time in neighbor sampling."

This reverts commit 63ae80c7b402bb626e24acbbc8fdfe9fffd0bc64.

* Revert "add timing in minibatch training of GraphSage."

This reverts commit e59dc8957a414c7df5c316f51d78bce822bdef5e.

* Revert "fix train_sampling."

This reverts commit ea6aea9a4aabb8ba0ff63070aa51e7ca81536ad9.

* fix lint.

* add comments and small update.

* add more comments.

* add more unit tests and fix bugs.

* check the existence of shared-mem graph index.

* use new partitioned graph storage.

* fix bugs.

* print error in fast pull.

* fix lint

* fix a compile error.

* save absolute path after partitioning.

* small fixes in the example

* Revert "[kvstore] support any data type for init_data() (#1465)"

This reverts commit 87b6997b

.

* fix a bug.

* disable evaluation.

* Revert "Revert "[kvstore] support any data type for init_data() (#1465)""

This reverts commit f5b8039c6326eb73bad8287db3d30d93175e5bee.

* support set and init data.

* support set and init data.

* Revert "Revert "[kvstore] support any data type for init_data() (#1465)""

This reverts commit f5b8039c6326eb73bad8287db3d30d93175e5bee.

* fix bugs.

* fix unit test.

* move to dgl.distributed.

* fix lint.

* fix lint.

* remove local_nids.

* fix lint.

* fix test.

* remove train_dist.

* revert train_sampling.

* rename funcs.

* address comments.

* address comments.

Use NodeDataView/EdgeDataView to keep track of data.

* address comments.

* address comments.

* revert.

* save data with DGL serializer.

* use the right way of getting shape.

* fix lint.

* address comments.

* address comments.

* fix an error in mxnet.

* address comments.

* add edge_map.

* add more test and fix bugs.
Co-authored-by: Zheng <dzzhen@186590dc80ff.ant.amazon.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-6-131.us-east-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-26-167.us-east-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-150.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-250.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-135.us-west-2.compute.internal>

2190c39d

02 May, 2020 2 commits

[KVStore] Fix bug of get_meta() API (#1491) · 5fc334fc

Chao Ma authored May 02, 2020

* fix get-shape

* update

* update

* update

* update

* update

* fix typo

* update

* update

5fc334fc

[Fix] Fix stream creation and empty dict in tensor serialization (#1489) · 30b8074a

Jinjing Zhou authored May 02, 2020

* add functions

* fix litn

* add unit test

* fix

* fix

* fix

* fix

* support empty dict

* simplify logic

* Update tensor_serialize.py

30b8074a

01 May, 2020 1 commit

[Hetero] Improve speed of several Hetero APIs (#1486) · f25bc176

Minjie Wang authored May 01, 2020

* add clone function to frame

* add utest

* replace all local_var with local_scope

* fix utest

* avoid creating canonical types in __getitem__

* lint

* try another utest  appraoch for mx

* utest

f25bc176

29 Apr, 2020 2 commits

[Bugfix] Add bool data type to backend. (#1487) · 3c4506e9
Minjie Wang authored Apr 29, 2020
```
* add bool to F.data_type_dict

* add utest

* skip bool test for mx
```
3c4506e9

Fix bug in using degree_as_nlabel for GIN dataset (#1451) · f1e4f378

Mohammad Amin Khashkhashi Moghaddam authored Apr 29, 2020



In order to make the `label2idx` dictionary, tensor values should be used and not the tensor itself.
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>

f1e4f378

28 Apr, 2020 2 commits

[Model] Shared memory history tensor for multi-gpu training of control variate methods (#1479) · 05b0f1ea
Quan (Andy) Gan authored Apr 28, 2020
```
* shared memory history for multi-gpu training

* reorg
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
```
05b0f1ea

[Model] Unsupervised learning with GraphSAGE (#1440) · dc0432e7

Quan (Andy) Gan authored Apr 28, 2020



* unsupervised graphsage first commit

* fix

* disable remove_edges and still got 0.90 performance

* optimize edgeids with multimap

* change hyperparams

* update README
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

dc0432e7

27 Apr, 2020 1 commit
- [Data] Add utils to save dict of tensors (#1481) · 1506560e
  Jinjing Zhou authored Apr 28, 2020
```
* add functions

* fix litn

* add unit test

* fix

* fix
```
  1506560e