GitLab

[Performance][Feature] Add `src_nodes` paramter to `to_block()` to avoid cost running unique() when available. (#2973)

* Add lhs_nodes are paremeter to to_block

* Update unit test

* Switch to simplified node conversion

* Switch lhs_nodes to be in/out parameter

* Update docs
Co-authored-by: Da Zheng <zhengda1936@gmail.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

* rgcn with new heterograph API

* apply_edge() forward for multi relation

* undoing changes from rgcn-hetero

* backward apply_edge(copy_u) added

* unittest for apply_edge(copy_e)

* Compatible with new PRs

* resolving conflict with master

* Bringing back change after resolving conflict

* minor

* minor
Co-authored-by: Israt Nisa <nisisrat@amazon.com>

* Added binary builtinMsgFunc forward() for heterograph

* Added backward for u_op_v

* Supports all binary builtin forward

* Supports binary message funcs with reduce func sum

* lint check

* removed import torch from unittest

* enabled GPU test

* lint check

* Fixed docstrings

* rename func get_hs_id

* edited comment
Co-authored-by: Israt Nisa <nisisrat@amazon.com>

* remove

* remove

* fix

* remove

* remove

* Make column double indices lazy

* Copy indices to proper contexts

* Fix initialization

* Add unit test

* Fix unit test for tensorflow

* Remove unused member
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* Implement range based NDArrayPartition

* Finish implement range based partition support

* Add unit test

* Fix whitepace

* Add Kernel suffix

* Fix argument passing

* Add doxygen docs and improve variable naming

* Add unit test

* Add function for converting a partition book

* Add example to partition_op docs

* Fix dtype conversion for mxnet and tensorflow

* Update filter code

* Add unit tests

* Fixes

* Switch to indices

* Rename functions

* Fix linting

* Fix whitespace

* Add doc

* Fix heterograph

* Change workspace allocation

* Fix linting

* Fix docs in filter.py

* Add todo
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* fix.

* fix.

* fix.

* fix.

* Fix test

* Deprecate old DistEmbedding impl, use synchronized embedding impl

* Basic imple of heterogeneous on homogenenous sampling

* make pass

* Pass C++ test

* Add python test code

* lint

* lint

* Add MultiLayerEtypeNeighborSampler

* Add unitest for single machine dataloader

* Add dist dataloader test for edge type sampler

* Fix lint

* fix

* support for per etype sample

* Fix some bug and enable distributed training with per edge sample

* fix

* Now distributed training works

* turn off some mxnet

* turn off mxnet for some dist test

* fix

* upd

* upd according to the comments

* Fix

* Fix test and now distributed works.

* upd

* upd

* Fix

* Fix bug

* remove dead code.

* upd

* Fix

* upd

* Fix
Co-authored-by: Ubuntu <ubuntu@ip-172-31-71-112.ec2.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-2-66.ec2.internal>
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Fix lint

* Fix

* Update

* Fix test cases

* Fix

* add docstrings

* Update
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-17.us-west-2.compute.internal>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>

* [Feature] apply dgl.reorder() onto several node classification datasets in DGL

* rebase on latest dgl.reorder_graph()

* [Feature] enable edge reorder in dgl.reorder_graph()

* refine doc string

* refine doc string for dgl.reorder_graph

* refine doc string further

Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

[Feature] Add Heterograph support on Python for builtin unary msg functions (copy_u, copy_e) (#2989)

* heterograph for binary func

* Added SDDMM support

* Added unittest

* added binary test cases

* unary mfuncs works

* Fixed lint err

* lint check and others

* link check

* fixed import *_hetero issue

* lint check

* replace torch with dgl backend

* lint cehck

* removed torch from test

* skip mxnet unittest

* skip gpu test

* Remove unused/duplicated code

* minor

* changed data structure of ndata and edata

* link check

* reorganized

* minor lint

* minor lint

* raise error for udf func

* lint check

* fix for CUDA 10.1

* add a note for future cross-type max/min reducing

* Add support CUDA < 11

* lint check

* tidied C code

* remove dummy GSDDMM_hetero backward implementation
Co-authored-by: Israt Nisa <nisisrat@amazon.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Quan Gan <coin2028@hotmail.com>

* add docstring for missing API; fix some docstring

* rename apis; address comments

* Add dgl.utils.is_sorted_srcdst

* Fix linting issues

* delete blank line

* Specify datatype to index tensor in test

* Force integer conversion
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

* [Feature] Add dgl.reorder() to re-order graph according to specified strategy

* fix unit test failure for metis reorder

* fix unit test failure on mxnet_cpu

* refine unit test for dgl.reorder

* fix unit test failure on mxnet

* fix array_equal error for mxnet unit test

* fix unit test failure for mxnet

* convert metis output to numpy array explicitly
Co-authored-by: Tong He <hetong007@gmail.com>

* csr and csc creation

* fix

* fix

* fixes to adj transpose

* fine

* raise error if indptr did not match number of nodes

* fix

* huh?

* oh
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

* update

* update

* update

* update

* lint

* lint

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* lint

* update

* clone

* update

* update

* update

* update

* replace idarray with ndarray

* refactor cpp part

* refactor python part

* debug

* refactor interface

* test and doc

* lint and test

* lint

* fix

* fix

* fix

* const

* doc

* fix

* fix

* fix

* fix

* fix & doc

* fix

* fix

* update

* update

* update

* merge

* doc

* doc

* lint

* fix

* more tests

* doc

* fix

* fix

* update

* update

* update

* fix

* fix
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

* add seal example

* 1. add paper infomation in examples/README
2. adjust codes
3. option test

* use latest `to_simple` to replace coalesce graph function

* remove outdated codes

* remove useless comment

* Node2vec
1.implement node2vec random walk c++ op
2.implement node2vec model
3.implement node2vec example

* add CMakeLists file modify

* refine c++ codes

* refine c++ codes

* add missing whitespace

* refine python codes

* add codes

* add node2vec_impl.h

* fix codes

* fix code style problem

* fixes

* remove

* lots of changes

* add benchmark

* fixes
Co-authored-by: smilexuhc <smile.xuhc@gmail.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Fix

* Update

* Fix subgraph tests

* Capture stdout for distributed test

* Capture stdout for distributed test

* Update

* Update

* Update

* Update subgraph.cc
Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-17.us-west-2.compute.internal>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>

* [Feature] add positive negative statistics

* [Feature] add permission information and fix import problem

* fix backend incompatible problem

* modify random split to remove sklearn usage

* modify file read to remove pandas usage

* add datasets into doc

* add random seed in data splitting

* add dataset unit test

* usage permission information update
Co-authored-by: zhjwy9343 <6593865@qq.com>

* fix #2870

* lint

* fix

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Update

* Update

* Add files via upload

* Add files via upload

* Update

* Lint

* Add files via upload

* Lint

* Update

* Update

* Update

* Update

* Update

* Lint Fix

* Lint
Co-authored-by: Ubuntu <ubuntu@ip-172-31-12-161.us-west-2.compute.internal>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>

* make to_heterogeneous and to_homogeneous invertible

* docstring

* oops
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>

* update

* update

* update

* update

* lint

* lint

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* lint

* update

* clone

* update

* update

* update

* update

* replace idarray with ndarray

* refactor cpp part

* refactor python part

* debug

* refactor interface

* test and doc

* lint and test

* lint

* fix

* fix

* fix

* const

* doc

* fix

* fix

* fix

* fix

* fix & doc

* fix

* fix

* fix

* fix

* fix

* fix

* update
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

* Fix #2808

* fix

* fix

* fix

* fix

* fix

* fix
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

[Feature][Performance] Implement NCCL wrapper for communicating NodeEmbeddings and sparse gradients. (#2825)

* Split NCCL wrapper from sparse optimizer and sparse embedding

* Add more unit tests for single node nccl

* Fix unit test for tf

* Switch to device histogram

* Fix histgram issues

* Finish migration to histogram

* Handle cases with zero send/recieve data

* Start on partition object

* Get compiling

* Updates

* Add unit tests

* Switch to partition object

* Fix linting issues

* Rename partition file

* Add python doc

* Fix python assert and finish doxygen comments

* Remove stubs for range based partition to satisfy pylint

* Wrap unit test in GPU only

* Wrap explicit cuda call in ifdef

* Merge with partition.py

* update docstrings

* Cleanup partition_op

* Add Workspace object

* Switch to using workspace object

* Move last remainder based function out of nccl_api

* Add error messages

* Update docs with examples

* Fix linting erros
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>

* add distributed in-degree and out-degree.

* update comments.

* fix a bug.

* add tests.

* add tests.

* fix a bug.

* fix docstring.

* update doc.

* fix

* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>

* test commit

* fixes

* oops

* add docs

* lint

* why does it say I have a trailing whitespace

* oh ok

* fixes

* why there's an invalid argument error

* address comments

* fix

* address comments

* Update heterograph.py

* remove unit tests

* replace tutorial

* init cuda support

* cuSPARSE err

* passed unittest for csr_mm/SpGEMM. int64 not supported

* Debugging cuSPARSE error 3

* csrgeam only supports int32?

* disabling int64 for cuda

* refactor and add CSRMask

* lint

* oops

* remove todo

* rewrite CSRMask with CSRGetData

* lint

* fix test

* address comments

* lint

* fix

* addresses comments and rename BUG_ON
Co-authored-by: Israt Nisa <nisisrat@amazon.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-30-71.ec2.internal>
Co-authored-by: Quan Gan <coin2028@hotmail.com>
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>

* return mapping.

* support heterogeneous graph.

* more test.

* fix lint.

* fix for diff backends.

* fix.

* fix.
Co-authored-by: Zheng <dzzhen@3c22fba32af5.ant.amazon.com>
Co-authored-by: xiang song(charlie.song) <classicxsong@gmail.com>

* Start on uniform GPU sampling

* Save more work

* Get cu file compiling

* Update sampling

* More changes

* Get GPU sampling for uniform probabilities solved

* Fix batch tensor migration

* Fix

* update kernels

* expand blocking

* Undo testing change

* Cut down on sampling overhead

* Fix replacement

* Update unit tests

* Add option to gpu sample in graphsage

* Copy only csc to gpu

* Add ogbn support

* Fix linting

* Remove nvtx from sample

* Improve documentation and error checking

* Expand documentation

* Update assert checking

* delete extra space

* Use standard dataloader when dataset is a dictionary

* ogb -> ogbn

* Fix edge selection determinism

* Fix typos

* Remove nvtx

* Add comment for self.fanout_arrays and assert

* Fix linting

* Migrate to scalarbatcher

* Fix indentation

* Fix batcher

* Fix indexing

* Only use databatcher for GPU

* Convert to DGL NDArray to PyTorch Tensor

* Add optimization for PyTorch's F.tensor() for list of GPU tensors
Co-authored-by: Da Zheng <zhengda1936@gmail.com>

* upd

* fix

* upd

* upd

* upd

* upd

* upd

* upd

* upd

* upd

* upd

* fix

* refactor

* upd test

* large feat_len or n in segment reduce

* lint

* Add start of to_block gpu implementation

* Pull in more changes from 0.4.2 cuda_to_block

* Move more code to IdArray

* Refactor DeviceNodeMapMaker

* Updates

* get compiling

* Integrate to_block

* Fix ID allocation

* Minor fixes

* Cleanup cuda calls to use cuda_common

* Reduce kernel calls

* Lint cleanup

* Expand documentation

* Remove unused function

* Rename variables for consistency

* Add doxygen comments

* Fix file extension

* Remove raw asynccopy for deviceapi

* Remove unused function

* Fix block/tile configuration

* Add cuda_device_common.cuh

* Add basic hashtable

* Migrate part of hashtable

* Refactor to use external hashtable

* Make functions members

* Format hash table functions

* Migrate duplicate filling

* Move last function over

* Refactor with cu file

* lint c++ code

* Move context check to C++ code

* Use macro switch

* Add missing files

* Update docstring

* update docs

* Move atomic functions

* Refactor hashtable

* Fix linting

* Expand docs

* Fix mismatched argument names

* Switch doxygen comments from using @param to \param
Co-authored-by: Jinjing Zhou <VoVAllen@users.noreply.github.com>
Co-authored-by: Minjie Wang <wmjlyjemaine@gmail.com>