GitLab

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

* Rehash code to optimize for loop

Reduced number of instructions in for loop, which exchanging edge features. This will reduce the number of times numpy's intersect1d is invoked (saving the runtime and memory overhead needs of numpy).

* Applying lintrunner patch to data_shuffle.py

* Edge Ownership processes are computed on the fly when required.

Earlier we were storing Edge ownership processes after the dataset was retrieved from the disk. For massively large datasets, each node can handle upto 5 Billion edges, this means storing owner process-ids will consume 5 * 8 = 40GB. This memory will be hanging around until the edges are exchanged.

To reduce the memory footprint of the pipeline, we no longer store the ownership process-ids in the 'edge_data' dictionary after reading the dataset from the disk. Instead, we compute them on the fly at the time of exchanging edges.

Another optimization is not to send/receive all the messages in a one single large message. Instead we now split the total number edges into chunks, limited by 8 GB per node. And we iterate until all the chunks are exchanged.

Once all the edges are exchanged, as a sanity check, we compute the total number of edges in the system and compare it with the original value before edge shuffling, in a final assert statement before return the result to the caller.

* Applying lintrunner patch.

Co-authored-by: songqing.zhang <songqing.zhang@shopee.com>

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

Co-authored-by: Ubuntu <ubuntu@ip-172-31-28-63.ap-northeast-1.compute.internal>

[bugfix] Fix assertions in /src/runtime/workspace.h and expand unit tests for sparse optimizer (#5299)

* Fix assertions for size 0 workspaces

* Expand unit test to cover case of communication

* Fixes

* Format

* Fix c++ formatting

* Set USE_LIBXSMM default to OFF.

* update

---------
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

* update cugraph_relgraphconv

* update equality test

* update cugraph rgcn example

* update RelGraphConvAgg based on latest API changes

* enable fallback option to fg when fanout is large

---------
Co-authored-by: Mufei Li <mufeili1996@gmail.com>

Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

* move example test from unit test to examples

* oops

* oops

* ok

* create_dev_conda_env

* fix

* polish

* nlabla

* cleanup

* more

* add_more

* polish

---------
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

Following changes are made in this PR.
1. In dataset_utils.py, when reading edges from disk we follow the order defined by the STR_EDGE_TYPE key in the metadata.json file. This order is implicitly used to assign edgeid to edge types. This same order is used to read edges from the disk as well.
2. Now the unit test framework will also randomize the order of edges read from the disk. This is done for the edges when reading from the disk for the unit tests.
Co-authored-by: Quan (Andy) Gan <coin2028@hotmail.com>

Co-authored-by: Mufei Li <mufeili1996@gmail.com>

Some weird formatting occurred in the documentation for EGATConv, so this quick commit should fix that.
Co-authored-by: Mufei Li <mufeili1996@gmail.com>

* enable sparse on windows and mac

* that was stupid

* let's see what's going on..

* [Sparse] Fix the import error on Mac OS.

When using template functions that are defined in source files from DGL,
the loader of MacOS somehow cannot find their definitions. This fix simply
avoids depending on template functions from DGL headers.

With this fix, the sparse tests all pass on the MAC environment.

* ok this is the problem

* make errors clearer

* uh

* test

* Update __init__.py

* disabling ddp on windows

---------
Co-authored-by: czkkkkkk <zekucai@gmail.com>

* Fix Column.record_stream(...) for unmaterialized tensors

* lint

* Record streams for indices on non-materialized columns

* add docstring

* fix for cpu index

* Record also stream on storage

* Always call self.storage.record_stream when storage is on GPU

* fix lint

---------
Co-authored-by: Xin Yao <xiny@nvidia.com>

Add Id hash map

* add SVD positional encoding

* modify importing module

* Fixed certain problems

* Change the test unit to a nonsigular one

* Fixed typo and make accord with lintrunner

* added svd_pe into dgl.rst

* Modified dgl.rst

* lintrunner patch for gloo_wrapper.py

* lintrunner changes to the tools directory.

* remove unnecessary clean up.

* more

---------
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

* [Dist] add input_dir for parmetis preprocess

* add support for parquet

* update parmetis_wrapper accordingly

* update pip install info.

* update

* clarify the version

* blabla

* blabl

---------
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

* remove_mock_sparse_example

* mock_sparse_test

* remove_mock_sparse

---------
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

* [Doc] Labor sampling improvement

When importance sampling option is used, the returned blocks include edge weights that need to be used during the message passing operation. So, I am including this in the documentation.

* Add documentation to weighted sampling case also

* swap-example

* size-shape
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

* part1

* more

* more

* spmatrix

* code
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>

* add_backslash

* fix

* more

* blabla
Co-authored-by: Steve <ubuntu@ip-172-31-34-29.ap-northeast-1.compute.internal>