Skip to content

GitLab

Switch branch/tag
  1. 11 Aug, 2022 3 commits
    • Minjie Wang's avatar
      Merge branch 'master' into dist_part · e9b624fe
      Minjie Wang authored
      e9b624fe
    • kylasa's avatar
      Adding launch script and wrapper script to trigger distributed graph … (#4276) · 8086d1ed
      kylasa authored 
      

* Adding launch script and wrapper script to trigger distributed graph partitioning pipeline as defined in the UX document

1. dispatch_data.py is a wrapper script which builds the command and triggers the distributed partitioning pipeline
2. distgraphlaunch.py is the main python script which triggers the pipeline and to simplify its usage dispatch_data.py is included as a wrapper script around it.

* Added code to auto-detect python version and retrieve some parameters from the input metadata json file

1. Auto detect python version
2. Read the metadata json file and extract some parameters to pass to the user defined command which is used to trigger the pipeline.

* Updated the json file name to metadata.json file per UX documentation

1. Renamed json file name per UX documentation.

* address comments

* fix

* fix doc

* use unbuffered logging to cure anxiety

* cure more anxiety

* Update tools/dispatch_data.py
Co-authored-by: default avatarMinjie Wang <minjie.wang@nyu.edu>

* oops
Co-authored-by: default avatarQuan Gan <coin2028@hotmail.com>
Co-authored-by: default avatarMinjie Wang <minjie.wang@nyu.edu>
      8086d1ed
    • Quan (Andy) Gan's avatar
      [Distributed] Graph chunking UX (#4365) · 067cd744
      Quan (Andy) Gan authored 
      * first commit

* update

* huh

* fix

* update

* revert core

* fix

* update

* rewrite

* oops

* address comments

* add graph name

* address comments

* remove sample metadata file

* address comments

* fix

* remove

* add docs
      067cd744
  3. 10 Aug, 2022 4 commits
  5. 09 Aug, 2022 2 commits
  7. 07 Aug, 2022 2 commits
    • kylasa's avatar
      [Distributed] reduce memory consumption in distributed graph partitioning. (#4338) · 60bc0b76
      kylasa authored 
      * Fix for node_subgraph function, which seems to generate segmentation fault for very large partitions

1. Removed three graph dgl objects and we create the final dgl object directly by maintaining the following constraints
a) nodes are reordered so that local nodes are placed in the beginning of the nodes list compared to non-local nodes.
b)Edges order are maintained as passed into this function.
c) src/dst end points are mapped to target values based on the reshuffle'd nodes order.

* Code changes addressing CI comments for this PR

1. Used Da's suggested map to map nodes from old to new order.
This is much simpler and mem. efficient.

* Addressing CI Comments.

1. Reduced the amount of documentation to reflect the actual implementation.
2. named the mapping object appropriately.
      60bc0b76
    • Chang Liu's avatar
      [Bugfix] Fix the default value of `num_bases` in RelGraphConv module (#4321) · 5ba5106a
      Chang Liu authored 
      * Fix doc and default settings for RelGraphConv

* Add unit test

* Split msg in two lines to pass CI-lint
      5ba5106a
  9. 06 Aug, 2022 1 commit
    • kylasa's avatar
      [Distributed] use alltoall fix to bypass gloo - alltoallv bug in distributed partitioning (#4311) · c1e01b1d
      kylasa authored 
      * Alltoall Fix to bypass gloo - alltoallv bug which is preventing further testing

1. Replaced alltoallv gloo wrapper call with alltoall message.
2. All the messages are padded to be of same length
3. Receiving side unpads the messages and continues processing.

* Code changes to address CI comments

1. Removed unused functions from gloo_wrapper.py
2. Changed the function signature of alltoallv_cpu_data as suggested.
3. Added docstring to include more description of the functionality inside alltoallv_cpu_data. Included more asserts to validate the assumptions.

* Changed the function name appropriately

Changed the function name from "alltoallv_cpu_data" to alltoallv_cpu which I believe is appropriate because underlying functionality is providing alltoallv which is basically alltoall_cpu + padding

* Added code and text to address the review comments.

1. Changed the function name to indicate the local use of this function.
2. Changed docstring to indicate the assumptions made by alltoallv_cpu function.

* Removed unused function from import statement

Removed unused/removed function from import statement.
      c1e01b1d
  11. 03 Aug, 2022 1 commit
  13. 02 Aug, 2022 1 commit
  15. 01 Aug, 2022 3 commits
  17. 30 Jul, 2022 2 commits
  19. 29 Jul, 2022 1 commit
    • Xin Yao's avatar
      [Feature] Add CUDA Weighted Neighborhood Sampling (#4064) · 86c81b4e
      Xin Yao authored 
      

* add weighted sampling without replacement (A-Chao)

* improve Algorithm A-Chao with block-wise prefix sum

* correctly fill out_idxs

* implement weighted sampling with replacement

* small fix

* merge host-side code of weighted/uniform sampling

* enable unit tests for cuda weighted sampling

* move thrust/cub wrapper to the cmake file

* update docs accordingly

* fix linting

* fix linting

* fix unit test

* Bump external CUB/Thrust versions

* Fix code style and update description of algorithm design

* [Feature] GPU support weighted graph neighbor sampling
commit by pengqirong(OPPO)

* merge pengqirong's implementation

* revert the change to cub and thrust

* fix linting

* use DeviceSegmentedSort for better performance

* add more comments

* add necessary notes

* add necessary notes

* resolve some comments

* define THRUST_CUB_WRAPPED_NAMESPACE

* fix doc
Co-authored-by: default avatar彭齐荣 <657017034@qq.com>
      86c81b4e
  21. 28 Jul, 2022 2 commits
  23. 27 Jul, 2022 3 commits
  25. 26 Jul, 2022 2 commits
  27. 25 Jul, 2022 2 commits
  29. 23 Jul, 2022 1 commit
    • kylasa's avatar
      [Distributed] Change for the new input format for distributed partitioning (#4273) · 7f8e1cf2
      kylasa authored 
      * Code changes to address the updated file format support for massively large graphs.

1. Updated the docstring for the starting function 'gen_dist_partitions" to describe the newly proposed file format for input dataset.
2. Code which was dependent on the structure of the old-metadata json object has been updated to read from the newly proposed metadata file.
3. Fixed some errors when appropriate functions were invoked and the calling function expects return values from the invoked furnction.
4. This modified code has been tested on "mag" dataset using 4-way partitions and verified the results

* Code changes to address the CI review comments

1. Improved docstrings for some functions.
2. Added a new function in the utils.py to compute the id ranges and this is used in multiple places.

* Added TODO to indicate the redundant data structure.

Because of the new file format changes, one of the dictionaries (node_feature_tids, node_tids) will be redundant. Added TODO text so that this will be removed in the next iteration of code changes.
      7f8e1cf2
  31. 22 Jul, 2022 1 commit
  33. 21 Jul, 2022 2 commits
  35. 20 Jul, 2022 1 commit
  37. 19 Jul, 2022 3 commits
  39. 15 Jul, 2022 1 commit
  41. 14 Jul, 2022 2 commits