Skip to content

GitLab

Switch branch/tag
  1. 06 Mar, 2023 1 commit
    • kylasa's avatar
      Support for no. of chunks smaller than no. of partitions. (#5390) · 894ad1e3
      kylasa authored 
      * Support for no. of chunks smaller than no. of partitions and Adding appropriate test cases.

Following changes are made with this PR.
1. Code changes for handling no. of chunks smaller than no. of partitions
2. Adding new test cases, which were previously deleted, for no. of chunks smaller than no. of partitions.
3. Also adding test cases, where multiple partitions are handled by a single process.

* Committing the missing files in this commit.

* lintrunner patch.

* lintrunner check

* lintrunner patch here.

* CI review comments.
      894ad1e3
  3. 25 Feb, 2023 1 commit
    • kylasa's avatar
      [DistDGL][Feature_Request]Changes in the metadata.json file for input graph dataset. (#5310) · a14f69c9
      kylasa authored 
      * Implemented the following changes.

* Remove NUM_NODES_PER_CHUNK
* Remove NUM_EDGES_PER_CHUNK
* Remove the dependency between no. of edge files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files and no. of edge files per edge type.
* Remove the dependency between no. of node feature files and no. of partitions
* Add “node_type_counts”. This will be a list of integers. Each integer will represent total count of a node-type. The index in this list and the index in the “node_type” will be the same for a given node-type.
* Add “edge_type_counts”. This will be a list of integers. Each integer will represent total count of an edge-type. The index in this list and the index in the “edge_type” list will be the same for a given edge-type.

* Applying lintrunner patch.

* Adding missing keys to the metadata in the unit test framework.

* lintrunner patch.

* Resolving CI test failures due to merge conflicts.

* Applying lintrunner patch

* applying lintrunner patch

* Replacing tabspace with spaces - to satisfy lintrunner

* Fixing the CI Test Failure cases.

* Applying lintrunner patch

* lintrunner complaining about a blank line.

* Resolving issues with print statement for NoneType

* Removed tests for the arbitrary chunks tests. Since this functionality is not supported anymore.

* Addressing CI review comments.

* addressing CI review comments

* lintrunner patch

* lintrunner patch.

* Addressing CI review comments.

* lintrunner patch.
      a14f69c9
  5. 23 Feb, 2023 1 commit
  7. 19 Feb, 2023 1 commit
  9. 05 Jan, 2023 1 commit
  11. 03 Jan, 2023 1 commit
  13. 15 Dec, 2022 1 commit
    • Rhett Ying's avatar
      [Dist] enable to chunk node/edge data into arbitrary number of chunks (#4930) · 9731e023
      Rhett Ying authored 
      

* [Dist] enable to chunk node/edge data into arbitrary number of chunks

* [Dist] enable to split node/edge data into arbitrary parts

* refine code

* Format boolean to uint8 forcely to avoid dist.scatter failure

* convert boolean to int8 before scatter and revert it after scatter

* refine code

* fix test

* refine code

* move test utilities into utils.py

* update comment

* fix empty data

* update

* update

* fix empty data issue

* release unnecessary mem

* release unnecessary mem

* release unnecessary mem

* release unnecessary mem

* release unnecessary mem

* remove unnecessary shuffle data

* separate array_split into standalone utility

* add example
Co-authored-by: default avatarxiang song(charlie.song) <classicxsong@gmail.com>
      9731e023
  15. 28 Nov, 2022 1 commit
  17. 18 Nov, 2022 1 commit
    • kylasa's avatar
      [Dist] Flexible pipeline - Initial commit (#4733) · c8ea9fa4
      kylasa authored 
      * Flexible pipeline - Initial commit

1. Implementation of flexible pipeline feature.
2. With this implementation, the pipeline now supports multiple partitions per process. And also assumes that num_partitions is always a multiple of num_processes.

* Update test_dist_part.py

* Code changes to address review comments

* Code refactoring of exchange_features function into two functions for better readability

* Upadting test_dist_part to fix merge issues with the master branch

* corrected variable names...

* Fixed code refactoring issues.

* Provide missing function arguments to exchange_feature function

* Providing the missing function argument to fix error.

* Provide missing function argument to 'get_shuffle_nids' function.

* Repositioned a variable within its scope.

* Removed tab space which is causing the indentation problem

* Fix issue with the CI test framework, which is the root cause for the failure of the CI tests.

1. Now we read files specific to the partition-id and store this data separately, identified by the local_part_id, in the local process.
2. Similarly as above, we also differentiate the node and edge features type_ids with the same keys as above.
3. These above two changes will help up to get the appropriate feature data during the feature exchange and send to the destination process correctly.

* Correct the parametrization for the CI unit test cases.

* Addressing Rui's code review comments.

* Addressing code review comments.
      c8ea9fa4
  19. 04 Nov, 2022 1 commit
  21. 27 Oct, 2022 1 commit
  23. 26 Oct, 2022 1 commit
  25. 19 Oct, 2022 1 commit
  27. 17 Oct, 2022 1 commit
    • Rhett Ying's avatar
      [Dist] Reduce peak memory in DistDGL (#4687) · b1309217
      Rhett Ying authored 
      * [Dist] Reduce peak memory in DistDGL: avoid validation, release memory once loaded

* remove orig_id from ndata/edata for partition_graph()

* delete orig_id from ndata/edata in dist part pipeline

* reduce dtype size and format before saving graphs

* fix lint

* ETYPE requires to be int32/64 for CSRSortByTag

* fix test failure

* refine
      b1309217
  29. 11 Oct, 2022 1 commit
  31. 03 Oct, 2022 2 commits
    • kylasa's avatar
      ParMETIS wrapper script to enable ParMETIS to process chunked dataset format (#4605) · eae6ce2a
      kylasa authored 
      * Creating ParMETIS wrapper script to run parmetis using one script from user perspective

* Addressed all the CI comments from PR https://github.com/dmlc/dgl/pull/4529

* Addressing CI comments.

* Isort, and black changes.

* Replaced python with python3

* Replaced single quote with double quotes per suggestion.

* Removed print statement

* Addressing CI Commets.

* Addressing CI review comments.

* Addressing CI comments as per chime discussion with Rui

* CI Comments, Black and isort changes

* Align with code refactoring, black, isort and code review comments.

* Addressing CI review comments, and fixing merge issues with the master branch

* Updated with proper unit test skip decorator
      eae6ce2a
    • kylasa's avatar
      Edge Feature support for input graph datasets for dist. graph partitioning pipeline (#4623) · 1f471396
      kylasa authored 
      * Added support for edge features.

* Added comments and removing unnecessary print statements.

* updated data_shuffle.py to remove compile error.

* Repaled python3 with python to match CI test framework.

* Removed unrelated files from the pull request.

* Isort changes.

* black changes on this file.

* Addressing CI review comments.

* Addressing CI comments.

* Removed duplicated and resolved merge conflict code.

* Addressing CI Comments from Rui.

* Addressing CI comments, and fixing merge issues.

* Addressing CI comments, code refactoring, isort and black
      1f471396
  33. 28 Sep, 2022 2 commits
  35. 20 Sep, 2022 1 commit
  37. 15 Sep, 2022 1 commit
  39. 22 Aug, 2022 1 commit
  41. 19 Aug, 2022 1 commit