Skip to content

GitLab

Switch branch/tag
  1. 08 Mar, 2023 1 commit
    • Xin Yao's avatar
      [Refactor] Replace third_party/nccl with PyTorch's NCCL backend (#4989) · 8d5d8962
      Xin Yao authored 
      * expose GeneratePermutation

* add sparse_all_to_all_push

* add sparse_all_to_all_pull

* add unit test

* handle world_size=1

* remove python nccl wrapper

* remove the nccl dependency

* use pinned memory to speedup D2H copy

* fix lint

* resolve comments

* fix lint

* fix ut

* resolve comments
      8d5d8962
  3. 06 Mar, 2023 3 commits
    • kylasa's avatar
      [DistDGL][UserEx]Sync parmetis_wrapper with changes in metadata.json (#5385) · 7b766393
      kylasa authored 
      * Sync parmetis_wrapper with changes in metadata.json

1. In the preprocess.py, make sure that num_partitions is defined as input argument. Also, align 'input_dir' with the input dataset. schema_file is assumed to be located inside the input_dir. Also, graph_stats.txt file is assumed to be present in the input_dir.

2. Use DGL_HOME environment variable so that parmetis_wrapper command can be run anywhere.

* Fix CI test failure cases.

* Addressing CI review comments.

* Addressing CI test failures.

* Applying lintrunner patch
      7b766393
    • kylasa's avatar
      Support for no. of chunks smaller than no. of partitions. (#5390) · 894ad1e3
      kylasa authored 
      * Support for no. of chunks smaller than no. of partitions and Adding appropriate test cases.

Following changes are made with this PR.
1. Code changes for handling no. of chunks smaller than no. of partitions
2. Adding new test cases, which were previously deleted, for no. of chunks smaller than no. of partitions.
3. Also adding test cases, where multiple partitions are handled by a single process.

* Committing the missing files in this commit.

* lintrunner patch.

* lintrunner check

* lintrunner patch here.

* CI review comments.
      894ad1e3
    • Quan (Andy) Gan's avatar
      [Bugfix] Fix duplicate worker_init_fn argument when provided in DataLoader (#5420) · 851d66fa
      Quan (Andy) Gan authored 
      * fix duplicate worker_init_fn

* lint

* lint again

* uugh
      851d66fa
  5. 04 Mar, 2023 1 commit
  7. 03 Mar, 2023 1 commit
  9. 01 Mar, 2023 2 commits
  11. 28 Feb, 2023 1 commit
  13. 25 Feb, 2023 1 commit
    • kylasa's avatar
      [DistDGL][Feature_Request]Changes in the metadata.json file for input graph dataset. (#5310) · a14f69c9
      kylasa authored 
      * Implemented the following changes.

* Remove NUM_NODES_PER_CHUNK
* Remove NUM_EDGES_PER_CHUNK
* Remove the dependency between no. of edge files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files per edge type and no. of partitions
* Remove the dependency between no. of edge feature files and no. of edge files per edge type.
* Remove the dependency between no. of node feature files and no. of partitions
* Add “node_type_counts”. This will be a list of integers. Each integer will represent total count of a node-type. The index in this list and the index in the “node_type” will be the same for a given node-type.
* Add “edge_type_counts”. This will be a list of integers. Each integer will represent total count of an edge-type. The index in this list and the index in the “edge_type” list will be the same for a given edge-type.

* Applying lintrunner patch.

* Adding missing keys to the metadata in the unit test framework.

* lintrunner patch.

* Resolving CI test failures due to merge conflicts.

* Applying lintrunner patch

* applying lintrunner patch

* Replacing tabspace with spaces - to satisfy lintrunner

* Fixing the CI Test Failure cases.

* Applying lintrunner patch

* lintrunner complaining about a blank line.

* Resolving issues with print statement for NoneType

* Removed tests for the arbitrary chunks tests. Since this functionality is not supported anymore.

* Addressing CI review comments.

* addressing CI review comments

* lintrunner patch

* lintrunner patch.

* Addressing CI review comments.

* lintrunner patch.
      a14f69c9
  15. 24 Feb, 2023 2 commits
  17. 23 Feb, 2023 2 commits
  19. 22 Feb, 2023 3 commits
  21. 21 Feb, 2023 12 commits
  23. 19 Feb, 2023 1 commit
  25. 17 Feb, 2023 4 commits
  27. 16 Feb, 2023 3 commits
  29. 15 Feb, 2023 1 commit
  31. 14 Feb, 2023 1 commit
  33. 13 Feb, 2023 1 commit
    • kylasa's avatar
      Code changes to fix order sensitivity of the pipeline (#5288) · 432c71ef
      kylasa authored 
      

Following changes are made in this PR.
1. In dataset_utils.py, when reading edges from disk we follow the order defined by the STR_EDGE_TYPE key in the metadata.json file. This order is implicitly used to assign edgeid to edge types. This same order is used to read edges from the disk as well.
2. Now the unit test framework will also randomize the order of edges read from the disk. This is done for the edges when reading from the disk for the unit tests.
Co-authored-by: default avatarQuan (Andy) Gan <coin2028@hotmail.com>
      432c71ef