• kylasa's avatar
    [Dist] Flexible pipeline - Initial commit (#4733) · c8ea9fa4
    kylasa authored
    * Flexible pipeline - Initial commit
    
    1. Implementation of flexible pipeline feature.
    2. With this implementation, the pipeline now supports multiple partitions per process. And also assumes that num_partitions is always a multiple of num_processes.
    
    * Update test_dist_part.py
    
    * Code changes to address review comments
    
    * Code refactoring of exchange_features function into two functions for better readability
    
    * Upadting test_dist_part to fix merge issues with the master branch
    
    * corrected variable names...
    
    * Fixed code refactoring issues.
    
    * Provide missing function arguments to exchange_feature function
    
    * Providing the missing function argument to fix error.
    
    * Provide missing function argument to 'get_shuffle_nids' function.
    
    * Repositioned a variable within its scope.
    
    * Removed tab space which is causing the indentation problem
    
    * Fix issue with the CI test framework, which is the root cause for the failure of the CI tests.
    
    1. Now we read files specific to the partition-id and store this data separately, identified by the local_part_id, in the local process.
    2. Similarly as above, we also differentiate the node and edge features type_ids with the same keys as above.
    3. These above two changes will help up to get the appropriate feature data during the feature exchange and send to the destination process correctly.
    
    * Correct the parametrization for the CI unit test cases.
    
    * Addressing Rui's code review comments.
    
    * Addressing code review comments.
    c8ea9fa4
test_dist_part.py 8.42 KB