• kylasa's avatar
    Adding launch script and wrapper script to trigger distributed graph … (#4276) · 8086d1ed
    kylasa authored
    
    
    * Adding launch script and wrapper script to trigger distributed graph partitioning pipeline as defined in the UX document
    
    1. dispatch_data.py is a wrapper script which builds the command and triggers the distributed partitioning pipeline
    2. distgraphlaunch.py is the main python script which triggers the pipeline and to simplify its usage dispatch_data.py is included as a wrapper script around it.
    
    * Added code to auto-detect python version and retrieve some parameters from the input metadata json file
    
    1. Auto detect python version
    2. Read the metadata json file and extract some parameters to pass to the user defined command which is used to trigger the pipeline.
    
    * Updated the json file name to metadata.json file per UX documentation
    
    1. Renamed json file name per UX documentation.
    
    * address comments
    
    * fix
    
    * fix doc
    
    * use unbuffered logging to cure anxiety
    
    * cure more anxiety
    
    * Update tools/dispatch_data.py
    Co-authored-by: default avatarMinjie Wang <minjie.wang@nyu.edu>
    
    * oops
    Co-authored-by: default avatarQuan Gan <coin2028@hotmail.com>
    Co-authored-by: default avatarMinjie Wang <minjie.wang@nyu.edu>
    8086d1ed
data_shuffle.py 33.2 KB