Skip to content

GitLab

Unverified Commit e57c6e35 authored by Jinjing Zhou's avatar Jinjing Zhou Committed by GitHub
Browse files

[Fix] Fix lint resource usage & Fix Docs (#3032) 

* fix

* remove nvidiasmi

* fix

* fix docs

* fix

* fix
parent 55e7796a
Hide whitespace changes
Inline Side-by-side
Showing with 426 additions and 363 deletions
benchmarks/benchmarks/utils.py
View file @ e57c6e35
...@@ -506,7 +506,7 @@ def skip_if_not_4gpu(): ...@@ -506,7 +506,7 @@ def skip_if_not_4gpu():
def _wrapper(func): def _wrapper(func):
if GPU_COUNT != 4: if GPU_COUNT != 4:
# skip if not enabled # skip if not enabled
print("Skip {}".format(func.benchmark_name)) print("Skip {}".format(func.__name__))
func.benchmark_name = "skip_" + func.__name__ func.benchmark_name = "skip_" + func.__name__
return func return func
return _wrapper return _wrapper
......
benchmarks/run.sh
View file @ e57c6e35
...@@ -12,8 +12,6 @@ pip install --upgrade pip ...@@ -12,8 +12,6 @@ pip install --upgrade pip
pip install asv pip install asv
pip uninstall -y dgl pip uninstall -y dgl
nvidia-smi
export DGL_BENCH_DEVICE=$DEVICE export DGL_BENCH_DEVICE=$DEVICE
echo "DGL_BENCH_DEVICE=$DGL_BENCH_DEVICE" echo "DGL_BENCH_DEVICE=$DGL_BENCH_DEVICE"
pushd $ROOT/benchmarks pushd $ROOT/benchmarks
......
docker/pods/ci-lint.yaml
View file @ e57c6e35
...@@ -10,5 +10,5 @@ spec: ...@@ -10,5 +10,5 @@ spec:
tty: true tty: true
resources: resources:
requests: requests:
cpu: 4 cpu: 1
serviceAccountName: dglciuser serviceAccountName: dglciuser
\ No newline at end of file
tutorials/dist/1_node_classification.py
View file @ e57c6e35
...@@ -8,363 +8,428 @@ read the tutorial of multi-GPU training first. This tutorial is developed on top ...@@ -8,363 +8,428 @@ read the tutorial of multi-GPU training first. This tutorial is developed on top
multi-GPU training by providing extra steps for partitioning a graph, modifying the training script multi-GPU training by providing extra steps for partitioning a graph, modifying the training script
and setting up the environment for distributed training. and setting up the environment for distributed training.
'''
###################################################### Partition a graph
#Partition a graph -----------------
#-----------------
# In this tutorial, we will use `OGBN products graph <https://ogb.stanford.edu/docs/nodeprop/#ogbn-products>`_
#In this tutorial, we will use `OGBN products graph <https://ogb.stanford.edu/docs/nodeprop/#ogbn-products>`_ as an example to illustrate the graph partitioning. Let's first load the graph into a DGL graph.
#as an example to illustrate the graph partitioning. Let's first load the graph into a DGL graph. Here we store the node labels as node data in the DGL Graph.
#Here we store the node labels as node data in the DGL Graph.
#
import dgl
import torch as th .. code-block:: python
from ogb.nodeproppred import DglNodePropPredDataset
data = DglNodePropPredDataset(name='ogbn-products')
graph, labels = data[0] import dgl
labels = labels[:, 0] import torch as th
graph.ndata['labels'] = labels from ogb.nodeproppred import DglNodePropPredDataset
data = DglNodePropPredDataset(name='ogbn-products')
###################################################### graph, labels = data[0]
#We need to split the data into training/validation/test set during the graph partitioning. labels = labels[:, 0]
#Because this is a node classification task, the training/validation/test sets contain node IDs. graph.ndata['labels'] = labels
#We recommend users to convert them as boolean arrays, in which True indicates the existence
#of the node ID in the set. In this way, we can store them as node data. After the partitioning,
#the boolean arrays will be stored with the graph partitions. We need to split the data into training/validation/test set during the graph partitioning.
# Because this is a node classification task, the training/validation/test sets contain node IDs.
We recommend users to convert them as boolean arrays, in which True indicates the existence
splitted_idx = data.get_idx_split() of the node ID in the set. In this way, we can store them as node data. After the partitioning,
train_nid, val_nid, test_nid = splitted_idx['train'], splitted_idx['valid'], splitted_idx['test'] the boolean arrays will be stored with the graph partitions.
train_mask = th.zeros((graph.number_of_nodes(),), dtype=th.bool)
train_mask[train_nid] = True
val_mask = th.zeros((graph.number_of_nodes(),), dtype=th.bool)
val_mask[val_nid] = True
test_mask = th.zeros((graph.number_of_nodes(),), dtype=th.bool) .. code-block:: python
test_mask[test_nid] = True
graph.ndata['train_mask'] = train_mask
graph.ndata['val_mask'] = val_mask splitted_idx = data.get_idx_split()
graph.ndata['test_mask'] = test_mask train_nid, val_nid, test_nid = splitted_idx['train'], splitted_idx['valid'], splitted_idx['test']
train_mask = th.zeros((graph.number_of_nodes(),), dtype=th.bool)
##################################################### train_mask[train_nid] = True
#Then we call the `partition_graph` function to partition the graph with val_mask = th.zeros((graph.number_of_nodes(),), dtype=th.bool)
#`METIS <http://glaros.dtc.umn.edu/gkhome/metis/metis/overview>`_ and save the partitioned results val_mask[val_nid] = True
#in the specified folder. **Note**: `partition_graph` runs on a single machine with a single thread. test_mask = th.zeros((graph.number_of_nodes(),), dtype=th.bool)
#You can go to `our user guide <https://docs.dgl.ai/en/latest/guide/distributed-preprocessing.html#distributed-partitioning>`_ test_mask[test_nid] = True
#to see more information on distributed graph partitioning. graph.ndata['train_mask'] = train_mask
# graph.ndata['val_mask'] = val_mask
#The code below shows an example of invoking the partitioning algorithm and generate four partitions. graph.ndata['test_mask'] = test_mask
#The partitioned results are stored in a folder called `4part_data`. While partitioning a graph,
#we allow users to specify how to balance the partitions. By default, the algorithm balances the number
#of nodes in each partition as much as possible. However, this balancing strategy is not sufficient Then we call the `partition_graph` function to partition the graph with
#for distributed GNN training because some partitions may have many more training nodes than other partitions `METIS <http://glaros.dtc.umn.edu/gkhome/metis/metis/overview>`_ and save the partitioned results
#or some partitions may have more edges than others. As such, `partition_graph` provides two additional arguments in the specified folder. **Note**: `partition_graph` runs on a single machine with a single thread.
#`balance_ntypes` and `balance_edges` to enforce more balancing criteria. For example, we can use the training mask You can go to `our user guide <https://docs.dgl.ai/en/latest/guide/distributed-preprocessing.html#distributed-partitioning>`_
#to balance the number of training nodes in each partition, as shown in the example below. We can also turn on to see more information on distributed graph partitioning.
#the `balance_edges` flag to ensure that all partitions have roughly the same number of edges.
# The code below shows an example of invoking the partitioning algorithm and generate four partitions.
The partitioned results are stored in a folder called `4part_data`. While partitioning a graph,
dgl.distributed.partition_graph(graph, graph_name='ogbn-products', num_parts=4, we allow users to specify how to balance the partitions. By default, the algorithm balances the number
out_path='4part_data', of nodes in each partition as much as possible. However, this balancing strategy is not sufficient
balance_ntypes=graph.ndata['train_mask'], for distributed GNN training because some partitions may have many more training nodes than other partitions
balance_edges=True) or some partitions may have more edges than others. As such, `partition_graph` provides two additional arguments
`balance_ntypes` and `balance_edges` to enforce more balancing criteria. For example, we can use the training mask
################################################### to balance the number of training nodes in each partition, as shown in the example below. We can also turn on
#When partitioning a graph, DGL shuffles node IDs and edge IDs so that nodes/edges assigned to the `balance_edges` flag to ensure that all partitions have roughly the same number of edges.
#a partition have contiguous IDs. This is necessary for DGL to maintain the mappings of global
#node/edge IDs and partition IDs. If a user needs to map the shuffled node/edge IDs to their original IDs,
#they can turn on the `return_mapping` flag of `partition_graph`, which returns a vector for the node ID mapping
#and edge ID mapping. Below shows an example of using the ID mapping to save the node embeddings after
#distributed training. This is a common use case when users want to use the trained node embeddings .. code-block:: python
#in their downstream task. Below let's assume that the trained node embeddings are stored in the `node_emb` tensor,
#which is indexed by the shuffled node IDs. We shuffle the embeddings again and store them in
#the `orig_node_emb` tensor, which is indexed by the original node IDs. dgl.distributed.partition_graph(graph, graph_name='ogbn-products', num_parts=4,
# out_path='4part_data',
balance_ntypes=graph.ndata['train_mask'],
nmap, emap = dgl.distributed.partition_graph(graph, graph_name='ogbn-products', balance_edges=True)
num_parts=4,
out_path='4part_data',
balance_ntypes=graph.ndata['train_mask'], When partitioning a graph, DGL shuffles node IDs and edge IDs so that nodes/edges assigned to
balance_edges=True, a partition have contiguous IDs. This is necessary for DGL to maintain the mappings of global
return_mapping=True) node/edge IDs and partition IDs. If a user needs to map the shuffled node/edge IDs to their original IDs,
orig_node_emb = th.zeros(node_emb.shape, dtype=node_emb.dtype) they can turn on the `return_mapping` flag of `partition_graph`, which returns a vector for the node ID mapping
orig_node_emb[nmap] = node_emb and edge ID mapping. Below shows an example of using the ID mapping to save the node embeddings after
distributed training. This is a common use case when users want to use the trained node embeddings
##################################################### in their downstream task. Below let's assume that the trained node embeddings are stored in the `node_emb` tensor,
#Distributed training script which is indexed by the shuffled node IDs. We shuffle the embeddings again and store them in
#--------------------------- the `orig_node_emb` tensor, which is indexed by the original node IDs.
#
#The distributed training script is very similar to multi-GPU training script with just a few modifications.
#It also relies on the Pytorch distributed component to exchange gradients and update model parameters.
#The distributed training script only contains the code of the trainers.
# .. code-block:: python
#Initialize network communication
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# nmap, emap = dgl.distributed.partition_graph(graph, graph_name='ogbn-products',
#Distributed GNN training requires to access the partitioned graph structure and node/edge features num_parts=4,
#as well as aggregating the gradients of model parameters from multiple trainers. DGL's distributed out_path='4part_data',
#component is responsible for accessing the distributed graph structure and distributed node features balance_ntypes=graph.ndata['train_mask'],
#and edge features while Pytorch distributed is responsible for exchanging the gradients of model parameters. balance_edges=True,
#As such, we need to initialize both DGL and Pytorch distributed components at the beginning of the training script. return_mapping=True)
# orig_node_emb = th.zeros(node_emb.shape, dtype=node_emb.dtype)
#We need to call DGL's initialize function to initialize the trainers' network communication and orig_node_emb[nmap] = node_emb
#connect with DGL's servers at the very beginning of the distributed training script. This function
#has an argument that accepts the path to the cluster configuration file.
# Distributed training script
import dgl ---------------------------
import torch as th
dgl.distributed.initialize(ip_config='ip_config.txt') The distributed training script is very similar to multi-GPU training script with just a few modifications.
It also relies on the Pytorch distributed component to exchange gradients and update model parameters.
##################################################################### The distributed training script only contains the code of the trainers.
#The configuration file `ip_config.txt` has the following format:
# Initialize network communication
#.. code-block:: shell ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#
# ip_addr1 [port1] Distributed GNN training requires to access the partitioned graph structure and node/edge features
# ip_addr2 [port2] as well as aggregating the gradients of model parameters from multiple trainers. DGL's distributed
# component is responsible for accessing the distributed graph structure and distributed node features
#Each row is a machine. The first column is the IP address and the second column is the port for and edge features while Pytorch distributed is responsible for exchanging the gradients of model parameters.
#connecting to the DGL server on the machine. The port is optional and the default port is 30050. As such, we need to initialize both DGL and Pytorch distributed components at the beginning of the training script.
#
#After initializing DGL's network communication, a user can initialize Pytorch's distributed communication. We need to call DGL's initialize function to initialize the trainers' network communication and
# connect with DGL's servers at the very beginning of the distributed training script. This function
th.distributed.init_process_group(backend='gloo') has an argument that accepts the path to the cluster configuration file.
#######################################################################
#Reference to the distributed graph
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# .. code-block:: python
#DGL's servers load the graph partitions automatically. After the servers load the partitions,
#trainers connect to the servers and can start to reference to the distributed graph in the cluster as below. import dgl
# import torch as th
g = dgl.distributed.DistGraph('ogbn-products') dgl.distributed.initialize(ip_config='ip_config.txt')
#######################################################################
#As shown in the code, we refer to a distributed graph by its name. This name is basically the one passed The configuration file `ip_config.txt` has the following format:
#to the `partition_graph` function as shown in the section above.
# .. code-block:: shell
#Get training and validation node IDs
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ip_addr1 [port1]
# ip_addr2 [port2]
#For distributed training, each trainer can run its own set of training nodes.
#The training nodes of the entire graph are stored in a distributed tensor as the `train_mask` node data, Each row is a machine. The first column is the IP address and the second column is the port for
#which was constructed before we partitioned the graph. Each trainer can invoke `node_split` to its set connecting to the DGL server on the machine. The port is optional and the default port is 30050.
#of training nodes. The `node_split` function splits the full training set evenly and returns
#the training nodes, majority of which are stored in the local partition, to ensure good data locality. After initializing DGL's network communication, a user can initialize Pytorch's distributed communication.
#
train_nid = dgl.distributed.node_split(g.ndata['train_mask'])
######################################################################
#We can split the validation nodes in the same way as above. In this case, each trainer gets .. code-block:: python
#a different set of validation nodes.
# th.distributed.init_process_group(backend='gloo')
valid_nid = dgl.distributed.node_split(g.ndata['val_mask'])
##################################################################### Reference to the distributed graph
#Define a GNN model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#^^^^^^^^^^^^^^^^^^
# DGL's servers load the graph partitions automatically. After the servers load the partitions,
#For distributed training, we define a GNN model exactly in the same way as trainers connect to the servers and can start to reference to the distributed graph in the cluster as below.
#`mini-batch training <https://doc.dgl.ai/guide/minibatch.html#>`_ or
#`full-graph training <https://doc.dgl.ai/guide/training-node.html#guide-training-node-classification>`_.
#The code below defines the GraphSage model.
#
import torch.nn as nn .. code-block:: python
import torch.nn.functional as F
import dgl.nn as dglnn g = dgl.distributed.DistGraph('ogbn-products')
import torch.optim as optim
class SAGE(nn.Module): As shown in the code, we refer to a distributed graph by its name. This name is basically the one passed
def __init__(self, in_feats, n_hidden, n_classes, n_layers): to the `partition_graph` function as shown in the section above.
super().__init__()
self.n_layers = n_layers Get training and validation node IDs
self.n_hidden = n_hidden ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
self.n_classes = n_classes
self.layers = nn.ModuleList() For distributed training, each trainer can run its own set of training nodes.
self.layers.append(dglnn.SAGEConv(in_feats, n_hidden, 'mean')) The training nodes of the entire graph are stored in a distributed tensor as the `train_mask` node data,
for i in range(1, n_layers - 1): which was constructed before we partitioned the graph. Each trainer can invoke `node_split` to its set
self.layers.append(dglnn.SAGEConv(n_hidden, n_hidden, 'mean')) of training nodes. The `node_split` function splits the full training set evenly and returns
self.layers.append(dglnn.SAGEConv(n_hidden, n_classes, 'mean')) the training nodes, majority of which are stored in the local partition, to ensure good data locality.
def forward(self, blocks, x):
for l, (layer, block) in enumerate(zip(self.layers, blocks)):
x = layer(block, x)
if l != self.n_layers - 1: .. code-block:: python
x = F.relu(x)
return x train_nid = dgl.distributed.node_split(g.ndata['train_mask'])
num_hidden = 256
num_labels = len(th.unique(g.ndata['labels'][0:g.number_of_nodes()])) We can split the validation nodes in the same way as above. In this case, each trainer gets
num_layers = 2 a different set of validation nodes.
lr = 0.001
model = SAGE(g.ndata['feat'].shape[1], num_hidden, num_labels, num_layers)
loss_fcn = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=lr)
.. code-block:: python
###################################################################
#For distributed training, we need to convert the model into a distributed model with valid_nid = dgl.distributed.node_split(g.ndata['val_mask'])
#Pytorch's `DistributedDataParallel`.
#
model = th.nn.parallel.DistributedDataParallel(model) Define a GNN model
^^^^^^^^^^^^^^^^^^
####################################################################
#Distributed mini-batch sampler For distributed training, we define a GNN model exactly in the same way as
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `mini-batch training <https://doc.dgl.ai/guide/minibatch.html#>`_ or
# `full-graph training <https://doc.dgl.ai/guide/training-node.html#guide-training-node-classification>`_.
#We can use the same `NodeDataLoader` to create a distributed mini-batch sampler for The code below defines the GraphSage model.
#node classification.
#
sampler = dgl.dataloading.MultiLayerNeighborSampler([25,10])
train_dataloader = dgl.dataloading.NodeDataLoader(
g, train_nid, sampler, batch_size=1024, .. code-block:: python
shuffle=True, drop_last=False)
valid_dataloader = dgl.dataloading.NodeDataLoader( import torch.nn as nn
g, valid_nid, sampler, batch_size=1024, import torch.nn.functional as F
shuffle=False, drop_last=False) import dgl.nn as dglnn
import torch.optim as optim
###################################################################
#Training loop class SAGE(nn.Module):
#^^^^^^^^^^^^^ def __init__(self, in_feats, n_hidden, n_classes, n_layers):
# super().__init__()
#The training loop for distributed training is also exactly the same as the single-process training. self.n_layers = n_layers
# self.n_hidden = n_hidden
import sklearn.metrics self.n_classes = n_classes
import numpy as np self.layers = nn.ModuleList()
self.layers.append(dglnn.SAGEConv(in_feats, n_hidden, 'mean'))
for epoch in range(10): for i in range(1, n_layers - 1):
# Loop over the dataloader to sample mini-batches. self.layers.append(dglnn.SAGEConv(n_hidden, n_hidden, 'mean'))
losses = [] self.layers.append(dglnn.SAGEConv(n_hidden, n_classes, 'mean'))
for step, (input_nodes, seeds, blocks) in enumerate(train_dataloader):
# Load the input features as well as output labels def forward(self, blocks, x):
batch_inputs = g.ndata['feat'][input_nodes] for l, (layer, block) in enumerate(zip(self.layers, blocks)):
batch_labels = g.ndata['labels'][seeds] x = layer(block, x)
if l != self.n_layers - 1:
# Compute loss and prediction x = F.relu(x)
batch_pred = model(blocks, batch_inputs) return x
loss = loss_fcn(batch_pred, batch_labels)
optimizer.zero_grad() num_hidden = 256
loss.backward() num_labels = len(th.unique(g.ndata['labels'][0:g.number_of_nodes()]))
losses.append(loss.detach().cpu().numpy()) num_layers = 2
optimizer.step() lr = 0.001
model = SAGE(g.ndata['feat'].shape[1], num_hidden, num_labels, num_layers)
# validation loss_fcn = nn.CrossEntropyLoss()
predictions = [] optimizer = optim.Adam(model.parameters(), lr=lr)
labels = []
with th.no_grad():
for step, (input_nodes, seeds, blocks) in enumerate(valid_dataloader): For distributed training, we need to convert the model into a distributed model with
inputs = g.ndata['feat'][input_nodes] Pytorch's `DistributedDataParallel`.
labels.append(g.ndata['labels'][seeds].numpy())
predictions.append(model(blocks, inputs).argmax(1).numpy())
predictions = np.concatenate(predictions)
labels = np.concatenate(labels)
accuracy = sklearn.metrics.accuracy_score(labels, predictions) .. code-block:: python
print('Epoch {}: Validation Accuracy {}'.format(epoch, accuracy))
model = th.nn.parallel.DistributedDataParallel(model)
######################################################################
#Set up distributed training environment
#--------------------------------------- Distributed mini-batch sampler
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#After partitioning a graph and preparing the training script, we now need to set up
#the distributed training environment and launch the training job. Basically, we need to We can use the same `NodeDataLoader` to create a distributed mini-batch sampler for
#create a cluster of machines and upload both the training script and the partitioned data node classification.
#to each machine in the cluster. A recommended solution of sharing the training script and
#the partitioned data in the cluster is to use NFS (Network File System).
#
#For any users who are not familiar with NFS, below is a small tutorial of setting up NFS
#in an existing cluster. .. code-block:: python
#
#NFS server side setup (ubuntu only) sampler = dgl.dataloading.MultiLayerNeighborSampler([25,10])
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ train_dataloader = dgl.dataloading.NodeDataLoader(
# g, train_nid, sampler, batch_size=1024,
#First, install essential libs on the storage server shuffle=True, drop_last=False)
# valid_dataloader = dgl.dataloading.NodeDataLoader(
#.. code-block:: shell g, valid_nid, sampler, batch_size=1024,
# shuffle=False, drop_last=False)
# sudo apt-get install nfs-kernel-server
#
#Below we assume the user account is ubuntu and we create a directory of workspace in the home directory. Training loop
# ^^^^^^^^^^^^^
#.. code-block:: shell
# The training loop for distributed training is also exactly the same as the single-process training.
# mkdir -p /home/ubuntu/workspace
#
#We assume that the all servers are under a subnet with ip range 192.168.0.0 to 192.168.255.255.
#We need to add the following line to `/etc/exports`
# .. code-block:: python
#.. code-block:: shell
# import sklearn.metrics
# /home/ubuntu/workspace 192.168.0.0/16(rw,sync,no_subtree_check) import numpy as np
#
#Then restart NFS, the setup on server side is finished. for epoch in range(10):
# # Loop over the dataloader to sample mini-batches.
#.. code-block:: shell losses = []
# for step, (input_nodes, seeds, blocks) in enumerate(train_dataloader):
# sudo systemctl restart nfs-kernel-server # Load the input features as well as output labels
# batch_inputs = g.ndata['feat'][input_nodes]
#For configuration details, please refer to NFS ArchWiki (https://wiki.archlinux.org/index.php/NFS). batch_labels = g.ndata['labels'][seeds]
#
#NFS client side setup (ubuntu only) # Compute loss and prediction
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ batch_pred = model(blocks, batch_inputs)
# loss = loss_fcn(batch_pred, batch_labels)
#To use NFS, clients also require to install essential packages optimizer.zero_grad()
# loss.backward()
#.. code-block:: shell losses.append(loss.detach().cpu().numpy())
# optimizer.step()
# sudo apt-get install nfs-common
# # validation
#You can either mount the NFS manually predictions = []
# labels = []
#.. code-block:: shell with th.no_grad():
# for step, (input_nodes, seeds, blocks) in enumerate(valid_dataloader):
# mkdir -p /home/ubuntu/workspace inputs = g.ndata['feat'][input_nodes]
# sudo mount -t nfs <nfs-server-ip>:/home/ubuntu/workspace /home/ubuntu/workspace labels.append(g.ndata['labels'][seeds].numpy())
# predictions.append(model(blocks, inputs).argmax(1).numpy())
#or add the following line to `/etc/fstab` so the folder will be mounted automatically predictions = np.concatenate(predictions)
# labels = np.concatenate(labels)
#.. code-block:: shell accuracy = sklearn.metrics.accuracy_score(labels, predictions)
# print('Epoch {}: Validation Accuracy {}'.format(epoch, accuracy))
# <nfs-server-ip>:/home/ubuntu/workspace /home/ubuntu/workspace nfs defaults 0 0
#
#Then run Set up distributed training environment
# ---------------------------------------
#.. code-block:: shell
# After partitioning a graph and preparing the training script, we now need to set up
# mount -a the distributed training environment and launch the training job. Basically, we need to
# create a cluster of machines and upload both the training script and the partitioned data
#Now go to `/home/ubuntu/workspace` and save the training script and the partitioned data in the folder. to each machine in the cluster. A recommended solution of sharing the training script and
# the partitioned data in the cluster is to use NFS (Network File System).
#SSH Access
#^^^^^^^^^^ For any users who are not familiar with NFS, below is a small tutorial of setting up NFS
# in an existing cluster.
#The launch script accesses the machines in the cluster via SSH. Users should follow the instruction
#in `this document <https://linuxize.com/post/how-to-setup-passwordless-ssh-login/>`_ to set up NFS server side setup (ubuntu only)
#the passwordless SSH login on every machine in the cluster. After setting up the passwordless SSH, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#users need to authenticate the connection to each machine and add their key fingerprints to `~/.ssh/known_hosts`.
#This can be done automatically when we ssh to a machine for the first time. First, install essential libs on the storage server
#
#Launch the distributed training job .. code-block:: shell
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# sudo apt-get install nfs-kernel-server
#After everything is ready, we can now use the launch script provided by DGL to launch the distributed
#training job in the cluster. We can run the launch script on any machine in the cluster. Below we assume the user account is ubuntu and we create a directory of workspace in the home directory.
#
#.. code-block:: shell .. code-block:: shell
#
# python3 ~/workspace/dgl/tools/launch.py \ mkdir -p /home/ubuntu/workspace
# --workspace ~/workspace/ \
# --num_trainers 1 \ We assume that the all servers are under a subnet with ip range 192.168.0.0 to 192.168.255.255.
# --num_samplers 0 \ We need to add the following line to `/etc/exports`
# --num_servers 1 \
# --part_config 4part_data/ogbn-products.json \ .. code-block:: shell
# --ip_config ip_config.txt \
# "python3 train_dist.py" /home/ubuntu/workspace 192.168.0.0/16(rw,sync,no_subtree_check)
#
#If we split the graph into four partitions as demonstrated at the beginning of the tutorial, the cluster has to have four machines. The command above launches one trainer and one server on each machine in the cluster. `ip_config.txt` lists the IP addresses of all machines in the cluster as follows: Then restart NFS, the setup on server side is finished.
#
#.. code-block:: shell .. code-block:: shell
#
# ip_addr1 sudo systemctl restart nfs-kernel-server
# ip_addr2
# ip_addr3 For configuration details, please refer to NFS ArchWiki (https://wiki.archlinux.org/index.php/NFS).
# ip_addr4
NFS client side setup (ubuntu only)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To use NFS, clients also require to install essential packages
.. code-block:: shell
sudo apt-get install nfs-common
You can either mount the NFS manually
.. code-block:: shell
mkdir -p /home/ubuntu/workspace
sudo mount -t nfs <nfs-server-ip>:/home/ubuntu/workspace /home/ubuntu/workspace
or add the following line to `/etc/fstab` so the folder will be mounted automatically
.. code-block:: shell
<nfs-server-ip>:/home/ubuntu/workspace /home/ubuntu/workspace nfs defaults 0 0
Then run
.. code-block:: shell
mount -a
Now go to `/home/ubuntu/workspace` and save the training script and the partitioned data in the folder.
SSH Access
^^^^^^^^^^
The launch script accesses the machines in the cluster via SSH. Users should follow the instruction
in `this document <https://linuxize.com/post/how-to-setup-passwordless-ssh-login/>`_ to set up
the passwordless SSH login on every machine in the cluster. After setting up the passwordless SSH,
users need to authenticate the connection to each machine and add their key fingerprints to `~/.ssh/known_hosts`.
This can be done automatically when we ssh to a machine for the first time.
Launch the distributed training job
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
After everything is ready, we can now use the launch script provided by DGL to launch the distributed
training job in the cluster. We can run the launch script on any machine in the cluster.
.. code-block:: shell
python3 ~/workspace/dgl/tools/launch.py \
--workspace ~/workspace/ \
--num_trainers 1 \
--num_samplers 0 \
--num_servers 1 \
--part_config 4part_data/ogbn-products.json \
--ip_config ip_config.txt \
"python3 train_dist.py"
If we split the graph into four partitions as demonstrated at the beginning of the tutorial, the cluster has to have four machines. The command above launches one trainer and one server on each machine in the cluster. `ip_config.txt` lists the IP addresses of all machines in the cluster as follows:
.. code-block:: shell
ip_addr1
ip_addr2
ip_addr3
ip_addr4
'''
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment