README.md 5.14 KB
Newer Older
1

2
# Stochastic Training for Graph Convolutional Networks Using Distributed Sampler
3

4
5
6
* Paper: [Control Variate](https://arxiv.org/abs/1710.10568)
* Paper: [Skip Connection](https://arxiv.org/abs/1809.05343)
* Author's code: [https://github.com/thu-ml/stochastic_gcn](https://github.com/thu-ml/stochastic_gcn)
7

8
9
10
11
12
13
14
15
### Dependencies

- MXNet nightly build

```bash
pip install mxnet --pre
```

16
17
18
19
### Usage

To run the following demos, you need to start trainer and sampler process on different machines by changing the `--ip`. You can also change the number of sampler by the `--num-sampler` option.

20
21
22
23
24
25
26
27
### Neighbor Sampling & Skip Connection

#### cora

Test accuracy ~83% with `--num-neighbors 2`, ~84% by training on the full graph

Trainer side:
```
28
DGLBACKEND=mxnet python3 train.py --model gcn_ns --dataset cora --self-loop --num-neighbors 2 --batch-size 1000 --test-batch-size 5000 --ip 127.0.0.1:50051 --num-sampler 1
29
30
31
```

Sampler side:
32
```
33
DGLBACKEND=mxnet python3 sampler.py --model gcn_ns --dataset cora --self-loop --num-neighbors 2 --batch-size 1000 --ip 127.0.0.1:50051 --num-sampler 1
34
35
```

36
37
38
39
40
41
#### citeseer 

Test accuracy ~69% with `--num-neighbors 2`, ~70% by training on the full graph

Trainer side:
```
42
DGLBACKEND=mxnet python3 train.py --model gcn_ns --dataset citeseer --self-loop --num-neighbors 2 --batch-size 1000 --test-batch-size 5000 --ip 127.0.0.1:50051 --num-sampler 1
43
```
44

45
Sampler side:
46
```
47
DGLBACKEND=mxnet python3 sampler.py --model gcn_ns --dataset citeseer --self-loop --num-neighbors 2 --batch-size 1000 --ip 127.0.0.1:50051 --num-sampler 1
48
49
```

50
51
52
53
54
55
#### pubmed

Test accuracy ~78% with `--num-neighbors 3`, ~77% by training on the full graph

Trainer side:
```
56
DGLBACKEND=mxnet python3 train.py --model gcn_ns --dataset pubmed --self-loop --num-neighbors 3 --batch-size 1000 --test-batch-size 5000 --ip 127.0.0.1:50051 --num-sampler 1
57
58
59
60
```

Sampler side:
```
61
DGLBACKEND=mxnet python3 sampler.py --model gcn_ns --dataset pubmed --self-loop --num-neighbors 3 --batch-size 1000 --ip 127.0.0.1:50051 --num-sampler 1
62
63
64
65
66
67
68
69
```

#### reddit

Test accuracy ~91% with `--num-neighbors 2` and `--batch-size 1000`, ~93% by training on the full graph

Trainer side:
```
70
DGLBACKEND=mxnet python3 train.py --model gcn_ns --dataset reddit-self-loop --num-neighbors 2 --batch-size 1000 --test-batch-size 5000 --n-hidden 64 --ip 127.0.0.1:2049 --num-sampler 1
71
72
73
74
```

Sampler side:
```
75
DGLBACKEND=mxnet python3 sampler.py --model gcn_ns --dataset reddit-self-loop --num-neighbors 2 --batch-size 1000 --ip 127.0.0.1:2049 --num-sampler 1
76
77
78
79
80
81
82
83
84
85
```

### Control Variate & Skip Connection

#### cora

Test accuracy ~84% with `--num-neighbors 1`, ~84% by training on the full graph

Trainer side:
```
86
DGLBACKEND=mxnet python3 train.py --model gcn_cv --dataset cora --self-loop --num-neighbors 1 --batch-size 1000000 --test-batch-size 1000000 --ip 127.0.0.1:50051 --num-sampler 1
87
```
88

89
90
Sampler side:
```
91
DGLBACKEND=mxnet python3 sampler.py --model gcn_cv --dataset cora --self-loop --num-neighbors 1 --batch-size 1000000 --ip 127.0.0.1:50051 --num-sampler 1
92
93
94
95
96
97
98
99
```

#### citeseer

Test accuracy ~69% with `--num-neighbors 1`, ~70% by training on the full graph

Trainer Side:
```
100
DGLBACKEND=mxnet python3 train.py --model gcn_cv --dataset citeseer --self-loop --num-neighbors 1 --batch-size 1000000 --test-batch-size 1000000 --ip 127.0.0.1:50051 --num-sampler 1
101
102
103
104
```

Sampler Side:
```
105
DGLBACKEND=mxnet python3 sampler.py --model gcn_cv --dataset citeseer --self-loop --num-neighbors 1 --batch-size 1000000 --ip 127.0.0.1:50051 --num-sampler 1
106
107
108
109
110
111
```

#### pubmed

Trainer Side:
```
112
DGLBACKEND=mxnet python3 train.py --model gcn_cv --dataset pubmed --self-loop --num-neighbors 1 --batch-size 1000000 --test-batch-size 1000000 --ip 127.0.0.1:50051 --num-sampler 1
113
114
115
116
```

Sampler Side:
```
117
DGLBACKEND=mxnet python3 sampler.py --model gcn_cv --dataset pubmed --self-loop --num-neighbors 1 --batch-size 1000000 --ip 127.0.0.1:50051 --num-sampler 1
118
119
120
121
122
123
124
125
```

#### reddit

Test accuracy ~93% with `--num-neighbors 1` and `--batch-size 1000`, ~93% by training on the full graph

Trainer Side:
```
126
DGLBACKEND=mxnet python3 train.py --model gcn_cv --dataset reddit-self-loop --num-neighbors 1 --batch-size 10000 --test-batch-size 5000 --n-hidden 64 --ip 127.0.0.1:50051 --num-sampler 1
127
128
129
130
```

Sampler Side:
```
131
DGLBACKEND=mxnet python3 sampler.py --model gcn_cv --dataset reddit-self-loop --num-neighbors 1 --batch-size 10000 --ip 127.0.0.1:50051 --num-sampler 1
132
133
134
135
136
137
138
139
140
141
142
143
```

### Control Variate & GraphSAGE-mean

Following [Control Variate](https://arxiv.org/abs/1710.10568), we use the mean pooling architecture GraphSAGE-mean, two linear layers and layer normalization per graph convolution layer.

#### reddit

Test accuracy 96.1% with `--num-neighbors 1` and `--batch-size 1000`, ~96.2% in [Control Variate](https://arxiv.org/abs/1710.10568) with `--num-neighbors 2` and `--batch-size 1000`

Trainer side:
```
144
DGLBACKEND=mxnet python3 train.py --model graphsage_cv --batch-size 1000 --test-batch-size 5000 --n-epochs 50 --dataset reddit --num-neighbors 1 --n-hidden 128 --dropout 0.2 --weight-decay 0 --ip 127.0.0.1:50051 --num-sampler 1
145
146
147
148
```

Sampler side:
```
149
DGLBACKEND=mxnet python3 sampler.py --model graphsage_cv --batch-size 1000 --dataset reddit --num-neighbors 1 --ip 127.0.0.1:50051 --num-sampler 1
150
```