ondisk_dataset_homograph.ipynb 20.7 KB
Newer Older
1
2
3
4
{
  "cells": [
    {
      "cell_type": "markdown",
5
6
7
      "metadata": {
        "id": "FnFhPMaAfLtJ"
      },
8
9
10
11
12
13
14
15
      "source": [
        "# OnDiskDataset for Homogeneous Graph\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/stochastic_training/ondisk_dataset_homograph.ipynb) [![GitHub](https://img.shields.io/badge/-View%20on%20GitHub-181717?logo=github&logoColor=ffffff)](https://github.com/dmlc/dgl/blob/master/notebooks/stochastic_training/ondisk_dataset_homograph.ipynb)\n",
        "\n",
        "This tutorial shows how to create `OnDiskDataset` for homogeneous graph that could be used in **GraphBolt** framework.\n",
        "\n",
        "By the end of this tutorial, you will be able to\n",
16
        "\n",
17
18
        "- organize graph structure data.\n",
        "- organize feature data.\n",
19
20
21
22
23
        "- organize training/validation/test set for specific tasks.\n",
        "\n",
        "To create an ``OnDiskDataset`` object, you need to organize all the data including graph structure, feature data and tasks into a directory. The directory should contain a ``metadata.yaml`` file that describes the metadata of the dataset.\n",
        "\n",
        "Now let's generate various data step by step and organize them together to instantiate `OnDiskDataset` finally."
24
      ]
25
26
27
28
29
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wlb19DtWgtzq"
30
31
32
33
      },
      "source": [
        "## Install DGL package"
      ]
34
35
36
    },
    {
      "cell_type": "code",
37
38
39
40
41
      "execution_count": null,
      "metadata": {
        "id": "UojlT9ZGgyr9"
      },
      "outputs": [],
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
      "source": [
        "# Install required packages.\n",
        "import os\n",
        "import torch\n",
        "import numpy as np\n",
        "os.environ['TORCH'] = torch.__version__\n",
        "os.environ['DGLBACKEND'] = \"pytorch\"\n",
        "\n",
        "# Install the CPU version.\n",
        "device = torch.device(\"cpu\")\n",
        "!pip install --pre dgl -f https://data.dgl.ai/wheels-test/repo.html\n",
        "\n",
        "try:\n",
        "    import dgl\n",
        "    import dgl.graphbolt as gb\n",
        "    installed = True\n",
        "except ImportError as error:\n",
        "    installed = False\n",
        "    print(error)\n",
        "print(\"DGL installed!\" if installed else \"DGL not found!\")"
62
      ]
63
64
65
    },
    {
      "cell_type": "markdown",
66
67
68
      "metadata": {
        "id": "2R7WnSbjsfbr"
      },
69
70
71
      "source": [
        "## Data preparation\n",
        "In order to demonstrate how to organize various data, let's create a base directory first."
72
      ]
73
74
75
    },
    {
      "cell_type": "code",
76
77
78
79
80
      "execution_count": null,
      "metadata": {
        "id": "SZipbzyltLfO"
      },
      "outputs": [],
81
82
83
84
      "source": [
        "base_dir = './ondisk_dataset_homograph'\n",
        "os.makedirs(base_dir, exist_ok=True)\n",
        "print(f\"Created base directory: {base_dir}\")"
85
      ]
86
87
88
    },
    {
      "cell_type": "markdown",
89
90
91
      "metadata": {
        "id": "qhNtIn_xhlnl"
      },
92
93
      "source": [
        "### Generate graph structure data\n",
94
        "For homogeneous graph, we just need to save edges(namely seeds) into  **Numpy** or **CSV** file.\n",
95
96
        "\n",
        "Note:\n",
97
98
        "- when saving to **Numpy**, the array requires to be in shape of `(2, N)`. This format is recommended as constructing graph from it is much faster than **CSV** file.\n",
        "- when saving to **CSV** file, do not save index and header.\n"
99
      ]
100
101
102
    },
    {
      "cell_type": "code",
103
104
105
106
107
      "execution_count": null,
      "metadata": {
        "id": "HcBt4G5BmSjr"
      },
      "outputs": [],
108
109
110
111
112
113
114
115
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "num_nodes = 1000\n",
        "num_edges = 10 * num_nodes\n",
        "edges_path = os.path.join(base_dir, \"edges.csv\")\n",
        "edges = np.random.randint(0, num_nodes, size=(num_edges, 2))\n",
        "\n",
116
        "print(f\"Part of edges: {edges[:5, :]}\")\n",
117
118
119
120
121
        "\n",
        "df = pd.DataFrame(edges)\n",
        "df.to_csv(edges_path, index=False, header=False)\n",
        "\n",
        "print(f\"Edges are saved into {edges_path}\")"
122
      ]
123
124
125
    },
    {
      "cell_type": "markdown",
126
127
128
      "metadata": {
        "id": "kh-4cPtzpcaH"
      },
129
130
131
      "source": [
        "### Generate feature data for graph\n",
        "For feature data, numpy arrays and torch tensors are supported for now."
132
      ]
133
134
135
    },
    {
      "cell_type": "code",
136
137
138
139
140
      "execution_count": null,
      "metadata": {
        "id": "_PVu1u5brBhF"
      },
      "outputs": [],
141
142
143
144
      "source": [
        "# Generate node feature in numpy array.\n",
        "node_feat_0_path = os.path.join(base_dir, \"node-feat-0.npy\")\n",
        "node_feat_0 = np.random.rand(num_nodes, 5)\n",
145
        "print(f\"Part of node feature [feat_0]: {node_feat_0[:3, :]}\")\n",
146
        "np.save(node_feat_0_path, node_feat_0)\n",
147
        "print(f\"Node feature [feat_0] is saved to {node_feat_0_path}\\n\")\n",
148
149
150
151
        "\n",
        "# Generate another node feature in torch tensor\n",
        "node_feat_1_path = os.path.join(base_dir, \"node-feat-1.pt\")\n",
        "node_feat_1 = torch.rand(num_nodes, 5)\n",
152
        "print(f\"Part of node feature [feat_1]: {node_feat_1[:3, :]}\")\n",
153
        "torch.save(node_feat_1, node_feat_1_path)\n",
154
        "print(f\"Node feature [feat_1] is saved to {node_feat_1_path}\\n\")\n",
155
156
157
158
        "\n",
        "# Generate edge feature in numpy array.\n",
        "edge_feat_0_path = os.path.join(base_dir, \"edge-feat-0.npy\")\n",
        "edge_feat_0 = np.random.rand(num_edges, 5)\n",
159
        "print(f\"Part of edge feature [feat_0]: {edge_feat_0[:3, :]}\")\n",
160
        "np.save(edge_feat_0_path, edge_feat_0)\n",
161
        "print(f\"Edge feature [feat_0] is saved to {edge_feat_0_path}\\n\")\n",
162
163
164
165
        "\n",
        "# Generate another edge feature in torch tensor\n",
        "edge_feat_1_path = os.path.join(base_dir, \"edge-feat-1.pt\")\n",
        "edge_feat_1 = torch.rand(num_edges, 5)\n",
166
        "print(f\"Part of edge feature [feat_1]: {edge_feat_1[:3, :]}\")\n",
167
        "torch.save(edge_feat_1, edge_feat_1_path)\n",
168
        "print(f\"Edge feature [feat_1] is saved to {edge_feat_1_path}\\n\")\n"
169
      ]
170
171
172
    },
    {
      "cell_type": "markdown",
173
174
175
      "metadata": {
        "id": "ZyqgOtsIwzh_"
      },
176
177
178
      "source": [
        "### Generate tasks\n",
        "`OnDiskDataset` supports multiple tasks. For each task, we need to prepare training/validation/test sets respectively. Such sets usually vary among different tasks. In this tutorial, let's create a **Node Classification** task and **Link Prediction** task."
179
      ]
180
181
182
    },
    {
      "cell_type": "markdown",
183
184
185
      "metadata": {
        "id": "hVxHaDIfzCkr"
      },
186
187
188
      "source": [
        "#### Node Classification Task\n",
        "For node classification task, we need **node IDs** and corresponding **labels** for each training/validation/test set. Like feature data, numpy arrays and torch tensors are supported for these sets."
189
      ]
190
191
192
    },
    {
      "cell_type": "code",
193
194
195
196
197
      "execution_count": null,
      "metadata": {
        "id": "S5-fyBbHzTCO"
      },
      "outputs": [],
198
199
200
201
202
203
204
205
206
207
      "source": [
        "num_trains = int(num_nodes * 0.6)\n",
        "num_vals = int(num_nodes * 0.2)\n",
        "num_tests = num_nodes - num_trains - num_vals\n",
        "\n",
        "ids = np.arange(num_nodes)\n",
        "np.random.shuffle(ids)\n",
        "\n",
        "nc_train_ids_path = os.path.join(base_dir, \"nc-train-ids.npy\")\n",
        "nc_train_ids = ids[:num_trains]\n",
208
        "print(f\"Part of train ids for node classification: {nc_train_ids[:3]}\")\n",
209
        "np.save(nc_train_ids_path, nc_train_ids)\n",
210
        "print(f\"NC train ids are saved to {nc_train_ids_path}\\n\")\n",
211
212
213
        "\n",
        "nc_train_labels_path = os.path.join(base_dir, \"nc-train-labels.pt\")\n",
        "nc_train_labels = torch.randint(0, 10, (num_trains,))\n",
214
        "print(f\"Part of train labels for node classification: {nc_train_labels[:3]}\")\n",
215
        "torch.save(nc_train_labels, nc_train_labels_path)\n",
216
        "print(f\"NC train labels are saved to {nc_train_labels_path}\\n\")\n",
217
218
219
        "\n",
        "nc_val_ids_path = os.path.join(base_dir, \"nc-val-ids.npy\")\n",
        "nc_val_ids = ids[num_trains:num_trains+num_vals]\n",
220
        "print(f\"Part of val ids for node classification: {nc_val_ids[:3]}\")\n",
221
        "np.save(nc_val_ids_path, nc_val_ids)\n",
222
        "print(f\"NC val ids are saved to {nc_val_ids_path}\\n\")\n",
223
224
225
        "\n",
        "nc_val_labels_path = os.path.join(base_dir, \"nc-val-labels.pt\")\n",
        "nc_val_labels = torch.randint(0, 10, (num_vals,))\n",
226
        "print(f\"Part of val labels for node classification: {nc_val_labels[:3]}\")\n",
227
        "torch.save(nc_val_labels, nc_val_labels_path)\n",
228
        "print(f\"NC val labels are saved to {nc_val_labels_path}\\n\")\n",
229
230
231
        "\n",
        "nc_test_ids_path = os.path.join(base_dir, \"nc-test-ids.npy\")\n",
        "nc_test_ids = ids[-num_tests:]\n",
232
        "print(f\"Part of test ids for node classification: {nc_test_ids[:3]}\")\n",
233
        "np.save(nc_test_ids_path, nc_test_ids)\n",
234
        "print(f\"NC test ids are saved to {nc_test_ids_path}\\n\")\n",
235
236
237
        "\n",
        "nc_test_labels_path = os.path.join(base_dir, \"nc-test-labels.pt\")\n",
        "nc_test_labels = torch.randint(0, 10, (num_tests,))\n",
238
        "print(f\"Part of test labels for node classification: {nc_test_labels[:3]}\")\n",
239
        "torch.save(nc_test_labels, nc_test_labels_path)\n",
240
        "print(f\"NC test labels are saved to {nc_test_labels_path}\\n\")"
241
      ]
242
243
244
245
246
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LhAcDCHQ_KJ0"
247
248
249
250
251
      },
      "source": [
        "#### Link Prediction Task\n",
        "For link prediction task, we need **seeds** or **corresponding labels and indexes** which representing the pos/neg property and group of the seeds for each training/validation/test set. Like feature data, numpy arrays and torch tensors are supported for these sets."
      ]
252
253
254
    },
    {
      "cell_type": "code",
255
256
257
258
259
      "execution_count": null,
      "metadata": {
        "id": "u0jCnXIcAQy4"
      },
      "outputs": [],
260
261
262
263
264
      "source": [
        "num_trains = int(num_edges * 0.6)\n",
        "num_vals = int(num_edges * 0.2)\n",
        "num_tests = num_edges - num_trains - num_vals\n",
        "\n",
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
        "lp_train_seeds_path = os.path.join(base_dir, \"lp-train-seeds.npy\")\n",
        "lp_train_seeds = edges[:num_trains, :]\n",
        "print(f\"Part of train seeds for link prediction: {lp_train_seeds[:3]}\")\n",
        "np.save(lp_train_seeds_path, lp_train_seeds)\n",
        "print(f\"LP train seeds are saved to {lp_train_seeds_path}\\n\")\n",
        "\n",
        "lp_val_seeds_path = os.path.join(base_dir, \"lp-val-seeds.npy\")\n",
        "lp_val_seeds = edges[num_trains:num_trains+num_vals, :]\n",
        "lp_val_neg_dsts = np.random.randint(0, num_nodes, (num_vals, 10)).reshape(-1)\n",
        "lp_val_neg_srcs = np.repeat(lp_val_seeds[:,0], 10)\n",
        "lp_val_neg_seeds = np.concatenate((lp_val_neg_srcs, lp_val_neg_dsts)).reshape(2,-1).T\n",
        "lp_val_seeds = np.concatenate((lp_val_seeds, lp_val_neg_seeds))\n",
        "print(f\"Part of val seeds for link prediction: {lp_val_seeds[:3]}\")\n",
        "np.save(lp_val_seeds_path, lp_val_seeds)\n",
        "print(f\"LP val seeds are saved to {lp_val_seeds_path}\\n\")\n",
        "\n",
        "lp_val_labels_path = os.path.join(base_dir, \"lp-val-labels.npy\")\n",
        "lp_val_labels = np.empty(num_vals * (10 + 1))\n",
        "lp_val_labels[:num_vals] = 1\n",
        "lp_val_labels[num_vals:] = 0\n",
        "print(f\"Part of val labels for link prediction: {lp_val_labels[:3]}\")\n",
        "np.save(lp_val_labels_path, lp_val_labels)\n",
        "print(f\"LP val labels are saved to {lp_val_labels_path}\\n\")\n",
        "\n",
        "lp_val_indexes_path = os.path.join(base_dir, \"lp-val-indexes.npy\")\n",
        "lp_val_indexes = np.arange(0, num_vals)\n",
        "lp_val_neg_indexes = np.repeat(lp_val_indexes, 10)\n",
        "lp_val_indexes = np.concatenate([lp_val_indexes, lp_val_neg_indexes])\n",
        "print(f\"Part of val indexes for link prediction: {lp_val_indexes[:3]}\")\n",
        "np.save(lp_val_indexes_path, lp_val_indexes)\n",
        "print(f\"LP val indexes are saved to {lp_val_indexes_path}\\n\")\n",
        "\n",
        "lp_test_seeds_path = os.path.join(base_dir, \"lp-test-seeds.npy\")\n",
        "lp_test_seeds = edges[-num_tests:, :]\n",
        "lp_test_neg_dsts = np.random.randint(0, num_nodes, (num_tests, 10)).reshape(-1)\n",
        "lp_test_neg_srcs = np.repeat(lp_test_seeds[:,0], 10)\n",
        "lp_test_neg_seeds = np.concatenate((lp_test_neg_srcs, lp_test_neg_dsts)).reshape(2,-1).T\n",
        "lp_test_seeds = np.concatenate((lp_test_seeds, lp_test_neg_seeds))\n",
        "print(f\"Part of test seeds for link prediction: {lp_test_seeds[:3]}\")\n",
        "np.save(lp_test_seeds_path, lp_test_seeds)\n",
        "print(f\"LP test seeds are saved to {lp_test_seeds_path}\\n\")\n",
        "\n",
        "lp_test_labels_path = os.path.join(base_dir, \"lp-test-labels.npy\")\n",
        "lp_test_labels = np.empty(num_tests * (10 + 1))\n",
        "lp_test_labels[:num_tests] = 1\n",
        "lp_test_labels[num_tests:] = 0\n",
        "print(f\"Part of val labels for link prediction: {lp_test_labels[:3]}\")\n",
        "np.save(lp_test_labels_path, lp_test_labels)\n",
        "print(f\"LP test labels are saved to {lp_test_labels_path}\\n\")\n",
        "\n",
        "lp_test_indexes_path = os.path.join(base_dir, \"lp-test-indexes.npy\")\n",
        "lp_test_indexes = np.arange(0, num_tests)\n",
        "lp_test_neg_indexes = np.repeat(lp_test_indexes, 10)\n",
        "lp_test_indexes = np.concatenate([lp_test_indexes, lp_test_neg_indexes])\n",
        "print(f\"Part of test indexes for link prediction: {lp_test_indexes[:3]}\")\n",
        "np.save(lp_test_indexes_path, lp_test_indexes)\n",
        "print(f\"LP test indexes are saved to {lp_test_indexes_path}\\n\")"
      ]
323
324
325
    },
    {
      "cell_type": "markdown",
326
327
328
      "metadata": {
        "id": "wbk6-wxRK-6S"
      },
329
330
      "source": [
        "## Organize Data into YAML File\n",
331
332
333
334
335
336
337
338
        "Now we need to create a `metadata.yaml` file which contains the paths, dadta types of graph structure, feature data, training/validation/test sets.\n",
        "\n",
        "Notes:\n",
        "- all path should be relative to `metadata.yaml`.\n",
        "- Below fields are optional and not specified in below example.\n",
        "  - `in_memory`: indicates whether to load dada into memory or `mmap`. Default is `True`.\n",
        "\n",
        "Please refer to [YAML specification](https://github.com/dmlc/dgl/blob/master/docs/source/stochastic_training/ondisk-dataset-specification.rst) for more details."
339
      ]
340
341
342
    },
    {
      "cell_type": "code",
343
344
345
346
347
      "execution_count": null,
      "metadata": {
        "id": "ddGTWW61Lpwp"
      },
      "outputs": [],
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
      "source": [
        "yaml_content = f\"\"\"\n",
        "    dataset_name: homogeneous_graph_nc_lp\n",
        "    graph:\n",
        "      nodes:\n",
        "        - num: {num_nodes}\n",
        "      edges:\n",
        "        - format: csv\n",
        "          path: {os.path.basename(edges_path)}\n",
        "    feature_data:\n",
        "      - domain: node\n",
        "        name: feat_0\n",
        "        format: numpy\n",
        "        path: {os.path.basename(node_feat_0_path)}\n",
        "      - domain: node\n",
        "        name: feat_1\n",
        "        format: torch\n",
        "        path: {os.path.basename(node_feat_1_path)}\n",
        "      - domain: edge\n",
        "        name: feat_0\n",
        "        format: numpy\n",
        "        path: {os.path.basename(edge_feat_0_path)}\n",
        "      - domain: edge\n",
        "        name: feat_1\n",
        "        format: torch\n",
        "        path: {os.path.basename(edge_feat_1_path)}\n",
        "    tasks:\n",
        "      - name: node_classification\n",
        "        num_classes: 10\n",
        "        train_set:\n",
        "          - data:\n",
379
        "              - name: seeds\n",
380
381
382
383
384
385
386
        "                format: numpy\n",
        "                path: {os.path.basename(nc_train_ids_path)}\n",
        "              - name: labels\n",
        "                format: torch\n",
        "                path: {os.path.basename(nc_train_labels_path)}\n",
        "        validation_set:\n",
        "          - data:\n",
387
        "              - name: seeds\n",
388
389
390
391
392
393
394
        "                format: numpy\n",
        "                path: {os.path.basename(nc_val_ids_path)}\n",
        "              - name: labels\n",
        "                format: torch\n",
        "                path: {os.path.basename(nc_val_labels_path)}\n",
        "        test_set:\n",
        "          - data:\n",
395
        "              - name: seeds\n",
396
397
398
399
400
401
402
403
404
        "                format: numpy\n",
        "                path: {os.path.basename(nc_test_ids_path)}\n",
        "              - name: labels\n",
        "                format: torch\n",
        "                path: {os.path.basename(nc_test_labels_path)}\n",
        "      - name: link_prediction\n",
        "        num_classes: 10\n",
        "        train_set:\n",
        "          - data:\n",
405
        "              - name: seeds\n",
406
        "                format: numpy\n",
407
        "                path: {os.path.basename(lp_train_seeds_path)}\n",
408
409
        "        validation_set:\n",
        "          - data:\n",
410
        "              - name: seeds\n",
411
        "                format: numpy\n",
412
413
414
415
416
417
418
        "                path: {os.path.basename(lp_val_seeds_path)}\n",
        "              - name: labels\n",
        "                format: numpy\n",
        "                path: {os.path.basename(lp_val_labels_path)}\n",
        "              - name: indexes\n",
        "                format: numpy\n",
        "                path: {os.path.basename(lp_val_indexes_path)}\n",
419
420
        "        test_set:\n",
        "          - data:\n",
421
        "              - name: seeds\n",
422
        "                format: numpy\n",
423
424
425
426
427
428
429
        "                path: {os.path.basename(lp_test_seeds_path)}\n",
        "              - name: labels\n",
        "                format: numpy\n",
        "                path: {os.path.basename(lp_test_labels_path)}\n",
        "              - name: indexes\n",
        "                format: numpy\n",
        "                path: {os.path.basename(lp_test_indexes_path)}\n",
430
431
432
433
        "\"\"\"\n",
        "metadata_path = os.path.join(base_dir, \"metadata.yaml\")\n",
        "with open(metadata_path, \"w\") as f:\n",
        "  f.write(yaml_content)"
434
      ]
435
436
437
    },
    {
      "cell_type": "markdown",
438
439
440
      "metadata": {
        "id": "kEfybHGhOW7O"
      },
441
442
443
444
445
446
447
      "source": [
        "## Instantiate `OnDiskDataset`\n",
        "Now we're ready to load dataset via `dgl.graphbolt.OnDiskDataset`. When instantiating, we just pass in the base directory where `metadata.yaml` file lies.\n",
        "\n",
        "During first instantiation, GraphBolt preprocesses the raw data such as constructing `FusedCSCSamplingGraph` from edges. All data including graph, feature data, training/validation/test sets are put into `preprocessed` directory after preprocessing. Any following dataset loading will skip the preprocess stage.\n",
        "\n",
        "After preprocessing, `load()` is required to be called explicitly in order to load graph, feature data and tasks."
448
      ]
449
450
451
    },
    {
      "cell_type": "code",
452
453
454
455
456
      "execution_count": null,
      "metadata": {
        "id": "W58CZoSzOiyo"
      },
      "outputs": [],
457
458
459
      "source": [
        "dataset = gb.OnDiskDataset(base_dir).load()\n",
        "graph = dataset.graph\n",
460
        "print(f\"Loaded graph: {graph}\\n\")\n",
461
462
        "\n",
        "feature = dataset.feature\n",
463
        "print(f\"Loaded feature store: {feature}\\n\")\n",
464
465
466
        "\n",
        "tasks = dataset.tasks\n",
        "nc_task = tasks[0]\n",
467
        "print(f\"Loaded node classification task: {nc_task}\\n\")\n",
468
        "lp_task = tasks[1]\n",
469
        "print(f\"Loaded link prediction task: {lp_task}\\n\")"
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
      ]
    }
  ],
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
486
      },
487
488
489
490
491
492
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
493
    }
494
495
496
497
  },
  "nbformat": 4,
  "nbformat_minor": 0
}