README.md 3.12 KB
Newer Older
Ruilong Li's avatar
readme  
Ruilong Li committed
1
# nerfacc
Matthew Tancik's avatar
Matthew Tancik committed
2
3
[![Core Tests.](https://github.com/KAIR-BAIR/nerfacc/actions/workflows/code_checks.yml/badge.svg)](https://github.com/KAIR-BAIR/nerfacc/actions/workflows/code_checks.yml)
[![Documentation Status](https://readthedocs.com/projects/plenoptix-nerfacc/badge/?version=latest)](https://plenoptix-nerfacc.readthedocs-hosted.com/en/latest/?badge=latest)
Ruilong Li's avatar
readme  
Ruilong Li committed
4

5
This is a **tiny** tootlbox  for **accelerating** NeRF training & rendering using PyTorch CUDA extensions. Plug-and-play for most of the NeRFs!
Ruilong Li's avatar
readme  
Ruilong Li committed
6

Ruilong Li's avatar
Ruilong Li committed
7
## Examples: Instant-NGP NeRF
Ruilong Li's avatar
readme  
Ruilong Li committed
8

Ruilong Li's avatar
Ruilong Li committed
9
10
``` bash
python examples/trainval.py ngp --train_split trainval
Ruilong Li's avatar
readme  
Ruilong Li committed
11
12
```

Ruilong Li's avatar
Ruilong Li committed
13
Performance on TITAN RTX :
14

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
15
16
| trainval | Lego | Mic | Materials | Chair | Hotdog |
| - | - | - | - | - | - |
17
18
19
| Time | 300s  | 274s  | 266s  | 341s  | 277s  |
| PSNR | 36.61 | 37.62 | 30.11 | 36.09 | 38.09 |
| FPS  | 12.87 | 23.67 | 9.33  | 16.91 | 7.48  |
Ruilong Li's avatar
Ruilong Li committed
20

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
21
Instant-NGP paper (5 min) on 3090 (w/ mask):
22

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
23
24
25
| trainval | Lego | Mic | Materials | Chair | Hotdog |
| - | - | - | - | - | - |
| PSNR | 36.39 | 36.22 | 29.78 | 35.00 | 37.40 |
26
27


Ruilong Li's avatar
Ruilong Li committed
28
29
30
31
32
33
34
35
## Examples: Vanilla MLP NeRF

``` bash
python examples/trainval.py vanilla --train_split train
```

Performance on test set:

Ruilong Li's avatar
Ruilong Li committed
36
37
38
39
40
41
|  | Lego | Mic | Materials | Chair | Hotdog |
| - | - | - | - | - | - |
| Paper PSNR (train set) | 32.54 | 32.91 | 29.62 | 33.00 | 36.18 |
| Our PSNR (train set) | 33.21 | 33.36 | 29.48 | 32.79 | 35.54 |
| Our PSNR (trainval set) | 33.66  | - | - | - | - | - |
| Our train time & test FPS | 45min; 0.43FPS | 44min; 1FPS | 37min; 0.33FPS* | 44min; 0.57FPS* | 50min; 0.15 FPS* |
Ruilong Li's avatar
Ruilong Li committed
42
43
44
45
46

For reference, vanilla NeRF paper trains on V100 GPU for 1-2 days per scene. Test time rendering takes about 30 secs to render a 800x800 image. Our model is trained on a TITAN X.

Note: We only use a single MLP with more samples (1024), instead of two MLPs with coarse-to-fine sampling as in the paper. Both ways share the same spirit to do dense sampling around the surface. Our fast rendering inheritly skip samples away from the surface so we can simplly increase the number of samples with a single MLP, to achieve the same goal with coarse-to-fine sampling, without runtime or memory issue.

Ruilong Li's avatar
Ruilong Li committed
47
48
*FPS for some scenes are tested under `--test_chunk_size=8192` (default is `81920`) to avoid OOM.

Ruilong Li's avatar
Ruilong Li committed
49

50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
## Examples: MLP NeRF on Dynamic objects

Here we trained something similar to D-NeRF on the dnerf dataset:

``` bash
python examples/trainval.py dnerf --train_split train --test_chunk_size=8192
```

Performance on test set:

|  | Lego | Stand Up |
| - | - | - |
| DNeRF paper PSNR (train set) | 21.64 | 32.79 |
| Our PSNR (train set) | 24.66 | 33.98 |
| Our train time & test FPS | 43min; 0.15FPS | 41min; 0.4FPS |
Ruilong Li's avatar
readme  
Ruilong Li committed
65

Ruilong Li's avatar
Ruilong Li committed
66

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
67
68
69
Note the numbers here are tested with version `v0.0.8`

<!-- 
Ruilong Li's avatar
Ruilong Li committed
70
71
72
73
74
## Tips:

1. sample rays over all images per iteration (`batch_over_images=True`) is better: `PSNR 33.31 -> 33.75`.
2. make use of scheduler (`MultiStepLR(optimizer, milestones=[20000, 30000], gamma=0.1)`) to adjust learning rate gives: `PSNR 33.75 -> 34.40`.
3. increasing chunk size (`chunk: 8192 -> 81920`) during inference gives speedup: `FPS 4.x -> 6.2`
Ruilong Li's avatar
Ruilong Li committed
75
4. random bkgd color (`color_bkgd_aug="random"`) for the `Lego` scene actually hurts: `PNSR 35.42 -> 34.38`
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
76
 -->