README.md 8.42 KB
Newer Older
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
1
2
3
4
5
6
7
8
9
10
11
<p>
  <!-- pypi-strip -->
  <picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://user-images.githubusercontent.com/3310961/199083722-881a2372-62c1-4255-8521-31a95a721851.png" />
  <source media="(prefers-color-scheme: light)" srcset="https://user-images.githubusercontent.com/3310961/199084143-0d63eb40-3f35-48d2-a9d5-78d1d60b7d66.png" />
  <!-- /pypi-strip -->
  <img alt="nerfacc logo" src="https://user-images.githubusercontent.com/3310961/199084143-0d63eb40-3f35-48d2-a9d5-78d1d60b7d66.png" width="350px" />
  <!-- pypi-strip -->
  </picture>
  <!-- /pypi-strip -->
</p>
Matthew Tancik's avatar
Matthew Tancik committed
12

Matthew Tancik's avatar
Matthew Tancik committed
13
[![Core Tests.](https://github.com/KAIR-BAIR/nerfacc/actions/workflows/code_checks.yml/badge.svg)](https://github.com/KAIR-BAIR/nerfacc/actions/workflows/code_checks.yml)
14
[![Documentation Status](https://readthedocs.com/projects/plenoptix-nerfacc/badge/?version=latest)](https://www.nerfacc.com/en/latest/?badge=latest)
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
15
[![Downloads](https://pepy.tech/badge/nerfacc)](https://pepy.tech/project/nerfacc)
Ruilong Li's avatar
readme  
Ruilong Li committed
16

17
https://www.nerfacc.com/
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
18

19
20
21
22
23
24
NerfAcc is a PyTorch Nerf acceleration toolbox for both training and inference. It focus on
efficient sampling in the volumetric rendering pipeline of radiance fields, which is 
universal and plug-and-play for most of the NeRFs.
With minimal modifications to the existing codebases, Nerfacc provides significant speedups 
in training various recent NeRF papers.
**And it is pure Python interface with flexible APIs!**
25

26
![Teaser](/docs/source/_static/images/teaser.jpg?raw=true)
Ruilong Li's avatar
readme  
Ruilong Li committed
27

28
29
## Installation

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
30
31
**Dependence**: Please install [Pytorch](https://pytorch.org/get-started/locally/) first.

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
32
The easist way is to install from PyPI. In this way it will build the CUDA code **on the first run** (JIT).
33
34
35
36
```
pip install nerfacc
```

Ruilong Li's avatar
Ruilong Li committed
37
Or install from source. In this way it will build the CUDA code during installation.
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
38
39
40
41
```
pip install git+https://github.com/KAIR-BAIR/nerfacc.git
```

42
43
44
We also provide pre-built wheels covering major combinations of Pytorch + CUDA supported by [official Pytorch](https://pytorch.org/get-started/previous-versions/).

```
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
45
# e.g., torch 1.13.0 + cu117
46
47
48
pip install nerfacc -f https://nerfacc-bucket.s3.us-west-2.amazonaws.com/whl/torch-1.13.0_cu117.html
```

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
49
50
51
52
53
54
| Windows & Linux | `cu102` | `cu113` | `cu116` | `cu117` |
|-----------------|---------|---------|---------|---------|
| torch 1.10.0    | ✅      | ✅      |         |         |
| torch 1.11.0    | ✅*     | ✅      |         |         |
| torch 1.12.0    | ✅*     | ✅      | ✅      |         |
| torch 1.13.0    |         |         | ✅      | ✅      |
55
56
57

\* Pytorch does not support Windows pre-built wheels for those combinations thus we do not support as well.

58
59
## Usage

60
61
62
63
64
65
66
67
68
The idea of NerfAcc is to perform efficient volumetric sampling with a computationally cheap estimator to discover surfaces.
So NerfAcc can work with any user-defined radiance field. To plug the NerfAcc rendering pipeline into your code and enjoy 
the acceleration, you only need to define two functions with your radience field.

- `sigma_fn`: Compute density at each sample. It will be used by the estimator
  (e.g., `nerfacc.OccGridEstimator`, `nerfacc.PropNetEstimator`) to discover surfaces. 
- `rgb_sigma_fn`: Compute color and density at each sample. It will be used by 
  `nerfacc.rendering` to conduct differentiable volumetric rendering. This function 
  will receive gradients to update your radiance field.
69

70
An simple example is like this:
71
72
73
74
75
76
77
78
79

``` python
import torch
from torch import Tensor
import nerfacc 

radiance_field = ...  # network: a NeRF model
rays_o: Tensor = ...  # ray origins. (n_rays, 3)
rays_d: Tensor = ...  # ray normalized directions. (n_rays, 3)
80
81
82
optimizer = ...       # optimizer

estimator = nerfacc.OccGridEstimator(...)
83
84
85
86

def sigma_fn(
    t_starts: Tensor, t_ends:Tensor, ray_indices: Tensor
) -> Tensor:
87
    """ Define how to query density for the estimator."""
88
89
    t_origins = rays_o[ray_indices]  # (n_samples, 3)
    t_dirs = rays_d[ray_indices]  # (n_samples, 3)
90
    positions = t_origins + t_dirs * (t_starts + t_ends)[:, None] / 2.0
91
    sigmas = radiance_field.query_density(positions) 
92
    return sigmas  # (n_samples,)
93
94
95
96

def rgb_sigma_fn(
    t_starts: Tensor, t_ends: Tensor, ray_indices: Tensor
) -> Tuple[Tensor, Tensor]:
97
    """ Query rgb and density values from a user-defined radiance field. """
98
99
    t_origins = rays_o[ray_indices]  # (n_samples, 3)
    t_dirs = rays_d[ray_indices]  # (n_samples, 3)
100
    positions = t_origins + t_dirs * (t_starts + t_ends)[:, None] / 2.0
101
    rgbs, sigmas = radiance_field(positions, condition=t_dirs)  
102
    return rgbs, sigmas  # (n_samples, 3), (n_samples,)
103

104
105
106
107
108
# Efficient Raymarching:
# ray_indices: (n_samples,). t_starts: (n_samples,). t_ends: (n_samples,).
ray_indices, t_starts, t_ends = estimator.sampling(
    rays_o, rays_d, sigma_fn=sigma_fn, near_plane=0.2, far_plane=1.0, early_stop_eps=1e-4, alpha_thre=1e-2, 
)
109
110
111

# Differentiable Volumetric Rendering.
# colors: (n_rays, 3). opaicity: (n_rays, 1). depth: (n_rays, 1).
112
color, opacity, depth, extras = nerfacc.rendering(
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
113
114
    t_starts, t_ends, ray_indices, n_rays=rays_o.shape[0], rgb_sigma_fn=rgb_sigma_fn
)
115

116
# Optimize: Both the network and rays will receive gradients
117
118
119
120
121
122
optimizer.zero_grad()
loss = F.mse_loss(color, color_gt)
loss.backward()
optimizer.step()
```

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
123
## Examples: 
Ruilong Li's avatar
readme  
Ruilong Li committed
124

125
126
See full benchmarking here: https://www.nerfacc.com/en/latest/examples/

Jingchen Ye's avatar
Jingchen Ye committed
127
Before running those example scripts, please check the script about which dataset is needed, and download the dataset first. You could use `--data_root` to specify the path.
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
128

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
129
```bash
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
130
131
# clone the repo with submodules.
git clone --recursive git://github.com/KAIR-BAIR/nerfacc/
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
132
```
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
133

134
135
136
### Static NeRFs

Instant-NGP on NeRF-Synthetic dataset with better performance in 4.5 minutes.
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
137
``` bash
138
139
140
141
# Occupancy Grid Estimator
python examples/train_ngp_nerf_occ.py --scene lego --data_root data/nerf_synthetic
# Proposal Net Estimator
python examples/train_ngp_nerf_prop.py --scene lego --data_root data/nerf_synthetic
Ruilong Li's avatar
readme  
Ruilong Li committed
142
143
```

144
Instant-NGP on Mip-NeRF 360 dataset with better performance in 5 minutes.
Ruilong Li's avatar
Ruilong Li committed
145
``` bash
146
147
148
149
# Occupancy Grid Estimator
python examples/train_ngp_nerf_occ.py --scene garden --data_root data/360_v2
# Proposal Net Estimator
python examples/train_ngp_nerf_prop.py --scene garden --data_root data/360_v2
Ruilong Li's avatar
Ruilong Li committed
150
151
```

152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
Vanilla MLP NeRF on NeRF-Synthetic dataset in an hour.
``` bash
# Occupancy Grid Estimator
python examples/train_mlp_nerf.py --scene lego --data_root data/nerf_synthetic
```

TensoRF on Tanks&Temple and NeRF-Synthetic datasets (plugin in the official codebase).
``` bash
cd benchmarks/tensorf/
# (set up the environment for that repo)
bash script.sh nerfsyn-nerfacc-occgrid 0
bash script.sh tt-nerfacc-occgrid 0
```

### Dynamic NeRFs
T-NeRF on D-NeRF dataset in an hour.
``` bash
# Occupancy Grid Estimator
python examples/train_mlp_tnerf.py --scene lego --data_root data/dnerf
```

K-Planes on D-NeRF dataset (plugin in the official codebase).
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
174
```bash
175
176
177
cd benchmarks/kplanes/
# (set up the environment for that repo)
bash script.sh dnerf-nerfacc-occgrid 0
178
179
```

180
TiNeuVox on HyperNeRF and D-NeRF datasets (plugin in the official codebase).
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
181
```bash
182
183
184
185
186
cd benchmarks/tineuvox/
# (set up the environment for that repo)
bash script.sh dnerf-nerfacc-occgrid 0
bash script.sh hypernerf-nerfacc-occgrid 0
bash script.sh hypernerf-nerfacc-propnet 0
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
187
```
188

189
190
191
192
193
194
195
196
197
198
### Camera Optimization NeRFs

BARF on the NeRF-Synthetic dataset (plugin in the official codebase).
```bash
cd benchmarks/barf/
# (set up the environment for that repo)
bash script.sh nerfsyn-nerfacc-occgrid 0
```

### 3rd-Party Usages:
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
199
- [nerfstudio](https://github.com/nerfstudio-project/nerfstudio): A collaboration friendly studio for NeRFs.
200
- [sdfstudio](https://autonomousvision.github.io/sdfstudio/): A unified framework for surface reconstruction..
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
201
- [instant-nsr-pl](https://github.com/bennyguo/instant-nsr-pl): NeuS in 10 minutes.
202
203
- [modelscope](https://github.com/modelscope/modelscope/blob/master/modelscope/models/cv/nerf_recon_acc/network/nerf.py): A collection of deep-learning algorithms.
- [Representing Volumetric Videos as Dynamic MLP Maps, CVPR 2023](https://github.com/zju3dv/mlp_maps)
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
204

205

Ruilong Li's avatar
Ruilong Li committed
206
## Common Installation Issues
207

Ruilong Li's avatar
Ruilong Li committed
208
209
210
211
<details>
    <summary>ImportError: .../csrc.so: undefined symbol</summary>
    If you are installing a pre-built wheel, make sure the Pytorch and CUDA version matchs with the nerfacc version (nerfacc.__version__).
</details>
212

213
214
215
## Citation

```bibtex
216
217
218
219
220
@article{li2023nerfacc,
  title={NerfAcc: Efficient Sampling Accelerates NeRFs.},
  author={Li, Ruilong and Hang Gao and Tancik, Matthew and Kanazawa, Angjoo},
  journal={TBD},
  year={2023}
221
}
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
222
```